@allansantos-dev/smart-tool 0.9.29 → 0.9.31

package/CHANGELOG.md

@@ -1,5 +1,50 @@
 # Changelog
 
+## 0.9.31 - beta
+
+- `scout_model` is filled by default. The setup screen (and the daemon, when it checks the models) now fills an empty
+  `scout_model` with Claude Haiku 5.5 when the configured gateway has it; a choice you made is never replaced, and a
+  gateway without that model leaves the field empty to choose (the scout then returns the literal lines, as before).
+- The other recommended models were re-measured by the same criterion (correctness first, price second; 3 rounds per
+  model; docs/research/2026-10-24-modelos-do-scout-e-dos-outros-papeis.md) and changed for new fills: `router_model`
+  Claude Haiku 5.5 (was GPT-4o-mini: it invented an answer to every request whose answer was not on the page),
+  `scope_model` GPT-5.4 nano (same accuracy as GPT-5-mini at 29% of the price), `research_model` GPT-5.4 nano (was
+  GPT-OSS 20B, which failed tool calls). Models already set are kept.
+  - Scout with an index (16 questions x 3 rounds x 15 models): Haiku 5.5 21/36 files found (the index's ceiling, equal
+    to no model), 12/12 on the questions with no answer, 0 failures; without an index, 20/36 against 12/36 for the
+    keyword scan and at most 17/36 for the other models.
+  - 8-question sessions with Sonnet: US$ 0.121 per session with Haiku 5.5 against 0.177 without a model, 73/80 against
+    75/80, and the agent never left the tool to read files itself (GPT-4o-mini: 10 times).
+- The client repeats a chat call once without `temperature` when the gateway answers 400 about it (Claude on Bedrock,
+  the GPT-5 family) and remembers the model; a second refusal still fails.
+- Tests: `tests/test_model_client.py`; probes `tests/scout_model_probe.py` and `tests/role_model_probe.py`;
+  `tests/scout_probe.py --scout-model`.
+
+## 0.9.30 - beta
+
+- New MCP tool `smart_scout(project_root, question, terms?, max_files?)`: answers a question about a project's code with
+  the lines that answer it, read inside Smart Tool so the files never enter the agent's context.
+  - Candidates come from the `smart_search` pipeline: the index when it is ready, the index plus a keyword read of the
+    files changed after it, or the keyword scan of the disk while the first index is built (answered at once, outside
+    the per-folder queue). With no index, no saved scope and no model, the scan covers the whole project (dependency,
+    hidden and generated folders excluded) instead of the root files only.
+  - For each of up to `max_files` files (default 5, at most 10) it reads the current lines from disk and keeps the
+    window with most of the question's words; the agent gets `path:lines`, the functions in it and those lines verbatim.
+  - Optional `scout_model` (setup screen, "Code scout"; empty by default): a small model of the configured gateway
+    reads the windows, writes up to 5 lines of answer and picks the line ranges to show, which are sliced from what
+    was read on disk. Without an index it first chooses the terms and folders to scan (two calls; one with an index).
+    A range outside what was read is clipped or dropped; an answer that cites no lines comes with the best keyword
+    windows; a model failure returns the lines with the error in a warning. Model calls are metered under the
+    purpose `scout` (`project_manage action=usage`).
+  - The server instructions point to `smart_scout` for questions that would mean reading several files, and the tool
+    loads upfront in Claude Code.
+  - Measured in isolated sessions (clean HOME and config, no CLAUDE.md, hooks, skills or plugins; 8 questions per
+    session, Sonnet, two repositories; docs/research/2026-10-24-smart-scout-construido-e-medido.md): with an index,
+    `smart_scout` with a `scout_model` cost US$ 0.154 per session against US$ 0.250 for `smart_search` alone, 31/32
+    right against 29/32, 90 s against 70 s; without a model, US$ 0.203, 29/32, 63 s. On a fresh install (no index)
+    it answered in 43 s (no model) and 117 s (model) against 256 s.
+- Tests: `tests/test_scout.py`; probe `tests/scout_probe.py`.
+
 ## 0.9.29 - beta
 
 - Registering the MCP server in Claude Code or Codex is idempotent. `claude mcp add` refuses a name that exists

package/README.md

@@ -5,6 +5,7 @@ A local MCP server that gives coding agents (Claude Code, Codex) cheaper, sharpe
 | Tool | What it does |
 |---|---|
 | `smart_search` / `smart_search_result` | Semantic + lexical search over an indexed project, results split into code, tests and docs, with the likeliest functions per chunk |
+| `smart_scout` | Answers a question about the code with the lines that answer it: reads the candidate files inside Smart Tool and returns `path:lines` with the literal text, so the files never enter the agent's context; with a `scout_model`, a small model also writes a short answer and keeps only the lines that support it |
 | `web_search` / `web_search_result` | Web search across several sources in parallel, in the query language and in English, with a cache shared across sessions; `depth="research"` runs a multi-round research loop |
 | `web_fetch` | Reads a known URL and returns only the answer to your prompt, written by a small model; pages are cached for 24 h. Pages over 60,000 characters are not cut: the model reads the parts closest to the prompt (needs an `embedding_model`) |
 | `project_manage` | Registers and indexes projects, shows coverage, code maps, duplicated functions and index storage; `graph` with `symbol` tells what changing a function touches: callers, calls, the tests that reach it and the files that import it |
@@ -15,7 +16,7 @@ registered as the `playwright` server where this machine lets a browser be drive
 An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
 are cheaper, or just tells the agent what Smart Tool would do.
 
-**Status:** 0.9.29 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
+**Status:** 0.9.31 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
 
 ## How it works
 
@@ -27,6 +28,7 @@ program that runs on your machine and offers the agent better tools for the same
 ```
 Claude Code / Codex ──MCP──▶ Smart Tool daemon (127.0.0.1, one per machine)
                                ├─ smart_search ──▶ local index of your project (SQLite)
+                               ├─ smart_scout ───▶ the files behind a search, read on disk (small model optional)
                                ├─ web_search ───▶ several search engines at once, shared cache
                                ├─ web_fetch ────▶ the page, read by a small model
                                └─ project_manage
@@ -41,6 +43,13 @@ rerank model, reorders the best candidates. The agent gets back a few chunks spl
 the functions most likely to matter, instead of every line that contains a word. The index follows the project:
 only files whose content changed are processed again, and each Git branch keeps its own view.
 
+**Reading code to answer a question.** `smart_scout` takes a question in plain language. It finds the candidates the
+same way as `smart_search` (or by keyword on disk while there is no index yet), reads the current lines of each one
+and keeps, per file, the window with most of the question's words. The agent receives `path:lines` and those lines
+verbatim. With a `scout_model` (the setup screen fills it with Claude Haiku 5.5 when the gateway has it), a small model reads those windows, writes a few lines of answer
+and picks which lines to show; without an index it first picks the words and folders to scan (two calls; one with an
+index). What the model writes never replaces the literal lines, and when it fails the lines come back with the error.
+
 **Searching the web.** `web_search` asks several free search sources in parallel (DuckDuckGo, Wikipedia, Yahoo,
 Exa, Google and others), in the language of the question and in English, and merges what comes back. Search APIs with a
 free quota (Tavily, Firecrawl) and a real browser are tried only when those results are not enough. Answers are cached and shared by every agent

package/config.py

@@ -23,6 +23,7 @@ _LOCK = threading.RLock()
 DEFAULT_CONFIG = {
     "router_model": "",
     "scope_model": "",
+    "scout_model": "",
     "embedding_model": "",
     "rerank_model": "",
     "research_model": "",

package/model_client.py

@@ -1,11 +1,16 @@
 """Smart Tool model client: talks to the configured model gateway through its adapter."""
 import contextvars
+import io
+import threading
 import time
+import urllib.error
 import usage_meter
 
 import gateway
 
 OFFLINE = contextvars.ContextVar("model_offline", default=None)
+_NO_TEMPERATURE = set()
+_no_temperature_lock = threading.Lock()
 
 
 def get_token():
@@ -32,7 +37,25 @@ def fetch(path, token, method="GET", body=None, timeout=15):
 
 
 def _fetch(path, _token, method="GET", body=None, timeout=15):
-    # Callers name routes as /v1/...; the adapter knows where its API starts.
+    """Routes are named /v1/...; the adapter knows where its API starts. Some models (a Claude on Bedrock, the GPT-5
+    family) reject `temperature` with HTTP 400; the request is repeated once without it and the model is remembered, so
+    any model of the gateway works with the same call. A second refusal propagates."""
     if not path.startswith("/v1/"):
         raise ValueError("Only /v1/ routes are accepted by this client.")
-    return gateway.request(path[len("/v1/"):], body=body, timeout=timeout)
+    route = path[len("/v1/"):]
+    model = body.get("model") if isinstance(body, dict) else None
+    if isinstance(body, dict) and "temperature" in body and model in _NO_TEMPERATURE:
+        body = {key: value for key, value in body.items() if key != "temperature"}
+    try:
+        return gateway.request(route, body=body, timeout=timeout)
+    except urllib.error.HTTPError as exc:
+        if exc.code != 400 or not isinstance(body, dict) or "temperature" not in body:
+            raise
+        detail = exc.read()
+        if b"temperature" not in detail.lower():
+            exc.read = io.BytesIO(detail).read
+            raise
+    with _no_temperature_lock:
+        _NO_TEMPERATURE.add(model)
+    return gateway.request(route, body={key: value for key, value in body.items() if key != "temperature"},
+                           timeout=timeout)

package/model_defaults.py

@@ -6,12 +6,16 @@ import config
 import model_catalog
 
 
-# Matched as the last path/dot segment of the catalog id, so "gpt-4o-mini" also finds "openai/gpt-4o-mini" (OpenRouter)
-# and "openai.gpt-4o-mini" (LiteLLM aliases). Router: small, no reasoning (it runs before tool calls).
+# Matched as the last path/dot segment of the catalog id, so "claude-haiku-5-5" also finds "vertex_ai.anthropic.claude-haiku-5-5"
+# (LiteLLM aliases) and "anthropic/claude-haiku-5-5" (OpenRouter). Chosen by measurement, 3 rounds per model, correctness
+# first and price second (docs/research/2026-10-24-modelos-do-scout-e-dos-outros-papeis.md): the router answers pages
+# and the scout reads code excerpts, where Claude Haiku 5.5 answered every request and led the no-index path; scope and
+# research pick the GPT-5.4 nano, as accurate there at about half the price and without the long reasoning of the mini.
 RECOMMENDED_MODELS = {
-    "router_model": ("chat", "gpt-4o-mini"),
-    "scope_model": ("chat", "gpt-5-mini"),
-    "research_model": ("chat", "gpt-oss-20b"),
+    "router_model": ("chat", "claude-haiku-5-5"),
+    "scope_model": ("chat", "gpt-5.4-nano"),
+    "scout_model": ("chat", "claude-haiku-5-5"),
+    "research_model": ("chat", "gpt-5.4-nano"),
     "embedding_model": ("embedding", "text-embedding-3-small"),
     "rerank_model": ("rerank", "rerank-3-5"),
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@allansantos-dev/smart-tool",
-  "version": "0.9.29",
+  "version": "0.9.31",
   "description": "Local MCP server that gives coding agents (Claude Code, Codex) cheaper, sharper tools than their built-in search. Windows.",
   "license": "Apache-2.0",
   "author": "Allan Santos",

package/scout.py

@@ -0,0 +1,307 @@
+"""smart_scout: answers a question about a project's code with the literal lines that answer it, read from disk.
+
+The heavy model of the session passes the question; the daemon finds the candidates (the index when it is ready, the
+keyword scan of the disk when it is not), reads the current lines of each candidate, keeps the window with most query
+terms and returns `path:lines` plus those lines verbatim. With a `scout_model` configured, a light model of the Smart
+Tool's own gateway also writes a short answer and picks which lines to show; what it writes never replaces the literal
+lines, which always come from the file on disk. Without an index the model gets two fixed calls (terms and folders to
+scan, then the answer); with an index, one.
+
+Market pattern copied (docs/research/2026-10-24-scout-modelo-leve-no-smart-tool.md): read-only, step limit, return
+`path:lines` with the excerpt (Windsurf Fast Context, Relace, Claude Code Explore); the light model is fed only the
+excerpts, never an agent prompt, so no agent context is reopened per call.
+"""
+import os
+import re
+
+import document_text
+import index_scope
+import indexer
+
+MAX_QUESTION_CHARS = 2000
+MAX_TERMS_CHARS = 400
+DEFAULT_MAX_FILES = 5
+MAX_FILES = 10
+WINDOWS_PER_FILE = 2
+LEXICAL_WINDOW_LINES = 24
+MODEL_WINDOW_LINES = 60
+CITED_MAX_LINES = 40
+UNCITED_WINDOWS = 2
+LITERAL_BUDGET_CHARS = 9000
+LINE_MAX_CHARS = 240
+ANSWER_MAX_CHARS = 1200
+PLAN_MAX_TERMS = 12
+PLAN_MAX_FOLDERS = 6
+OUTLINE_MAX_DIRS = 300
+OUTLINE_MAX_ROOT_FILES = 200
+_FIELDS = {"project_root", "question", "terms", "max_files"}
+_WORD = re.compile(r"[\w]{3,}", re.UNICODE)
+_STOPWORDS = {
+    "the", "and", "for", "how", "what", "where", "which", "does", "with", "from", "that", "this", "are", "when", "who",
+    "why", "into", "used", "use", "uses", "its", "has", "have", "code", "project", "file", "files",
+    "como", "onde", "qual", "quais", "que", "quem", "por", "para", "com", "uma", "dos", "das", "nos", "nas", "pelo",
+    "pela", "ele", "ela", "isso", "esse", "essa", "este", "esta", "fica", "ficam", "usa", "usado", "usada", "sobre",
+    "entre", "depois", "antes", "quando", "cada", "seu", "sua", "seus", "suas", "não", "sem", "mais", "código",
+    "projeto", "arquivo", "arquivos", "decide", "funciona",
+}
+
+PLAN_SYSTEM = (
+    "You plan a keyword scan of a code project that has no search index yet. Given a question and the project's "
+    "folders and root files, return the identifier-style terms most likely written in the code that answers it "
+    "(function, class, variable, config names; split camelCase is fine; both the code's language and the question's) "
+    "and the folders worth scanning (only names from the list; the root files are always scanned). Call submit_scan_plan."
+)
+PLAN_TOOL = {"type": "function", "function": {
+    "name": "submit_scan_plan",
+    "description": "Terms to look for in file contents and the folders to scan.",
+    "parameters": {"type": "object", "additionalProperties": False, "required": ["terms", "folders"], "properties": {
+        "terms": {"type": "array", "items": {"type": "string"}, "maxItems": PLAN_MAX_TERMS},
+        "folders": {"type": "array", "items": {"type": "string"}, "maxItems": PLAN_MAX_FOLDERS}}}}}
+ANSWER_SYSTEM = (
+    "You answer a question about a code project using ONLY the numbered excerpts provided (each line starts with its "
+    "line number). Write at most 5 short lines, citing excerpts as [E1], [E2]. Then cite the excerpts that answer the "
+    "question with the narrowest line range inside each one (at most 40 lines). If no excerpt answers it, set found to "
+    "false and say what is missing; never guess beyond the excerpts. Answer in the language of the question. "
+    "Call submit_scout_answer."
+)
+ANSWER_TOOL = {"type": "function", "function": {
+    "name": "submit_scout_answer",
+    "description": "Short answer and the excerpt line ranges that support it.",
+    "parameters": {"type": "object", "additionalProperties": False, "required": ["found", "answer", "citations"],
+                   "properties": {
+                       "found": {"type": "boolean"},
+                       "answer": {"type": "string"},
+                       "citations": {"type": "array", "maxItems": MAX_FILES * WINDOWS_PER_FILE, "items": {
+                           "type": "object", "additionalProperties": False, "required": ["id", "start_line", "end_line"],
+                           "properties": {"id": {"type": "string"}, "start_line": {"type": "integer"},
+                                          "end_line": {"type": "integer"}}}}}}}}
+
+
+def arguments(raw):
+    """Validated `{"project_root", "question", "terms", "max_files"}` from the tool call; raises ValueError naming the
+    field. `terms` (identifier words the caller already knows) is optional and joins the question in the search."""
+    if not isinstance(raw, dict):
+        raise ValueError("smart_scout requires its arguments as a JSON object")
+    unknown = set(raw) - _FIELDS
+    if unknown:
+        raise ValueError(f"Unknown fields in smart_scout: {', '.join(sorted(unknown))}. Accepted: {', '.join(sorted(_FIELDS))}.")
+    root = raw.get("project_root")
+    if not isinstance(root, str) or not os.path.isdir(root):
+        raise ValueError("smart_scout requires 'project_root' pointing to an existing folder")
+    question = raw.get("question")
+    if not isinstance(question, str) or not question.strip() or len(question) > MAX_QUESTION_CHARS:
+        raise ValueError(f"smart_scout requires 'question' (non-empty text of at most {MAX_QUESTION_CHARS} characters)")
+    terms = raw.get("terms")
+    if terms is not None and (not isinstance(terms, str) or len(terms) > MAX_TERMS_CHARS):
+        raise ValueError(f"terms must be text of at most {MAX_TERMS_CHARS} characters")
+    max_files = raw.get("max_files", DEFAULT_MAX_FILES)
+    if type(max_files) is not int or not 1 <= max_files <= MAX_FILES:
+        raise ValueError(f"max_files must be an integer from 1 to {MAX_FILES}")
+    return {"project_root": root, "question": question.strip(), "terms": (terms or "").strip(), "max_files": max_files}
+
+
+def focus_terms(*texts):
+    """Casefolded content words of the texts (3+ characters, common question words dropped), camelCase split, in
+    order: what a window is scored by."""
+    found = []
+    for text in texts:
+        spaced = re.sub(r"([a-z])([A-Z])", r"\1 \2", text or "")
+        for word in _WORD.findall(spaced):
+            folded = word.casefold()
+            if folded not in _STOPWORDS and folded not in found:
+                found.append(folded)
+    return found[:24]
+
+
+def disk_lines(root, path, start, end):
+    """`[(line_number, text)]` of lines `start..end` of the file as it is on disk now, or None when the file cannot be
+    read by the index rules (secret, dotfile, too large, binary, gone) or holds no line text (docx)."""
+    if document_text.location_kind(path) != "source_line":
+        return None
+    data, encoding = indexer.read_indexable(root, path)
+    if data is None:
+        return None
+    try:
+        lines = data.decode(encoding or "utf-8", "replace").splitlines()
+    except LookupError:
+        return None
+    start = max(1, start)
+    end = min(len(lines), end)
+    return [(number, lines[number - 1]) for number in range(start, end + 1)] if start <= end else None
+
+
+def best_window(lines, terms, size):
+    """The run of at most `size` consecutive `(number, text)` lines holding most occurrences of `terms` (first run on a
+    tie), trimmed of blank edges."""
+    if len(lines) <= size:
+        window = lines
+    else:
+        hits = [sum(text.casefold().count(term) for term in terms) for _number, text in lines]
+        best, best_at, current = -1, 0, sum(hits[:size])
+        for at in range(len(lines) - size + 1):
+            if at:
+                current += hits[at + size - 1] - hits[at - 1]
+            if current > best:
+                best, best_at = current, at
+        window = lines[best_at:best_at + size]
+    while window and not window[0][1].strip():
+        window = window[1:]
+    while window and not window[-1][1].strip():
+        window = window[:-1]
+    return window
+
+
+def order_candidates(blocks, max_files):
+    """Search rows of the code, doc and test blocks in reading order: code first, with the best doc and test rows kept
+    inside the first `max_files` files when the code rows would crowd them out."""
+    code, doc, test = (list(blocks.get(name) or []) for name in ("code", "doc", "test"))
+    reserved = [rows[0] for rows in (doc, test) if rows]
+    head = code[:max(1, max_files - len(reserved))]
+    rest = code[len(head):] + doc[1:] + test[1:]
+    return head + reserved + rest
+
+
+def evidence(root, rows, terms, max_files, window_lines):
+    """Evidence items `{id, path, start, end, lines, symbols}` for the first `max_files` files of `rows` (search rows
+    `{path, lines: "a-b", snippet, symbols}`), at most WINDOWS_PER_FILE non-overlapping windows per file, each the
+    `window_lines` lines of the chunk with most `terms`, read from disk now. A file that cannot be read from disk keeps
+    the indexed snippet with its chunk range."""
+    items, files = [], []
+    for row in rows:
+        path = row["path"]
+        try:
+            start, end = (int(part) for part in str(row["lines"]).split("-", 1))
+        except ValueError:
+            continue
+        if path not in files:
+            if len(files) >= max_files:
+                continue
+            files.append(path)
+        mine = [item for item in items if item["path"] == path]
+        if len(mine) >= WINDOWS_PER_FILE or any(item["start"] <= end and start <= item["end"] for item in mine):
+            continue
+        lines = disk_lines(root, path, start, end)
+        if lines:
+            window = best_window(lines, terms, window_lines)
+            if not window:
+                continue
+            start, end = window[0][0], window[-1][0]
+        else:
+            window = [(None, text) for text in str(row.get("snippet") or "").splitlines()[:window_lines]]
+        items.append({"id": f"E{len(items) + 1}", "path": path, "start": start, "end": end, "lines": window,
+                      "symbols": row.get("symbols") or ""})
+    return items
+
+
+def numbered(lines):
+    """Lines as `  123| text`, long lines cut at LINE_MAX_CHARS."""
+    out = []
+    for number, text in lines:
+        text = text if len(text) <= LINE_MAX_CHARS else text[:LINE_MAX_CHARS] + " …"
+        out.append(f"{number:>6}| {text}" if number is not None else f"      | {text}")
+    return "\n".join(out)
+
+
+def outline(root):
+    """`(folders, root_files)` shown to the plan call: folders up to two levels deep and loose files at the root, with
+    the index's mechanical exclusions (dependency caches, hidden folders, generated output) already applied."""
+    skipped = set(index_scope.mechanical_excludes(root))
+    folders = index_scope._structure(root, max_dirs=OUTLINE_MAX_DIRS,
+                                     keep=lambda rel: rel.count("/") < 2 and rel.split("/")[0] not in skipped)
+    try:
+        names = sorted(os.listdir(root))
+    except OSError:
+        names = []
+    files = [name for name in names if not name.startswith(".") and os.path.isfile(os.path.join(root, name))]
+    return folders, files[:OUTLINE_MAX_ROOT_FILES]
+
+
+def plan_prompt(question, folders, root_files):
+    """User message of the plan call."""
+    return ("Question: " + question + "\n\nFolders:\n" + "\n".join(folders or ["(none)"]) +
+            "\n\nRoot files:\n" + "\n".join(root_files or ["(none)"]))
+
+
+def valid_plan(raw, folders):
+    """`(terms, folders)` from the plan call: terms as non-empty words, folders only those listed (none means only the
+    root files, which `index_scope.resolve_included_files` always adds); raises ValueError when no usable term
+    remains."""
+    if not isinstance(raw, dict) or not isinstance(raw.get("terms"), list) or not isinstance(raw.get("folders"), list):
+        raise ValueError("the scan plan came back without terms and folders")
+    terms = [term.strip() for term in raw["terms"] if isinstance(term, str) and term.strip()][:PLAN_MAX_TERMS]
+    if not terms:
+        raise ValueError("the scan plan came back without terms")
+    allowed = set(folders)
+    picked = [folder.strip().strip("/") for folder in raw["folders"] if isinstance(folder, str)]
+    return terms, [folder for folder in dict.fromkeys(picked) if folder in allowed][:PLAN_MAX_FOLDERS]
+
+
+def answer_prompt(question, items):
+    """User message of the answer call: the question and each excerpt with its id, path and numbered lines."""
+    parts = [f"Question: {question}"]
+    for item in items:
+        parts.append(f"[{item['id']}] {item['path']}:{item['start']}-{item['end']}"
+                     + (f" ({item['symbols']})" if item["symbols"] else "") + "\n" + numbered(item["lines"]))
+    return "\n\n".join(parts)
+
+
+def valid_answer(raw, items):
+    """`(found, answer, citations)` from the answer call. Citations name an excerpt id and a range clipped to that
+    excerpt and to CITED_MAX_LINES; unknown ids and empty ranges are dropped. Raises ValueError on a malformed reply."""
+    if (not isinstance(raw, dict) or type(raw.get("found")) is not bool or not isinstance(raw.get("answer"), str)
+            or not isinstance(raw.get("citations"), list)):
+        raise ValueError("the answer came back outside the expected fields")
+    by_id = {item["id"]: item for item in items}
+    citations = []
+    for cite in raw["citations"]:
+        if not isinstance(cite, dict) or cite.get("id") not in by_id:
+            continue
+        item = by_id[cite["id"]]
+        try:
+            start, end = int(cite.get("start_line")), int(cite.get("end_line"))
+        except (TypeError, ValueError):
+            continue
+        if item["lines"][0][0] is None:
+            start, end = item["start"], item["end"]
+        start, end = max(start, item["start"]), min(end, item["end"], max(start, item["start"]) + CITED_MAX_LINES - 1)
+        if start <= end and not any(c["id"] == item["id"] and c["start"] <= end and start <= c["end"] for c in citations):
+            citations.append({"id": item["id"], "start": start, "end": end})
+    answer = raw["answer"].strip()[:ANSWER_MAX_CHARS]
+    if not answer:
+        raise ValueError("the answer came back empty")
+    return raw["found"], answer, citations
+
+
+def render(question, source, items, citations=None, answer=None, found=None, model_note="", warnings=(), others=()):
+    """The text returned to the agent. Lexical mode (`citations` None): every evidence window, within
+    LITERAL_BUDGET_CHARS, after the model's answer when there is one (a model that cited nothing never leaves the
+    answer without lines). Model mode: the model's answer, then only the cited ranges (sliced from the windows read
+    from disk), and the uncited excerpts as `path:lines` pointers. `others` are candidates not shown, listed as
+    pointers."""
+    out = [f"question: {question}", f"source: {source}", f"model: {model_note or 'none (literal lines only)'}"]
+    if answer is not None:
+        out.append(("answer: " if found else "answer (not found in the excerpts): ") + answer)
+    shown, budget = [], LITERAL_BUDGET_CHARS
+    if citations is None:
+        picks = [(item, item["start"], item["end"]) for item in items]
+    else:
+        by_id = {item["id"]: item for item in items}
+        picks = [(by_id[c["id"]], c["start"], c["end"]) for c in citations]
+    out.append("evidence:" if picks else "evidence: none")
+    for item, start, end in picks:
+        lines = [(n, t) for n, t in item["lines"] if n is None or start <= n <= end]
+        block = numbered(lines)
+        head = f"- [{item['id']}] {item['path']}:{start}-{end}" + (f" ({item['symbols']})" if item["symbols"] else "")
+        if len(block) > budget:
+            out.append(head + " (lines not shown: output budget reached)")
+            continue
+        budget -= len(block)
+        out += [head, block]
+        shown.append(item["id"])
+    pointers = [f"{item['path']}:{item['start']}-{item['end']}" for item in items if item["id"] not in shown
+                and citations is not None] + list(others)
+    if pointers:
+        out.append("other candidates (not shown): " + ", ".join(pointers))
+    for warning in warnings:
+        out.append(f"warning: {warning}")
+    return "\n".join(out) + "\n"

package/setup_ui.py

@@ -79,6 +79,7 @@ def status_payload():
         "missing_models": models["missing"],
         "router_model": cfg.get("router_model", ""),
         "scope_model": cfg.get("scope_model", ""),
+        "scout_model": cfg.get("scout_model", ""),
         "research_model": cfg.get("research_model", ""),
         "research_max_iterations": cfg.get("research_max_iterations", config.DEFAULT_CONFIG["research_max_iterations"]),
         "quick_search_timeout_s": cfg.get("quick_search_timeout_s", config.DEFAULT_CONFIG["quick_search_timeout_s"]),
@@ -132,6 +133,7 @@ def handle_models_get():
         "current": {
             "router_model": cfg.get("router_model", ""),
             "scope_model": cfg.get("scope_model", ""),
+            "scout_model": cfg.get("scout_model", ""),
             "research_model": cfg.get("research_model", ""),
             "research_max_iterations": cfg.get("research_max_iterations", config.DEFAULT_CONFIG["research_max_iterations"]),
             "quick_search_timeout_s": cfg.get("quick_search_timeout_s", config.DEFAULT_CONFIG["quick_search_timeout_s"]),
@@ -369,6 +371,7 @@ def handle_autostart_action(body):
 _MODEL_FIELDS = {
     "router_model": "chat",
     "scope_model": "chat",
+    "scout_model": "chat",
     "research_model": "chat",
     "embedding_model": "embedding",
     "rerank_model": "rerank",
@@ -681,6 +684,7 @@ _PAGE_TEMPLATE = """<!doctype html>
             <div class="field"><label for="sel-scope">Index classification</label><span class="explain">Decides scope, code or documentation, and languages while indexing. Runs once per project.</span><input id="sel-scope" list="dl-scope" autocomplete="off" spellcheck="false"><datalist id="dl-scope"></datalist></div>
             <div class="field"><label for="sel-router">Router</label><span class="explain">Decides before each agent Grep, Read or Bash. Use a small model without reasoning: above 10 s routing turns off.</span><input id="sel-router" list="dl-router" autocomplete="off" spellcheck="false"><datalist id="dl-router"></datalist></div>
           </div>
+          <div class="field"><label for="sel-scout">Code scout (optional)</label><span class="explain">smart_scout: writes a short answer over the lines it read and keeps only the ones that answer. Empty returns the literal lines without a model.</span><input id="sel-scout" list="dl-scout" autocomplete="off" spellcheck="false"><datalist id="dl-scout"></datalist></div>
           <div class="field"><label for="sel-research">Web research</label><span class="explain">Used by web_search with depth=research. Empty uses the router model.</span><input id="sel-research" list="dl-research" autocomplete="off" spellcheck="false"><datalist id="dl-research"></datalist></div>
           <details class="more">
             <summary>Advanced web research limits</summary>
@@ -1031,6 +1035,7 @@ async function loadModels() {
     const data = await getJSON("/setup/models");
     fillSuggestions($("sel-router"), data.chat_lightweight.length ? data.chat_lightweight : data.chat, data.current.router_model);
     fillSuggestions($("sel-scope"), data.chat, data.current.scope_model);
+    fillSuggestions($("sel-scout"), data.chat_lightweight.length ? data.chat_lightweight : data.chat, data.current.scout_model);
     fillSuggestions($("sel-research"), data.chat, data.current.research_model);
     fillSuggestions($("sel-embedding"), data.embedding, data.current.embedding_model);
     fillSuggestions($("sel-rerank"), data.rerank, data.current.rerank_model);
@@ -1048,7 +1053,7 @@ $("btn-save-models").addEventListener("click", async () => {
   setMsg("models-msg", "", "Saving…");
   try {
     await post("/setup/models", {
-      router_model: $("sel-router").value, scope_model: $("sel-scope").value, research_model: $("sel-research").value,
+      router_model: $("sel-router").value, scope_model: $("sel-scope").value, scout_model: $("sel-scout").value, research_model: $("sel-research").value,
       research_max_iterations: Number($("num-research-iterations").value), quick_search_timeout_s: Number($("num-quick-timeout").value),
       site_memory_similarity_threshold: Number($("num-site-memory-threshold").value),
       embedding_model: $("sel-embedding").value, rerank_model: $("sel-rerank").value,

package/smart_tool_daemon.py

@@ -52,6 +52,7 @@ import index_inventory
 import usage_meter
 import document_text
 import dossier
+import scout
 import embedding_cache
 import model_defaults
 import project_identity
@@ -334,7 +335,8 @@ def _smart_search_job_payload(job_id):
     return payload
 
 # Claude Code defers MCP tools behind tool search; the tools the hook redirects to (and their job polling) load
-# upfront, so an agent denied by the hook already has the replacement in its tool list.
+# upfront, so an agent denied by the hook already has the replacement in its tool list. smart_scout loads upfront too:
+# the server instructions point to it in place of reading several files.
 ALWAYS_LOAD = {"anthropic/alwaysLoad": True}
 SERVER_INSTRUCTIONS = (
     "Smart Tool serves an indexed semantic search over the user's projects and a cheaper web reader. Use smart_search "
@@ -345,9 +347,9 @@ SERVER_INSTRUCTIONS = (
     "node, PowerShell, curl or wget. "
     "Before changing a function, project_manage action=graph with symbol lists its callers and tests; after editing, "
     "project_manage action=affected_tests lists the tests to run. When smart_search returns nothing, retry smart_search with "
-    "other terms; do not switch to Grep, Read of folders or shell search. A helper agent (smart-scout, a cheap model) is "
-    "optional: use it only for a task that chains several Smart Tool calls or while the index is still loading; a plain "
-    "search is faster called directly. A helper that returns empty means retry with the tool or report not found."
+    "other terms; do not switch to Grep, Read of folders or shell search. To answer a question that would mean reading "
+    "several files, call smart_scout: it reads them inside Smart Tool and returns only the lines that answer, as "
+    "path:lines with the literal text."
 )
 
 TOOLS = [
@@ -433,6 +435,30 @@ TOOLS = [
             "required": ["project_root"],
         },
     },
+    {
+        "name": "smart_scout",
+        "_meta": ALWAYS_LOAD,
+        "description": (
+            "Answers a question about a project's code with the lines that answer it, read inside Smart Tool so the "
+            "files never enter your context: path:lines plus the literal text of each excerpt, read from disk now. "
+            "Use it instead of reading several files one by one (how a flow works, where something is decided). "
+            "Candidates come from the index when it is ready and from a keyword scan of the disk when it is not. "
+            "When a scout_model is configured, a light model also writes a short answer and keeps only the lines that "
+            "support it; its text never replaces the literal lines. For a plain lookup of a name, smart_search is enough."
+        ),
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "project_root": {"type": "string", "description": "Absolute path to the project root"},
+                "question": {"type": "string", "description": "The question in natural language, as you would ask a colleague"},
+                "terms": {"type": "string", "description": "Optional identifier words you already know (function, class, config names)"},
+                "max_files": {"type": "integer", "minimum": 1, "maximum": scout.MAX_FILES, "default": scout.DEFAULT_MAX_FILES,
+                              "description": "Files read and shown at most"},
+            },
+            "required": ["project_root", "question"],
+            "additionalProperties": False,
+        },
+    },
     {
         "name": "smart_search_result",
         "_meta": ALWAYS_LOAD,
@@ -1102,6 +1128,7 @@ def _perform_smart_search_isolated(arguments, job_id):
     vectors = {}
     overlay = None
     search_model = model
+    source = "index"
     if problem:
         model_client.OFFLINE.set(problem)
     token = "" if problem else model_client.get_token()
@@ -1120,6 +1147,7 @@ def _perform_smart_search_isolated(arguments, job_id):
     elif _cold_index(root, cfg, problem):
         overlay, notice = _cold_overlay(root, texts)
         search_model = None
+        source = "cold"
         warnings += [text for text in (notice, _refresh_in_background(root, {"refresh": True})) if text]
     elif problem and not local_embedder.is_local(model):
         if not indexer.info(root).get("chunks"):
@@ -1144,6 +1172,7 @@ def _perform_smart_search_isolated(arguments, job_id):
             lag = _index_lag(root, model, cfg, problem, job_id)
             if lag is not None:
                 overlay, notice = _lag_overlay(root, lag)
+                source = "index+disk" if overlay else source
                 warnings += [text for text in (notice, _refresh_in_background(root, lag)) if text]
             else:
                 try:
@@ -1231,6 +1260,9 @@ def _perform_smart_search_isolated(arguments, job_id):
     warning = " ".join(warnings)
     if warning:
         _update_job(job_id, warning=warning)
+    if arguments.get("_structured"):
+        return {"source": source if source == "cold" or vectors else "lexical index", "blocks": blocks,
+                "warnings": warnings}
     return _grouped_yaml(index_profile.public(profile), blocks, " ".join(warnings + notes))
 
 
@@ -1614,6 +1646,152 @@ def _agent_search_answer(payload):
     return _to_json_sanitized(slim)
 
 
+SCOUT_SEARCH_WAIT_S = 20
+SCOUT_MODEL_TIMEOUT_S = 30
+SCOUT_SCAN_MAX_FILES = 4000
+SCOUT_MAX_POINTERS = 8
+_SCOUT_SOURCES = {
+    "index": "index (semantic search and rerank)",
+    "index+disk": "index, with the files changed after it searched by keyword on disk",
+    "lexical index": "index by keyword only (no embedding)",
+    "cold": "keyword scan of the disk (no index yet; it is being built in the background)",
+    "pending": "keyword scan of the disk (the index is being updated in the background)",
+    "failed": "keyword scan of the disk (the search failed)",
+}
+
+
+def _handle_smart_scout(arguments):
+    """smart_scout: the literal lines that answer a question about a project, read from disk, with a short answer by
+    the configured `scout_model` when there is one. Candidates come from the smart_search pipeline (index when ready,
+    keyword scan of the disk when not); without an index the model first plans terms and folders to scan (two calls in
+    all, one with an index). A model failure is reported in a warning and the literal lines are returned without a
+    summary; model calls are recorded in the usage meter under the purpose `scout`."""
+    started = time.monotonic()
+    args = scout.arguments(arguments)
+    root = project_identity.display_root(args["project_root"])
+    question, max_files = args["question"], args["max_files"]
+    query = " ".join(part for part in (args["terms"], question) if part)
+    model = config.load_config().get("scout_model") or ""
+    problem = _gateway_problem() if model else None
+    warnings, calls, failure = [], 0, None
+    if problem:
+        warnings.append(f"scout_model {model} unavailable ({problem}): literal lines only, no answer.")
+        failure, model = "gateway", ""
+    with usage_meter.job(root, index_views.describe(root), "scout"):
+        found = _scout_search(root, query, question, max_files)
+        usage_meter.phase("scout")
+        warnings += found["warnings"]
+        rows = scout.order_candidates(found["blocks"], max_files)
+        plan_terms = []
+        if model and found["source"] in ("cold", "pending", "failed"):
+            calls += 1
+            try:
+                plan_terms, planned = _scout_plan_scan(root, question, model)
+                rows = planned + rows
+            except Exception as exc:
+                warnings.append(f"scout_model {model} failed planning the scan ({_project_error(exc)}): literal lines "
+                                "only, no answer.")
+                failure, model = "plan", ""
+        if (not model and found["source"] in ("cold", "pending", "failed")
+                and not index_scope.valid_scope(index_scope.load_scope(root))):
+            wide, scanned, total = _scout_scan(root, [""], " ".join(scout.focus_terms(args["terms"], question)))
+            rows = wide + rows
+            warnings.append(f"No index and no saved scope yet: keyword scan of the whole project in {scanned} of {total} "
+                            "files (dependency, hidden and generated folders excluded).")
+        terms = scout.focus_terms(args["terms"], question, " ".join(plan_terms))
+        items = scout.evidence(root, rows, terms, max_files,
+                               scout.MODEL_WINDOW_LINES if model else scout.LEXICAL_WINDOW_LINES)
+        shown = {item["path"] for item in items}
+        others = list(dict.fromkeys(f"{row['path']}:{row['lines']}" for row in rows
+                                    if row["path"] not in shown))[:SCOUT_MAX_POINTERS]
+        source = _SCOUT_SOURCES[found["source"]]
+        text = None
+        if model and items:
+            calls += 1
+            try:
+                raw = _research_llm_call(model, model_client.get_token(), scout.ANSWER_SYSTEM,
+                                         scout.answer_prompt(question, items), scout.ANSWER_TOOL,
+                                         timeout=SCOUT_MODEL_TIMEOUT_S)
+                found_answer, answer, citations = scout.valid_answer(raw, items)
+                note = f"{model} ({calls} call{'s' if calls > 1 else ''})"
+                if citations:
+                    text = scout.render(question, source, items, citations, answer, found_answer, note, warnings, others)
+                else:
+                    shown_items = scout.evidence(root, rows, terms, max_files,
+                                                 scout.LEXICAL_WINDOW_LINES)[:scout.UNCITED_WINDOWS]
+                    rest = [f"{item['path']}:{item['start']}-{item['end']}" for item in items
+                            if item["path"] not in {kept["path"] for kept in shown_items}]
+                    text = scout.render(question, source, shown_items, None, answer, found_answer, note,
+                                        warnings + ["the scout_model cited no lines; the best keyword windows are shown "
+                                                    "instead."], rest + others)
+            except Exception as exc:
+                warnings.append(f"scout_model {model} failed writing the answer ({_project_error(exc)}): literal lines "
+                                "only, no answer.")
+                failure = "answer"
+                items = scout.evidence(root, rows, terms, max_files, scout.LEXICAL_WINDOW_LINES)
+        if text is None:
+            text = scout.render(question, source, items, warnings=warnings, others=others)
+    _metric("scout", source=found["source"], model=bool(model) and failure is None, model_calls=calls, failure=failure,
+            files=len({item["path"] for item in items}), chars=len(text), elapsed_s=round(time.monotonic() - started, 2))
+    return _sanitize_text(text)
+
+
+def _scout_search(root, query, question, max_files):
+    """`{"source", "blocks", "warnings"}` for smart_scout: the smart_search pipeline (structured result) when it answers
+    within SCOUT_SEARCH_WAIT_S; otherwise, or when it fails, a keyword scan of the disk in the searched scope while the
+    search job keeps updating the index in the background. While the first index of the folder is being built the cold
+    path runs at once, outside the per-folder queue, as for a redirected search."""
+    limits = {"code": min(SEARCH_LIMIT_TOTAL - 6, max_files * 2), "test": 3, "doc": 3}
+    search = {"project_root": root, "query": query, "doc_query": question, "group": "all", "top_k": limits,
+              "_structured": True}
+    if _cold_index_building(root):
+        return _perform_smart_search(search, "")
+    job_id, future = _submit_project_job("smart_search", search)
+    deadline = time.monotonic() + SCOUT_SEARCH_WAIT_S
+    if future is not None:
+        with contextlib.suppress(Exception):
+            future.result(timeout=SCOUT_SEARCH_WAIT_S)
+    while True:
+        with _JOBS_LOCK:
+            job = dict(_jobs.get(job_id) or {})
+        if job.get("status") != "pending" or time.monotonic() >= deadline:
+            break
+        time.sleep(0.2)
+    if job.get("status") == "done" and isinstance(job.get("result"), dict):
+        return job["result"]
+    state, warnings = ("pending", []) if job.get("status") == "pending" else (
+        "failed", [f"smart_search failed ({job.get('error') or 'no result'}); keyword scan of the disk instead."])
+    overlay, _notice = _cold_overlay(root, [query])
+    found = indexer.search(root, None, top_k=max_files * 3, query=query, overlay=overlay)
+    return {"source": state, "warnings": warnings,
+            "blocks": {"code": [{"path": path, "lines": f"{start}-{end}", "snippet": text[:500]}
+                                for _score, path, start, end, text in found]}}
+
+
+def _scout_plan_scan(root, question, model):
+    """First model call of smart_scout without an index: the model names identifier terms and folders from the
+    project outline, and the daemon scans those folders (plus the root files) by keyword within COLD_SCAN_BUDGET_S.
+    Returns `(terms, rows)` with the rows ranked like the cold search."""
+    folders, root_files = scout.outline(root)
+    raw = _research_llm_call(model, model_client.get_token(), scout.PLAN_SYSTEM,
+                             scout.plan_prompt(question, folders, root_files), scout.PLAN_TOOL,
+                             timeout=SCOUT_MODEL_TIMEOUT_S)
+    terms, picked = scout.valid_plan(raw, folders)
+    rows, _scanned, _total = _scout_scan(root, picked, " ".join(terms))
+    return terms, rows
+
+
+def _scout_scan(root, include, query):
+    """`(rows, scanned, total)`: the files under the `include` folders (index exclusions applied, root files always
+    in, '' = the whole project) holding a term of `query`, read from disk within COLD_SCAN_BUDGET_S and ranked by
+    keyword like the cold search."""
+    files = index_scope.resolve_included_files(root, {"include": include})[:SCOUT_SCAN_MAX_FILES]
+    chunks, scanned = indexer.keyword_chunks(root, files, [query], COLD_SCAN_BUDGET_S)
+    found = indexer.search(root, None, top_k=scout.MAX_FILES * 2, query=query, overlay={"drop": [], "chunks": chunks})
+    return ([{"path": path, "lines": f"{start}-{end}", "snippet": text[:500]} for _score, path, start, end, text in found],
+            scanned, len(files))
+
+
 def _project_payload(project):
     data = dict(project)
     data['dismissed_duplicates'] = len(data.get('dismissed_duplicates') or {})
@@ -3773,6 +3951,7 @@ _TOOL_HANDLERS = {
     "project_manage": _handle_project_manage,
     "smart_search": _handle_smart_search,
     "smart_search_result": _handle_smart_search_result,
+    "smart_scout": _handle_smart_scout,
     "web_search": _handle_web_search,
     "web_search_result": _handle_web_search_result,
 }

package/usage_meter.py

@@ -11,7 +11,7 @@ import project_store
 
 _CONTEXT=contextvars.ContextVar('smart_usage',default=None)
 _PHASES={'scope':'scope','embedding_query':'query_embedding','model_probe':'embedding_probe',
-         'indexing':'index_embedding','reranking':'rerank'}
+         'indexing':'index_embedding','reranking':'rerank','scout':'scout'}
 
 
 @contextlib.contextmanager

package/version.py

@@ -1,7 +1,7 @@
 """Single source of the Smart Tool version and of the User-Agent sent to public APIs."""
 import re
 
-VERSION = "0.9.29"
+VERSION = "0.9.31"
 _CONTACT_RE = re.compile(r"^(?:[^\s@()<>;]+@[^\s@()<>;]+\.[^\s@()<>;]+|https?://[^\s()<>;]+)$")