@allansantos-dev/smart-tool 0.9.40 → 0.9.41

package/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## 0.9.41 - beta
+
+- Far fewer calls to the embedding and rerank models. Measured on a real machine (24 h): 8303 embedding and 1473 rerank
+  calls, 0.3% of them failing; the volume came from four places, all cut: (1) an indexing job no longer embeds a fixed
+  text to learn the vector size when the index already stores it for the same model (1300 calls a day); (2) embedding
+  batches carry up to 128 chunks and 120,000 characters instead of 32,000 characters (3.8 chunks per call before); a
+  request the gateway refuses for its size is split in two halves once; (3) one rerank call per search instead of one
+  per block (code, test, doc); the blocks keep the order the separate calls gave; (4) a project in `eager` mode whose
+  files change all the time waits longer between jobs (60, 120, 240, 480 up to 900 s, back to 60 s after a quiet
+  period), and `*.tmp`, `*.bak`, `*.swp` and `*~` files no longer mark a project as changed or enter the index.
+  Measurement before and after in `docs/research/data/call-volume-*.json`, findings and sources in
+  `docs/research/2026-10-27-excesso-de-chamadas-embed-rerank.md`. Tests in `tests/test_call_volume.py`.
+- A model that answers 429 or 503 three times in a row opens a circuit for 30 s: calls to it fail at once and the search
+  answers from the lexical index with the usual warning, instead of repeating requests. At most 6 requests are in
+  flight per model. Rerank and the model check wait the gateway's `Retry-After` (or a jittered exponential delay) before
+  asking again, like embeddings.
+- The setup Models page groups the failures by type (429, 503, other 5xx, timeout, connection, authentication, other):
+  one row per type and role with the count, first and last time, one sample message, and the latest 20 lines on
+  expand. The summary shows the calls per purpose (indexing, query, rerank, model check). `dependency_mode` has its own
+  card in the setup rules.
+- The dependency notice also reads `requirements*.txt` (PyPI, `==` pins and the lower bound of `>=` and `~=`) and `go.mod`
+  (Go, `require` lines with a plain `vX.Y.Z`). `web_fetch` reads PDFs with the same page and character caps as the
+  indexer (`document_text`).
+
 ## 0.9.40 - beta
 
 - PDFs of the project are searchable. `smart_search` indexes the text layer of a `.pdf` inside the project scope and

package/README.md

@@ -16,7 +16,7 @@ registered as the `playwright` server where this machine lets a browser be drive
 An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
 are cheaper, or just tells the agent what Smart Tool would do.
 
-**Status:** 0.9.40 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
+**Status:** 0.9.41 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
 
 ## How it works
 
@@ -129,7 +129,7 @@ Redirects) and Maintenance (Setup audit, Startup).
    When a new version recommends other models than the ones you use, a Windows notification (**Apply now**, **Don't
    show again**) and the **New recommended models** card on this page offer them: nothing changes until you apply, the
    models you had are kept, and **Switch back to my previous models** undoes it. The **Model failures (last 24 h)** table
-   lists every failed gateway call by role, model and tool; a model that fails half of its calls (5 or more) also raises a
+   groups the failed gateway calls by type (429, 503, other 5xx, timeout, connection, authentication) and role, with the count, the first and last time and one sample, and shows the calls per purpose; a model that fails half of its calls (5 or more) also raises a
    notification and a tray alert. A failure inside a tool call also comes back as a warning line in the tool answer;
    calls made by a hook or by background work (index scope and profile, block-pattern proposals) have no answer to
    carry it, so they show only in the table, the log and the notification. The local embedder is not in that table or in
@@ -189,7 +189,7 @@ Redirects) and Maintenance (Setup audit, Startup).
    A documented function edited only in its body is not mentioned. `project_manage` with `action=docs` lists the
    public functions still without a docstring, for documenting a project that started without them.
 
-   **Dependency check on edit**: an edit of `package.json` that adds or changes a dependency lists the OSV advisory IDs of up to 5 of them (exact version, or the lower bound of `^`, `~` and `>=`), answered from a 6 h disk cache or one 2.5 s batch call that sends only package names and versions; when OSV cannot answer, the reminder says the check was skipped and the edit goes on. `dependency_mode` in `config.json`: `warn` (default) or `off`.
+   **Dependency check on edit**: an edit of `package.json`, `requirements*.txt` or `go.mod` that adds or changes a dependency lists the OSV advisory IDs of up to 5 of them (exact version, or the lower bound of `^`, `~`, `~=` and `>=`), answered from a 6 h disk cache or one 2.5 s batch call that sends only package names and versions; when OSV cannot answer, the reminder says the check was skipped and the edit goes on. `dependency_mode` in `config.json`: `warn` (default) or `off`.
 
    **PDFs** inside a project are indexed as text with `[page N]` markers (20 MiB, 300 pages and 20 s per file; a scanned, password-protected or corrupt PDF is skipped with the reason), so `smart_search` returns the page that holds a passage.
 

package/config.py

@@ -62,7 +62,7 @@ DOC_MODES = ("require", "remind", "off")
 # Edit hook on functions that copy one already indexed (exact or near-identical body): warn = let the edit run and
 # point to the existing function; off = say nothing. Never blocks: near matches are leads, not defects.
 DUPLICATE_MODES = ("warn", "off")
-# Edit hook on package.json: warn = let the edit run and list the added or changed dependencies that have OSV advisories;
+# Edit hook on package.json, requirements*.txt and go.mod: warn = let the edit run and list the added or changed dependencies that have OSV advisories;
 # off = say nothing and make no network call.
 DEPENDENCY_MODES = ("warn", "off")
 # Router model reviewing, in the background, searches the deterministic redirect rule let run: on = propose block

package/dependency_notice.py

@@ -1,10 +1,12 @@
-"""Vulnerable-dependency notice for an edit of package.json, built before the edit runs.
+"""Vulnerable-dependency notice for an edit of a dependency manifest, built before the edit runs.
 
-Compares the dependency tables of the file before and after the edit, asks OSV (one `querybatch` call, short timeout)
-about the added or changed ones only, and answers from a disk cache of (package, version) -> advisory IDs for 6 hours.
-Only package names and versions leave the machine. The caller (`hook_decision`) turns a `DependencyCheckError` into
+Manifests: package.json (OSV ecosystem npm), requirements*.txt (PyPI: `==` pins and the lower bound of `>=` and `~=`)
+and go.mod (Go: `require` lines with a plain vX.Y.Z). Compares the dependencies of the file before and after the edit,
+asks OSV (one `querybatch` call, short timeout) about the added or changed ones only, and answers from a disk cache of
+(ecosystem, package, version) -> advisory IDs for 6 hours. Only package names and versions leave the machine. The caller (`hook_decision`) turns a `DependencyCheckError` into
 "dependency check skipped: <reason>" and lets the edit go on.
 """
+import collections
 import json
 import os
 import re
@@ -23,13 +25,17 @@ TABLES = ("dependencies", "devDependencies", "peerDependencies", "optionalDepend
 PREFIX = "Smart Tool dependencies: "
 _SPECIFIER_RE = re.compile(r"(\^|~|>=|>|=)?\s*v?(\d+)(?:\.(\d+))?(?:\.(\d+))?"
                            r"((?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?)")
+_REQUIREMENTS_FILE_RE = re.compile(r"requirements[A-Za-z0-9._-]*\.txt")
+_REQUIREMENT_RE = re.compile(r"([A-Za-z0-9][A-Za-z0-9._-]*)\s*(?:\[[^\]]*\])?\s*([^;]*)")
+_PIP_VERSION_RE = re.compile(r"[0-9][0-9A-Za-z.+!_-]*")
+_GO_VERSION_RE = re.compile(r"v\d+\.\d+\.\d+")
 
 
 class DependencyCheckError(RuntimeError):
     """The check could not run (OSV unreachable or answering badly, unreadable cache, current file not JSON)."""
 
 
-def _tables(text):
+def _npm_pairs(text):
     """The (name, specifier) pairs of every dependency table in a package.json text.
 
     Parameters: text (str) file content.
@@ -49,32 +55,91 @@ def _tables(text):
     return pairs
 
 
-def changed_dependencies(before, after):
-    """Dependencies added or whose specifier changed between two package.json texts.
+def _requirements_pairs(text):
+    """The (name, specifier) pairs of a requirements.txt text; options, includes, editable and URL lines are skipped.
 
-    Parameters: before (str or None) text before the edit, None when the file did not exist; after (str) text after.
-    Returns: list of (name, specifier); empty when `after` is not valid JSON (the user is mid-edit).
-    Raises: DependencyCheckError when `before` exists but is not valid JSON.
+    Parameters: text (str) file content.
+    Returns: dict (name, specifier) -> None in file order; the specifier is the version clauses without spaces ("" when
+    the line only names the package).
+    """
+    pairs = {}
+    for raw in text.splitlines():
+        line = raw.split(" #")[0].split(";")[0].strip().rstrip("\\").strip()
+        if not line or line.startswith(("#", "-")):
+            continue
+        line = " ".join(word for word in line.split() if not word.startswith("--"))
+        match = _REQUIREMENT_RE.fullmatch(line)
+        if not match:
+            continue
+        specifier = "".join(match.group(2).split())
+        if specifier and specifier[0] not in "=<>!~":
+            continue
+        pairs[(match.group(1), specifier)] = None
+    return pairs
+
+
+def _go_pairs(text):
+    """The (module, version) pairs of the `require` lines of a go.mod text, single-line and block; replace, exclude and
+    the other directives are skipped.
+
+    Parameters: text (str) file content.
+    Returns: dict (module, version) -> None in file order.
     """
-    now = _tables(after)
+    pairs, block = {}, None
+    for raw in text.splitlines():
+        tokens = raw.split("//")[0].split()
+        if not tokens:
+            continue
+        if block is None:
+            if tokens[0] == "require" and len(tokens) == 3:
+                pairs[(tokens[1], tokens[2])] = None
+            elif tokens[-1] == "(":
+                block = tokens[0]
+        elif tokens == [")"]:
+            block = None
+        elif block == "require" and len(tokens) == 2:
+            pairs[(tokens[0], tokens[1])] = None
+    return pairs
+
+
+def ecosystem_of(path):
+    """The OSV ecosystem of a manifest file name.
+
+    Parameters: path (str) file path.
+    Returns: "npm" for package.json, "PyPI" for requirements*.txt, "Go" for go.mod; None for any other file.
+    """
+    name = os.path.basename(path)
+    if name == "package.json":
+        return "npm"
+    if name == "go.mod":
+        return "Go"
+    return "PyPI" if _REQUIREMENTS_FILE_RE.fullmatch(name) else None
+
+
+def changed_dependencies(before, after, ecosystem="npm"):
+    """Dependencies added or whose specifier changed between two manifest texts.
+
+    Parameters: before (str or None) text before the edit, None when the file did not exist; after (str) text after;
+    ecosystem (str) one of ECOSYSTEMS.
+    Returns: list of (name, specifier); empty when `after` is a package.json that is not valid JSON (mid-edit).
+    Raises: DependencyCheckError when `before` exists but is a package.json that is not valid JSON.
+    """
+    parse = _ECOSYSTEMS[ecosystem].parse
+    now = parse(after)
     if now is None:
         return []
     if before is None or not before.strip():
         old = {}
     else:
-        old = _tables(before)
+        old = parse(before)
         if old is None:
             raise DependencyCheckError("current package.json is not valid JSON")
     return [pair for pair in now if pair not in old]
 
 
-def queried_version(specifier):
-    """The version to ask OSV about for a specifier.
-
-    Parameters: specifier (str) the value in the dependency table.
-    Returns: str, the exact version or the lower bound of a ^, ~, >= or > range; None when the specifier cannot be
-    reduced safely (tags, `*`, URLs, file:, git, workspace:, npm: aliases, x-ranges, compound ranges).
-    """
+def _npm_version(specifier):
+    """The npm case of `queried_version`: the exact version or the lower bound of a ^, ~, >= or > range; None for tags,
+    `*`, URLs, file:, git, workspace:, npm: aliases, x-ranges and compound ranges."""
     match = _SPECIFIER_RE.fullmatch(specifier.strip())
     if not match:
         return None
@@ -87,6 +152,42 @@ def queried_version(specifier):
     return ".".join(parts) + (tail or "")
 
 
+def _pip_version(specifier):
+    """The PyPI case of `queried_version`: the `==` version, else the first `>=` or `~=` lower bound; None for no
+    specifier, `===`, wildcards, `<`, `>`, `!=` alone and anything that is not a version."""
+    clauses = specifier.split(",")
+    for operators in (("==",), (">=", "~=")):
+        for clause in clauses:
+            for operator in operators:
+                if clause.startswith(operator) and _PIP_VERSION_RE.fullmatch(clause[len(operator):]):
+                    return clause[len(operator):]
+    return None
+
+
+def _go_version(specifier):
+    """The Go case of `queried_version`: a plain vX.Y.Z; None for pseudo-versions, pre-releases and +incompatible."""
+    return specifier if _GO_VERSION_RE.fullmatch(specifier) else None
+
+
+def queried_version(specifier, ecosystem="npm"):
+    """The version to ask OSV about for a specifier.
+
+    Parameters: specifier (str) the value in the dependency table; ecosystem (str) one of ECOSYSTEMS.
+    Returns: str, the exact version or the lower bound of the range; None when the specifier cannot be reduced safely.
+    """
+    return _ECOSYSTEMS[ecosystem].version(specifier)
+
+
+_Ecosystem = collections.namedtuple("_Ecosystem", "parse version label exact")
+_ECOSYSTEMS = {
+    "npm": _Ecosystem(_npm_pairs, _npm_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
+    "PyPI": _Ecosystem(_requirements_pairs, _pip_version, lambda name, spec: f"{name}{spec}",
+                       lambda spec, version: spec == "==" + version),
+    "Go": _Ecosystem(_go_pairs, _go_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
+}
+ECOSYSTEMS = tuple(_ECOSYSTEMS)
+
+
 def _load_cache():
     """The cache as {"name@version": {"ids": [...], "at": epoch}}; empty when the file does not exist.
 
@@ -111,26 +212,27 @@ def _fresh_ids(entry, now):
     return entry["ids"] if now - entry["at"] < CACHE_TTL_S else None
 
 
-def _advisories(queries):
+def _advisories(ecosystem, queries):
     """Advisory IDs for (name, version) pairs, from the cache and one OSV batch for the rest.
 
-    Parameters: queries (list of (name, version)).
+    Parameters: ecosystem (str) one of ECOSYSTEMS; queries (list of (name, version)).
     Returns: tuple (list of IDs or None per query in order, str cache-write problem or "").
     Raises: DependencyCheckError when the cache is corrupt or OSV cannot answer.
     """
     cache, now = _load_cache(), time.time()
-    found = {query: _fresh_ids(cache.get("@".join(query)), now) for query in queries}
+    key = lambda query: f"{ecosystem}:{query[0]}@{query[1]}"  # noqa: E731
+    found = {query: _fresh_ids(cache.get(key(query)), now) for query in queries}
     missing = [query for query in queries if found[query] is None]
     if not missing:
         return [found[query] for query in queries], ""
     try:
-        answered = freshness.npm_advisories_batch(missing, OSV_TIMEOUT_S)
+        answered = freshness.advisories_batch(ecosystem, missing, OSV_TIMEOUT_S)
     except RuntimeError as exc:
         raise DependencyCheckError(str(exc)) from exc
     for query, ids in zip(missing, answered):
         found[query] = ids
         if ids is not None:
-            cache["@".join(query)] = {"ids": ids, "at": now}
+            cache[key(query)] = {"ids": ids, "at": now}
     cache = {key: entry for key, entry in cache.items() if _fresh_ids(entry, now) is not None}
     try:
         os.makedirs(os.path.dirname(CACHE_PATH), exist_ok=True)
@@ -140,28 +242,30 @@ def _advisories(queries):
     return [found[query] for query in queries], ""
 
 
-def notice(before, after):
-    """The dependency notice for one package.json edit.
+def notice(before, after, ecosystem="npm"):
+    """The dependency notice for one manifest edit.
 
-    Parameters: before (str or None) file text before the edit; after (str) text after it.
+    Parameters: before (str or None) file text before the edit; after (str) text after it; ecosystem (str) one of
+    ECOSYSTEMS, from `ecosystem_of`.
     Returns: str, "Smart Tool dependencies: ..." lines for the changed dependencies that have advisories or could not
     be verified, "" when there is nothing to say.
     Raises: DependencyCheckError when the check could not run.
     """
-    changed = changed_dependencies(before, after)
+    spec = _ECOSYSTEMS[ecosystem]
+    changed = changed_dependencies(before, after, ecosystem)
     handled, extra = changed[:MAX_PACKAGES], len(changed) - MAX_PACKAGES
     lines, queries = {}, []
     for name, specifier in handled:
-        version = queried_version(specifier) if freshness.NPM_NAME_RE.fullmatch(name) else None
+        version = spec.version(specifier) if freshness.NAME_RES[ecosystem].fullmatch(name) else None
         if version is None:
-            lines[(name, specifier)] = f"`{name}@{specifier}` specifier not verified."
+            lines[(name, specifier)] = f"`{spec.label(name, specifier)}` specifier not verified."
         else:
             queries.append((name, specifier, version))
     problem = ""
     if queries:
-        ids, problem = _advisories([(name, version) for name, _specifier, version in queries])
+        ids, problem = _advisories(ecosystem, [(name, version) for name, _specifier, version in queries])
         for (name, specifier, version), advisories in zip(queries, ids):
-            label = f"`{name}@{specifier}`" + ("" if version == specifier else f" (checked {version})")
+            label = f"`{spec.label(name, specifier)}`" + ("" if spec.exact(specifier, version) else f" (checked {version})")
             if advisories is None:
                 lines[(name, specifier)] = f"{label} advisory check incomplete, not verified."
             elif advisories:

package/freshness.py

@@ -73,11 +73,13 @@ def npm_registry_evidence(package):
 
 
 NPM_NAME_RE = re.compile(r"(?:@[a-z0-9._-]+/)?[a-z0-9._-]+")
+NAME_RES = {"npm": NPM_NAME_RE, "PyPI": re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*"),
+            "Go": re.compile(r"[A-Za-z0-9][A-Za-z0-9._~/-]*")}
 
 
-def _osv_query(package, package_version):
-    """One OSV query object: the package (npm) and the version asked about."""
-    return {"package": {"name": package, "ecosystem": "npm"}, "version": package_version}
+def _osv_query(ecosystem, package, package_version):
+    """One OSV query object: the package in `ecosystem` and the version asked about."""
+    return {"package": {"name": package, "ecosystem": ecosystem}, "version": package_version}
 
 
 def _osv_request(endpoint, payload):
@@ -107,7 +109,7 @@ def npm_advisories(package, package_version):
     """IDs conhecidos pela OSV para a versão; None significa checagem indisponível."""
     if not package or not package_version or not NPM_NAME_RE.fullmatch(package):
         return None
-    request = _osv_request("query", _osv_query(package, package_version))
+    request = _osv_request("query", _osv_query("npm", package, package_version))
     try:
         with urllib.request.urlopen(request, timeout=7) as response:
             return _osv_ids(json.load(response))
@@ -115,17 +117,21 @@ def npm_advisories(package, package_version):
         return None
 
 
-def npm_advisories_batch(packages, timeout):
+def advisories_batch(ecosystem, packages, timeout):
     """Advisory IDs for each (package, version) of `packages` from one OSV `querybatch` call.
 
-    Parameters: packages (list of (name, version)); timeout (float) seconds for the request.
+    Parameters: ecosystem (str) "npm", "PyPI" or "Go"; packages (list of (name, version)); timeout (float) seconds for
+    the request.
     Returns: list in the same order, each the sorted ID list or None when that package's answer is incomplete
     (next_page_token) or malformed, so it is unverified and not clean.
-    Raises: RuntimeError when the service cannot be reached, answers badly or answers a different number of results.
+    Raises: ValueError for an unknown ecosystem or an invalid name or version; RuntimeError when the service cannot be
+    reached, answers badly or answers a different number of results.
     """
-    if any(not NPM_NAME_RE.fullmatch(name) or not package_version for name, package_version in packages):
-        raise ValueError(f"Invalid npm package query: {packages!r}")
-    request = _osv_request("querybatch", {"queries": [_osv_query(name, package_version)
+    if ecosystem not in NAME_RES:
+        raise ValueError(f"Unknown OSV ecosystem: {ecosystem!r}")
+    if any(not NAME_RES[ecosystem].fullmatch(name) or not package_version for name, package_version in packages):
+        raise ValueError(f"Invalid {ecosystem} package query: {packages!r}")
+    request = _osv_request("querybatch", {"queries": [_osv_query(ecosystem, name, package_version)
                                                       for name, package_version in packages]})
     try:
         with urllib.request.urlopen(request, timeout=timeout) as response:

package/hook_decision.py

@@ -230,20 +230,21 @@ def _edit_review(payload, doc_mode, duplicate_mode, dependency_mode):
     """Checks of an edit before it runs. Docstrings follow doc_mode, in code files of registered projects only (tests,
     docs and files outside a project are left out, as in the docs coverage): missing ones deny the edit in require mode and become a reminder in remind mode; a
     documented function whose signature changed always gets a reminder. Copies of indexed functions follow
-    duplicate_mode and only add context, never deny. An edit of a package.json follows dependency_mode: advisories
+    duplicate_mode and only add context, never deny. An edit of a package.json, requirements*.txt or go.mod follows dependency_mode: advisories
     of the added or changed dependencies are added as context, never a denial."""
     with code_graph.pending_scope():
         return _edit_review_scoped(payload, doc_mode, duplicate_mode, dependency_mode)
 
 
-def _dependency_part(before, after):
-    """The dependency notice of one package.json edit, or the "dependency check skipped" line when it cannot run.
+def _dependency_part(before, after, ecosystem):
+    """The dependency notice of one manifest edit, or the "dependency check skipped" line when it cannot run.
 
-    Parameters: before (str or None) text before the edit; after (str) text after it.
+    Parameters: before (str or None) text before the edit; after (str) text after it; ecosystem (str) from
+    `dependency_notice.ecosystem_of`.
     Returns: str, empty when there is nothing to say.
     """
     try:
-        return dependency_notice.notice(before, after)
+        return dependency_notice.notice(before, after, ecosystem)
     except dependency_notice.DependencyCheckError as exc:
         return f"{dependency_notice.PREFIX}dependency check skipped: {exc}"
 
@@ -252,8 +253,9 @@ def _edit_review_scoped(payload, doc_mode, duplicate_mode, dependency_mode):
     """`_edit_review` inside one `code_graph.pending_scope`, so the pending-changes scan runs once per review."""
     doc_findings, duplicate_findings, failures, dependency_parts = [], [], [], []
     for path, before, after in edit_preview.preview(payload.get("tool_name"), payload.get("tool_input"), payload.get("cwd")):
-        if dependency_mode != "off" and os.path.basename(path) == "package.json":
-            dependency_parts += [part for part in (_dependency_part(before, after),) if part]
+        ecosystem = dependency_notice.ecosystem_of(path)
+        if dependency_mode != "off" and ecosystem:
+            dependency_parts += [part for part in (_dependency_part(before, after, ecosystem),) if part]
             continue
         if not doc_check.supported(path):
             continue

package/index_scope.py

@@ -40,10 +40,13 @@ _PROJECT_TYPE_EXCLUDES = (
     (("Podfile",), ("Pods",)),
 )
 
+# Files a tool writes while it works and replaces or deletes right after (atomic-write temp, backup, editor swap).
+TRANSIENT_FILES = {"*.tmp", "*.bak", "*.swp", "*~"}
+
 # Mesmo critério do `_ALWAYS_EXCLUDE`, para arquivo: o significado não varia por
 # projeto. Lockfile é grafo de dependência resolvido por ferramenta — nunca é fonte de
 # resposta numa busca semântica, e é grande o bastante pra dominar o índice.
-_ALWAYS_EXCLUDE_FILES = {
+_ALWAYS_EXCLUDE_FILES = TRANSIENT_FILES | {
     "package-lock.json", "yarn.lock", "pnpm-lock.yaml", "npm-shrinkwrap.json",
     "poetry.lock", "pdm.lock", "uv.lock", "Pipfile.lock", "Cargo.lock",
     "composer.lock", "go.sum", "Gemfile.lock", "packages.lock.json",
@@ -110,6 +113,11 @@ def normalize_include(include):
     return bases, dropped
 
 
+def is_transient(name):
+    """Whether a file name is a temporary, backup or editor-swap file (`*.tmp`, `*.bak`, `*.swp`, `*~`)."""
+    return any(fnmatch.fnmatch(name.casefold(), pattern) for pattern in TRANSIENT_FILES)
+
+
 def always_excluded(name):
     """Directory name in _ALWAYS_EXCLUDE, entries may be globs (e.g. *.egg-info)."""
     folded = name.casefold()

package/indexer.py

@@ -31,8 +31,10 @@ CHUNK_MAX_CHARS = 8000
 CHUNK_OVERLAP_LINES = 12
 CHUNKER_VERSION = 3
 SCHEMA_VERSION = 2
-REINDEX_BATCH_MAX_CHARS = 32_000
-REINDEX_BATCH_MAX_ITEMS = 32
+# An embeddings request holds up to 128 items and 120,000 characters (about 30,000 tokens): far under the 2048 items and
+# 300,000 tokens per request that OpenAI and Azure accept (docs/research/2026-10-27-excesso-de-chamadas-embed-rerank.md).
+REINDEX_BATCH_MAX_CHARS = 120_000
+REINDEX_BATCH_MAX_ITEMS = 128
 # Embedding requests in flight per flush. Measured 2026-10-03 (docs/spikes/embed_parallel_spike.py, 30 batches,
 # 3 interleaved rounds): 1 worker 33-50 s, 2 workers 19-27 s, 4 workers 11-20 s (~2.8x), no 429 from the gateway.
 EMBED_WORKERS = 4

package/model_breaker.py

@@ -0,0 +1,70 @@
+"""Protection of the model gateway, per model id: a circuit that opens after consecutive 429/503 answers and a cap on
+the requests in flight.
+
+Pattern: a circuit breaker (closed, open for a pause, then one trial call decides). Retrying a gateway that answers
+busy only adds load (AWS Bedrock throughput guide: honor Retry-After, back off with jitter); once it keeps answering
+busy the caller stops asking for a while and serves what it can without the model.
+"""
+import contextlib
+import threading
+import time
+import urllib.error
+
+FAILURES_TO_OPEN = 3
+PAUSE_S = 30
+MAX_IN_FLIGHT = 6
+_BUSY_CODES = (429, 503)
+_lock = threading.Lock()
+_failures = {}
+_open_until = {}
+_slots = {}
+
+
+class CircuitOpen(RuntimeError):
+    """The model is paused after repeated busy answers; no request was sent."""
+
+
+def guard(model):
+    """Raises CircuitOpen while `model` is paused. Nothing happens for a call without a model id."""
+    if not model:
+        return
+    with _lock:
+        left = _open_until.get(model, 0) - time.monotonic()
+    if left > 0:
+        raise CircuitOpen(f"Model {model} is paused for {int(left) + 1} s after {FAILURES_TO_OPEN} consecutive "
+                          "HTTP 429/503 answers from the gateway; no request was sent.")
+
+
+def note(model, failure=None):
+    """Records the outcome of a request: a success closes the circuit, a 429/503 counts toward opening it, any other
+    failure changes nothing. After the pause the first failure opens it again."""
+    if not model:
+        return
+    busy = isinstance(failure, urllib.error.HTTPError) and failure.code in _BUSY_CODES
+    with _lock:
+        if failure is None:
+            _failures[model] = 0
+        elif busy:
+            _failures[model] = _failures.get(model, 0) + 1
+            if _failures[model] >= FAILURES_TO_OPEN:
+                _open_until[model] = time.monotonic() + PAUSE_S
+
+
+@contextlib.contextmanager
+def slot(model):
+    """Holds one of the `MAX_IN_FLIGHT` places of `model` for the duration of a request."""
+    if not model:
+        yield
+        return
+    with _lock:
+        semaphore = _slots.setdefault(model, threading.BoundedSemaphore(MAX_IN_FLIGHT))
+    with semaphore:
+        yield
+
+
+def reset():
+    """Forgets every circuit and cap (tests)."""
+    with _lock:
+        _failures.clear()
+        _open_until.clear()
+        _slots.clear()

package/model_client.py

@@ -9,6 +9,7 @@ import usage_meter
 
 import config
 import gateway
+import model_breaker
 import model_health
 
 OFFLINE = contextvars.ContextVar("model_offline", default=None)
@@ -26,18 +27,24 @@ def get_token():
 
 
 def fetch(path, token, method="GET", body=None, timeout=15):
+    """One gateway request; returns the parsed JSON. A call that names a model first passes that model's circuit and
+    request cap (model_breaker, CircuitOpen when paused), and every attempt is metered. HTTP errors propagate."""
     if OFFLINE.get():
         raise RuntimeError(OFFLINE.get())
-    started=time.monotonic();result=None;headers=None;failure=None
-    try:
-        result,headers=_fetch(path,token,method,body,timeout)
-        return result
-    except Exception as exc:
-        failure=exc
-        raise
-    finally:
-        usage_meter.record(path,body,result,headers,failure,time.monotonic()-started)
-        model_health.record(path,body,failure)
+    model=body.get("model") if isinstance(body,dict) else None
+    model_breaker.guard(model)
+    with model_breaker.slot(model):
+        started=time.monotonic();result=None;headers=None;failure=None
+        try:
+            result,headers=_fetch(path,token,method,body,timeout)
+            return result
+        except Exception as exc:
+            failure=exc
+            raise
+        finally:
+            model_breaker.note(model,failure)
+            usage_meter.record(path,body,result,headers,failure,time.monotonic()-started)
+            model_health.record(path,body,failure)
 
 
 def _fetch(path, _token, method="GET", body=None, timeout=15):

package/model_health.py

@@ -3,17 +3,19 @@ rerank). Never stores request content: the model, the role, the tool that was ru
 text and the time.
 
 Three consumers read it: the tool answer (a warning line when a call of that very tool failed), the setup screen
-(rate table and recent failures) and the tray (one toast per model per day when the rate passes the limit).
+(rate table, failures by type and calls per purpose) and the tray (one toast per model per day when the rate passes the limit).
 """
 import contextlib
 import contextvars
 import io
 import os
+import re
 import sqlite3
 import time
 
 import config
 import project_store
+import usage_meter
 
 WINDOW_S = 24 * 3600
 ALERT_MIN_CALLS = 5
@@ -23,10 +25,17 @@ MUTE_S = 7 * 24 * 3600
 KEEP_S = 7 * 24 * 3600
 KEEP_FAILURES = 500
 ERROR_CHARS = 300
+SAMPLE_CHARS = 160
+GROUP_ROWS = 20
+PURPOSES = ("index_embedding", "query_embedding", "rerank", "embedding_probe", "scope")
 _PATHS = ("/v1/chat/completions", "/v1/embeddings", "/v1/rerank")
 _TOOL = contextvars.ContextVar("model_health_tool", default=None)
 _COLLECTED = contextvars.ContextVar("model_health_collected", default=None)
 _SCHEMA = set()
+_STATUS = re.compile(r"HTTP(?: Error)? (\d{3})")
+_TIMEOUT = re.compile(r"timed out|timeout|WinError 10060|Errno 110\b", re.I)
+_CONNECTION = re.compile(r"URLError|ConnectionError|ConnectionRefused|ConnectionReset|refused|reset|getaddrinfo|"
+                         r"WinError 1006[01]|WinError 10054|Name or service|unreachable", re.I)
 
 
 @contextlib.contextmanager
@@ -56,6 +65,22 @@ def _tables(db):
     _SCHEMA.add(project_store.DB_PATH)
 
 
+def classify(error):
+    """Failure type of a stored error text: '429', '503', '5xx' (any other 5xx), 'auth' (401/403), 'timeout',
+    'connection' or 'other'. An HTTP status in the text wins over words in the response body."""
+    status = _STATUS.search(error or "")
+    if status:
+        code = int(status.group(1))
+        if code in (429, 503):
+            return str(code)
+        if code in (401, 403):
+            return "auth"
+        return "5xx" if code >= 500 else "other"
+    if _TIMEOUT.search(error or ""):
+        return "timeout"
+    return "connection" if _CONNECTION.search(error or "") else "other"
+
+
 def role_of(model, cfg=None):
     """Name of the config field that selects `model` ('other' when none does)."""
     cfg = cfg if cfg is not None else config.load_config()
@@ -111,8 +136,24 @@ def warning_text(failures, setup_url):
             f"{first['error']}. Pick another model: {setup_url}/setup#models")
 
 
+def _groups(rows):
+    """Failure rows (latest first) as one entry per (type, role): count, first and last time, the latest message cut to
+    SAMPLE_CHARS and the latest GROUP_ROWS raw rows. Largest group first."""
+    groups = {}
+    for row in rows:
+        group = groups.setdefault((classify(row["error"]), row["role"]), {
+            "type": classify(row["error"]), "role": row["role"], "count": 0, "first_at": row["at"], "last_at": row["at"],
+            "sample": row["error"][:SAMPLE_CHARS], "rows": []})
+        group["count"] += 1
+        group["first_at"] = row["at"]
+        if len(group["rows"]) < GROUP_ROWS:
+            group["rows"].append(row)
+    return sorted(groups.values(), key=lambda group: (-group["count"], -group["last_at"]))
+
+
 def summary(now=None):
-    """Per model and role over the last 24 h, plus the 20 latest failures."""
+    """Per model and role over the last 24 h, every kept failure of the window grouped by type and role, and the calls
+    per purpose of the usage meter (its days are UTC, so that part covers the window's days, not its exact hours)."""
     now = now or time.time()
     with project_store._LOCK, _db() as db:
         _tables(db)
@@ -121,7 +162,7 @@ def summary(now=None):
             "SELECT model,role,SUM(calls) AS calls,SUM(errors) AS errors FROM model_calls WHERE hour>=? "
             "GROUP BY model,role ORDER BY SUM(errors) DESC,SUM(calls) DESC", (int((now - WINDOW_S) // 3600),))]
         failures = [dict(row) for row in db.execute(
-            "SELECT at,model,role,tool,path,error FROM model_failures WHERE at>=? ORDER BY at DESC LIMIT 20", (now - WINDOW_S,))]
+            "SELECT at,model,role,tool,path,error FROM model_failures WHERE at>=? ORDER BY at DESC", (now - WINDOW_S,))]
         for row in models:
             last = db.execute("SELECT error FROM model_failures WHERE model=? AND role=? ORDER BY at DESC LIMIT 1",
                               (row["model"], row["role"])).fetchone()
@@ -129,7 +170,8 @@ def summary(now=None):
     for row in models:
         row["rate"] = round(row["errors"] / row["calls"], 3) if row["calls"] else 0.0
         row["failing"] = row["calls"] >= ALERT_MIN_CALLS and row["rate"] >= ALERT_RATE
-    return {"window_hours": WINDOW_S // 3600, "models": models, "failures": failures}
+    return {"window_hours": WINDOW_S // 3600, "models": models, "groups": _groups(failures),
+            "calls_by_purpose": usage_meter.calls_by_purpose(now - WINDOW_S, PURPOSES)}
 
 
 def alerts_due(now=None):

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@allansantos-dev/smart-tool",
-  "version": "0.9.40",
+  "version": "0.9.41",
   "description": "Local MCP server that gives coding agents (Claude Code, Codex) cheaper, sharper tools than their built-in search. Windows.",
   "license": "Apache-2.0",
   "author": "Allan Santos",

package/project_monitor.py

@@ -20,6 +20,8 @@ IDLE_TICK_S = 5
 # A folder that changes all the time (a tool's state folder) is indexed at most this often by the monitor; the changes
 # keep accumulating and one job covers them, and a search meanwhile reads the stale files straight from disk.
 MIN_JOB_GAP_S = 60
+# While the changes do not stop, each job lengthens the wait for the next one; a wait that passes unused resets it.
+JOB_GAP_STEPS_S = (MIN_JOB_GAP_S, 120, 240, 480, 900)
 FULL_CHECK_S = 86400
 MAX_EVENT_PATHS = 1000
 
@@ -27,7 +29,8 @@ MAX_EVENT_PATHS = 1000
 def relevant_path(path):
     parts = path.replace("\\", "/").strip("/").split("/")
     return bool(parts and all(part not in ("", ".", "..") and not part.startswith(".")
-                              and not index_scope.always_excluded(part) for part in parts))
+                              and not index_scope.always_excluded(part) for part in parts)
+                and not index_scope.is_transient(parts[-1]))
 
 
 def snapshot(root, scope):
@@ -58,6 +61,7 @@ class ProjectMonitor:
         self._git_checked = {}
         self._pending = {}
         self._next_job = {}
+        self._gap_level = {}
         self._filters = {}
         self._views = {}
         self._lock = threading.RLock()
@@ -155,6 +159,17 @@ class ProjectMonitor:
             project_store.update(key, watch_backend="windows_events", watch_error="")
             watcher.start()
 
+    def _job_gap(self, key, now):
+        """Wait before the next job of a project, called when a job is enqueued at `now`: one step longer than the last
+        one when the previous wait was used up at once (at most twice its length went by), the first step otherwise."""
+        level, at = self._gap_level.get(key, (0, None))
+        if at is None or now - at >= 2 * JOB_GAP_STEPS_S[level]:
+            level = 0
+        else:
+            level = min(level + 1, len(JOB_GAP_STEPS_S) - 1)
+        self._gap_level[key] = (level, now)
+        return JOB_GAP_STEPS_S[level]
+
     def tick(self):
         now = time.monotonic()
         projects = project_store.all_projects()
@@ -223,7 +238,7 @@ class ProjectMonitor:
             except Exception as exc:
                 project_store.update(key, status="error", last_error=str(exc)[:300], next_retry_at=time.time() + 60)
                 continue
-            self._next_job[key] = now + MIN_JOB_GAP_S
+            self._next_job[key] = now + self._job_gap(key, now)
             with self._lock:
                 # Um evento recebido durante o submit continua pendente para a próxima geração.
                 if self._pending.get(key) is pending and pending.get("revision", 0) == revision:

package/setup_ui.py

@@ -255,6 +255,10 @@ def handle_integration_get():
         duplicate_mode, duplicate_error = config.duplicate_mode(), ""
     except ValueError as exc:
         duplicate_mode, duplicate_error = "", str(exc)
+    try:
+        dependency_mode, dependency_error = config.dependency_mode(), ""
+    except ValueError as exc:
+        dependency_mode, dependency_error = "", str(exc)
     try:
         proposals_mode, proposals_error = config.pattern_proposals(), ""
     except ValueError as exc:
@@ -267,6 +271,8 @@ def handle_integration_get():
             "hook_modes": list(config.HOOK_MODES), "doc_mode": doc_mode, "doc_mode_error": doc_error,
             "doc_modes": list(config.DOC_MODES), "duplicate_mode": duplicate_mode,
             "duplicate_mode_error": duplicate_error, "duplicate_modes": list(config.DUPLICATE_MODES),
+            "dependency_mode": dependency_mode, "dependency_mode_error": dependency_error,
+            "dependency_modes": list(config.DEPENDENCY_MODES),
             "pattern_proposals": proposals_mode, "pattern_proposals_error": proposals_error,
             "block_patterns": {"proposals": stored["proposals"], "accepted": stored["accepted"],
                                "rejected": len(stored["rejected"]), "error": patterns_error},
@@ -285,10 +291,11 @@ def handle_integration_save(body):
         except (OSError, ValueError) as exc:
             raise SetupError(str(exc))
         return handle_integration_get()
-    if isinstance(body, dict) and body.get("action") in ("hook_mode", "doc_mode", "duplicate_mode", "pattern_proposals"):
+    if isinstance(body, dict) and body.get("action") in ("hook_mode", "doc_mode", "duplicate_mode", "dependency_mode", "pattern_proposals"):
         action = body["action"]
         allowed = {"hook_mode": config.HOOK_MODES, "doc_mode": config.DOC_MODES,
-                   "duplicate_mode": config.DUPLICATE_MODES, "pattern_proposals": config.PATTERN_PROPOSAL_MODES}[action]
+                   "duplicate_mode": config.DUPLICATE_MODES, "dependency_mode": config.DEPENDENCY_MODES,
+                   "pattern_proposals": config.PATTERN_PROPOSAL_MODES}[action]
         if body.get("mode") not in allowed:
             raise SetupError(f"Invalid mode: use {', '.join(allowed)}.")
         try:
@@ -740,7 +747,7 @@ _PAGE_TEMPLATE = """<!doctype html>
 
       <div class="card" id="health-card">
         <div class="card-head"><h3>Model failures (last 24 h)</h3><span class="pill" id="health-pill" data-tone="busy">Checking</span></div>
-        <p class="lede">Every failed call to a gateway model, by role and tool. A model that fails half of its calls (5 or more) also raises a Windows notification.</p>
+        <p class="lede">Failed calls to a gateway model, grouped by type (429, 503, timeout, ...) and role, with the latest lines of each group. A model that fails half of its calls (5 or more) also raises a Windows notification.</p>
         <div id="health-table"></div>
       </div>
 
@@ -1153,15 +1160,29 @@ for (const [id, action] of [["btn-upgrade-apply", "apply"], ["btn-upgrade-declin
     }
   });
 }
+const FAILURE_TYPES = { "429": "429 Too Many Requests", "503": "503 Service Unavailable", "5xx": "Other 5xx", timeout: "Timeout",
+  connection: "Connection", auth: "Authentication (401/403)", other: "Other" };
+const PURPOSE_LABELS = { index_embedding: "Indexing", query_embedding: "Query", rerank: "Rerank", embedding_probe: "Model check", scope: "Scope" };
+function healthHtml(data) {
+  const when = (at) => esc(new Date(at * 1000).toLocaleString());
+  const models = data.models.map((row) => `<tr><td>${esc(row.role)}</td><td>${esc(row.model)}</td><td>${row.errors} / ${row.calls}</td><td>${Math.round(row.rate * 100)}%</td></tr>`).join("");
+  const purposes = data.calls_by_purpose.map((row) => `<tr><td>${esc(PURPOSE_LABELS[row.purpose] || row.purpose)}</td><td>${row.attempts}</td><td>${row.errors}</td></tr>`).join("");
+  const groups = data.groups.map((group) => {
+    const rows = group.rows.map((row) => `<tr><td>${when(row.at)}</td><td>${esc(row.model)}</td><td>${esc(row.tool || "-")}</td><td>${esc(row.error)}</td></tr>`).join("");
+    return `<details class="more health-group" data-type="${esc(group.type)}"><summary>${esc(FAILURE_TYPES[group.type] || group.type)} · ${esc(group.role)} · ${plural(group.count, "failure")} · ${when(group.first_at)} to ${when(group.last_at)}</summary>`
+      + `<p class="mono">${esc(group.sample)}</p><table><tr><th>When</th><th>Model</th><th>Tool</th><th>Error</th></tr>${rows}</table>`
+      + (group.count > group.rows.length ? `<p>Showing the latest ${group.rows.length} of ${group.count}.</p>` : "") + "</details>";
+  }).join("");
+  return (models ? `<table><tr><th>Role</th><th>Model</th><th>Failed / calls</th><th>Rate</th></tr>${models}</table>` : "<p>No model call in the last 24 hours.</p>")
+    + (purposes ? `<h4>Calls by purpose</h4><table><tr><th>Purpose</th><th>Calls</th><th>Failed</th></tr>${purposes}</table>` : "")
+    + (groups ? `<h4>Failures by type</h4>${groups}` : "");
+}
 async function loadHealth() {
   try {
     const data = await getJSON("/setup/models/health");
     const failing = data.models.filter((row) => row.failing).length;
     setPill("health-pill", failing ? "bad" : "ok", failing ? plural(failing, "model") + " failing" : "No problem");
-    const models = data.models.map((row) => `<tr><td>${esc(row.role)}</td><td>${esc(row.model)}</td><td>${row.errors} / ${row.calls}</td><td>${Math.round(row.rate * 100)}%</td></tr>`).join("");
-    const failures = data.failures.map((row) => `<tr><td>${esc(new Date(row.at * 1000).toLocaleString())}</td><td>${esc(row.role)}</td><td>${esc(row.model)}</td><td>${esc(row.tool || "-")}</td><td>${esc(row.error)}</td></tr>`).join("");
-    $("health-table").innerHTML = (models ? `<table><tr><th>Role</th><th>Model</th><th>Failed / calls</th><th>Rate</th></tr>${models}</table>` : "<p>No model call in the last 24 hours.</p>")
-      + (failures ? `<table><tr><th>When</th><th>Role</th><th>Model</th><th>Tool</th><th>Error</th></tr>${failures}</table>` : "");
+    $("health-table").innerHTML = healthHtml(data);
   } catch (error) {
     setPill("health-pill", "bad", "Unavailable");
     $("health-table").textContent = "Could not read the failure log: " + error.message;
@@ -1270,6 +1291,10 @@ const RULES = [
     intro: "Compares the functions an edit writes with the indexed code: identical bodies, and near-identical ones with only local names changed. It never blocks.",
     options: [["warn", "Warn", "The edit runs; the agent is told when a function copies one already in the project.", true],
               ["off", "Off", "No duplicate checks."]] },
+  { key: "dependency_mode", title: "Vulnerable dependencies on edit", applies: "Applies on the next edit.",
+    intro: "Looks up the dependencies an edit of a dependency manifest adds or changes in the OSV advisory database. It never blocks.",
+    options: [["warn", "Warn", "The edit runs; the agent is told which added or changed dependencies have known advisories.", true],
+              ["off", "Off", "No dependency checks and no network call."]] },
   { key: "pattern_proposals", title: "Block patterns proposed by a model", applies: "Applies to the next review.",
     intro: "Afterwards, never delaying a call, the scope model (else the router model) reviews searches the fixed rule let through and may propose block patterns. Nothing blocks until you accept it on the Redirects page.",
     options: [["on", "On", "Review in the background and list proposals for you.", true],

package/smart_tool_daemon.py

@@ -676,69 +676,97 @@ def _embed_retry_delay(exc, attempt):
     return min(8, 2 ** (attempt - 1)) + random.uniform(0, .25)
 
 
+def _wait_to_retry(exc, attempt, deadline=None, cancel_check=None):
+    """Sleeps the wait the gateway asked for (`Retry-After`, else exponential with jitter) before the next attempt;
+    raises TimeoutError when that wait does not fit in `deadline`."""
+    delay = _embed_retry_delay(exc, attempt)
+    if deadline is not None and delay >= _research_time_left(deadline):
+        raise TimeoutError("Not enough time left to honor the gateway backoff.")
+    if cancel_check:
+        until = time.monotonic() + delay
+        while time.monotonic() < until:
+            cancel_check()
+            time.sleep(min(.2, max(0, until - time.monotonic())))
+    else:
+        time.sleep(delay)
+
+
+def _embed_request(model, batch, token, deadline=None, on_retry=None, cancel_check=None):
+    """Vectors of one embeddings request in input order, repeating a transient failure up to EMBED_MAX_ATTEMPTS times."""
+    for attempt in range(1, EMBED_MAX_ATTEMPTS + 1):
+        if cancel_check:
+            cancel_check()
+        timeout = (min(EMBED_REQUEST_TIMEOUT_S, _research_time_left(deadline))
+                   if deadline is not None else EMBED_REQUEST_TIMEOUT_S)
+        try:
+            response = model_client.fetch(
+                "/v1/embeddings", token, method="POST", body={"model": model, "input": batch},
+                timeout=timeout,
+            )
+        except Exception as exc:
+            if attempt >= EMBED_MAX_ATTEMPTS or not _transient_embed_error(exc):
+                raise
+            if on_retry:
+                on_retry(attempt + 1, len(batch))
+            _wait_to_retry(exc, attempt, deadline, cancel_check)
+            continue
+        if cancel_check:
+            cancel_check()
+        rows = response.get("data", []) if isinstance(response, dict) else []
+        if not isinstance(rows, list) or len(rows) != len(batch):
+            raise RuntimeError("The gateway returned a different number of embeddings than the input.")
+        ordered = [None] * len(batch)
+        for row in rows:
+            if not isinstance(row, dict):
+                raise indexer.IndexVectorError("Invalid embedding row.")
+            idx = row.get("index", 0 if len(batch) == 1 else None)
+            if type(idx) is not int or not 0 <= idx < len(batch) or ordered[idx] is not None:
+                raise indexer.IndexVectorError("Embedding indexes missing, duplicated or outside the batch.")
+            ordered[idx] = row.get("embedding")
+        return ordered
+
+
 def _embed(model, texts, token, deadline=None, on_retry=None, cancel_check=None):
+    """One vector per text. A request the gateway refuses for its size (HTTP 400 or 413) is split in two halves, each
+    sent once; a half that is refused again raises."""
     if local_embedder.is_local(model):
         return local_embedder.embed(model, texts, deadline=deadline, cancel_check=cancel_check, on_retry=on_retry)
     vectors = []
     dimensions = None
     for batch in _embed_batches(texts):
-        max_attempts = EMBED_MAX_ATTEMPTS if deadline is None or on_retry is not None else 1
-        for attempt in range(1, max_attempts + 1):
-            if cancel_check:
-                cancel_check()
-            timeout = (min(EMBED_REQUEST_TIMEOUT_S, _research_time_left(deadline))
-                       if deadline is not None else EMBED_REQUEST_TIMEOUT_S)
-            try:
-                response = model_client.fetch(
-                    "/v1/embeddings", token, method="POST", body={"model": model, "input": batch},
-                    timeout=timeout,
-                )
-            except Exception as exc:
-                if attempt >= max_attempts or not _transient_embed_error(exc):
-                    raise
-                if on_retry:
-                    on_retry(attempt + 1, len(batch))
-                delay = _embed_retry_delay(exc, attempt)
-                if deadline is not None and delay >= _research_time_left(deadline):
-                    raise TimeoutError("Not enough time left to honor the gateway backoff.")
-                if cancel_check:
-                    until = time.monotonic() + delay
-                    while time.monotonic() < until:
-                        cancel_check()
-                        time.sleep(min(.2, max(0, until - time.monotonic())))
-                else:
-                    time.sleep(delay)
-                continue
-            if cancel_check:
-                cancel_check()
-            rows = response.get("data", []) if isinstance(response, dict) else []
-            if not isinstance(rows, list) or len(rows) != len(batch):
-                raise RuntimeError("The gateway returned a different number of embeddings than the input.")
-            ordered = [None] * len(batch)
-            for row in rows:
-                if not isinstance(row, dict):
-                    raise indexer.IndexVectorError("Invalid embedding row.")
-                idx = row.get("index", 0 if len(batch) == 1 else None)
-                if type(idx) is not int or not 0 <= idx < len(batch) or ordered[idx] is not None:
-                    raise indexer.IndexVectorError("Embedding indexes missing, duplicated or outside the batch.")
-                vector = row.get("embedding")
-                dimensions = indexer.validate_vector(vector, dimensions)
-                ordered[idx] = vector
-            vectors.extend(ordered)
-            break
+        try:
+            ordered = _embed_request(model, batch, token, deadline, on_retry, cancel_check)
+        except urllib.error.HTTPError as exc:
+            if exc.code not in (400, 413) or len(batch) < 2:
+                raise
+            half = len(batch) // 2
+            ordered = (_embed_request(model, batch[:half], token, deadline, on_retry, cancel_check)
+                       + _embed_request(model, batch[half:], token, deadline, on_retry, cancel_check))
+        for vector in ordered:
+            dimensions = indexer.validate_vector(vector, dimensions)
+        vectors.extend(ordered)
     return vectors
 
 
 def _rerank(model, query, documents, top_n, token, deadline=None):
+    """Rows `{index, relevance_score}` of the best `top_n` documents. Only an HTTP answer that is worth repeating
+    (429, 5xx) is asked again, after the wait of `_wait_to_retry`; a timeout fails at once so a search does not stall."""
     if not documents:
         return []
     top_n = min(top_n, len(documents))
-    timeout = min(15, _research_time_left(deadline)) if deadline is not None else 15
-    response = model_client.fetch(
-        "/v1/rerank", token, method="POST",
-        body={"model": model, "query": query, "documents": documents, "top_n": top_n},
-        timeout=timeout,
-    )
+    for attempt in range(1, EMBED_MAX_ATTEMPTS + 1):
+        timeout = min(15, _research_time_left(deadline)) if deadline is not None else 15
+        try:
+            response = model_client.fetch(
+                "/v1/rerank", token, method="POST",
+                body={"model": model, "query": query, "documents": documents, "top_n": top_n},
+                timeout=timeout,
+            )
+            break
+        except urllib.error.HTTPError as exc:
+            if attempt >= EMBED_MAX_ATTEMPTS or not _transient_embed_error(exc):
+                raise
+            _wait_to_retry(exc, attempt, deadline)
     rows = response.get("results", []) if isinstance(response, dict) else []
     if not isinstance(rows, list) or not rows:
         raise ValueError("The reranker returned no valid results.")
@@ -1054,7 +1082,9 @@ def _prepare_project_index(arguments, job_id, cfg, token, query_vector=None, pre
         return result
     model = model or local_embedder.resolve(cfg.get("embedding_model"), indexer.info(root).get("model_id"))
     lexical = model == indexer.LEXICAL_MODEL
-    if not lexical and query_vector is None:
+    current = indexer.info(root)
+    known_dimensions = current.get("dimensions") if current.get("model_id") == model else None
+    if not lexical and query_vector is None and not known_dimensions:
         _update_job(job_id, phase="model_probe")
         query_vector = _embed(model, ["Smart Tool: indexing dimension check."], token,
                               deadline=_job_deadline(job_id), cancel_check=checkpoint,
@@ -1062,7 +1092,6 @@ def _prepare_project_index(arguments, job_id, cfg, token, query_vector=None, pre
                                   retry_attempt=attempt, batch_items=items))[0]
     checkpoint()
     _update_job(job_id, phase="indexing")
-    current = indexer.info(root)
     full_check = bool(arguments.get("full_check") or project.get("needs_full_check") or
                       time.time() - current.get("full_scan_at", 0) >= project_monitor.FULL_CHECK_S)
     stats = indexer.reindex(
@@ -1072,7 +1101,7 @@ def _prepare_project_index(arguments, job_id, cfg, token, query_vector=None, pre
                                  job_id, phase="indexing_retry", retry_attempt=attempt, batch_items=items)),
         on_progress=lambda processed, total: _update_job(
             job_id, phase="indexing", processed_files=processed, total_files=total),
-        model_id=model, expected_dimensions=None if lexical else len(query_vector),
+        model_id=model, expected_dimensions=None if lexical else known_dimensions or len(query_vector),
         force_rebuild=bool(arguments.get("force_rebuild")), full_check=full_check,
         changed_paths=set(arguments.get("changed_paths") or []) | set(project.get('dirty_paths', [])), cancel_check=checkpoint,
     )
@@ -1110,6 +1139,33 @@ LEXICAL_WARNING = ("No embedding model: index and search are lexical only (exact
                    "Configure an embedding_model for semantic search.")
 
 
+def _rerank_blocks(model, groups, queries, found, vectors, token, deadline=None):
+    """`({group: rows}, {group: error})` of the blocks that have candidates and a query vector. One rerank request goes
+    out per distinct query, not per block: the candidates of the blocks that share a query travel together and come back
+    split by block (`index` is the position inside the block's candidates). A reranker scores each document against the
+    query on its own, so a block gets the order and scores a request of its own would give."""
+    ranked, failed = {}, {}
+    by_query = {}
+    for group in groups:
+        if found[group] and vectors.get(queries[group]) is not None:
+            by_query.setdefault(queries[group], []).append(group)
+    for query, members in by_query.items():
+        owners = [(group, position) for group in members for position in range(len(found[group]))]
+        documents = [found[group][position][4] for group, position in owners]
+        try:
+            rows = _rerank(model, query, documents, len(documents), token, deadline=deadline)
+        except Exception as exc:
+            failed.update({group: _project_error(exc) for group in members})
+            continue
+        ranked.update({group: [] for group in members})
+        for row in rows:
+            group, position = owners[row["index"]]
+            ranked[group].append({"index": position, "relevance_score": row["relevance_score"]})
+        failed.update({group: "ValueError: The reranker returned no result for this block."
+                       for group in members if not ranked[group]})
+    return ranked, failed
+
+
 def _perform_smart_search(arguments, job_id):
     return contextvars.copy_context().run(_perform_smart_search_isolated, arguments, job_id)
 
@@ -1220,22 +1276,22 @@ def _perform_smart_search_isolated(arguments, job_id):
                                    include=lambda path, group=group: index_profile.kind(path, profile) == group)
              for group in groups}
 
+    reranked, rerank_failed = {}, {}
+    if cfg.get("rerank_model"):
+        _update_job(job_id, phase="reranking")
+        reranked, rerank_failed = _rerank_blocks(cfg["rerank_model"], groups, queries, found, vectors, token,
+                                                 deadline=_job_deadline(job_id))
+
     def search_block(group):
         top_k = limits[group]
         query, vector, candidates = queries[group], vectors.get(queries[group]), found[group]
         ordered = [(score, "hybrid" if vector is not None else "lexical", row) for score, *row in candidates[:top_k]]
         problem = None
-        if cfg.get("rerank_model") and candidates and vector is not None:
-            _update_job(job_id, phase="reranking")
-            try:
-                reranked = _rerank(cfg["rerank_model"], query, [c[4] for c in candidates], top_k, token,
-                                   deadline=_job_deadline(job_id))
-                if not reranked or any(type(item.get("index")) is not int or not 0 <= item["index"] < len(candidates)
-                                       for item in reranked):
-                    raise ValueError("Invalid index in the reranker response.")
-                ordered = [(float(item["relevance_score"]), "rerank", candidates[item["index"]][1:]) for item in reranked]
-            except Exception as exc:
-                problem = f"Rerank unavailable for {group}; hybrid lexical/vector order kept. " + _project_error(exc)
+        if group in rerank_failed:
+            problem = f"Rerank unavailable for {group}; hybrid lexical/vector order kept. " + rerank_failed[group]
+        elif group in reranked:
+            ordered = [(float(item["relevance_score"]), "rerank", candidates[item["index"]][1:])
+                       for item in reranked[group][:top_k]]
         exact = indexer.overlay_exact_keys(overlay, query)
         if exact:
             pinned = [(c[0], "overlay", c[1:]) for c in candidates if tuple(c[1:4]) in exact][:2]

package/usage_meter.py

@@ -97,6 +97,19 @@ def _record_job(job_id,payload):
         db.execute('DELETE FROM usage_index_jobs WHERE at<?',(time.time()-366*86400,))
 
 
+def calls_by_purpose(since,purposes):
+    """Attempts and errors of every project per purpose in `purposes`, from the UTC day of the epoch time `since` on
+    (the table keeps days, not hours). Purposes with no call are left out."""
+    day=datetime.datetime.fromtimestamp(since,datetime.timezone.utc).date().isoformat()
+    marks=','.join('?' for _ in purposes)
+    with project_store._LOCK,project_store._db() as db:
+        _tables(db)
+        rows=db.execute('SELECT purpose,SUM(attempts),SUM(errors) FROM usage_daily WHERE day>=? AND purpose IN ('+marks+') GROUP BY purpose',
+                        (day,*purposes)).fetchall()
+    found={purpose:{'purpose':purpose,'attempts':attempts,'errors':errors} for purpose,attempts,errors in rows}
+    return [found[purpose] for purpose in purposes if purpose in found]
+
+
 def summary(project_id,days=30,view_id=None):
     if type(days) is not int or not 1<=days<=366:raise ValueError('Choose a period from 1 to 366 days.')
     with project_store._LOCK,project_store._db() as db:

package/version.py

@@ -1,7 +1,7 @@
 """Single source of the Smart Tool version and of the User-Agent sent to public APIs."""
 import re
 
-VERSION = "0.9.40"
+VERSION = "0.9.41"
 _CONTACT_RE = re.compile(r"^(?:[^\s@()<>;]+@[^\s@()<>;]+\.[^\s@()<>;]+|https?://[^\s()<>;]+)$")
 
 

package/web_fetch.py

@@ -38,8 +38,7 @@ SELECT_BUDGET_CHARS = 15_000
 CHUNK_CHARS = 2_000
 PAGE_FORMAT = 2
 MAX_DOWNLOAD_BYTES = 5 * 1024 * 1024
-PDF_MAX_PAGES = 300
-PDF_TIMEOUT_S = 30
+PDF_INTERACTIVE_TIMEOUT_S = 30  # Override of document_text.PDF_TIMEOUT_S: a person waits for this answer, the indexer does not.
 MAX_REDIRECTS = 5
 HTTP_TIMEOUT_S = 20
 READ_CHUNK = 64 * 1024
@@ -212,12 +211,13 @@ def thin(text):
 
 
 def pdf_text(body, url, deadline):
-    """Text of a PDF (pypdf, pure Python), page by page up to PDF_MAX_PAGES. The native WebFetch reads PDFs and the hook
-    sends it here, so a PDF used to be a dead end (9 times in 10 days of transcripts). A hostile PDF can make pypdf
-    parse for minutes, so it runs in a subprocess killed at the request's deadline (at most PDF_TIMEOUT_S)."""
-    timeout = min(PDF_TIMEOUT_S, _left(deadline))
+    """Text of a PDF (pypdf, pure Python), page by page, with the page and character caps of document_text. The native
+    WebFetch reads PDFs and the hook sends it here, so a PDF used to be a dead end (9 times in 10 days of transcripts).
+    A hostile PDF can make pypdf parse for minutes, so it runs in a subprocess killed at the request's deadline (at most
+    PDF_INTERACTIVE_TIMEOUT_S, passed to document_text.run_pdf instead of the indexer's document_text.PDF_TIMEOUT_S)."""
+    timeout = min(PDF_INTERACTIVE_TIMEOUT_S, _left(deadline))
     try:
-        return document_text.run_pdf(body, timeout, PDF_MAX_PAGES, PAGE_STORE_MAX_CHARS)
+        return document_text.run_pdf(body, timeout, document_text.PDF_MAX_PAGES, document_text.PDF_MAX_CHARS)
     except document_text.PdfTimeout:
         raise FetchError(f"PDF not read in {timeout:.0f}s ({url}).") from None
     except document_text.PdfUnreadable as exc: