@allansantos-dev/smart-tool 0.9.39 → 0.9.41

package/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## 0.9.41 - beta
+
+- Far fewer calls to the embedding and rerank models. Measured on a real machine (24 h): 8303 embedding and 1473 rerank
+  calls, 0.3% of them failing; the volume came from four places, all cut: (1) an indexing job no longer embeds a fixed
+  text to learn the vector size when the index already stores it for the same model (1300 calls a day); (2) embedding
+  batches carry up to 128 chunks and 120,000 characters instead of 32,000 characters (3.8 chunks per call before); a
+  request the gateway refuses for its size is split in two halves once; (3) one rerank call per search instead of one
+  per block (code, test, doc); the blocks keep the order the separate calls gave; (4) a project in `eager` mode whose
+  files change all the time waits longer between jobs (60, 120, 240, 480 up to 900 s, back to 60 s after a quiet
+  period), and `*.tmp`, `*.bak`, `*.swp` and `*~` files no longer mark a project as changed or enter the index.
+  Measurement before and after in `docs/research/data/call-volume-*.json`, findings and sources in
+  `docs/research/2026-10-27-excesso-de-chamadas-embed-rerank.md`. Tests in `tests/test_call_volume.py`.
+- A model that answers 429 or 503 three times in a row opens a circuit for 30 s: calls to it fail at once and the search
+  answers from the lexical index with the usual warning, instead of repeating requests. At most 6 requests are in
+  flight per model. Rerank and the model check wait the gateway's `Retry-After` (or a jittered exponential delay) before
+  asking again, like embeddings.
+- The setup Models page groups the failures by type (429, 503, other 5xx, timeout, connection, authentication, other):
+  one row per type and role with the count, first and last time, one sample message, and the latest 20 lines on
+  expand. The summary shows the calls per purpose (indexing, query, rerank, model check). `dependency_mode` has its own
+  card in the setup rules.
+- The dependency notice also reads `requirements*.txt` (PyPI, `==` pins and the lower bound of `>=` and `~=`) and `go.mod`
+  (Go, `require` lines with a plain `vX.Y.Z`). `web_fetch` reads PDFs with the same page and character caps as the
+  indexer (`document_text`).
+
+## 0.9.40 - beta
+
+- PDFs of the project are searchable. `smart_search` indexes the text layer of a `.pdf` inside the project scope and
+  returns the page (`[page N]`) with the passage. The extractor is the one `web_fetch` already used, now in
+  `document_text`: `pypdf` in an isolated subprocess killed after 20 s, at most 20 MiB, 300 pages and 1,000,000
+  characters per file. A scanned PDF (no text layer), a password-protected one, a corrupt one or one that overruns the
+  deadline is skipped with the reason in the index stats and the rest of the project is indexed as usual. The secret,
+  minified and long-line guards apply to the extracted text. `document_text.VERSION` is 2: every `.docx` already indexed
+  is read once more. Tests in `tests/test_pdf_index.py`; measurement before and after in
+  `docs/research/data/pdf-probe-*.json`.
+- Editing `package.json` warns about known-vulnerable dependencies. Added or changed entries of `dependencies`,
+  `devDependencies`, `peerDependencies` and `optionalDependencies` (at most 5 per edit) are checked against OSV with one
+  `querybatch` call (2.5 s timeout, answers cached on disk for 6 h); the edit reminder lists the advisory IDs per
+  `package@version`, or says `dependency check skipped: <reason>` when OSV cannot be reached, and the edit goes on.
+  Only package names and versions leave the machine. `dependency_mode` in `config.json` (`warn` by default, `off`)
+  turns it off. Tests in `tests/test_dependency_notice.py`; measurement in `docs/research/data/dependency-probe-*.json`.
+- A scheduled setup audit that finds new errors or warnings raises one Windows toast per project with an "Open the
+  audit" button; the first audit, manual runs and info findings stay silent (`tests/test_audit_toast.py`).
+- `smart_scout` answers a question about intent (why, reason, decided) with up to three decision notes from the project
+  notes next to the code lines (`tests/test_scout.py`).
+- The edit reminder gives the command that runs the tests calling the edited function (`tests/test_edit_review_cache.py`).
+
 ## 0.9.39 - beta
 
 - Two-letter technical terms (`ui`, `db`, `id`) count again in the shared `focus_terms`, used by `smart_search

package/README.md

@@ -5,7 +5,7 @@ A local MCP server that gives coding agents (Claude Code, Codex) cheaper, sharpe
 | Tool | What it does |
 |---|---|
 | `smart_search` / `smart_search_result` | Semantic + lexical search over an indexed project, results split into code, tests and docs, with the likeliest functions per chunk |
-| `smart_scout` | Answers a question about the code with the lines that answer it: reads the candidate files inside Smart Tool and returns `path:lines` with the literal text, so the files never enter the agent's context; with a `scout_model`, a small model also writes a short answer and keeps only the lines that support it |
+| `smart_scout` | Answers a question about the code with the lines that answer it: reads the candidate files inside Smart Tool and returns `path:lines` with the literal text, so the files never enter the agent's context; with a `scout_model`, a small model also writes a short answer and keeps only the lines that support it; a question about why something is as it is (why, reason, decided) also lists up to three decision notes from the notes the project keeps about its intent, as ids with one excerpt each |
 | `web_search` / `web_search_result` | Web search across several sources in parallel, in the query language and in English, with a cache shared across sessions; `depth="research"` runs a multi-round research loop |
 | `web_fetch` | Reads a known URL and returns only the answer to your prompt, written by a small model; pages are cached for 24 h. Pages over 60,000 characters are not cut: the model reads the parts closest to the prompt (needs an `embedding_model`) |
 | `project_manage` | Registers and indexes projects, shows coverage, code maps, duplicated functions and index storage; `graph` with `symbol` tells what changing a function touches: callers, calls, the tests that reach it and the files that import it |
@@ -16,7 +16,7 @@ registered as the `playwright` server where this machine lets a browser be drive
 An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
 are cheaper, or just tells the agent what Smart Tool would do.
 
-**Status:** 0.9.39 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
+**Status:** 0.9.41 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
 
 ## How it works
 
@@ -129,7 +129,7 @@ Redirects) and Maintenance (Setup audit, Startup).
    When a new version recommends other models than the ones you use, a Windows notification (**Apply now**, **Don't
    show again**) and the **New recommended models** card on this page offer them: nothing changes until you apply, the
    models you had are kept, and **Switch back to my previous models** undoes it. The **Model failures (last 24 h)** table
-   lists every failed gateway call by role, model and tool; a model that fails half of its calls (5 or more) also raises a
+   groups the failed gateway calls by type (429, 503, other 5xx, timeout, connection, authentication) and role, with the count, the first and last time and one sample, and shows the calls per purpose; a model that fails half of its calls (5 or more) also raises a
    notification and a tray alert. A failure inside a tool call also comes back as a warning line in the tool answer;
    calls made by a hook or by background work (index scope and profile, block-pattern proposals) have no answer to
    carry it, so they show only in the table, the log and the notification. The local embedder is not in that table or in
@@ -181,13 +181,18 @@ Redirects) and Maintenance (Setup audit, Startup).
    `@override`) functions are exempt, as in pydocstyle, eslint-plugin-jsdoc and Checkstyle, and a JSDoc above a set of
    TypeScript overloads documents the implementation:
    - **Remind** (default): the edit runs and the agent is told which touched functions have no docstring, and which
-     documented ones changed their signature, with their callers.
+     documented ones changed their signature, with their callers, the tests calling them and the command that runs those tests
+  (from the code graph, only when it is already cached: the hook never waits for an analysis).
    - **Require**: an edit that leaves a touched function without a docstring is blocked until it adds one.
    - **Off**: no checks.
 
    A documented function edited only in its body is not mentioned. `project_manage` with `action=docs` lists the
    public functions still without a docstring, for documenting a project that started without them.
 
+   **Dependency check on edit**: an edit of `package.json`, `requirements*.txt` or `go.mod` that adds or changes a dependency lists the OSV advisory IDs of up to 5 of them (exact version, or the lower bound of `^`, `~`, `~=` and `>=`), answered from a 6 h disk cache or one 2.5 s batch call that sends only package names and versions; when OSV cannot answer, the reminder says the check was skipped and the edit goes on. `dependency_mode` in `config.json`: `warn` (default) or `off`.
+
+   **PDFs** inside a project are indexed as text with `[page N]` markers (20 MiB, 300 pages and 20 s per file; a scanned, password-protected or corrupt PDF is skipped with the reason), so `smart_search` returns the page that holds a passage.
+
    **Block patterns waiting for you** are shown in one approval panel: 8 per page, the ones that would have blocked
    the most logged calls first, a filter over pattern, reason and example, and bulk accept or reject of the selected
    rows; each row opens to its reason and example.
@@ -219,7 +224,9 @@ Redirects) and Maintenance (Setup audit, Startup).
    two minutes after it starts and then every 30 minutes (MCP servers shared by several projects are contacted once
    and their result reused for 6 hours; Run now always contacts them again), and the setup screen's **Setup audit**
    panel shows each project's last report: findings by severity, what is new, what was resolved, the tokens each MCP
-   server's tools cost and what was not verified.
+   server's tools cost and what was not verified. When a scheduled run finds new errors or warnings, a Windows
+   notification from the tray names the folder and up to three of them, once per finding, with a button that opens the
+   panel; manual runs and info findings stay silent.
 
    The agent hears about it without asking for it. `setup_audit` with `mode=last` returns the stored report for the
    folder (or the nearest audited parent) without auditing again, and fails with what to do when none is stored. On

package/config.py

@@ -37,6 +37,7 @@ DEFAULT_CONFIG = {
     "hook_mode": "redirect",
     "doc_mode": "remind",
     "duplicate_mode": "warn",
+    "dependency_mode": "warn",
     "auto_update": True,
     "pattern_proposals": "on",
     "contact": "",
@@ -61,6 +62,9 @@ DOC_MODES = ("require", "remind", "off")
 # Edit hook on functions that copy one already indexed (exact or near-identical body): warn = let the edit run and
 # point to the existing function; off = say nothing. Never blocks: near matches are leads, not defects.
 DUPLICATE_MODES = ("warn", "off")
+# Edit hook on package.json, requirements*.txt and go.mod: warn = let the edit run and list the added or changed dependencies that have OSV advisories;
+# off = say nothing and make no network call.
+DEPENDENCY_MODES = ("warn", "off")
 # Router model reviewing, in the background, searches the deterministic redirect rule let run: on = propose block
 # patterns for the user to accept in the setup screen; off = no review.
 PATTERN_PROPOSAL_MODES = ("on", "off")
@@ -85,6 +89,13 @@ def duplicate_mode(cfg=None):
     return value
 
 
+def dependency_mode(cfg=None):
+    value = (cfg if cfg is not None else load_config()).get("dependency_mode") or DEFAULT_CONFIG["dependency_mode"]
+    if value not in DEPENDENCY_MODES:
+        raise ValueError(f"Invalid dependency_mode ({value!r}) in {CONFIG_PATH}: use warn or off.")
+    return value
+
+
 def pattern_proposals(cfg=None):
     """on: the router model reviews searches the redirect rule let run and proposes block patterns for the user to
     accept; off: no review."""

package/dependency_notice.py

@@ -0,0 +1,280 @@
+"""Vulnerable-dependency notice for an edit of a dependency manifest, built before the edit runs.
+
+Manifests: package.json (OSV ecosystem npm), requirements*.txt (PyPI: `==` pins and the lower bound of `>=` and `~=`)
+and go.mod (Go: `require` lines with a plain vX.Y.Z). Compares the dependencies of the file before and after the edit,
+asks OSV (one `querybatch` call, short timeout) about the added or changed ones only, and answers from a disk cache of
+(ecosystem, package, version) -> advisory IDs for 6 hours. Only package names and versions leave the machine. The caller (`hook_decision`) turns a `DependencyCheckError` into
+"dependency check skipped: <reason>" and lets the edit go on.
+"""
+import collections
+import json
+import os
+import re
+import time
+
+import atomic_io
+import freshness
+import paths
+
+CACHE_PATH = os.path.join(paths.DATA_DIR, "dependency_advisories.json")
+CACHE_TTL_S = 6 * 3600
+OSV_TIMEOUT_S = 2.5
+MAX_PACKAGES = 5
+MAX_IDS_SHOWN = 5
+TABLES = ("dependencies", "devDependencies", "peerDependencies", "optionalDependencies")
+PREFIX = "Smart Tool dependencies: "
+_SPECIFIER_RE = re.compile(r"(\^|~|>=|>|=)?\s*v?(\d+)(?:\.(\d+))?(?:\.(\d+))?"
+                           r"((?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?)")
+_REQUIREMENTS_FILE_RE = re.compile(r"requirements[A-Za-z0-9._-]*\.txt")
+_REQUIREMENT_RE = re.compile(r"([A-Za-z0-9][A-Za-z0-9._-]*)\s*(?:\[[^\]]*\])?\s*([^;]*)")
+_PIP_VERSION_RE = re.compile(r"[0-9][0-9A-Za-z.+!_-]*")
+_GO_VERSION_RE = re.compile(r"v\d+\.\d+\.\d+")
+
+
+class DependencyCheckError(RuntimeError):
+    """The check could not run (OSV unreachable or answering badly, unreadable cache, current file not JSON)."""
+
+
+def _npm_pairs(text):
+    """The (name, specifier) pairs of every dependency table in a package.json text.
+
+    Parameters: text (str) file content.
+    Returns: dict (name, specifier) -> None, in file order; None when the text is not a JSON object.
+    """
+    try:
+        data = json.loads(text)
+    except ValueError:
+        return None
+    if not isinstance(data, dict):
+        return None
+    pairs = {}
+    for table in TABLES:
+        entries = data.get(table)
+        if isinstance(entries, dict):
+            pairs.update({(name, spec): None for name, spec in entries.items() if isinstance(spec, str)})
+    return pairs
+
+
+def _requirements_pairs(text):
+    """The (name, specifier) pairs of a requirements.txt text; options, includes, editable and URL lines are skipped.
+
+    Parameters: text (str) file content.
+    Returns: dict (name, specifier) -> None in file order; the specifier is the version clauses without spaces ("" when
+    the line only names the package).
+    """
+    pairs = {}
+    for raw in text.splitlines():
+        line = raw.split(" #")[0].split(";")[0].strip().rstrip("\\").strip()
+        if not line or line.startswith(("#", "-")):
+            continue
+        line = " ".join(word for word in line.split() if not word.startswith("--"))
+        match = _REQUIREMENT_RE.fullmatch(line)
+        if not match:
+            continue
+        specifier = "".join(match.group(2).split())
+        if specifier and specifier[0] not in "=<>!~":
+            continue
+        pairs[(match.group(1), specifier)] = None
+    return pairs
+
+
+def _go_pairs(text):
+    """The (module, version) pairs of the `require` lines of a go.mod text, single-line and block; replace, exclude and
+    the other directives are skipped.
+
+    Parameters: text (str) file content.
+    Returns: dict (module, version) -> None in file order.
+    """
+    pairs, block = {}, None
+    for raw in text.splitlines():
+        tokens = raw.split("//")[0].split()
+        if not tokens:
+            continue
+        if block is None:
+            if tokens[0] == "require" and len(tokens) == 3:
+                pairs[(tokens[1], tokens[2])] = None
+            elif tokens[-1] == "(":
+                block = tokens[0]
+        elif tokens == [")"]:
+            block = None
+        elif block == "require" and len(tokens) == 2:
+            pairs[(tokens[0], tokens[1])] = None
+    return pairs
+
+
+def ecosystem_of(path):
+    """The OSV ecosystem of a manifest file name.
+
+    Parameters: path (str) file path.
+    Returns: "npm" for package.json, "PyPI" for requirements*.txt, "Go" for go.mod; None for any other file.
+    """
+    name = os.path.basename(path)
+    if name == "package.json":
+        return "npm"
+    if name == "go.mod":
+        return "Go"
+    return "PyPI" if _REQUIREMENTS_FILE_RE.fullmatch(name) else None
+
+
+def changed_dependencies(before, after, ecosystem="npm"):
+    """Dependencies added or whose specifier changed between two manifest texts.
+
+    Parameters: before (str or None) text before the edit, None when the file did not exist; after (str) text after;
+    ecosystem (str) one of ECOSYSTEMS.
+    Returns: list of (name, specifier); empty when `after` is a package.json that is not valid JSON (mid-edit).
+    Raises: DependencyCheckError when `before` exists but is a package.json that is not valid JSON.
+    """
+    parse = _ECOSYSTEMS[ecosystem].parse
+    now = parse(after)
+    if now is None:
+        return []
+    if before is None or not before.strip():
+        old = {}
+    else:
+        old = parse(before)
+        if old is None:
+            raise DependencyCheckError("current package.json is not valid JSON")
+    return [pair for pair in now if pair not in old]
+
+
+def _npm_version(specifier):
+    """The npm case of `queried_version`: the exact version or the lower bound of a ^, ~, >= or > range; None for tags,
+    `*`, URLs, file:, git, workspace:, npm: aliases, x-ranges and compound ranges."""
+    match = _SPECIFIER_RE.fullmatch(specifier.strip())
+    if not match:
+        return None
+    operator, major, minor, patch, tail = match.groups()
+    parts = [major, minor, patch]
+    if operator in ("^", "~", ">=", ">"):
+        return ".".join(part or "0" for part in parts) + (tail or "")
+    if None in parts:
+        return None
+    return ".".join(parts) + (tail or "")
+
+
+def _pip_version(specifier):
+    """The PyPI case of `queried_version`: the `==` version, else the first `>=` or `~=` lower bound; None for no
+    specifier, `===`, wildcards, `<`, `>`, `!=` alone and anything that is not a version."""
+    clauses = specifier.split(",")
+    for operators in (("==",), (">=", "~=")):
+        for clause in clauses:
+            for operator in operators:
+                if clause.startswith(operator) and _PIP_VERSION_RE.fullmatch(clause[len(operator):]):
+                    return clause[len(operator):]
+    return None
+
+
+def _go_version(specifier):
+    """The Go case of `queried_version`: a plain vX.Y.Z; None for pseudo-versions, pre-releases and +incompatible."""
+    return specifier if _GO_VERSION_RE.fullmatch(specifier) else None
+
+
+def queried_version(specifier, ecosystem="npm"):
+    """The version to ask OSV about for a specifier.
+
+    Parameters: specifier (str) the value in the dependency table; ecosystem (str) one of ECOSYSTEMS.
+    Returns: str, the exact version or the lower bound of the range; None when the specifier cannot be reduced safely.
+    """
+    return _ECOSYSTEMS[ecosystem].version(specifier)
+
+
+_Ecosystem = collections.namedtuple("_Ecosystem", "parse version label exact")
+_ECOSYSTEMS = {
+    "npm": _Ecosystem(_npm_pairs, _npm_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
+    "PyPI": _Ecosystem(_requirements_pairs, _pip_version, lambda name, spec: f"{name}{spec}",
+                       lambda spec, version: spec == "==" + version),
+    "Go": _Ecosystem(_go_pairs, _go_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
+}
+ECOSYSTEMS = tuple(_ECOSYSTEMS)
+
+
+def _load_cache():
+    """The cache as {"name@version": {"ids": [...], "at": epoch}}; empty when the file does not exist.
+
+    Raises: DependencyCheckError when the file exists and is unreadable, not JSON or not an object (never reset).
+    """
+    if not os.path.exists(CACHE_PATH):
+        return {}
+    try:
+        with open(CACHE_PATH, encoding="utf-8") as stream:
+            data = json.load(stream)
+    except (OSError, ValueError) as exc:
+        raise DependencyCheckError(f"advisory cache {CACHE_PATH} is unreadable: {exc}") from exc
+    if not isinstance(data, dict):
+        raise DependencyCheckError(f"advisory cache {CACHE_PATH} is not a JSON object")
+    return data
+
+
+def _fresh_ids(entry, now):
+    """The cached IDs of an entry, or None when it is missing, malformed or older than CACHE_TTL_S."""
+    if not isinstance(entry, dict) or not isinstance(entry.get("ids"), list) or not isinstance(entry.get("at"), (int, float)):
+        return None
+    return entry["ids"] if now - entry["at"] < CACHE_TTL_S else None
+
+
+def _advisories(ecosystem, queries):
+    """Advisory IDs for (name, version) pairs, from the cache and one OSV batch for the rest.
+
+    Parameters: ecosystem (str) one of ECOSYSTEMS; queries (list of (name, version)).
+    Returns: tuple (list of IDs or None per query in order, str cache-write problem or "").
+    Raises: DependencyCheckError when the cache is corrupt or OSV cannot answer.
+    """
+    cache, now = _load_cache(), time.time()
+    key = lambda query: f"{ecosystem}:{query[0]}@{query[1]}"  # noqa: E731
+    found = {query: _fresh_ids(cache.get(key(query)), now) for query in queries}
+    missing = [query for query in queries if found[query] is None]
+    if not missing:
+        return [found[query] for query in queries], ""
+    try:
+        answered = freshness.advisories_batch(ecosystem, missing, OSV_TIMEOUT_S)
+    except RuntimeError as exc:
+        raise DependencyCheckError(str(exc)) from exc
+    for query, ids in zip(missing, answered):
+        found[query] = ids
+        if ids is not None:
+            cache[key(query)] = {"ids": ids, "at": now}
+    cache = {key: entry for key, entry in cache.items() if _fresh_ids(entry, now) is not None}
+    try:
+        os.makedirs(os.path.dirname(CACHE_PATH), exist_ok=True)
+        atomic_io.write_secret_text(CACHE_PATH, json.dumps(cache, indent=1))
+    except OSError as exc:
+        return [found[query] for query in queries], f"advisory cache not saved: {exc}"
+    return [found[query] for query in queries], ""
+
+
+def notice(before, after, ecosystem="npm"):
+    """The dependency notice for one manifest edit.
+
+    Parameters: before (str or None) file text before the edit; after (str) text after it; ecosystem (str) one of
+    ECOSYSTEMS, from `ecosystem_of`.
+    Returns: str, "Smart Tool dependencies: ..." lines for the changed dependencies that have advisories or could not
+    be verified, "" when there is nothing to say.
+    Raises: DependencyCheckError when the check could not run.
+    """
+    spec = _ECOSYSTEMS[ecosystem]
+    changed = changed_dependencies(before, after, ecosystem)
+    handled, extra = changed[:MAX_PACKAGES], len(changed) - MAX_PACKAGES
+    lines, queries = {}, []
+    for name, specifier in handled:
+        version = spec.version(specifier) if freshness.NAME_RES[ecosystem].fullmatch(name) else None
+        if version is None:
+            lines[(name, specifier)] = f"`{spec.label(name, specifier)}` specifier not verified."
+        else:
+            queries.append((name, specifier, version))
+    problem = ""
+    if queries:
+        ids, problem = _advisories(ecosystem, [(name, version) for name, _specifier, version in queries])
+        for (name, specifier, version), advisories in zip(queries, ids):
+            label = f"`{spec.label(name, specifier)}`" + ("" if spec.exact(specifier, version) else f" (checked {version})")
+            if advisories is None:
+                lines[(name, specifier)] = f"{label} advisory check incomplete, not verified."
+            elif advisories:
+                shown = ", ".join(advisories[:MAX_IDS_SHOWN])
+                more = f" and {len(advisories) - MAX_IDS_SHOWN} more" if len(advisories) > MAX_IDS_SHOWN else ""
+                lines[(name, specifier)] = f"{label} has advisory IDs {shown}{more}."
+    parts = [lines[pair] for pair in handled if pair in lines]
+    if extra > 0:
+        parts.append(f"{extra} more not checked.")
+    if problem and parts:
+        parts.append(f"({problem})")
+    return PREFIX + " ".join(parts) if parts else ""

package/document_text.py

@@ -1,10 +1,17 @@
-"""Extração local de texto DOCX, sem Office, execução de macros ou acesso externo."""
+"""Extração local de texto DOCX e PDF, sem Office, execução de macros ou acesso externo."""
 import io
+import json
 import re
+import subprocess
+import sys
 import zipfile
 import xml.etree.ElementTree as ET
 
-VERSION = 1
+VERSION = 2
+PDF_MAX_BYTES = 20 * 1024 * 1024
+PDF_TIMEOUT_S = 20
+PDF_MAX_PAGES = 300
+PDF_MAX_CHARS = 1_000_000
 MAX_ARCHIVE_BYTES = 10 * 1024 * 1024
 MAX_XML_BYTES = 8 * 1024 * 1024
 MAX_TEXT_BYTES = 1024 * 1024
@@ -15,6 +22,20 @@ class DocumentError(ValueError):
     pass
 
 
+class PdfTimeout(DocumentError):
+    pass
+
+
+class PdfEncrypted(DocumentError):
+    pass
+
+
+class PdfUnreadable(DocumentError):
+    def __init__(self, message, detail):
+        super().__init__(message)
+        self.detail = detail
+
+
 class _DocumentTreeBuilder(ET.TreeBuilder):
     def doctype(self, name, pubid, system):
         # O callback também recusa DTD em XML UTF-16, antes de expandir entidades.
@@ -65,5 +86,95 @@ def extract_docx(data):
         raise DocumentError('Could not extract text from this DOCX.') from exc
 
 
+PDF_SCRIPT = """
+import io, json, sys
+import pypdf
+reader = pypdf.PdfReader(io.BytesIO(sys.stdin.buffer.read()))
+if reader.is_encrypted and reader.decrypt("") == pypdf.PasswordType.NOT_DECRYPTED:
+    print(json.dumps({"encrypted": True}))
+    sys.exit()
+limit, budget = int(sys.argv[1]), int(sys.argv[2])
+pages, size = [], 0
+for page in reader.pages[:limit]:
+    pages.append((page.extract_text() or "").strip()[:budget - size])
+    size += len(pages[-1])
+    if size >= budget:
+        break
+print(json.dumps({"pages": pages, "count": len(reader.pages)}))
+"""
+
+
+def run_pdf(data, timeout, max_pages, max_chars):
+    """Text of a PDF (pypdf, pure Python), page by page, with `[page N]` markers. A hostile PDF can make pypdf parse for
+    minutes, so it runs in an isolated subprocess (`-I`) killed at the deadline.
+
+    Parameters: data (PDF bytes), timeout (seconds before the subprocess is killed), max_pages, max_chars (cap on the
+    text read, over all pages). Returns: the marked text, with a `[pages A-B not read]` line when a cap cut it; empty
+    pages are left out. Raises: PdfTimeout, PdfEncrypted (a password is needed), PdfUnreadable (corrupt; `detail` has
+    the last line pypdf reported), all DocumentError."""
+    try:
+        done = subprocess.run([sys.executable, "-I", "-c", PDF_SCRIPT, str(max_pages), str(max_chars)],
+                              input=data, capture_output=True, timeout=timeout,
+                              creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
+    except subprocess.TimeoutExpired:
+        raise PdfTimeout(f"PDF not read in {timeout:.0f}s.") from None
+    if done.returncode:
+        detail = (done.stderr.decode("utf-8", "replace").strip().splitlines() or ["no output"])[-1]
+        raise PdfUnreadable("PDF could not be read.", detail)
+    try:
+        result = json.loads(done.stdout)
+        if result.get("encrypted"):
+            raise PdfEncrypted("PDF protected by a password.")
+        pages, count = result["pages"], result["count"]
+    except (ValueError, KeyError, AttributeError) as exc:
+        if isinstance(exc, PdfEncrypted):
+            raise
+        raise PdfUnreadable("PDF could not be read.", f"unexpected extractor output ({type(exc).__name__})") from None
+    text = "\n\n".join(f"[page {number}]\n{page}" for number, page in enumerate(pages, 1) if page)
+    if count > len(pages):
+        text += f"\n\n[pages {len(pages) + 1}-{count} not read]"
+    return text
+
+
+def extract_pdf(data):
+    """Indexable text of a PDF through run_pdf with the indexer's deadline and caps.
+
+    Parameters: data (PDF bytes). Returns: UTF-8 bytes of the marked text. Raises: DocumentError with a readable reason
+    for a timeout, a password, a corrupt file or a PDF without a text layer (scanned; there is no OCR)."""
+    try:
+        text = run_pdf(data, PDF_TIMEOUT_S, PDF_MAX_PAGES, PDF_MAX_CHARS)
+    except PdfUnreadable as exc:
+        raise PdfUnreadable(f"PDF could not be read ({exc.detail})"[:200], exc.detail) from None
+    if "[page " not in text:
+        raise DocumentError("PDF has no text layer (scanned pages are not read).")
+    return (text + "\n").encode("utf-8", "replace")
+
+
+def is_extracted(path):
+    """Whether the file type goes through a text extractor before indexing.
+
+    Parameters: path (any file name or path). Returns: True for `.docx` and `.pdf`, case-insensitive."""
+    return str(path).lower().endswith(('.docx', '.pdf'))
+
+
+def size_limit(path):
+    """Largest file size, in bytes, the extractor of this type accepts.
+
+    Parameters: path of an extracted type (see is_extracted). Returns: the byte limit of its extractor."""
+    return PDF_MAX_BYTES if str(path).lower().endswith('.pdf') else MAX_ARCHIVE_BYTES
+
+
+def extract(path, data):
+    """Text of an extracted-type file.
+
+    Parameters: path (decides the type, see is_extracted), data (file bytes). Returns: UTF-8 text bytes. Raises:
+    DocumentError with a readable reason."""
+    return extract_pdf(data) if str(path).lower().endswith('.pdf') else extract_docx(data)
+
+
 def location_kind(path):
-    return 'extracted_text_line' if str(path).lower().endswith('.docx') else 'source_line'
+    """Kind of line numbers a result of this file carries.
+
+    Parameters: path. Returns: `extracted_text_line` (lines of the extracted text, not of the file) for extracted
+    types, else `source_line`."""
+    return 'extracted_text_line' if is_extracted(path) else 'source_line'

package/freshness.py

@@ -72,34 +72,79 @@ def npm_registry_evidence(package):
         return None
 
 
-def npm_advisories(package, package_version):
-    """IDs conhecidos pela OSV para a versão; None significa checagem indisponível."""
-    if (not package or not package_version
-            or not re.fullmatch(r"(?:@[a-z0-9._-]+/)?[a-z0-9._-]+", package)):
-        return None
-    body = json.dumps({"package": {"name": package, "ecosystem": "npm"},
-                       "version": package_version}).encode("utf-8")
-    request = urllib.request.Request(
-        "https://api.osv.dev/v1/query", data=body, method="POST",
+NPM_NAME_RE = re.compile(r"(?:@[a-z0-9._-]+/)?[a-z0-9._-]+")
+NAME_RES = {"npm": NPM_NAME_RE, "PyPI": re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*"),
+            "Go": re.compile(r"[A-Za-z0-9][A-Za-z0-9._~/-]*")}
+
+
+def _osv_query(ecosystem, package, package_version):
+    """One OSV query object: the package in `ecosystem` and the version asked about."""
+    return {"package": {"name": package, "ecosystem": ecosystem}, "version": package_version}
+
+
+def _osv_request(endpoint, payload):
+    """The POST request to an OSV endpoint (`query` or `querybatch`) with the product's headers and User-Agent."""
+    return urllib.request.Request(
+        "https://api.osv.dev/v1/" + endpoint, data=json.dumps(payload).encode("utf-8"), method="POST",
         headers={"Content-Type": "application/json", "Accept": "application/json",
                  "User-Agent": version.user_agent(config.load_config().get("contact"))},
     )
+
+
+def _osv_ids(data):
+    """Sorted advisory IDs of one OSV result object; None when it is incomplete (next_page_token) or malformed, since
+    neither confirms the absence of a record."""
+    if isinstance(data, dict) and data.get("next_page_token"):
+        return None  # Página incompleta não confirma ausência de aviso novo.
+    rows = data.get("vulns", []) if isinstance(data, dict) else None
+    if not isinstance(rows, list):
+        return None
+    ids = [row.get("id") for row in rows if isinstance(row, dict)]
+    if any(not isinstance(advisory, str) or not advisory for advisory in ids):
+        return None
+    return sorted(set(ids))
+
+
+def npm_advisories(package, package_version):
+    """IDs conhecidos pela OSV para a versão; None significa checagem indisponível."""
+    if not package or not package_version or not NPM_NAME_RE.fullmatch(package):
+        return None
+    request = _osv_request("query", _osv_query("npm", package, package_version))
     try:
         with urllib.request.urlopen(request, timeout=7) as response:
-            data = json.load(response)
-        if isinstance(data, dict) and data.get("next_page_token"):
-            return None  # Página incompleta não confirma ausência de aviso novo.
-        rows = data.get("vulns", []) if isinstance(data, dict) else None
-        if not isinstance(rows, list):
-            return None
-        ids = [row.get("id") for row in rows if isinstance(row, dict)]
-        if any(not isinstance(advisory, str) or not advisory for advisory in ids):
-            return None
-        return sorted(set(ids))
+            return _osv_ids(json.load(response))
     except Exception:
         return None
 
 
+def advisories_batch(ecosystem, packages, timeout):
+    """Advisory IDs for each (package, version) of `packages` from one OSV `querybatch` call.
+
+    Parameters: ecosystem (str) "npm", "PyPI" or "Go"; packages (list of (name, version)); timeout (float) seconds for
+    the request.
+    Returns: list in the same order, each the sorted ID list or None when that package's answer is incomplete
+    (next_page_token) or malformed, so it is unverified and not clean.
+    Raises: ValueError for an unknown ecosystem or an invalid name or version; RuntimeError when the service cannot be
+    reached, answers badly or answers a different number of results.
+    """
+    if ecosystem not in NAME_RES:
+        raise ValueError(f"Unknown OSV ecosystem: {ecosystem!r}")
+    if any(not NAME_RES[ecosystem].fullmatch(name) or not package_version for name, package_version in packages):
+        raise ValueError(f"Invalid {ecosystem} package query: {packages!r}")
+    request = _osv_request("querybatch", {"queries": [_osv_query(ecosystem, name, package_version)
+                                                      for name, package_version in packages]})
+    try:
+        with urllib.request.urlopen(request, timeout=timeout) as response:
+            data = json.load(response)
+    except Exception as exc:
+        raise RuntimeError(f"OSV querybatch failed: {exc}") from exc
+    results = data.get("results") if isinstance(data, dict) else None
+    if not isinstance(results, list) or len(results) != len(packages):
+        raise RuntimeError(f"OSV querybatch answered {len(results) if isinstance(results, list) else 'no'} results "
+                           f"for {len(packages)} queries")
+    return [_osv_ids(result) for result in results]
+
+
 def npm_osv_evidence(package, version):
     ids = npm_advisories(package, version)
     if ids is None: