@allansantos-dev/smart-tool 0.9.40 → 0.9.42

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,48 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.9.42 - beta
4
+
5
+ - Presentations and workbooks of the project are searchable. `.pptx` text comes out per slide under `[slide N]`, with the
6
+ speaker notes under `[notes N]`; `.xlsx` per sheet under `[sheet NAME]`, one tab-separated line per row, cached formula
7
+ values (nothing is calculated), shared and inline strings resolved. They go through the same zip and XML guards as
8
+ `.docx` and `.pdf` (10 MiB, 2000 entries, no encryption, no DOCTYPE or ENTITY, 8 MiB of XML, 1 MiB of text), plus 100,000
9
+ rows and 16,384 columns for a workbook; a corrupt, hostile or oversized file is skipped with the reason and the rest of the
10
+ project is indexed. `document_text.VERSION` is 3: `.docx` and `.pdf` already indexed are read once more. Measurement
11
+ before and after in `docs/research/data/office-probe-*.json`; tests in `tests/test_office_index.py`. A CSV of 1 MiB is
12
+ indexed and searched in 4 ms; one of 20 MiB is refused as `over size limit` in the index stats, without stalling.
13
+ - The session-start audit report works in Codex. The Setup audit page turns on the SessionStart hook for Claude Code or
14
+ Codex (preview, confirm, backup, remove only ours); Codex runs a new hook only after you approve it in `/hooks`, so the page
15
+ shows "installed, waiting for your approval in /hooks" until the approval is found in `config.toml`, and the first-search
16
+ audit line keeps showing until then. Research with the primary sources in
17
+ `docs/research/2026-10-27-codex-sessionstart-hook.md`; tests in `tests/test_codex_audit_hook.py`.
18
+ - Review fixes: a workbook cell reference past column XFD, a negative shared-string index, a corrupt deflate stream and
19
+ a newline in a sheet name no longer escape as `MemoryError`, `zlib.error`, a wrong text or a forged `[sheet ...]` line;
20
+ the tests of the Codex hook ignore `CODEX_HOME` and `CLAUDE_CONFIG_DIR` so they never write outside their temporary home.
21
+
22
+ ## 0.9.41 - beta
23
+
24
+ - Far fewer calls to the embedding and rerank models. Measured on a real machine (24 h): 8303 embedding and 1473 rerank
25
+ calls, 0.3% of them failing; the volume came from four places, all cut: (1) an indexing job no longer embeds a fixed
26
+ text to learn the vector size when the index already stores it for the same model (1300 calls a day); (2) embedding
27
+ batches carry up to 128 chunks and 120,000 characters instead of 32,000 characters (3.8 chunks per call before); a
28
+ request the gateway refuses for its size is split in two halves once; (3) one rerank call per search instead of one
29
+ per block (code, test, doc); the blocks keep the order the separate calls gave; (4) a project in `eager` mode whose
30
+ files change all the time waits longer between jobs (60, 120, 240, 480 up to 900 s, back to 60 s after a quiet
31
+ period), and `*.tmp`, `*.bak`, `*.swp` and `*~` files no longer mark a project as changed or enter the index.
32
+ Measurement before and after in `docs/research/data/call-volume-*.json`, findings and sources in
33
+ `docs/research/2026-10-27-excesso-de-chamadas-embed-rerank.md`. Tests in `tests/test_call_volume.py`.
34
+ - A model that answers 429 or 503 three times in a row opens a circuit for 30 s: calls to it fail at once and the search
35
+ answers from the lexical index with the usual warning, instead of repeating requests. At most 6 requests are in
36
+ flight per model. Rerank and the model check wait the gateway's `Retry-After` (or a jittered exponential delay) before
37
+ asking again, like embeddings.
38
+ - The setup Models page groups the failures by type (429, 503, other 5xx, timeout, connection, authentication, other):
39
+ one row per type and role with the count, first and last time, one sample message, and the latest 20 lines on
40
+ expand. The summary shows the calls per purpose (indexing, query, rerank, model check). `dependency_mode` has its own
41
+ card in the setup rules.
42
+ - The dependency notice also reads `requirements*.txt` (PyPI, `==` pins and the lower bound of `>=` and `~=`) and `go.mod`
43
+ (Go, `require` lines with a plain `vX.Y.Z`). `web_fetch` reads PDFs with the same page and character caps as the
44
+ indexer (`document_text`).
45
+
3
46
  ## 0.9.40 - beta
4
47
 
5
48
  - PDFs of the project are searchable. `smart_search` indexes the text layer of a `.pdf` inside the project scope and
package/README.md CHANGED
@@ -16,7 +16,7 @@ registered as the `playwright` server where this machine lets a browser be drive
16
16
  An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
17
17
  are cheaper, or just tells the agent what Smart Tool would do.
18
18
 
19
- **Status:** 0.9.40 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
19
+ **Status:** 0.9.42 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
20
20
 
21
21
  ## How it works
22
22
 
@@ -129,7 +129,7 @@ Redirects) and Maintenance (Setup audit, Startup).
129
129
  When a new version recommends other models than the ones you use, a Windows notification (**Apply now**, **Don't
130
130
  show again**) and the **New recommended models** card on this page offer them: nothing changes until you apply, the
131
131
  models you had are kept, and **Switch back to my previous models** undoes it. The **Model failures (last 24 h)** table
132
- lists every failed gateway call by role, model and tool; a model that fails half of its calls (5 or more) also raises a
132
+ groups the failed gateway calls by type (429, 503, other 5xx, timeout, connection, authentication) and role, with the count, the first and last time and one sample, and shows the calls per purpose; a model that fails half of its calls (5 or more) also raises a
133
133
  notification and a tray alert. A failure inside a tool call also comes back as a warning line in the tool answer;
134
134
  calls made by a hook or by background work (index scope and profile, block-pattern proposals) have no answer to
135
135
  carry it, so they show only in the table, the log and the notification. The local embedder is not in that table or in
@@ -189,9 +189,12 @@ Redirects) and Maintenance (Setup audit, Startup).
189
189
  A documented function edited only in its body is not mentioned. `project_manage` with `action=docs` lists the
190
190
  public functions still without a docstring, for documenting a project that started without them.
191
191
 
192
- **Dependency check on edit**: an edit of `package.json` that adds or changes a dependency lists the OSV advisory IDs of up to 5 of them (exact version, or the lower bound of `^`, `~` and `>=`), answered from a 6 h disk cache or one 2.5 s batch call that sends only package names and versions; when OSV cannot answer, the reminder says the check was skipped and the edit goes on. `dependency_mode` in `config.json`: `warn` (default) or `off`.
192
+ **Dependency check on edit**: an edit of `package.json`, `requirements*.txt` or `go.mod` that adds or changes a dependency lists the OSV advisory IDs of up to 5 of them (exact version, or the lower bound of `^`, `~`, `~=` and `>=`), answered from a 6 h disk cache or one 2.5 s batch call that sends only package names and versions; when OSV cannot answer, the reminder says the check was skipped and the edit goes on. `dependency_mode` in `config.json`: `warn` (default) or `off`.
193
193
 
194
- **PDFs** inside a project are indexed as text with `[page N]` markers (20 MiB, 300 pages and 20 s per file; a scanned, password-protected or corrupt PDF is skipped with the reason), so `smart_search` returns the page that holds a passage.
194
+ **PDFs** inside a project are indexed as text with `[page N]` markers (20 MiB, 300 pages and 20 s per file; a scanned, password-protected or corrupt PDF is skipped with the reason), so `smart_search` returns the page that holds a passage. **Presentations and workbooks** (`.pptx`, `.xlsx`) are indexed the
195
+ same way: text of each slide under `[slide N]` with the speaker notes under `[notes N]`, each sheet under `[sheet NAME]` with one
196
+ tab-separated line per row (cached formula values; 100,000 rows or 1 MiB of text at most). Older formats (`.ppt`, `.xls`,
197
+ `.xlsm`) are not read.
195
198
 
196
199
  **Block patterns waiting for you** are shown in one approval panel: 8 per page, the ones that would have blocked
197
200
  the most logged calls first, a filter over pattern, reason and example, and bulk accept or reject of the selected
@@ -232,9 +235,10 @@ Redirects) and Maintenance (Setup audit, Startup).
232
235
  folder (or the nearest audited parent) without auditing again, and fails with what to do when none is stored. On
233
236
  its first `smart_search` of a session, and once per folder, the agent also gets one short line (under 100 tokens)
234
237
  when the stored report holds errors or warnings; a clean report adds nothing. Optionally, **Report at session
235
- start (Claude Code)** on the Setup audit page moves that line to the start of every new session: it previews the
236
- exact SessionStart entry, adds it to `~/.claude/settings.json` only after you confirm (backup first), and **Turn
237
- off** removes only that entry. It is off by default; while it is on, the search no longer repeats the line. The
238
+ start** on the Setup audit page (Claude Code or Codex) moves that line to the start of every new session: it
239
+ previews the exact SessionStart entry, adds it to `~/.claude/settings.json` or `~/.codex/hooks.json` only after you
240
+ confirm (backup first), and **Turn off** removes only that entry. Codex runs a new hook only after you approve it in
241
+ `/hooks`; until then the page shows "waiting for your approval" and the search keeps repeating the line. It is off by default; while it is on, the search no longer repeats the line. The
238
242
  hook script (`hooks/session_audit.py`) only reads the stored report from the running service and prints nothing
239
243
  when the service is down.
240
244
  4. **Local models** (optional): if the mcp-memory embedding sidecar is installed on the machine, its local embedder
package/client_hooks.py CHANGED
@@ -1,9 +1,11 @@
1
+ import hashlib
1
2
  import json
2
3
  import os
3
4
  import secrets
4
5
  import shutil
5
6
  import sys
6
7
  import time
8
+ import tomllib
7
9
 
8
10
  import daemon_launcher
9
11
  import endpoint_sync
@@ -14,6 +16,9 @@ AUDIT_SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "hooks",
14
16
  AUDIT_MARKER = "session_audit.py"
15
17
  AUDIT_MATCHER = "startup|resume|clear"
16
18
  AUDIT_TIMEOUT_S = 10
19
+ AUDIT_CLIENTS = ("claude", "codex")
20
+ AUDIT_ON = "current"
21
+ AUDIT_WAITING = "waiting_approval"
17
22
  HTTP_PATH = "/hooks/pretool"
18
23
  POST_PATH = "/hooks/posttool"
19
24
  # A redirected shell search waits for its answer in the PreToolUse (smart_tool_daemon.HOOK_SHELL_BUDGET_S).
@@ -282,9 +287,16 @@ def _audit_entry():
282
287
  "timeout": AUDIT_TIMEOUT_S}]}
283
288
 
284
289
 
285
- def _audit_plan():
286
- """(path, data, groups) of Claude Code's settings with our SessionStart entry already removed from groups."""
287
- path = os.path.realpath(CLIENTS["claude"]["files"][0])
290
+ def _audit_client(client):
291
+ """The client's config spec; only Claude Code and Codex fire a SessionStart hook whose output the model reads."""
292
+ if client not in AUDIT_CLIENTS:
293
+ raise ValueError(f"The session-start audit report is only available for Claude Code or Codex, not {client!r}.")
294
+ return CLIENTS[client]
295
+
296
+
297
+ def _audit_plan(client="claude"):
298
+ """(path, data, groups) of the client's hooks file with our SessionStart entry already removed from groups."""
299
+ path = os.path.realpath(_audit_client(client)["files"][0])
288
300
  try:
289
301
  with open(path, encoding="utf-8") as stream:
290
302
  data = json.load(stream)
@@ -299,10 +311,52 @@ def _audit_plan():
299
311
  return path, data, groups
300
312
 
301
313
 
302
- def audit_state():
303
- """The SessionStart audit hook in Claude Code's settings: `current` (our entry equals this version's), `outdated`
304
- (an entry of ours that differs) or `missing`."""
305
- path = CLIENTS["claude"]["files"][0]
314
+ def _codex_hash(event, group, index=0):
315
+ """Codex's approval hash (`sha256:` of the key-sorted compact JSON of the normalized hook identity, as in
316
+ openai/codex hooks/src/engine/discovery.rs `hook_hash`) of the handler at `index` in a hooks.json group."""
317
+ handler = group["hooks"][index]
318
+ normalized = {"type": handler["type"], "command": handler["command"], "timeout": handler.get("timeout", 600),
319
+ "async": bool(handler.get("async", False))}
320
+ if handler.get("statusMessage") is not None:
321
+ normalized["statusMessage"] = handler["statusMessage"]
322
+ identity = {"event_name": event, "hooks": [normalized]}
323
+ if group.get("matcher") is not None:
324
+ identity["matcher"] = group["matcher"]
325
+ text = json.dumps(identity, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
326
+ return "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest()
327
+
328
+
329
+ def _codex_approved(groups):
330
+ """Our SessionStart handler is trusted in Codex's config.toml: the entry keyed by hooks.json path, event, group
331
+ and handler index holds the hash of the definition as it is now (a missing or different hash means /hooks has not
332
+ approved it, so Codex does not run it). An unreadable config.toml raises ValueError."""
333
+ spec = CLIENTS["codex"]
334
+ try:
335
+ with open(spec["files"][1], encoding="utf-8") as stream:
336
+ hooks = tomllib.loads(stream.read()).get("hooks")
337
+ states = hooks.get("state") if isinstance(hooks, dict) else None
338
+ states = states if isinstance(states, dict) else {}
339
+ except FileNotFoundError:
340
+ return False
341
+ except tomllib.TOMLDecodeError as exc:
342
+ raise ValueError(f"{spec['files'][1]} is not valid TOML: {exc}")
343
+ paths = {spec["files"][0], os.path.realpath(spec["files"][0])}
344
+ for group_index, group in enumerate(groups):
345
+ if not _ours(group, _audit_hook):
346
+ continue
347
+ handler_index = next(i for i, hook in enumerate(group["hooks"]) if _audit_hook(hook))
348
+ for path in paths:
349
+ trusted = (states.get(f"{path}:session_start:{group_index}:{handler_index}") or {}).get("trusted_hash")
350
+ if trusted == _codex_hash("session_start", group, handler_index):
351
+ return True
352
+ return False
353
+
354
+
355
+ def audit_state(client="claude"):
356
+ """The SessionStart audit hook in the client's hooks file: `current` (our entry equals this version's and, in
357
+ Codex, was approved in /hooks), `waiting_approval` (Codex only: written but not approved, so it does not run yet),
358
+ `outdated` (an entry of ours that differs) or `missing`."""
359
+ path = _audit_client(client)["files"][0]
306
360
  try:
307
361
  with open(path, encoding="utf-8") as stream:
308
362
  data = json.load(stream)
@@ -312,52 +366,61 @@ def audit_state():
312
366
  found = [group for group in groups if _ours(group, _audit_hook)] if isinstance(groups, list) else []
313
367
  if not found:
314
368
  return "missing"
315
- return "current" if _audit_entry() in found else "outdated"
369
+ if _audit_entry() not in found:
370
+ return "outdated"
371
+ return "current" if client != "codex" or _codex_approved(groups) else AUDIT_WAITING
316
372
 
317
373
 
318
374
  def audit_hook_installed(client):
319
- """The Claude Code session-start audit report is on, so a search need not repeat it. An unreadable settings file
320
- counts as off here; `audit_state` raises for it on the setup screen."""
321
- if client != "claude":
375
+ """The session-start audit report is on and running (Codex: approved in /hooks), so a search need not repeat it.
376
+ An unreadable config file counts as off here; `audit_state` raises for it on the setup screen."""
377
+ if client not in AUDIT_CLIENTS:
322
378
  return False
323
379
  try:
324
- return audit_state() == "current"
380
+ return audit_state(client) == "current"
325
381
  except (OSError, ValueError):
326
382
  return False
327
383
 
328
384
 
329
- def audit_preview():
330
- """The change that turns the SessionStart audit report on in Claude Code, with a token for `audit_install`."""
331
- path, _data, _groups = _audit_plan()
385
+ def _audit_after(client):
386
+ """{after_install: ...} for a client whose new hooks need the user's approval, else nothing."""
387
+ note = CLIENTS[client].get("after_install")
388
+ return {"after_install": note} if note else {}
389
+
390
+
391
+ def audit_preview(client="claude"):
392
+ """The change that turns the SessionStart audit report on in the client, with a token for `audit_install`."""
393
+ path, _data, _groups = _audit_plan(client)
332
394
  entry = _audit_entry()
333
- if audit_state() == "current":
395
+ if audit_state(client) in (AUDIT_ON, AUDIT_WAITING):
334
396
  return {"installed": True, "file": path, "change": None}
335
397
  token = secrets.token_urlsafe(12)
336
- _PREVIEWS[token] = {"client": "audit", "at": time.time()}
398
+ _PREVIEWS[token] = {"client": f"audit:{client}", "at": time.time()}
337
399
  return {"installed": False, "file": path, "event": "SessionStart", "change": entry, "token": token,
338
- "expires_s": PREVIEW_TTL_S}
400
+ "expires_s": PREVIEW_TTL_S, **_audit_after(client)}
339
401
 
340
402
 
341
- def audit_install(token, confirm):
342
- """Writes the SessionStart audit entry into Claude Code's settings with a backup; needs the `audit_preview` token
343
- (10 min) and confirm=True. Returns {installed, changed, file, backup}."""
403
+ def audit_install(token, confirm, client="claude"):
404
+ """Writes the SessionStart audit entry into the client's hooks file with a backup; needs the `audit_preview` token
405
+ (10 min) and confirm=True. Returns {installed, changed, file, backup}, plus `after_install` in Codex: the hook
406
+ runs only after the user approves it in /hooks."""
344
407
  planned = _PREVIEWS.pop(token or "", None)
345
- if confirm is not True or not planned or planned["client"] != "audit" or time.time() - planned["at"] > PREVIEW_TTL_S:
408
+ if confirm is not True or not planned or planned["client"] != f"audit:{client}" or time.time() - planned["at"] > PREVIEW_TTL_S:
346
409
  raise ValueError("Turning the audit report on requires the preview token (valid for 10 min) and confirm=true.")
347
- path, data, groups = _audit_plan()
348
- if audit_state() == "current":
410
+ path, data, groups = _audit_plan(client)
411
+ if audit_state(client) in (AUDIT_ON, AUDIT_WAITING):
349
412
  return {"installed": True, "changed": False, "file": path}
350
413
  groups.append(_audit_entry())
351
- backup = _write_checked(path, data, lambda: audit_state() == "current", "The audit hook")
352
- return {"installed": True, "changed": True, "file": path, "backup": backup}
414
+ backup = _write_checked(path, data, lambda: audit_state(client) in (AUDIT_ON, AUDIT_WAITING), "The audit hook")
415
+ return {"installed": True, "changed": True, "file": path, "backup": backup, **_audit_after(client)}
353
416
 
354
417
 
355
- def audit_remove():
356
- """Takes only our SessionStart entry out of Claude Code's settings (after a backup); nothing else is touched."""
357
- path, data, groups = _audit_plan()
358
- if audit_state() == "missing":
418
+ def audit_remove(client="claude"):
419
+ """Takes only our SessionStart entry out of the client's hooks file (after a backup); nothing else is touched."""
420
+ path, data, groups = _audit_plan(client)
421
+ if audit_state(client) == "missing":
359
422
  return {"installed": False, "changed": False, "file": path}
360
423
  if not groups:
361
424
  del data["hooks"]["SessionStart"]
362
- backup = _write_checked(path, data, lambda: audit_state() == "missing", "The removal")
425
+ backup = _write_checked(path, data, lambda: audit_state(client) == "missing", "The removal")
363
426
  return {"installed": False, "changed": True, "file": path, "backup": backup}
package/config.py CHANGED
@@ -62,7 +62,7 @@ DOC_MODES = ("require", "remind", "off")
62
62
  # Edit hook on functions that copy one already indexed (exact or near-identical body): warn = let the edit run and
63
63
  # point to the existing function; off = say nothing. Never blocks: near matches are leads, not defects.
64
64
  DUPLICATE_MODES = ("warn", "off")
65
- # Edit hook on package.json: warn = let the edit run and list the added or changed dependencies that have OSV advisories;
65
+ # Edit hook on package.json, requirements*.txt and go.mod: warn = let the edit run and list the added or changed dependencies that have OSV advisories;
66
66
  # off = say nothing and make no network call.
67
67
  DEPENDENCY_MODES = ("warn", "off")
68
68
  # Router model reviewing, in the background, searches the deterministic redirect rule let run: on = propose block
@@ -1,10 +1,12 @@
1
- """Vulnerable-dependency notice for an edit of package.json, built before the edit runs.
1
+ """Vulnerable-dependency notice for an edit of a dependency manifest, built before the edit runs.
2
2
 
3
- Compares the dependency tables of the file before and after the edit, asks OSV (one `querybatch` call, short timeout)
4
- about the added or changed ones only, and answers from a disk cache of (package, version) -> advisory IDs for 6 hours.
5
- Only package names and versions leave the machine. The caller (`hook_decision`) turns a `DependencyCheckError` into
3
+ Manifests: package.json (OSV ecosystem npm), requirements*.txt (PyPI: `==` pins and the lower bound of `>=` and `~=`)
4
+ and go.mod (Go: `require` lines with a plain vX.Y.Z). Compares the dependencies of the file before and after the edit,
5
+ asks OSV (one `querybatch` call, short timeout) about the added or changed ones only, and answers from a disk cache of
6
+ (ecosystem, package, version) -> advisory IDs for 6 hours. Only package names and versions leave the machine. The caller (`hook_decision`) turns a `DependencyCheckError` into
6
7
  "dependency check skipped: <reason>" and lets the edit go on.
7
8
  """
9
+ import collections
8
10
  import json
9
11
  import os
10
12
  import re
@@ -23,13 +25,17 @@ TABLES = ("dependencies", "devDependencies", "peerDependencies", "optionalDepend
23
25
  PREFIX = "Smart Tool dependencies: "
24
26
  _SPECIFIER_RE = re.compile(r"(\^|~|>=|>|=)?\s*v?(\d+)(?:\.(\d+))?(?:\.(\d+))?"
25
27
  r"((?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?)")
28
+ _REQUIREMENTS_FILE_RE = re.compile(r"requirements[A-Za-z0-9._-]*\.txt")
29
+ _REQUIREMENT_RE = re.compile(r"([A-Za-z0-9][A-Za-z0-9._-]*)\s*(?:\[[^\]]*\])?\s*([^;]*)")
30
+ _PIP_VERSION_RE = re.compile(r"[0-9][0-9A-Za-z.+!_-]*")
31
+ _GO_VERSION_RE = re.compile(r"v\d+\.\d+\.\d+")
26
32
 
27
33
 
28
34
  class DependencyCheckError(RuntimeError):
29
35
  """The check could not run (OSV unreachable or answering badly, unreadable cache, current file not JSON)."""
30
36
 
31
37
 
32
- def _tables(text):
38
+ def _npm_pairs(text):
33
39
  """The (name, specifier) pairs of every dependency table in a package.json text.
34
40
 
35
41
  Parameters: text (str) file content.
@@ -49,32 +55,91 @@ def _tables(text):
49
55
  return pairs
50
56
 
51
57
 
52
- def changed_dependencies(before, after):
53
- """Dependencies added or whose specifier changed between two package.json texts.
58
+ def _requirements_pairs(text):
59
+ """The (name, specifier) pairs of a requirements.txt text; options, includes, editable and URL lines are skipped.
54
60
 
55
- Parameters: before (str or None) text before the edit, None when the file did not exist; after (str) text after.
56
- Returns: list of (name, specifier); empty when `after` is not valid JSON (the user is mid-edit).
57
- Raises: DependencyCheckError when `before` exists but is not valid JSON.
61
+ Parameters: text (str) file content.
62
+ Returns: dict (name, specifier) -> None in file order; the specifier is the version clauses without spaces ("" when
63
+ the line only names the package).
64
+ """
65
+ pairs = {}
66
+ for raw in text.splitlines():
67
+ line = raw.split(" #")[0].split(";")[0].strip().rstrip("\\").strip()
68
+ if not line or line.startswith(("#", "-")):
69
+ continue
70
+ line = " ".join(word for word in line.split() if not word.startswith("--"))
71
+ match = _REQUIREMENT_RE.fullmatch(line)
72
+ if not match:
73
+ continue
74
+ specifier = "".join(match.group(2).split())
75
+ if specifier and specifier[0] not in "=<>!~":
76
+ continue
77
+ pairs[(match.group(1), specifier)] = None
78
+ return pairs
79
+
80
+
81
+ def _go_pairs(text):
82
+ """The (module, version) pairs of the `require` lines of a go.mod text, single-line and block; replace, exclude and
83
+ the other directives are skipped.
84
+
85
+ Parameters: text (str) file content.
86
+ Returns: dict (module, version) -> None in file order.
58
87
  """
59
- now = _tables(after)
88
+ pairs, block = {}, None
89
+ for raw in text.splitlines():
90
+ tokens = raw.split("//")[0].split()
91
+ if not tokens:
92
+ continue
93
+ if block is None:
94
+ if tokens[0] == "require" and len(tokens) == 3:
95
+ pairs[(tokens[1], tokens[2])] = None
96
+ elif tokens[-1] == "(":
97
+ block = tokens[0]
98
+ elif tokens == [")"]:
99
+ block = None
100
+ elif block == "require" and len(tokens) == 2:
101
+ pairs[(tokens[0], tokens[1])] = None
102
+ return pairs
103
+
104
+
105
+ def ecosystem_of(path):
106
+ """The OSV ecosystem of a manifest file name.
107
+
108
+ Parameters: path (str) file path.
109
+ Returns: "npm" for package.json, "PyPI" for requirements*.txt, "Go" for go.mod; None for any other file.
110
+ """
111
+ name = os.path.basename(path)
112
+ if name == "package.json":
113
+ return "npm"
114
+ if name == "go.mod":
115
+ return "Go"
116
+ return "PyPI" if _REQUIREMENTS_FILE_RE.fullmatch(name) else None
117
+
118
+
119
+ def changed_dependencies(before, after, ecosystem="npm"):
120
+ """Dependencies added or whose specifier changed between two manifest texts.
121
+
122
+ Parameters: before (str or None) text before the edit, None when the file did not exist; after (str) text after;
123
+ ecosystem (str) one of ECOSYSTEMS.
124
+ Returns: list of (name, specifier); empty when `after` is a package.json that is not valid JSON (mid-edit).
125
+ Raises: DependencyCheckError when `before` exists but is a package.json that is not valid JSON.
126
+ """
127
+ parse = _ECOSYSTEMS[ecosystem].parse
128
+ now = parse(after)
60
129
  if now is None:
61
130
  return []
62
131
  if before is None or not before.strip():
63
132
  old = {}
64
133
  else:
65
- old = _tables(before)
134
+ old = parse(before)
66
135
  if old is None:
67
136
  raise DependencyCheckError("current package.json is not valid JSON")
68
137
  return [pair for pair in now if pair not in old]
69
138
 
70
139
 
71
- def queried_version(specifier):
72
- """The version to ask OSV about for a specifier.
73
-
74
- Parameters: specifier (str) the value in the dependency table.
75
- Returns: str, the exact version or the lower bound of a ^, ~, >= or > range; None when the specifier cannot be
76
- reduced safely (tags, `*`, URLs, file:, git, workspace:, npm: aliases, x-ranges, compound ranges).
77
- """
140
+ def _npm_version(specifier):
141
+ """The npm case of `queried_version`: the exact version or the lower bound of a ^, ~, >= or > range; None for tags,
142
+ `*`, URLs, file:, git, workspace:, npm: aliases, x-ranges and compound ranges."""
78
143
  match = _SPECIFIER_RE.fullmatch(specifier.strip())
79
144
  if not match:
80
145
  return None
@@ -87,6 +152,42 @@ def queried_version(specifier):
87
152
  return ".".join(parts) + (tail or "")
88
153
 
89
154
 
155
+ def _pip_version(specifier):
156
+ """The PyPI case of `queried_version`: the `==` version, else the first `>=` or `~=` lower bound; None for no
157
+ specifier, `===`, wildcards, `<`, `>`, `!=` alone and anything that is not a version."""
158
+ clauses = specifier.split(",")
159
+ for operators in (("==",), (">=", "~=")):
160
+ for clause in clauses:
161
+ for operator in operators:
162
+ if clause.startswith(operator) and _PIP_VERSION_RE.fullmatch(clause[len(operator):]):
163
+ return clause[len(operator):]
164
+ return None
165
+
166
+
167
+ def _go_version(specifier):
168
+ """The Go case of `queried_version`: a plain vX.Y.Z; None for pseudo-versions, pre-releases and +incompatible."""
169
+ return specifier if _GO_VERSION_RE.fullmatch(specifier) else None
170
+
171
+
172
+ def queried_version(specifier, ecosystem="npm"):
173
+ """The version to ask OSV about for a specifier.
174
+
175
+ Parameters: specifier (str) the value in the dependency table; ecosystem (str) one of ECOSYSTEMS.
176
+ Returns: str, the exact version or the lower bound of the range; None when the specifier cannot be reduced safely.
177
+ """
178
+ return _ECOSYSTEMS[ecosystem].version(specifier)
179
+
180
+
181
+ _Ecosystem = collections.namedtuple("_Ecosystem", "parse version label exact")
182
+ _ECOSYSTEMS = {
183
+ "npm": _Ecosystem(_npm_pairs, _npm_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
184
+ "PyPI": _Ecosystem(_requirements_pairs, _pip_version, lambda name, spec: f"{name}{spec}",
185
+ lambda spec, version: spec == "==" + version),
186
+ "Go": _Ecosystem(_go_pairs, _go_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
187
+ }
188
+ ECOSYSTEMS = tuple(_ECOSYSTEMS)
189
+
190
+
90
191
  def _load_cache():
91
192
  """The cache as {"name@version": {"ids": [...], "at": epoch}}; empty when the file does not exist.
92
193
 
@@ -111,26 +212,27 @@ def _fresh_ids(entry, now):
111
212
  return entry["ids"] if now - entry["at"] < CACHE_TTL_S else None
112
213
 
113
214
 
114
- def _advisories(queries):
215
+ def _advisories(ecosystem, queries):
115
216
  """Advisory IDs for (name, version) pairs, from the cache and one OSV batch for the rest.
116
217
 
117
- Parameters: queries (list of (name, version)).
218
+ Parameters: ecosystem (str) one of ECOSYSTEMS; queries (list of (name, version)).
118
219
  Returns: tuple (list of IDs or None per query in order, str cache-write problem or "").
119
220
  Raises: DependencyCheckError when the cache is corrupt or OSV cannot answer.
120
221
  """
121
222
  cache, now = _load_cache(), time.time()
122
- found = {query: _fresh_ids(cache.get("@".join(query)), now) for query in queries}
223
+ key = lambda query: f"{ecosystem}:{query[0]}@{query[1]}" # noqa: E731
224
+ found = {query: _fresh_ids(cache.get(key(query)), now) for query in queries}
123
225
  missing = [query for query in queries if found[query] is None]
124
226
  if not missing:
125
227
  return [found[query] for query in queries], ""
126
228
  try:
127
- answered = freshness.npm_advisories_batch(missing, OSV_TIMEOUT_S)
229
+ answered = freshness.advisories_batch(ecosystem, missing, OSV_TIMEOUT_S)
128
230
  except RuntimeError as exc:
129
231
  raise DependencyCheckError(str(exc)) from exc
130
232
  for query, ids in zip(missing, answered):
131
233
  found[query] = ids
132
234
  if ids is not None:
133
- cache["@".join(query)] = {"ids": ids, "at": now}
235
+ cache[key(query)] = {"ids": ids, "at": now}
134
236
  cache = {key: entry for key, entry in cache.items() if _fresh_ids(entry, now) is not None}
135
237
  try:
136
238
  os.makedirs(os.path.dirname(CACHE_PATH), exist_ok=True)
@@ -140,28 +242,30 @@ def _advisories(queries):
140
242
  return [found[query] for query in queries], ""
141
243
 
142
244
 
143
- def notice(before, after):
144
- """The dependency notice for one package.json edit.
245
+ def notice(before, after, ecosystem="npm"):
246
+ """The dependency notice for one manifest edit.
145
247
 
146
- Parameters: before (str or None) file text before the edit; after (str) text after it.
248
+ Parameters: before (str or None) file text before the edit; after (str) text after it; ecosystem (str) one of
249
+ ECOSYSTEMS, from `ecosystem_of`.
147
250
  Returns: str, "Smart Tool dependencies: ..." lines for the changed dependencies that have advisories or could not
148
251
  be verified, "" when there is nothing to say.
149
252
  Raises: DependencyCheckError when the check could not run.
150
253
  """
151
- changed = changed_dependencies(before, after)
254
+ spec = _ECOSYSTEMS[ecosystem]
255
+ changed = changed_dependencies(before, after, ecosystem)
152
256
  handled, extra = changed[:MAX_PACKAGES], len(changed) - MAX_PACKAGES
153
257
  lines, queries = {}, []
154
258
  for name, specifier in handled:
155
- version = queried_version(specifier) if freshness.NPM_NAME_RE.fullmatch(name) else None
259
+ version = spec.version(specifier) if freshness.NAME_RES[ecosystem].fullmatch(name) else None
156
260
  if version is None:
157
- lines[(name, specifier)] = f"`{name}@{specifier}` specifier not verified."
261
+ lines[(name, specifier)] = f"`{spec.label(name, specifier)}` specifier not verified."
158
262
  else:
159
263
  queries.append((name, specifier, version))
160
264
  problem = ""
161
265
  if queries:
162
- ids, problem = _advisories([(name, version) for name, _specifier, version in queries])
266
+ ids, problem = _advisories(ecosystem, [(name, version) for name, _specifier, version in queries])
163
267
  for (name, specifier, version), advisories in zip(queries, ids):
164
- label = f"`{name}@{specifier}`" + ("" if version == specifier else f" (checked {version})")
268
+ label = f"`{spec.label(name, specifier)}`" + ("" if spec.exact(specifier, version) else f" (checked {version})")
165
269
  if advisories is None:
166
270
  lines[(name, specifier)] = f"{label} advisory check incomplete, not verified."
167
271
  elif advisories: