@allansantos-dev/smart-tool 0.9.40 → 0.9.42
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +43 -0
- package/README.md +11 -7
- package/client_hooks.py +94 -31
- package/config.py +1 -1
- package/dependency_notice.py +136 -32
- package/document_text.py +274 -47
- package/freshness.py +16 -10
- package/hook_decision.py +9 -7
- package/hooks/session_audit.py +1 -1
- package/index_profile.py +1 -1
- package/index_scope.py +9 -1
- package/indexer.py +4 -2
- package/model_breaker.py +70 -0
- package/model_client.py +17 -10
- package/model_health.py +46 -4
- package/package.json +1 -1
- package/project_monitor.py +17 -2
- package/scout.py +1 -1
- package/setup_ui.py +55 -22
- package/smart_tool_daemon.py +120 -64
- package/usage_meter.py +13 -0
- package/version.py +1 -1
- package/web_fetch.py +7 -7
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,48 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.9.42 - beta
|
|
4
|
+
|
|
5
|
+
- Presentations and workbooks of the project are searchable. `.pptx` text comes out per slide under `[slide N]`, with the
|
|
6
|
+
speaker notes under `[notes N]`; `.xlsx` per sheet under `[sheet NAME]`, one tab-separated line per row, cached formula
|
|
7
|
+
values (nothing is calculated), shared and inline strings resolved. They go through the same zip and XML guards as
|
|
8
|
+
`.docx` and `.pdf` (10 MiB, 2000 entries, no encryption, no DOCTYPE or ENTITY, 8 MiB of XML, 1 MiB of text), plus 100,000
|
|
9
|
+
rows and 16,384 columns for a workbook; a corrupt, hostile or oversized file is skipped with the reason and the rest of the
|
|
10
|
+
project is indexed. `document_text.VERSION` is 3: `.docx` and `.pdf` already indexed are read once more. Measurement
|
|
11
|
+
before and after in `docs/research/data/office-probe-*.json`; tests in `tests/test_office_index.py`. A CSV of 1 MiB is
|
|
12
|
+
indexed and searched in 4 ms; one of 20 MiB is refused as `over size limit` in the index stats, without stalling.
|
|
13
|
+
- The session-start audit report works in Codex. The Setup audit page turns on the SessionStart hook for Claude Code or
|
|
14
|
+
Codex (preview, confirm, backup, remove only ours); Codex runs a new hook only after you approve it in `/hooks`, so the page
|
|
15
|
+
shows "installed, waiting for your approval in /hooks" until the approval is found in `config.toml`, and the first-search
|
|
16
|
+
audit line keeps showing until then. Research with the primary sources in
|
|
17
|
+
`docs/research/2026-10-27-codex-sessionstart-hook.md`; tests in `tests/test_codex_audit_hook.py`.
|
|
18
|
+
- Review fixes: a workbook cell reference past column XFD, a negative shared-string index, a corrupt deflate stream and
|
|
19
|
+
a newline in a sheet name no longer escape as `MemoryError`, `zlib.error`, a wrong text or a forged `[sheet ...]` line;
|
|
20
|
+
the tests of the Codex hook ignore `CODEX_HOME` and `CLAUDE_CONFIG_DIR` so they never write outside their temporary home.
|
|
21
|
+
|
|
22
|
+
## 0.9.41 - beta
|
|
23
|
+
|
|
24
|
+
- Far fewer calls to the embedding and rerank models. Measured on a real machine (24 h): 8303 embedding and 1473 rerank
|
|
25
|
+
calls, 0.3% of them failing; the volume came from four places, all cut: (1) an indexing job no longer embeds a fixed
|
|
26
|
+
text to learn the vector size when the index already stores it for the same model (1300 calls a day); (2) embedding
|
|
27
|
+
batches carry up to 128 chunks and 120,000 characters instead of 32,000 characters (3.8 chunks per call before); a
|
|
28
|
+
request the gateway refuses for its size is split in two halves once; (3) one rerank call per search instead of one
|
|
29
|
+
per block (code, test, doc); the blocks keep the order the separate calls gave; (4) a project in `eager` mode whose
|
|
30
|
+
files change all the time waits longer between jobs (60, 120, 240, 480 up to 900 s, back to 60 s after a quiet
|
|
31
|
+
period), and `*.tmp`, `*.bak`, `*.swp` and `*~` files no longer mark a project as changed or enter the index.
|
|
32
|
+
Measurement before and after in `docs/research/data/call-volume-*.json`, findings and sources in
|
|
33
|
+
`docs/research/2026-10-27-excesso-de-chamadas-embed-rerank.md`. Tests in `tests/test_call_volume.py`.
|
|
34
|
+
- A model that answers 429 or 503 three times in a row opens a circuit for 30 s: calls to it fail at once and the search
|
|
35
|
+
answers from the lexical index with the usual warning, instead of repeating requests. At most 6 requests are in
|
|
36
|
+
flight per model. Rerank and the model check wait the gateway's `Retry-After` (or a jittered exponential delay) before
|
|
37
|
+
asking again, like embeddings.
|
|
38
|
+
- The setup Models page groups the failures by type (429, 503, other 5xx, timeout, connection, authentication, other):
|
|
39
|
+
one row per type and role with the count, first and last time, one sample message, and the latest 20 lines on
|
|
40
|
+
expand. The summary shows the calls per purpose (indexing, query, rerank, model check). `dependency_mode` has its own
|
|
41
|
+
card in the setup rules.
|
|
42
|
+
- The dependency notice also reads `requirements*.txt` (PyPI, `==` pins and the lower bound of `>=` and `~=`) and `go.mod`
|
|
43
|
+
(Go, `require` lines with a plain `vX.Y.Z`). `web_fetch` reads PDFs with the same page and character caps as the
|
|
44
|
+
indexer (`document_text`).
|
|
45
|
+
|
|
3
46
|
## 0.9.40 - beta
|
|
4
47
|
|
|
5
48
|
- PDFs of the project are searchable. `smart_search` indexes the text layer of a `.pdf` inside the project scope and
|
package/README.md
CHANGED
|
@@ -16,7 +16,7 @@ registered as the `playwright` server where this machine lets a browser be drive
|
|
|
16
16
|
An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
|
|
17
17
|
are cheaper, or just tells the agent what Smart Tool would do.
|
|
18
18
|
|
|
19
|
-
**Status:** 0.9.
|
|
19
|
+
**Status:** 0.9.42 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
|
|
20
20
|
|
|
21
21
|
## How it works
|
|
22
22
|
|
|
@@ -129,7 +129,7 @@ Redirects) and Maintenance (Setup audit, Startup).
|
|
|
129
129
|
When a new version recommends other models than the ones you use, a Windows notification (**Apply now**, **Don't
|
|
130
130
|
show again**) and the **New recommended models** card on this page offer them: nothing changes until you apply, the
|
|
131
131
|
models you had are kept, and **Switch back to my previous models** undoes it. The **Model failures (last 24 h)** table
|
|
132
|
-
|
|
132
|
+
groups the failed gateway calls by type (429, 503, other 5xx, timeout, connection, authentication) and role, with the count, the first and last time and one sample, and shows the calls per purpose; a model that fails half of its calls (5 or more) also raises a
|
|
133
133
|
notification and a tray alert. A failure inside a tool call also comes back as a warning line in the tool answer;
|
|
134
134
|
calls made by a hook or by background work (index scope and profile, block-pattern proposals) have no answer to
|
|
135
135
|
carry it, so they show only in the table, the log and the notification. The local embedder is not in that table or in
|
|
@@ -189,9 +189,12 @@ Redirects) and Maintenance (Setup audit, Startup).
|
|
|
189
189
|
A documented function edited only in its body is not mentioned. `project_manage` with `action=docs` lists the
|
|
190
190
|
public functions still without a docstring, for documenting a project that started without them.
|
|
191
191
|
|
|
192
|
-
**Dependency check on edit**: an edit of `package.json` that adds or changes a dependency lists the OSV advisory IDs of up to 5 of them (exact version, or the lower bound of `^`,
|
|
192
|
+
**Dependency check on edit**: an edit of `package.json`, `requirements*.txt` or `go.mod` that adds or changes a dependency lists the OSV advisory IDs of up to 5 of them (exact version, or the lower bound of `^`, `~`, `~=` and `>=`), answered from a 6 h disk cache or one 2.5 s batch call that sends only package names and versions; when OSV cannot answer, the reminder says the check was skipped and the edit goes on. `dependency_mode` in `config.json`: `warn` (default) or `off`.
|
|
193
193
|
|
|
194
|
-
**PDFs** inside a project are indexed as text with `[page N]` markers (20 MiB, 300 pages and 20 s per file; a scanned, password-protected or corrupt PDF is skipped with the reason), so `smart_search` returns the page that holds a passage.
|
|
194
|
+
**PDFs** inside a project are indexed as text with `[page N]` markers (20 MiB, 300 pages and 20 s per file; a scanned, password-protected or corrupt PDF is skipped with the reason), so `smart_search` returns the page that holds a passage. **Presentations and workbooks** (`.pptx`, `.xlsx`) are indexed the
|
|
195
|
+
same way: text of each slide under `[slide N]` with the speaker notes under `[notes N]`, each sheet under `[sheet NAME]` with one
|
|
196
|
+
tab-separated line per row (cached formula values; 100,000 rows or 1 MiB of text at most). Older formats (`.ppt`, `.xls`,
|
|
197
|
+
`.xlsm`) are not read.
|
|
195
198
|
|
|
196
199
|
**Block patterns waiting for you** are shown in one approval panel: 8 per page, the ones that would have blocked
|
|
197
200
|
the most logged calls first, a filter over pattern, reason and example, and bulk accept or reject of the selected
|
|
@@ -232,9 +235,10 @@ Redirects) and Maintenance (Setup audit, Startup).
|
|
|
232
235
|
folder (or the nearest audited parent) without auditing again, and fails with what to do when none is stored. On
|
|
233
236
|
its first `smart_search` of a session, and once per folder, the agent also gets one short line (under 100 tokens)
|
|
234
237
|
when the stored report holds errors or warnings; a clean report adds nothing. Optionally, **Report at session
|
|
235
|
-
start
|
|
236
|
-
exact SessionStart entry, adds it to `~/.claude/settings.json` only after you
|
|
237
|
-
off** removes only that entry.
|
|
238
|
+
start** on the Setup audit page (Claude Code or Codex) moves that line to the start of every new session: it
|
|
239
|
+
previews the exact SessionStart entry, adds it to `~/.claude/settings.json` or `~/.codex/hooks.json` only after you
|
|
240
|
+
confirm (backup first), and **Turn off** removes only that entry. Codex runs a new hook only after you approve it in
|
|
241
|
+
`/hooks`; until then the page shows "waiting for your approval" and the search keeps repeating the line. It is off by default; while it is on, the search no longer repeats the line. The
|
|
238
242
|
hook script (`hooks/session_audit.py`) only reads the stored report from the running service and prints nothing
|
|
239
243
|
when the service is down.
|
|
240
244
|
4. **Local models** (optional): if the mcp-memory embedding sidecar is installed on the machine, its local embedder
|
package/client_hooks.py
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
|
+
import hashlib
|
|
1
2
|
import json
|
|
2
3
|
import os
|
|
3
4
|
import secrets
|
|
4
5
|
import shutil
|
|
5
6
|
import sys
|
|
6
7
|
import time
|
|
8
|
+
import tomllib
|
|
7
9
|
|
|
8
10
|
import daemon_launcher
|
|
9
11
|
import endpoint_sync
|
|
@@ -14,6 +16,9 @@ AUDIT_SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "hooks",
|
|
|
14
16
|
AUDIT_MARKER = "session_audit.py"
|
|
15
17
|
AUDIT_MATCHER = "startup|resume|clear"
|
|
16
18
|
AUDIT_TIMEOUT_S = 10
|
|
19
|
+
AUDIT_CLIENTS = ("claude", "codex")
|
|
20
|
+
AUDIT_ON = "current"
|
|
21
|
+
AUDIT_WAITING = "waiting_approval"
|
|
17
22
|
HTTP_PATH = "/hooks/pretool"
|
|
18
23
|
POST_PATH = "/hooks/posttool"
|
|
19
24
|
# A redirected shell search waits for its answer in the PreToolUse (smart_tool_daemon.HOOK_SHELL_BUDGET_S).
|
|
@@ -282,9 +287,16 @@ def _audit_entry():
|
|
|
282
287
|
"timeout": AUDIT_TIMEOUT_S}]}
|
|
283
288
|
|
|
284
289
|
|
|
285
|
-
def
|
|
286
|
-
"""
|
|
287
|
-
|
|
290
|
+
def _audit_client(client):
|
|
291
|
+
"""The client's config spec; only Claude Code and Codex fire a SessionStart hook whose output the model reads."""
|
|
292
|
+
if client not in AUDIT_CLIENTS:
|
|
293
|
+
raise ValueError(f"The session-start audit report is only available for Claude Code or Codex, not {client!r}.")
|
|
294
|
+
return CLIENTS[client]
|
|
295
|
+
|
|
296
|
+
|
|
297
|
+
def _audit_plan(client="claude"):
|
|
298
|
+
"""(path, data, groups) of the client's hooks file with our SessionStart entry already removed from groups."""
|
|
299
|
+
path = os.path.realpath(_audit_client(client)["files"][0])
|
|
288
300
|
try:
|
|
289
301
|
with open(path, encoding="utf-8") as stream:
|
|
290
302
|
data = json.load(stream)
|
|
@@ -299,10 +311,52 @@ def _audit_plan():
|
|
|
299
311
|
return path, data, groups
|
|
300
312
|
|
|
301
313
|
|
|
302
|
-
def
|
|
303
|
-
"""
|
|
304
|
-
|
|
305
|
-
|
|
314
|
+
def _codex_hash(event, group, index=0):
|
|
315
|
+
"""Codex's approval hash (`sha256:` of the key-sorted compact JSON of the normalized hook identity, as in
|
|
316
|
+
openai/codex hooks/src/engine/discovery.rs `hook_hash`) of the handler at `index` in a hooks.json group."""
|
|
317
|
+
handler = group["hooks"][index]
|
|
318
|
+
normalized = {"type": handler["type"], "command": handler["command"], "timeout": handler.get("timeout", 600),
|
|
319
|
+
"async": bool(handler.get("async", False))}
|
|
320
|
+
if handler.get("statusMessage") is not None:
|
|
321
|
+
normalized["statusMessage"] = handler["statusMessage"]
|
|
322
|
+
identity = {"event_name": event, "hooks": [normalized]}
|
|
323
|
+
if group.get("matcher") is not None:
|
|
324
|
+
identity["matcher"] = group["matcher"]
|
|
325
|
+
text = json.dumps(identity, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
|
|
326
|
+
return "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest()
|
|
327
|
+
|
|
328
|
+
|
|
329
|
+
def _codex_approved(groups):
|
|
330
|
+
"""Our SessionStart handler is trusted in Codex's config.toml: the entry keyed by hooks.json path, event, group
|
|
331
|
+
and handler index holds the hash of the definition as it is now (a missing or different hash means /hooks has not
|
|
332
|
+
approved it, so Codex does not run it). An unreadable config.toml raises ValueError."""
|
|
333
|
+
spec = CLIENTS["codex"]
|
|
334
|
+
try:
|
|
335
|
+
with open(spec["files"][1], encoding="utf-8") as stream:
|
|
336
|
+
hooks = tomllib.loads(stream.read()).get("hooks")
|
|
337
|
+
states = hooks.get("state") if isinstance(hooks, dict) else None
|
|
338
|
+
states = states if isinstance(states, dict) else {}
|
|
339
|
+
except FileNotFoundError:
|
|
340
|
+
return False
|
|
341
|
+
except tomllib.TOMLDecodeError as exc:
|
|
342
|
+
raise ValueError(f"{spec['files'][1]} is not valid TOML: {exc}")
|
|
343
|
+
paths = {spec["files"][0], os.path.realpath(spec["files"][0])}
|
|
344
|
+
for group_index, group in enumerate(groups):
|
|
345
|
+
if not _ours(group, _audit_hook):
|
|
346
|
+
continue
|
|
347
|
+
handler_index = next(i for i, hook in enumerate(group["hooks"]) if _audit_hook(hook))
|
|
348
|
+
for path in paths:
|
|
349
|
+
trusted = (states.get(f"{path}:session_start:{group_index}:{handler_index}") or {}).get("trusted_hash")
|
|
350
|
+
if trusted == _codex_hash("session_start", group, handler_index):
|
|
351
|
+
return True
|
|
352
|
+
return False
|
|
353
|
+
|
|
354
|
+
|
|
355
|
+
def audit_state(client="claude"):
|
|
356
|
+
"""The SessionStart audit hook in the client's hooks file: `current` (our entry equals this version's and, in
|
|
357
|
+
Codex, was approved in /hooks), `waiting_approval` (Codex only: written but not approved, so it does not run yet),
|
|
358
|
+
`outdated` (an entry of ours that differs) or `missing`."""
|
|
359
|
+
path = _audit_client(client)["files"][0]
|
|
306
360
|
try:
|
|
307
361
|
with open(path, encoding="utf-8") as stream:
|
|
308
362
|
data = json.load(stream)
|
|
@@ -312,52 +366,61 @@ def audit_state():
|
|
|
312
366
|
found = [group for group in groups if _ours(group, _audit_hook)] if isinstance(groups, list) else []
|
|
313
367
|
if not found:
|
|
314
368
|
return "missing"
|
|
315
|
-
|
|
369
|
+
if _audit_entry() not in found:
|
|
370
|
+
return "outdated"
|
|
371
|
+
return "current" if client != "codex" or _codex_approved(groups) else AUDIT_WAITING
|
|
316
372
|
|
|
317
373
|
|
|
318
374
|
def audit_hook_installed(client):
|
|
319
|
-
"""The
|
|
320
|
-
counts as off here; `audit_state` raises for it on the setup screen."""
|
|
321
|
-
if client
|
|
375
|
+
"""The session-start audit report is on and running (Codex: approved in /hooks), so a search need not repeat it.
|
|
376
|
+
An unreadable config file counts as off here; `audit_state` raises for it on the setup screen."""
|
|
377
|
+
if client not in AUDIT_CLIENTS:
|
|
322
378
|
return False
|
|
323
379
|
try:
|
|
324
|
-
return audit_state() == "current"
|
|
380
|
+
return audit_state(client) == "current"
|
|
325
381
|
except (OSError, ValueError):
|
|
326
382
|
return False
|
|
327
383
|
|
|
328
384
|
|
|
329
|
-
def
|
|
330
|
-
"""
|
|
331
|
-
|
|
385
|
+
def _audit_after(client):
|
|
386
|
+
"""{after_install: ...} for a client whose new hooks need the user's approval, else nothing."""
|
|
387
|
+
note = CLIENTS[client].get("after_install")
|
|
388
|
+
return {"after_install": note} if note else {}
|
|
389
|
+
|
|
390
|
+
|
|
391
|
+
def audit_preview(client="claude"):
|
|
392
|
+
"""The change that turns the SessionStart audit report on in the client, with a token for `audit_install`."""
|
|
393
|
+
path, _data, _groups = _audit_plan(client)
|
|
332
394
|
entry = _audit_entry()
|
|
333
|
-
if audit_state()
|
|
395
|
+
if audit_state(client) in (AUDIT_ON, AUDIT_WAITING):
|
|
334
396
|
return {"installed": True, "file": path, "change": None}
|
|
335
397
|
token = secrets.token_urlsafe(12)
|
|
336
|
-
_PREVIEWS[token] = {"client": "audit", "at": time.time()}
|
|
398
|
+
_PREVIEWS[token] = {"client": f"audit:{client}", "at": time.time()}
|
|
337
399
|
return {"installed": False, "file": path, "event": "SessionStart", "change": entry, "token": token,
|
|
338
|
-
"expires_s": PREVIEW_TTL_S}
|
|
400
|
+
"expires_s": PREVIEW_TTL_S, **_audit_after(client)}
|
|
339
401
|
|
|
340
402
|
|
|
341
|
-
def audit_install(token, confirm):
|
|
342
|
-
"""Writes the SessionStart audit entry into
|
|
343
|
-
(10 min) and confirm=True. Returns {installed, changed, file, backup}
|
|
403
|
+
def audit_install(token, confirm, client="claude"):
|
|
404
|
+
"""Writes the SessionStart audit entry into the client's hooks file with a backup; needs the `audit_preview` token
|
|
405
|
+
(10 min) and confirm=True. Returns {installed, changed, file, backup}, plus `after_install` in Codex: the hook
|
|
406
|
+
runs only after the user approves it in /hooks."""
|
|
344
407
|
planned = _PREVIEWS.pop(token or "", None)
|
|
345
|
-
if confirm is not True or not planned or planned["client"] != "audit" or time.time() - planned["at"] > PREVIEW_TTL_S:
|
|
408
|
+
if confirm is not True or not planned or planned["client"] != f"audit:{client}" or time.time() - planned["at"] > PREVIEW_TTL_S:
|
|
346
409
|
raise ValueError("Turning the audit report on requires the preview token (valid for 10 min) and confirm=true.")
|
|
347
|
-
path, data, groups = _audit_plan()
|
|
348
|
-
if audit_state()
|
|
410
|
+
path, data, groups = _audit_plan(client)
|
|
411
|
+
if audit_state(client) in (AUDIT_ON, AUDIT_WAITING):
|
|
349
412
|
return {"installed": True, "changed": False, "file": path}
|
|
350
413
|
groups.append(_audit_entry())
|
|
351
|
-
backup = _write_checked(path, data, lambda: audit_state()
|
|
352
|
-
return {"installed": True, "changed": True, "file": path, "backup": backup}
|
|
414
|
+
backup = _write_checked(path, data, lambda: audit_state(client) in (AUDIT_ON, AUDIT_WAITING), "The audit hook")
|
|
415
|
+
return {"installed": True, "changed": True, "file": path, "backup": backup, **_audit_after(client)}
|
|
353
416
|
|
|
354
417
|
|
|
355
|
-
def audit_remove():
|
|
356
|
-
"""Takes only our SessionStart entry out of
|
|
357
|
-
path, data, groups = _audit_plan()
|
|
358
|
-
if audit_state() == "missing":
|
|
418
|
+
def audit_remove(client="claude"):
|
|
419
|
+
"""Takes only our SessionStart entry out of the client's hooks file (after a backup); nothing else is touched."""
|
|
420
|
+
path, data, groups = _audit_plan(client)
|
|
421
|
+
if audit_state(client) == "missing":
|
|
359
422
|
return {"installed": False, "changed": False, "file": path}
|
|
360
423
|
if not groups:
|
|
361
424
|
del data["hooks"]["SessionStart"]
|
|
362
|
-
backup = _write_checked(path, data, lambda: audit_state() == "missing", "The removal")
|
|
425
|
+
backup = _write_checked(path, data, lambda: audit_state(client) == "missing", "The removal")
|
|
363
426
|
return {"installed": False, "changed": True, "file": path, "backup": backup}
|
package/config.py
CHANGED
|
@@ -62,7 +62,7 @@ DOC_MODES = ("require", "remind", "off")
|
|
|
62
62
|
# Edit hook on functions that copy one already indexed (exact or near-identical body): warn = let the edit run and
|
|
63
63
|
# point to the existing function; off = say nothing. Never blocks: near matches are leads, not defects.
|
|
64
64
|
DUPLICATE_MODES = ("warn", "off")
|
|
65
|
-
# Edit hook on package.json: warn = let the edit run and list the added or changed dependencies that have OSV advisories;
|
|
65
|
+
# Edit hook on package.json, requirements*.txt and go.mod: warn = let the edit run and list the added or changed dependencies that have OSV advisories;
|
|
66
66
|
# off = say nothing and make no network call.
|
|
67
67
|
DEPENDENCY_MODES = ("warn", "off")
|
|
68
68
|
# Router model reviewing, in the background, searches the deterministic redirect rule let run: on = propose block
|
package/dependency_notice.py
CHANGED
|
@@ -1,10 +1,12 @@
|
|
|
1
|
-
"""Vulnerable-dependency notice for an edit of
|
|
1
|
+
"""Vulnerable-dependency notice for an edit of a dependency manifest, built before the edit runs.
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
Manifests: package.json (OSV ecosystem npm), requirements*.txt (PyPI: `==` pins and the lower bound of `>=` and `~=`)
|
|
4
|
+
and go.mod (Go: `require` lines with a plain vX.Y.Z). Compares the dependencies of the file before and after the edit,
|
|
5
|
+
asks OSV (one `querybatch` call, short timeout) about the added or changed ones only, and answers from a disk cache of
|
|
6
|
+
(ecosystem, package, version) -> advisory IDs for 6 hours. Only package names and versions leave the machine. The caller (`hook_decision`) turns a `DependencyCheckError` into
|
|
6
7
|
"dependency check skipped: <reason>" and lets the edit go on.
|
|
7
8
|
"""
|
|
9
|
+
import collections
|
|
8
10
|
import json
|
|
9
11
|
import os
|
|
10
12
|
import re
|
|
@@ -23,13 +25,17 @@ TABLES = ("dependencies", "devDependencies", "peerDependencies", "optionalDepend
|
|
|
23
25
|
PREFIX = "Smart Tool dependencies: "
|
|
24
26
|
_SPECIFIER_RE = re.compile(r"(\^|~|>=|>|=)?\s*v?(\d+)(?:\.(\d+))?(?:\.(\d+))?"
|
|
25
27
|
r"((?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?)")
|
|
28
|
+
_REQUIREMENTS_FILE_RE = re.compile(r"requirements[A-Za-z0-9._-]*\.txt")
|
|
29
|
+
_REQUIREMENT_RE = re.compile(r"([A-Za-z0-9][A-Za-z0-9._-]*)\s*(?:\[[^\]]*\])?\s*([^;]*)")
|
|
30
|
+
_PIP_VERSION_RE = re.compile(r"[0-9][0-9A-Za-z.+!_-]*")
|
|
31
|
+
_GO_VERSION_RE = re.compile(r"v\d+\.\d+\.\d+")
|
|
26
32
|
|
|
27
33
|
|
|
28
34
|
class DependencyCheckError(RuntimeError):
|
|
29
35
|
"""The check could not run (OSV unreachable or answering badly, unreadable cache, current file not JSON)."""
|
|
30
36
|
|
|
31
37
|
|
|
32
|
-
def
|
|
38
|
+
def _npm_pairs(text):
|
|
33
39
|
"""The (name, specifier) pairs of every dependency table in a package.json text.
|
|
34
40
|
|
|
35
41
|
Parameters: text (str) file content.
|
|
@@ -49,32 +55,91 @@ def _tables(text):
|
|
|
49
55
|
return pairs
|
|
50
56
|
|
|
51
57
|
|
|
52
|
-
def
|
|
53
|
-
"""
|
|
58
|
+
def _requirements_pairs(text):
|
|
59
|
+
"""The (name, specifier) pairs of a requirements.txt text; options, includes, editable and URL lines are skipped.
|
|
54
60
|
|
|
55
|
-
Parameters:
|
|
56
|
-
Returns:
|
|
57
|
-
|
|
61
|
+
Parameters: text (str) file content.
|
|
62
|
+
Returns: dict (name, specifier) -> None in file order; the specifier is the version clauses without spaces ("" when
|
|
63
|
+
the line only names the package).
|
|
64
|
+
"""
|
|
65
|
+
pairs = {}
|
|
66
|
+
for raw in text.splitlines():
|
|
67
|
+
line = raw.split(" #")[0].split(";")[0].strip().rstrip("\\").strip()
|
|
68
|
+
if not line or line.startswith(("#", "-")):
|
|
69
|
+
continue
|
|
70
|
+
line = " ".join(word for word in line.split() if not word.startswith("--"))
|
|
71
|
+
match = _REQUIREMENT_RE.fullmatch(line)
|
|
72
|
+
if not match:
|
|
73
|
+
continue
|
|
74
|
+
specifier = "".join(match.group(2).split())
|
|
75
|
+
if specifier and specifier[0] not in "=<>!~":
|
|
76
|
+
continue
|
|
77
|
+
pairs[(match.group(1), specifier)] = None
|
|
78
|
+
return pairs
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def _go_pairs(text):
|
|
82
|
+
"""The (module, version) pairs of the `require` lines of a go.mod text, single-line and block; replace, exclude and
|
|
83
|
+
the other directives are skipped.
|
|
84
|
+
|
|
85
|
+
Parameters: text (str) file content.
|
|
86
|
+
Returns: dict (module, version) -> None in file order.
|
|
58
87
|
"""
|
|
59
|
-
|
|
88
|
+
pairs, block = {}, None
|
|
89
|
+
for raw in text.splitlines():
|
|
90
|
+
tokens = raw.split("//")[0].split()
|
|
91
|
+
if not tokens:
|
|
92
|
+
continue
|
|
93
|
+
if block is None:
|
|
94
|
+
if tokens[0] == "require" and len(tokens) == 3:
|
|
95
|
+
pairs[(tokens[1], tokens[2])] = None
|
|
96
|
+
elif tokens[-1] == "(":
|
|
97
|
+
block = tokens[0]
|
|
98
|
+
elif tokens == [")"]:
|
|
99
|
+
block = None
|
|
100
|
+
elif block == "require" and len(tokens) == 2:
|
|
101
|
+
pairs[(tokens[0], tokens[1])] = None
|
|
102
|
+
return pairs
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
def ecosystem_of(path):
|
|
106
|
+
"""The OSV ecosystem of a manifest file name.
|
|
107
|
+
|
|
108
|
+
Parameters: path (str) file path.
|
|
109
|
+
Returns: "npm" for package.json, "PyPI" for requirements*.txt, "Go" for go.mod; None for any other file.
|
|
110
|
+
"""
|
|
111
|
+
name = os.path.basename(path)
|
|
112
|
+
if name == "package.json":
|
|
113
|
+
return "npm"
|
|
114
|
+
if name == "go.mod":
|
|
115
|
+
return "Go"
|
|
116
|
+
return "PyPI" if _REQUIREMENTS_FILE_RE.fullmatch(name) else None
|
|
117
|
+
|
|
118
|
+
|
|
119
|
+
def changed_dependencies(before, after, ecosystem="npm"):
|
|
120
|
+
"""Dependencies added or whose specifier changed between two manifest texts.
|
|
121
|
+
|
|
122
|
+
Parameters: before (str or None) text before the edit, None when the file did not exist; after (str) text after;
|
|
123
|
+
ecosystem (str) one of ECOSYSTEMS.
|
|
124
|
+
Returns: list of (name, specifier); empty when `after` is a package.json that is not valid JSON (mid-edit).
|
|
125
|
+
Raises: DependencyCheckError when `before` exists but is a package.json that is not valid JSON.
|
|
126
|
+
"""
|
|
127
|
+
parse = _ECOSYSTEMS[ecosystem].parse
|
|
128
|
+
now = parse(after)
|
|
60
129
|
if now is None:
|
|
61
130
|
return []
|
|
62
131
|
if before is None or not before.strip():
|
|
63
132
|
old = {}
|
|
64
133
|
else:
|
|
65
|
-
old =
|
|
134
|
+
old = parse(before)
|
|
66
135
|
if old is None:
|
|
67
136
|
raise DependencyCheckError("current package.json is not valid JSON")
|
|
68
137
|
return [pair for pair in now if pair not in old]
|
|
69
138
|
|
|
70
139
|
|
|
71
|
-
def
|
|
72
|
-
"""The version
|
|
73
|
-
|
|
74
|
-
Parameters: specifier (str) the value in the dependency table.
|
|
75
|
-
Returns: str, the exact version or the lower bound of a ^, ~, >= or > range; None when the specifier cannot be
|
|
76
|
-
reduced safely (tags, `*`, URLs, file:, git, workspace:, npm: aliases, x-ranges, compound ranges).
|
|
77
|
-
"""
|
|
140
|
+
def _npm_version(specifier):
|
|
141
|
+
"""The npm case of `queried_version`: the exact version or the lower bound of a ^, ~, >= or > range; None for tags,
|
|
142
|
+
`*`, URLs, file:, git, workspace:, npm: aliases, x-ranges and compound ranges."""
|
|
78
143
|
match = _SPECIFIER_RE.fullmatch(specifier.strip())
|
|
79
144
|
if not match:
|
|
80
145
|
return None
|
|
@@ -87,6 +152,42 @@ def queried_version(specifier):
|
|
|
87
152
|
return ".".join(parts) + (tail or "")
|
|
88
153
|
|
|
89
154
|
|
|
155
|
+
def _pip_version(specifier):
|
|
156
|
+
"""The PyPI case of `queried_version`: the `==` version, else the first `>=` or `~=` lower bound; None for no
|
|
157
|
+
specifier, `===`, wildcards, `<`, `>`, `!=` alone and anything that is not a version."""
|
|
158
|
+
clauses = specifier.split(",")
|
|
159
|
+
for operators in (("==",), (">=", "~=")):
|
|
160
|
+
for clause in clauses:
|
|
161
|
+
for operator in operators:
|
|
162
|
+
if clause.startswith(operator) and _PIP_VERSION_RE.fullmatch(clause[len(operator):]):
|
|
163
|
+
return clause[len(operator):]
|
|
164
|
+
return None
|
|
165
|
+
|
|
166
|
+
|
|
167
|
+
def _go_version(specifier):
|
|
168
|
+
"""The Go case of `queried_version`: a plain vX.Y.Z; None for pseudo-versions, pre-releases and +incompatible."""
|
|
169
|
+
return specifier if _GO_VERSION_RE.fullmatch(specifier) else None
|
|
170
|
+
|
|
171
|
+
|
|
172
|
+
def queried_version(specifier, ecosystem="npm"):
|
|
173
|
+
"""The version to ask OSV about for a specifier.
|
|
174
|
+
|
|
175
|
+
Parameters: specifier (str) the value in the dependency table; ecosystem (str) one of ECOSYSTEMS.
|
|
176
|
+
Returns: str, the exact version or the lower bound of the range; None when the specifier cannot be reduced safely.
|
|
177
|
+
"""
|
|
178
|
+
return _ECOSYSTEMS[ecosystem].version(specifier)
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
_Ecosystem = collections.namedtuple("_Ecosystem", "parse version label exact")
|
|
182
|
+
_ECOSYSTEMS = {
|
|
183
|
+
"npm": _Ecosystem(_npm_pairs, _npm_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
|
|
184
|
+
"PyPI": _Ecosystem(_requirements_pairs, _pip_version, lambda name, spec: f"{name}{spec}",
|
|
185
|
+
lambda spec, version: spec == "==" + version),
|
|
186
|
+
"Go": _Ecosystem(_go_pairs, _go_version, lambda name, spec: f"{name}@{spec}", lambda spec, version: spec == version),
|
|
187
|
+
}
|
|
188
|
+
ECOSYSTEMS = tuple(_ECOSYSTEMS)
|
|
189
|
+
|
|
190
|
+
|
|
90
191
|
def _load_cache():
|
|
91
192
|
"""The cache as {"name@version": {"ids": [...], "at": epoch}}; empty when the file does not exist.
|
|
92
193
|
|
|
@@ -111,26 +212,27 @@ def _fresh_ids(entry, now):
|
|
|
111
212
|
return entry["ids"] if now - entry["at"] < CACHE_TTL_S else None
|
|
112
213
|
|
|
113
214
|
|
|
114
|
-
def _advisories(queries):
|
|
215
|
+
def _advisories(ecosystem, queries):
|
|
115
216
|
"""Advisory IDs for (name, version) pairs, from the cache and one OSV batch for the rest.
|
|
116
217
|
|
|
117
|
-
Parameters: queries (list of (name, version)).
|
|
218
|
+
Parameters: ecosystem (str) one of ECOSYSTEMS; queries (list of (name, version)).
|
|
118
219
|
Returns: tuple (list of IDs or None per query in order, str cache-write problem or "").
|
|
119
220
|
Raises: DependencyCheckError when the cache is corrupt or OSV cannot answer.
|
|
120
221
|
"""
|
|
121
222
|
cache, now = _load_cache(), time.time()
|
|
122
|
-
|
|
223
|
+
key = lambda query: f"{ecosystem}:{query[0]}@{query[1]}" # noqa: E731
|
|
224
|
+
found = {query: _fresh_ids(cache.get(key(query)), now) for query in queries}
|
|
123
225
|
missing = [query for query in queries if found[query] is None]
|
|
124
226
|
if not missing:
|
|
125
227
|
return [found[query] for query in queries], ""
|
|
126
228
|
try:
|
|
127
|
-
answered = freshness.
|
|
229
|
+
answered = freshness.advisories_batch(ecosystem, missing, OSV_TIMEOUT_S)
|
|
128
230
|
except RuntimeError as exc:
|
|
129
231
|
raise DependencyCheckError(str(exc)) from exc
|
|
130
232
|
for query, ids in zip(missing, answered):
|
|
131
233
|
found[query] = ids
|
|
132
234
|
if ids is not None:
|
|
133
|
-
cache[
|
|
235
|
+
cache[key(query)] = {"ids": ids, "at": now}
|
|
134
236
|
cache = {key: entry for key, entry in cache.items() if _fresh_ids(entry, now) is not None}
|
|
135
237
|
try:
|
|
136
238
|
os.makedirs(os.path.dirname(CACHE_PATH), exist_ok=True)
|
|
@@ -140,28 +242,30 @@ def _advisories(queries):
|
|
|
140
242
|
return [found[query] for query in queries], ""
|
|
141
243
|
|
|
142
244
|
|
|
143
|
-
def notice(before, after):
|
|
144
|
-
"""The dependency notice for one
|
|
245
|
+
def notice(before, after, ecosystem="npm"):
|
|
246
|
+
"""The dependency notice for one manifest edit.
|
|
145
247
|
|
|
146
|
-
Parameters: before (str or None) file text before the edit; after (str) text after it
|
|
248
|
+
Parameters: before (str or None) file text before the edit; after (str) text after it; ecosystem (str) one of
|
|
249
|
+
ECOSYSTEMS, from `ecosystem_of`.
|
|
147
250
|
Returns: str, "Smart Tool dependencies: ..." lines for the changed dependencies that have advisories or could not
|
|
148
251
|
be verified, "" when there is nothing to say.
|
|
149
252
|
Raises: DependencyCheckError when the check could not run.
|
|
150
253
|
"""
|
|
151
|
-
|
|
254
|
+
spec = _ECOSYSTEMS[ecosystem]
|
|
255
|
+
changed = changed_dependencies(before, after, ecosystem)
|
|
152
256
|
handled, extra = changed[:MAX_PACKAGES], len(changed) - MAX_PACKAGES
|
|
153
257
|
lines, queries = {}, []
|
|
154
258
|
for name, specifier in handled:
|
|
155
|
-
version =
|
|
259
|
+
version = spec.version(specifier) if freshness.NAME_RES[ecosystem].fullmatch(name) else None
|
|
156
260
|
if version is None:
|
|
157
|
-
lines[(name, specifier)] = f"`{name
|
|
261
|
+
lines[(name, specifier)] = f"`{spec.label(name, specifier)}` specifier not verified."
|
|
158
262
|
else:
|
|
159
263
|
queries.append((name, specifier, version))
|
|
160
264
|
problem = ""
|
|
161
265
|
if queries:
|
|
162
|
-
ids, problem = _advisories([(name, version) for name, _specifier, version in queries])
|
|
266
|
+
ids, problem = _advisories(ecosystem, [(name, version) for name, _specifier, version in queries])
|
|
163
267
|
for (name, specifier, version), advisories in zip(queries, ids):
|
|
164
|
-
label = f"`{name
|
|
268
|
+
label = f"`{spec.label(name, specifier)}`" + ("" if spec.exact(specifier, version) else f" (checked {version})")
|
|
165
269
|
if advisories is None:
|
|
166
270
|
lines[(name, specifier)] = f"{label} advisory check incomplete, not verified."
|
|
167
271
|
elif advisories:
|