@allansantos-dev/smart-tool 0.9.17 → 0.9.19

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,35 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.9.19 - beta
4
+
5
+ - A search no longer waits for the index to catch up. It answers from the existing index at once, searches the files
6
+ edited since the last index by keyword straight from disk, and says "Indexing in progress: N file(s) changed..." in
7
+ the answer while the update runs in the background. An index that does not exist yet, or was built by another model,
8
+ is still built first.
9
+ - A search waits up to 25 s for its own result before answering with a `job_id` (it was 3 s, below the 3.3 s a warm
10
+ search takes at best, so nearly every call needed a second `smart_search_result`).
11
+ - The scope and the language profile are no longer asked of the model inside a search that has an index: the saved ones
12
+ are used and refreshed in the background.
13
+ - New folders are registered watched and eager: the index follows edits by itself. A PreToolUse Edit/Write of an eager
14
+ folder without a running watcher queues one debounced update with the edited paths. Folders already registered keep
15
+ their policy.
16
+ - A `top_k` whose total is above 20 is scaled down in proportion instead of rejecting the call (the rejection cost the
17
+ agent a turn on its first search).
18
+ - A search made right after an edit keeps the just-edited chunk in the answer: the reranker can no longer push it out.
19
+ - An edit whose path comes in the Windows 8.3 short form (`ALLANS~1`-style folder names) is matched to its registered
20
+ project; before, the project lookup compared it with the long form and found nothing.
21
+ - The daemon log prints the seconds spent in each phase of a job (`phases=queued:0.02,auth:0.01,...`).
22
+
23
+ ## 0.9.18 - beta
24
+
25
+ - Updates are no longer applied only at logon. The tray checks npm at start and every 24 h and installs a newer
26
+ version as soon as it sees it (on by default); a version that failed to install is tried again 24 h later instead of
27
+ never. `SMART_TOOL_AUTO_UPDATE=off` turns automatic updates off, like the tray switch.
28
+ - Nothing about updates is hidden. The tray notifies when a newer version is available, when it starts updating, when
29
+ Smart Tool was updated (from which version to which) and when an update failed. Agent sessions get the same line once
30
+ on their first `smart_search`, with the link to the changes, and the SessionStart report (when on) shows it to you
31
+ at session start.
32
+
3
33
  ## 0.9.17 - beta
4
34
 
5
35
  - The agent learns the setup audit result without asking for it. On its first `smart_search` of a session, once per
package/README.md CHANGED
@@ -15,7 +15,7 @@ registered as the `playwright` server where this machine lets a browser be drive
15
15
  An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
16
16
  are cheaper, or just tells the agent what Smart Tool would do.
17
17
 
18
- **Status:** 0.9.17 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
18
+ **Status:** 0.9.19 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
19
19
 
20
20
  ## How it works
21
21
 
@@ -76,9 +76,13 @@ gateway such as Ollama, nothing about your code leaves the machine.
76
76
  npx @allansantos-dev/smart-tool@latest install
77
77
  ```
78
78
 
79
- Updates install themselves: when the tray starts (at logon) and npm has a newer version, it runs this same command in
80
- a visible console, once per version; the tray menu has "Update to X" (or "Check for updates") on demand and an
81
- "Update automatically" switch. Running the command by hand also updates; `~/.smart-tool` is kept. It downloads [uv](https://docs.astral.sh/uv/) (checksum
79
+ Updates install themselves, on by default: the tray checks npm when it starts (at logon) and every 24 h, and when there
80
+ is a newer version it runs this same command in a visible console (once per version; a failed version is tried again
81
+ after 24 h, with the previous version kept). You are told each time: a Windows notification from the tray (newer version,
82
+ updating, updated, update failed), and agent sessions get the same line on their first `smart_search` (and at session
83
+ start when the SessionStart report is on, shown to you too) with the link to the changes. The tray menu has "Update to
84
+ X" (or "Check for updates") on demand and an "Update automatically" switch; `SMART_TOOL_AUTO_UPDATE=off` also turns
85
+ automatic updates off, and the newer-version notice then carries the command to run. Running the command by hand also updates; `~/.smart-tool` is kept. It downloads [uv](https://docs.astral.sh/uv/) (checksum
82
86
  verified) and a private Python 3.12 into `~/.smart-tool/runtime`, with nothing added to `PATH` or the registry, then
83
87
  copies Smart Tool to `%LOCALAPPDATA%\Programs\SmartTool` with its own virtual environment,
84
88
  installs the web runtime, registers a per-user Scheduled Task that starts the tray at logon and starts the daemon on
@@ -0,0 +1,13 @@
1
+ ---
2
+ name: smart-scout
3
+ description: Optional helper for the Smart Tool. Use it only for a task that chains several Smart Tool calls (search, then graph, then affected_tests) or while the index is still loading. A plain search is faster called directly.
4
+ model: haiku
5
+ tools: mcp__smart-tool__smart_search, mcp__smart-tool__smart_search_result, mcp__smart-tool__project_manage, Read
6
+ ---
7
+ You run Smart Tool calls for another session and return only the result.
8
+
9
+ Use smart_search with project_root equal to the working directory (forward slashes), query_identifiers in English identifier terms, query_comments in the language of the comments, and top_k {"code": 6, "test": 2, "doc": 2}. When it answers pending, call smart_search_result with the job_id until it finishes. project_manage action=graph (symbol) and action=affected_tests are yours to chain when the task asks for them. Open at most 3 files with Read and make at most 8 tool calls in total.
10
+
11
+ Reply ONLY with a list of up to 8 items, each `path:first_line-last_line: reason in up to 15 words`, no code excerpts and no introduction.
12
+
13
+ If nothing relevant came back after two different queries, reply exactly `EMPTY: <the two queries you tried>`. Never invent a path. Never use Grep, shell search or Read of folders to compensate: an empty answer is a valid answer, and the session that called you decides whether to retry with smart_search or to report that the code was not found.
package/hook_decision.py CHANGED
@@ -171,10 +171,8 @@ def advise(reason):
171
171
 
172
172
 
173
173
  def _project_of(path):
174
- target = os.path.normcase(os.path.abspath(path))
175
- roots = [p["root"] for p in project_store.all_projects()
176
- if target.startswith(os.path.normcase(os.path.abspath(p["root"])) + os.sep)]
177
- return max(roots, key=len) if roots else None
174
+ project = project_store.containing(path)
175
+ return project["root"] if project else None
178
176
 
179
177
 
180
178
  def _impact_line(root, rel, name):
package/indexer.py CHANGED
@@ -779,9 +779,14 @@ def _vector_rows(root, path):
779
779
  return rows, meta, norms
780
780
 
781
781
 
782
+ def _query_terms(query):
783
+ """Up to 24 distinct word terms of the query, in order."""
784
+ return list(dict.fromkeys(re.findall(r"[\w]{2,}", query or "", re.UNICODE)))[:24]
785
+
786
+
782
787
  def _lexical_scored(path, query, rows, limit, offset=0):
783
788
  """(chunk_id, score) best first; higher score is a better lexical match."""
784
- terms = list(dict.fromkeys(re.findall(r"[\w]{2,}", query or "", re.UNICODE)))[:24]
789
+ terms = _query_terms(query)
785
790
  if not terms:
786
791
  return []
787
792
  expression = " OR ".join('"' + term.replace('"', '""') + '"' for term in terms)
@@ -808,11 +813,72 @@ def _min_max(pairs):
808
813
  return {key: (value - low) / (high - low) if high > low else 1.0 for key, value in pairs}
809
814
 
810
815
 
811
- def search(root, query_vector=None, top_k=20, query=None, model_id=None, include=None):
812
- """Recuperação vetorial + lexical por fusão de notas relativas, sem rede."""
816
+ def stale_files(root, included_paths):
817
+ """`(changed, removed)` against the indexed state: `changed` lists in-scope files whose `(mtime_ns, size)` differs
818
+ from the stored one (or that were never indexed), `removed` lists indexed files that left the scope. It only
819
+ stats files, so it is the cheap answer to "what would a reindex have to touch"."""
820
+ snapshot = manifest_snapshot(root)
821
+ changed, removed = [], set(snapshot)
822
+ for rel in sorted(set(included_paths)):
823
+ stat, _reason = _stat_candidate(root, rel)
824
+ if stat is None:
825
+ continue
826
+ removed.discard(rel)
827
+ if snapshot.get(rel) != stat:
828
+ changed.append(rel)
829
+ return changed, sorted(removed)
830
+
831
+
832
+ def overlay_chunks(root, rel_paths):
833
+ """`[(path, start_line, end_line, text)]` read from disk right now for the given files, chunked like the index.
834
+ Files that cannot be read consistently are skipped: the overlay only complements the index, it never fails a
835
+ search."""
836
+ chunks = []
837
+ for rel in rel_paths:
838
+ data, encoding, _stat = _stable_read(root, rel)
839
+ if data is None:
840
+ continue
841
+ pieces = chunk_file(root, rel, CHUNK_LINES, data=data, encoding=encoding)
842
+ chunks.extend((rel, start, end, text) for start, end, text in pieces or [])
843
+ return chunks
844
+
845
+
846
+ def _overlay_scored(chunks, query, alpha):
847
+ """`[(score, path, start, end, text)]` for overlay chunks matching the query terms by keyword. A chunk that holds
848
+ every term scores 1.0, the top of the fused index scores (`search` ranks it first on a tie): the edited file is read
849
+ from disk, so it is the current answer for an exact match. A partial match stays on the 0-1 scale of the lexical
850
+ share of the fused score."""
851
+ terms = _query_terms(query)
852
+ scored = []
853
+ for path, start, end, text in chunks:
854
+ haystack = (path + " " + text).casefold()
855
+ matched = sum(term.casefold() in haystack for term in terms)
856
+ if matched:
857
+ score = 1.0 if matched == len(terms) else (1 - alpha) * matched / len(terms)
858
+ scored.append((score, path, start, end, text))
859
+ return scored
860
+
861
+
862
+ def overlay_exact_keys(overlay, query):
863
+ """`{(path, start, end)}` of the overlay chunks that hold every query term. A reranker must not push them below
864
+ indexed chunks: they come from the file as it is on disk now."""
865
+ if not overlay or not overlay["chunks"] or not query:
866
+ return set()
867
+ return {(path, start, end) for score, path, start, end, _text in _overlay_scored(overlay["chunks"], query, 0.0)
868
+ if score == 1.0}
869
+
870
+
871
+ def search(root, query_vector=None, top_k=20, query=None, model_id=None, include=None, overlay=None):
872
+ """Vector plus lexical retrieval by fusion of relative scores, without network. `overlay` is
873
+ `{"drop": paths, "chunks": overlay_chunks(...)}`: the indexed chunks of the `drop` paths (stale or removed
874
+ files) are ignored and the overlay chunks, read from disk, take part by keyword instead."""
813
875
  path = existing_db_path(root)
814
876
  if not path:
815
877
  return []
878
+ kind_filter = include
879
+ if overlay and overlay["drop"]:
880
+ dropped = set(overlay["drop"])
881
+ include = lambda rel: rel not in dropped and (kind_filter is None or kind_filter(rel))
816
882
  rows, meta, norms = _vector_rows(root, path)
817
883
  if include is not None:
818
884
  rows = [row for row in rows if include(row[1])]
@@ -854,7 +920,14 @@ def search(root, query_vector=None, top_k=20, query=None, model_id=None, include
854
920
  else:
855
921
  scores, ids = vector_scores, vector_ids
856
922
  by_id = {row[0]: row for row in rows}
857
- return [(scores[chunk_id], *by_id[chunk_id][1:5]) for chunk_id in ids[:top_k]]
923
+ found = [(scores[chunk_id], *by_id[chunk_id][1:5]) for chunk_id in ids[:top_k]]
924
+ if overlay and overlay["chunks"]:
925
+ extra = _overlay_scored([chunk for chunk in overlay["chunks"] if kind_filter is None or kind_filter(chunk[0])],
926
+ query, FUSION_ALPHA if query_vector is not None else 0.0)
927
+ found = [item for _score, item in sorted(
928
+ [((item[0], 1), item) for item in extra] + [((item[0], 0), item) for item in found],
929
+ key=lambda pair: pair[0], reverse=True)][:top_k]
930
+ return found
858
931
 
859
932
 
860
933
  def graph_cache_path(view_path):
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@allansantos-dev/smart-tool",
3
- "version": "0.9.17",
3
+ "version": "0.9.19",
4
4
  "description": "Local MCP server that gives coding agents (Claude Code, Codex) cheaper, sharper tools than their built-in search. Windows.",
5
5
  "license": "Apache-2.0",
6
6
  "author": "Allan Santos",
package/project_store.py CHANGED
@@ -40,8 +40,8 @@ def register(root, watch=None):
40
40
  row = db.execute("SELECT data FROM projects WHERE id=?", (key,)).fetchone()
41
41
  data = json.loads(row[0]) if row else {
42
42
  "id": key, "root": display, "name": os.path.basename(display) or display,
43
- "status": "registered", "watch": False, "paused": False, "enabled": False,
44
- "update_mode": "on_search",
43
+ "status": "registered", "watch": True, "paused": False, "enabled": False,
44
+ "update_mode": "eager",
45
45
  "dirty_seq": 0, "indexed_seq": 0, "created_at": time.time(),
46
46
  "last_indexed": None, "last_checked": None, "last_error": "",
47
47
  "last_job_id": None, "watch_backend": "off", "next_retry_at": 0,
@@ -60,6 +60,13 @@ def get(project_id):
60
60
  return json.loads(row[0]) if row else None
61
61
 
62
62
 
63
+ def containing(path):
64
+ """The registered project whose folder holds `path` (the deepest one when projects nest), or `None`."""
65
+ target = project_identity.canonical_root(path)
66
+ inside = [p for p in all_projects() if target.startswith(project_identity.canonical_root(p["root"]) + os.sep)]
67
+ return max(inside, key=lambda p: len(p["root"])) if inside else None
68
+
69
+
63
70
  def all_projects():
64
71
  with _LOCK, _db() as db:
65
72
  rows = db.execute("SELECT data FROM projects ORDER BY root COLLATE NOCASE").fetchall()
@@ -68,6 +68,7 @@ import local_embedder
68
68
  import web_search_health
69
69
  import web_fetch
70
70
  import version
71
+ import update_check
71
72
  import gateway
72
73
  import hook_decision
73
74
  import endpoint_sync
@@ -171,7 +172,11 @@ _capabilities_cache = None
171
172
  _jobs = {}
172
173
  _JOBS_LOCK = threading.Lock()
173
174
  _MAX_JOBS_RETAINED = 200
174
- SMART_SEARCH_SYNC_WAIT_S = 3
175
+ # A warm search never took less than 3.3 s at its fastest 5% and the median is 8.6 s: waiting less answers pending to
176
+ # nearly every call and costs the agent extra smart_search_result round trips. Codex gives an MCP call 60 s by default.
177
+ SMART_SEARCH_SYNC_WAIT_S = 25
178
+ OVERLAY_MAX_FILES = 40
179
+ EDIT_INDEX_DELAY_S = 8
175
180
  # smart_search_result waits for the job this long before answering pending: answering at once had an agent poll the
176
181
  # same job 7 times in 16 s and then give up on Smart Tool (claude -p session, 2026-10-09; job durations median 6 s,
177
182
  # p75 10.7 s).
@@ -231,6 +236,8 @@ def _update_job(job_id, **fields):
231
236
  current = _jobs.get(job_id)
232
237
  if current is not None and current.get("status") == "pending":
233
238
  updated = {**current, **fields, "updated_at": time.time()}
239
+ if fields.get("phase") and fields["phase"] != current.get("phase"):
240
+ updated["phases"] = [*current.get("phases", []), [fields["phase"], time.time()]]
234
241
  _jobs[job_id] = updated
235
242
  if (fields.get("phase") != current.get("phase") or
236
243
  time.monotonic() - _PERSIST_LAST.get(job_id, 0) > .5 or fields.get("cancel_requested")):
@@ -240,6 +247,20 @@ def _update_job(job_id, **fields):
240
247
  _persist_project_job(job_id, persistent)
241
248
 
242
249
 
250
+ def _phase_timings(job):
251
+ """"phase:seconds" for each phase of a job in the order it ran; the last phase lasts until the job finished."""
252
+ marks = [("queued", job.get("created_at") or 0), *[tuple(mark) for mark in job.get("phases", [])]]
253
+ end = job.get("finished_at") or time.time()
254
+ spans = {}
255
+ for (name, start), following in zip(marks, [*marks[1:], ("", end)]):
256
+ spans[name] = spans.get(name, 0.0) + max(following[1] - start, 0.0)
257
+ return ",".join(f"{name}:{seconds:.2f}" for name, seconds in spans.items())
258
+
259
+
260
+ _EDIT_PENDING = {}
261
+ _EDIT_LOCK = threading.Lock()
262
+
263
+
243
264
  def _project_lock(root):
244
265
  key = project_identity.canonical_root(root)
245
266
  with _PROJECT_LOCKS_LOCK:
@@ -321,7 +342,10 @@ SERVER_INSTRUCTIONS = (
321
342
  "with query). Never retry the same search or download through Bash, python, node, PowerShell, curl or wget: that "
322
343
  "bypasses the rule the user set. Reading one known file, or searching inside one known file, is never blocked. "
323
344
  "Before changing a function, project_manage action=graph with symbol lists its callers and tests; after editing, "
324
- "project_manage action=affected_tests lists the tests to run."
345
+ "project_manage action=affected_tests lists the tests to run. When smart_search returns nothing, retry smart_search with "
346
+ "other terms; do not switch to Grep, Read of folders or shell search. A helper agent (smart-scout, a cheap model) is "
347
+ "optional: use it only for a task that chains several Smart Tool calls or while the index is still loading; a plain "
348
+ "search is faster called directly. A helper that returns empty means retry with the tool or report not found."
325
349
  )
326
350
 
327
351
  TOOLS = [
@@ -376,7 +400,10 @@ TOOLS = [
376
400
  "query_comments with the same request in the comments' language; when the profile shows two "
377
401
  "languages both are required, or explain in single_language_reason. Put documentation searches in "
378
402
  "doc_query. Code and test results list the likeliest functions as name:line in symbols. Tune top_k per block for the project (project_manage search_limits). Fast searches return results directly. If work takes longer than a few seconds, "
379
- "a pending job_id is returned; call smart_search_result with that ID until done."
403
+ "a pending job_id is returned; call smart_search_result with that ID until done. A warning that starts "
404
+ "with 'Indexing in progress' means the files named there changed after the last index: they were "
405
+ "searched by keyword from disk and the rest of the answer is current; repeat the search later for their "
406
+ "semantic ranking."
380
407
  ),
381
408
  "inputSchema": {
382
409
  "type": "object",
@@ -813,6 +840,110 @@ def _agent_scope_request(root, scope, problem):
813
840
  "(always included unless listed in exclude).")
814
841
 
815
842
 
843
+ def _profile_due(scope, scope_model):
844
+ """Whether the saved scope lacks a current language profile and the model may be asked for one now, that is, a
845
+ scope model is configured and the pause after a failed attempt has passed."""
846
+ failed_at = (scope.get("profile_error") or {}).get("at", 0)
847
+ return bool(scope_model and (scope.get("profile") or {}).get("version") != index_profile.PROFILE_VERSION
848
+ and time.time() - failed_at >= index_profile.RETRY_AFTER_S)
849
+
850
+
851
+ def _index_lag(root, model, cfg, problem, job_id):
852
+ """What a search can serve without waiting for the index. `None` when the existing index cannot be used as it is
853
+ (missing, built by another model, chunker or view, no saved scope, or a scope that needs the model while it is
854
+ unavailable): the caller then updates it before searching. Otherwise `{"changed", "removed", "refresh"}` with the
855
+ in-scope files that differ from the indexed state, the indexed files that left the scope, and whether a
856
+ background update is due (stale files, or a scope or profile the model has to refresh)."""
857
+ project = project_store.get(project_identity.project_id(root)) or {}
858
+ info = indexer.info(root)
859
+ view = index_views.describe(root)
860
+ scope = index_scope.load_scope(root)
861
+ if (not info.get("chunks") or info.get("model_id") != model or not index_scope.valid_scope(scope)
862
+ or info.get("chunker_version") != indexer.CHUNKER_VERSION
863
+ or info.get("schema_version") != indexer.SCHEMA_VERSION
864
+ or info.get("chunk_lines") != indexer.CHUNK_LINES
865
+ or (view.get("git") and info.get("view_id") != view.get("view_id"))
866
+ or project.get("scope_dirty")):
867
+ return None
868
+ scope_model = cfg.get("scope_model") or cfg.get("router_model")
869
+ rescan = index_scope.needs_rescan(root, scope)
870
+ if rescan and (not scope_model or problem):
871
+ return None
872
+ _update_job(job_id, phase="listing")
873
+ changed, removed = indexer.stale_files(root, index_scope.resolve_included_files(root, scope))
874
+ return {"changed": changed, "removed": removed,
875
+ "refresh": bool(changed or removed or rescan or _profile_due(scope, scope_model))}
876
+
877
+
878
+ def _lag_overlay(root, lag):
879
+ """`(overlay, warning)` for `indexer.search`: the stale files read from disk, searched by keyword in place of their
880
+ outdated chunks, and the notice telling the agent that indexing is in progress. `(None, "")` when the index is
881
+ current."""
882
+ stale = lag["changed"] + lag["removed"]
883
+ if not stale:
884
+ return None, ""
885
+ shown = lag["changed"][:OVERLAY_MAX_FILES]
886
+ skipped = len(lag["changed"]) - len(shown)
887
+ names = ", ".join(stale[:3]) + (f" (+{len(stale) - 3})" if len(stale) > 3 else "")
888
+ warning = (f"Indexing in progress: {len(stale)} file(s) changed after the last index ({names}). Results come from "
889
+ "the existing index; those files were searched by keyword straight from disk, without semantic "
890
+ "ranking" + (f", and {skipped} of them were not searched" if skipped else "") +
891
+ ". The index updates in the background: repeat the search later for the updated ranking.")
892
+ return {"drop": stale, "chunks": indexer.overlay_chunks(root, shown)}, warning
893
+
894
+
895
+ def _refresh_in_background(root, lag):
896
+ """Queues the index update that the search did not wait for. Returns the notice to show when it could not be
897
+ queued, `""` when it was queued or none was due."""
898
+ if not lag["refresh"]:
899
+ return ""
900
+ try:
901
+ _submit_project_job("project_index", {"project_root": root, "_origin": "search"})
902
+ except RuntimeError as exc:
903
+ return f"The background index update could not start ({exc}); the next search tries again."
904
+ return ""
905
+
906
+
907
+ def _note_edit(payload):
908
+ """Queues a debounced background index update for a file an agent is about to write, in an eager project whose
909
+ watcher is not running. The PreToolUse hook sees the edit before it lands, so the update waits
910
+ `EDIT_INDEX_DELAY_S` and groups the edits of a burst into one job with `changed_paths`."""
911
+ if not isinstance(payload, dict) or payload.get("tool_name") not in ("Edit", "MultiEdit", "Write"):
912
+ return
913
+ target = (payload.get("tool_input") or {}).get("file_path")
914
+ if not isinstance(target, str) or not os.path.isabs(target):
915
+ return
916
+ project = project_store.containing(target)
917
+ if (not project or not project.get("enabled") or project.get("paused") or project.get("update_mode") != "eager"
918
+ or project.get("watch_backend", "off") != "off"):
919
+ return
920
+ rel = os.path.relpath(project_identity.display_root(target), project["root"]).replace(os.sep, "/")
921
+ with _EDIT_LOCK:
922
+ pending = _EDIT_PENDING.get(project["id"])
923
+ if pending is not None:
924
+ pending.add(rel)
925
+ return
926
+ _EDIT_PENDING[project["id"]] = {rel}
927
+ timer = threading.Timer(EDIT_INDEX_DELAY_S, _flush_edits, args=(project["id"],))
928
+ timer.daemon = True
929
+ timer.start()
930
+
931
+
932
+ def _flush_edits(project_id):
933
+ """Submits the index update for the files `_note_edit` collected; a failure is logged because it runs on a timer
934
+ thread nobody waits for."""
935
+ with _EDIT_LOCK:
936
+ paths = _EDIT_PENDING.pop(project_id, set())
937
+ project = project_store.get(project_id)
938
+ if not paths or not project or project.get("paused") or not project.get("enabled"):
939
+ return
940
+ try:
941
+ _submit_project_job("project_index", {"project_root": project["root"], "changed_paths": sorted(paths),
942
+ "_origin": "edit"})
943
+ except Exception as exc:
944
+ _log(f"edit index update not queued: {type(exc).__name__}: {exc}")
945
+
946
+
816
947
  def _prepare_project_index(arguments, job_id, cfg, token, query_vector=None, preview=False, model=None):
817
948
  root = arguments["project_root"]
818
949
  project = project_store.get(project_identity.project_id(root)) or {}
@@ -834,9 +965,7 @@ def _prepare_project_index(arguments, job_id, cfg, token, query_vector=None, pre
834
965
  deadline=_job_deadline(job_id), cancel_check=checkpoint)
835
966
  _update_job(job_id, phase="listing")
836
967
  included = index_scope.resolve_included_files(root, scope, cancel_check=checkpoint)
837
- failed_at = (scope.get("profile_error") or {}).get("at", 0)
838
- if ((scope.get("profile") or {}).get("version") != index_profile.PROFILE_VERSION
839
- and time.time() - failed_at >= index_profile.RETRY_AFTER_S and scope_model):
968
+ if _profile_due(scope, scope_model):
840
969
  _update_job(job_id, phase="profile")
841
970
  try:
842
971
  profile = index_profile.decide(root, included, scope_model, deadline=_job_deadline(job_id))
@@ -917,6 +1046,8 @@ def _grouped_yaml(profile, blocks, warning):
917
1046
 
918
1047
  LOCAL_EMBEDDER_DOWN_WARNING = ("The mcp-memory local embedder is configured but not responding: index and search are lexical only "
919
1048
  "until it is back (check the mcp-memory embedding service).")
1049
+ EMPTY_RESULT_NOTICE = ("No matches. Do not fall back to Grep, Read of folders or shell search: retry smart_search with other "
1050
+ "identifier terms, another group, or a wider top_k, and if it stays empty report that the code was not found.")
920
1051
  LEXICAL_WARNING = ("No embedding model: index and search are lexical only (exact words, no synonyms or cross-language matches). "
921
1052
  "Configure an embedding_model for semantic search.")
922
1053
 
@@ -938,6 +1069,7 @@ def _perform_smart_search_isolated(arguments, job_id):
938
1069
  model = local_embedder.resolve(cfg.get("embedding_model"), indexer.info(root).get("model_id"))
939
1070
  warnings = []
940
1071
  vectors = {}
1072
+ overlay = None
941
1073
  search_model = model
942
1074
  if problem:
943
1075
  model_client.OFFLINE.set(problem)
@@ -974,18 +1106,23 @@ def _perform_smart_search_isolated(arguments, job_id):
974
1106
  warnings.append("Embedding unavailable; lexical results from the last ready index. The update was not confirmed. " + _project_error(exc))
975
1107
  search_model = None
976
1108
  if vectors:
977
- try:
978
- _prepare_project_index(arguments, job_id, cfg, token, query_vector=next(iter(vectors.values())),
979
- model=model)
980
- except ProjectJobCancelled:
981
- raise
982
- except Exception as exc:
983
- if not indexer.info(root).get("chunks"):
1109
+ lag = _index_lag(root, model, cfg, problem, job_id)
1110
+ if lag is not None:
1111
+ overlay, notice = _lag_overlay(root, lag)
1112
+ warnings += [text for text in (notice, _refresh_in_background(root, lag)) if text]
1113
+ else:
1114
+ try:
1115
+ _prepare_project_index(arguments, job_id, cfg, token, query_vector=next(iter(vectors.values())),
1116
+ model=model)
1117
+ except ProjectJobCancelled:
984
1118
  raise
985
- warnings.append("Index update failed; results from the last ready index. " + _project_error(exc))
986
- search_model = None
987
- if indexer.info(root).get("model_id") != model:
988
- vectors = {}
1119
+ except Exception as exc:
1120
+ if not indexer.info(root).get("chunks"):
1121
+ raise
1122
+ warnings.append("Index update failed; results from the last ready index. " + _project_error(exc))
1123
+ search_model = None
1124
+ if indexer.info(root).get("model_id") != model:
1125
+ vectors = {}
989
1126
  if vectors and not cfg.get("rerank_model"):
990
1127
  warnings.append("Rerank not configured: hybrid lexical/vector order, less precise at the top. Configure a rerank_model.")
991
1128
  _job_checkpoint(job_id)
@@ -1014,7 +1151,7 @@ def _perform_smart_search_isolated(arguments, job_id):
1014
1151
  query_terms = _symbol_terms(" ".join(queries[group] for group in groups if group != "doc"))
1015
1152
  _update_job(job_id, phase="searching")
1016
1153
  found = {group: indexer.search(root, vectors.get(queries[group]), top_k=max(limits[group] * 4, candidate_pool),
1017
- query=queries[group], model_id=search_model,
1154
+ query=queries[group], model_id=search_model, overlay=overlay,
1018
1155
  include=lambda path, group=group: index_profile.kind(path, profile) == group)
1019
1156
  for group in groups}
1020
1157
 
@@ -1034,7 +1171,12 @@ def _perform_smart_search_isolated(arguments, job_id):
1034
1171
  ordered = [(float(item["relevance_score"]), "rerank", candidates[item["index"]][1:]) for item in reranked]
1035
1172
  except Exception as exc:
1036
1173
  problem = f"Rerank unavailable for {group}; hybrid lexical/vector order kept. " + _project_error(exc)
1037
- rows = [{"path": path, "lines": f"{start}-{end}", "score": round(score, 6), "snippet": text[:500],
1174
+ exact = indexer.overlay_exact_keys(overlay, query)
1175
+ if exact:
1176
+ pinned = [(c[0], "overlay", c[1:]) for c in candidates if tuple(c[1:4]) in exact][:2]
1177
+ pinned_keys = {tuple(item[2][:3]) for item in pinned}
1178
+ ordered = pinned + [item for item in ordered if tuple(item[2][:3]) not in pinned_keys]
1179
+ rows =[{"path": path, "lines": f"{start}-{end}", "score": round(score, 6), "snippet": text[:500],
1038
1180
  "ranking": ranking, "location_kind": document_text.location_kind(path),
1039
1181
  **({"symbols": _chunk_symbols(symbol_index, path, start, end, text, query_terms)}
1040
1182
  if symbol_index and group != "doc" else {})}
@@ -1049,6 +1191,8 @@ def _perform_smart_search_isolated(arguments, job_id):
1049
1191
  if problem:
1050
1192
  warnings.append(problem)
1051
1193
  _job_checkpoint(job_id)
1194
+ if not any(blocks.values()):
1195
+ notes.append(EMPTY_RESULT_NOTICE)
1052
1196
  warning = " ".join(warnings)
1053
1197
  if warning:
1054
1198
  _update_job(job_id, warning=warning)
@@ -1119,7 +1263,8 @@ def _run_project_job_in_view(job_id, arguments, kind):
1119
1263
  if updated:
1120
1264
  fields.update(indexed_seq=captured_seq, last_indexed=time.time())
1121
1265
  project_store.update(project_id, **fields)
1122
- _log(f"project_job {job_id} {project_id} {kind} done elapsed={time.time()-job['created_at']:.2f}s")
1266
+ _log(f"project_job {job_id} {project_id} {kind} done elapsed={time.time()-job['created_at']:.2f}s "
1267
+ f"phases={_phase_timings(final)}")
1123
1268
  return result
1124
1269
  except Exception as exc:
1125
1270
  with _JOBS_LOCK:
@@ -1214,8 +1359,13 @@ def _search_limits(value, groups, saved=None):
1214
1359
  for group in groups:
1215
1360
  raw = value if not isinstance(value, dict) else value.get(group, (saved or {}).get(group, SEARCH_LIMIT_DEFAULT))
1216
1361
  limits[group] = _limit_value(raw, f"top_k.{group}")
1217
- if sum(limits.values()) > SEARCH_LIMIT_TOTAL:
1218
- raise ValueError(f"The top_k total across the requested blocks is {sum(limits.values())}; the maximum is {SEARCH_LIMIT_TOTAL}.")
1362
+ total = sum(limits.values())
1363
+ if total > SEARCH_LIMIT_TOTAL:
1364
+ # A first call that asks for more than the total is scaled down in proportion (a rejection costs the agent a turn).
1365
+ scaled = {group: max(1, limit * SEARCH_LIMIT_TOTAL // total) for group, limit in limits.items()}
1366
+ while sum(scaled.values()) > SEARCH_LIMIT_TOTAL:
1367
+ scaled[max(scaled, key=scaled.get)] -= 1
1368
+ limits = scaled
1219
1369
  return limits
1220
1370
 
1221
1371
 
@@ -1313,6 +1463,7 @@ def _pending_notices(session, client, project_root):
1313
1463
  pending = []
1314
1464
  if session["hook_notice"] and "hook" not in delivered:
1315
1465
  pending.append(("hook", session["hook_notice"]))
1466
+ pending += [(key, f"{text} Tell the user.") for key, text in _update_notices() if key not in delivered]
1316
1467
  key = "audit:" + setup_audit.index_key(project_root)
1317
1468
  if key not in delivered and not client_hooks.audit_hook_installed(client):
1318
1469
  text = setup_audit.session_notice(project_root)
@@ -1321,6 +1472,16 @@ def _pending_notices(session, client, project_root):
1321
1472
  return pending
1322
1473
 
1323
1474
 
1475
+ def _update_notices():
1476
+ """Update notices (updated, failed, newer version on npm); an unreadable state file is logged, not raised, so a
1477
+ search never fails over it."""
1478
+ try:
1479
+ return update_check.notices()
1480
+ except Exception as exc:
1481
+ _log(f"update notices unavailable: {type(exc).__name__}: {exc}")
1482
+ return []
1483
+
1484
+
1324
1485
  def _smart_search_arguments(arguments):
1325
1486
  if not isinstance(arguments, dict):
1326
1487
  raise ValueError("smart_search requires its arguments as a JSON object")
@@ -3376,6 +3537,10 @@ def _pretool_hook(handler):
3376
3537
  client = (urllib.parse.parse_qs(urllib.parse.urlsplit(handler.path).query).get("client") or [""])[0]
3377
3538
  try:
3378
3539
  payload = json.loads(handler._read_body(HOOK_MAX_BODY_BYTES) or b"{}")
3540
+ try:
3541
+ _note_edit(payload)
3542
+ except Exception as exc:
3543
+ _log(f"edit index update skipped: {type(exc).__name__}: {exc}")
3379
3544
  output = hook_decision.decide(payload, client or None, _web_route)
3380
3545
  except Exception as exc:
3381
3546
  _log(f"PreToolUse hook without a decision: {type(exc).__name__}: {exc}")
@@ -3388,17 +3553,23 @@ def _pretool_hook(handler):
3388
3553
 
3389
3554
  def _session_audit_hook(handler):
3390
3555
  """SessionStart output for the audit report hook: the stored audit notice of the session's folder as
3391
- additionalContext, or `{}` when nothing is stored or it holds no error or warning. Never runs an audit."""
3556
+ additionalContext, and the update notices both as systemMessage (shown to the user) and as context; `{}` when
3557
+ there is nothing to say. Never runs an audit."""
3392
3558
  try:
3393
3559
  payload = json.loads(handler._read_body(HOOK_MAX_BODY_BYTES) or b"{}")
3394
3560
  cwd = payload.get("cwd")
3395
3561
  text = setup_audit.session_notice(cwd) if isinstance(cwd, str) and os.path.isdir(cwd) else ""
3396
3562
  except Exception as exc:
3397
3563
  _log(f"SessionStart audit hook without a report: {type(exc).__name__}: {exc}")
3564
+ text = ""
3565
+ updates = " ".join(notice for _key, notice in _update_notices())
3566
+ context = " ".join(part for part in (updates, text) if part)
3567
+ if not context:
3398
3568
  return {}
3399
- if not text:
3400
- return {}
3401
- return {"hookSpecificOutput": {"hookEventName": "SessionStart", "additionalContext": text}}
3569
+ output = {"hookSpecificOutput": {"hookEventName": "SessionStart", "additionalContext": context}}
3570
+ if updates:
3571
+ output["systemMessage"] = updates
3572
+ return output
3402
3573
 
3403
3574
 
3404
3575
  _TOOL_HANDLERS = {
@@ -3823,6 +3994,10 @@ def main():
3823
3994
  server = _bind_server()
3824
3995
  _BOUND_PORT = server.server_address[1]
3825
3996
  threading.Thread(target=_probe_capabilities, daemon=True).start()
3997
+ try:
3998
+ update_check.note_running_version()
3999
+ except Exception as exc:
4000
+ _log(f"running version not recorded: {type(exc).__name__}: {exc}")
3826
4001
  _write_registry(_BOUND_PORT)
3827
4002
  atexit.register(_remove_registry)
3828
4003
  try:
package/tray.py CHANGED
@@ -111,24 +111,48 @@ def _poll_loop(icon):
111
111
  time.sleep(POLL_INTERVAL_S)
112
112
 
113
113
 
114
+ def _announce_start(icon):
115
+ """Tells the user, once per start, that Smart Tool was just updated or that the last automatic update failed."""
116
+ updated = update_check.note_running_version()
117
+ if updated:
118
+ icon.notify(f"Smart Tool updated from {updated['from']} to {updated['to']}.", "Smart Tool")
119
+ failed = update_check.failed_update()
120
+ if failed:
121
+ icon.notify(f"The update to {failed} failed; Smart Tool {version.VERSION} is still installed. "
122
+ "It is tried again in 24 h, or now from this menu.", "Smart Tool")
123
+
124
+
114
125
  def _update_loop(icon):
115
- """Checks npm at start and every 24 h. At start (logon, before agent sessions use the daemon) a newer version is
116
- installed right away when auto_update is on; later checks only notify and offer it in the menu."""
117
- first = True
126
+ """Checks npm at start and every 24 h. A newer version is installed right away when automatic updates are on (once
127
+ per version, again 24 h after a failed try); otherwise the user is told and the menu offers it."""
128
+ try:
129
+ _announce_start(icon)
130
+ except Exception as exc:
131
+ icon.notify(f"Could not read the update state: {exc}", "Smart Tool")
118
132
  while True:
119
133
  try:
120
134
  latest = update_check.newer_version()
121
135
  except Exception:
122
136
  latest = None
123
- if latest and latest != _state["update"]:
137
+ else:
138
+ try:
139
+ update_check.record_check(latest)
140
+ except Exception as exc:
141
+ icon.notify(f"Could not save the update check: {exc}", "Smart Tool")
142
+ if latest:
143
+ known = latest == _state["update"]
124
144
  _state["update"] = latest
125
145
  icon.update_menu()
126
- if first and config.auto_update() and update_check.should_auto_update(latest):
127
- icon.notify(f"Updating Smart Tool to {latest}...", "Smart Tool")
146
+ try:
147
+ automatic = update_check.automatic_enabled() and update_check.should_auto_update(latest)
148
+ except Exception as exc:
149
+ automatic = False
150
+ icon.notify(f"Automatic update skipped: {exc}", "Smart Tool")
151
+ if automatic:
152
+ icon.notify(f"Updating Smart Tool from {version.VERSION} to {latest}...", "Smart Tool")
128
153
  update_check.run_update(latest, automatic=True)
129
- else:
154
+ elif not known:
130
155
  icon.notify(f"Smart Tool {latest} is available: use \"Update to {latest}\" in this menu.", "Smart Tool")
131
- first = False
132
156
  time.sleep(UPDATE_CHECK_INTERVAL_S)
133
157
 
134
158
 
@@ -162,12 +186,16 @@ def _update_now(icon, item):
162
186
 
163
187
  def _auto_update_checked(item):
164
188
  try:
165
- return config.auto_update()
189
+ return update_check.automatic_enabled()
166
190
  except Exception:
167
191
  return False
168
192
 
169
193
 
170
194
  def _toggle_auto_update(icon, item):
195
+ if update_check.env_opted_out():
196
+ icon.notify(f"{update_check.ENV_OPT_OUT}=off is set: automatic updates stay off until it is removed.",
197
+ "Smart Tool")
198
+ return
171
199
  try:
172
200
  config.update_config(lambda cfg: cfg.__setitem__("auto_update", not config.auto_update(cfg)))
173
201
  except Exception as exc:
package/update_check.py CHANGED
@@ -1,5 +1,10 @@
1
- """Newer Smart Tool release on npm and the update itself: the tray runs the same install command a user would, in a
2
- visible console, at startup (once per version, when auto_update is on) or on demand from its menu."""
1
+ """Newer Smart Tool release on npm and the update itself: the tray checks npm at start and every 24 h and, when
2
+ automatic updates are on (the default), runs the same install command a user would, in a visible console. A failed
3
+ version is tried again after 24 h. What happened (newer version available, updated, update failed) is told to the
4
+ user by the tray and to agent sessions by the daemon, from the state file, without contacting npm again.
5
+
6
+ Automatic updates are turned off by the tray switch (config `auto_update`) or by SMART_TOOL_AUTO_UPDATE=off; the
7
+ newer-version notice still shows then, with the command to update by hand."""
3
8
  import json
4
9
  import os
5
10
  import subprocess
@@ -7,13 +12,18 @@ import time
7
12
  import urllib.request
8
13
 
9
14
  import atomic_io
15
+ import config
10
16
  import paths
11
17
  import version
12
18
 
13
19
  PACKAGE = "@allansantos-dev/smart-tool"
14
20
  REGISTRY_URL = "https://registry.npmjs.org/@allansantos-dev%2Fsmart-tool/latest"
15
21
  UPDATE_COMMAND = f"npx --yes {PACKAGE}@latest install"
22
+ CHANGELOG_URL = "https://github.com/AllanSantos-DV/smart-tool-dist/blob/main/CHANGELOG.md"
16
23
  STATE_PATH = os.path.join(paths.DATA_DIR, "update-state.json")
24
+ ENV_OPT_OUT = "SMART_TOOL_AUTO_UPDATE"
25
+ RETRY_AFTER_S = 24 * 3600
26
+ UPDATED_NOTICE_S = 24 * 3600
17
27
 
18
28
 
19
29
  def _parts(value):
@@ -29,27 +39,108 @@ def newer_version(timeout=10):
29
39
  return latest if _parts(latest) > _parts(version.VERSION) else None
30
40
 
31
41
 
32
- def attempted():
33
- """Version the last automatic update went for, or None."""
42
+ def env_opted_out():
43
+ """SMART_TOOL_AUTO_UPDATE set to off, 0, false or no."""
44
+ return os.environ.get(ENV_OPT_OUT, "").strip().lower() in ("off", "0", "false", "no")
45
+
46
+
47
+ def automatic_enabled(cfg=None):
48
+ """Automatic updates are on: the tray switch (config `auto_update`, on by default) and no environment opt-out."""
49
+ return not env_opted_out() and config.auto_update(cfg)
50
+
51
+
52
+ def _state():
34
53
  try:
35
54
  with open(STATE_PATH, encoding="utf-8") as stream:
36
- return json.load(stream).get("attempted")
55
+ return json.load(stream)
37
56
  except FileNotFoundError:
57
+ return {}
58
+
59
+
60
+ def _save(state):
61
+ os.makedirs(os.path.dirname(STATE_PATH), exist_ok=True)
62
+ atomic_io.write_secret_text(STATE_PATH, json.dumps(state))
63
+
64
+
65
+ def attempted():
66
+ """Version the last automatic update went for, or None."""
67
+ return _state().get("attempted")
68
+
69
+
70
+ def should_auto_update(latest, now=None):
71
+ """An automatic update runs once per version and again 24 h after a failed try: when it fails, the installer
72
+ restores the previous version and restarts the tray, which would otherwise try the same version on every start."""
73
+ if not latest:
74
+ return False
75
+ state = _state()
76
+ if state.get("attempted") != latest:
77
+ return True
78
+ return (now or time.time()) - state.get("at", 0) >= RETRY_AFTER_S
79
+
80
+
81
+ def record_check(latest):
82
+ """Stores the result of an npm check (the newer version, or None) for the notices read without network."""
83
+ state = _state()
84
+ state.update(latest=latest, checked_at=time.time())
85
+ _save(state)
86
+
87
+
88
+ def note_running_version(now=None):
89
+ """Called when the tray and the daemon start: a version different from the one that ran last is recorded as an
90
+ update (from, to, at) for the updated notice; the very first run records nothing. Returns the update record when
91
+ this call found it, else None."""
92
+ state = _state()
93
+ running = state.get("running")
94
+ if running == version.VERSION:
38
95
  return None
96
+ state["running"] = version.VERSION
97
+ found = None
98
+ if running:
99
+ found = state["updated"] = {"from": running, "to": version.VERSION, "at": now or time.time()}
100
+ _save(state)
101
+ return found
102
+
103
+
104
+ def failed_update():
105
+ """The version an automatic update went for and did not install (the previous version is still running), else
106
+ None."""
107
+ target = _state().get("attempted")
108
+ return target if target and _parts(target) > _parts(version.VERSION) else None
39
109
 
40
110
 
41
- def should_auto_update(latest):
42
- """An automatic update runs once per version: when it fails, the installer restores the previous version and
43
- restarts the tray, which would otherwise try the same version again on every start."""
44
- return bool(latest) and attempted() != latest
111
+ def notices(now=None):
112
+ """[(key, text)] about updates, for the user and the agent: Smart Tool updated in the last 24 h, the last
113
+ automatic update failed, or a newer version is on npm. Reads only the state file."""
114
+ now = now or time.time()
115
+ state = _state()
116
+ found = []
117
+ updated = state.get("updated") or {}
118
+ if updated.get("to") == version.VERSION and now - updated.get("at", 0) < UPDATED_NOTICE_S:
119
+ found.append((f"updated:{version.VERSION}",
120
+ f"Smart Tool was updated from {updated['from']} to {version.VERSION}. Changes: {CHANGELOG_URL}"))
121
+ failed = failed_update()
122
+ latest = state.get("latest")
123
+ if failed:
124
+ retry = ("the tray tries again 24 h after the failed try" if automatic_enabled()
125
+ else "automatic updates are now off")
126
+ found.append((f"failed:{failed}",
127
+ f"The automatic update of Smart Tool to {failed} failed; {version.VERSION} is still installed "
128
+ f"and {retry}. To update now, run: {UPDATE_COMMAND}"))
129
+ elif latest and _parts(latest) > _parts(version.VERSION):
130
+ how = ("the tray installs it automatically" if automatic_enabled()
131
+ else f"automatic updates are off (tray switch or {ENV_OPT_OUT}=off); to update, run: {UPDATE_COMMAND}")
132
+ found.append((f"available:{latest}",
133
+ f"Smart Tool {latest} is available on npm (installed: {version.VERSION}); {how}."))
134
+ return found
45
135
 
46
136
 
47
137
  def run_update(latest, automatic):
48
138
  """Starts the install command in its own visible console, detached from the tray (the installer stops and restarts
49
139
  the tray). The console closes on success and stays open with the error on failure."""
50
140
  if automatic:
51
- os.makedirs(os.path.dirname(STATE_PATH), exist_ok=True)
52
- atomic_io.write_secret_text(STATE_PATH, json.dumps({"attempted": latest, "at": time.time()}))
141
+ state = _state()
142
+ state.update(attempted=latest, at=time.time())
143
+ _save(state)
53
144
  flags = getattr(subprocess, "CREATE_NEW_CONSOLE", 0) | getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0)
54
145
  command = f'title Smart Tool update to {latest} && {UPDATE_COMMAND} || pause'
55
146
  subprocess.Popen(["cmd.exe", "/c", command], creationflags=flags, close_fds=True,
package/version.py CHANGED
@@ -1,7 +1,7 @@
1
1
  """Single source of the Smart Tool version and of the User-Agent sent to public APIs."""
2
2
  import re
3
3
 
4
- VERSION = "0.9.17"
4
+ VERSION = "0.9.19"
5
5
  _CONTACT_RE = re.compile(r"^(?:[^\s@()<>;]+@[^\s@()<>;]+\.[^\s@()<>;]+|https?://[^\s()<>;]+)$")
6
6
 
7
7