@allansantos-dev/smart-tool 0.9.25 → 0.9.27

package/CHANGELOG.md

@@ -1,5 +1,57 @@
 # Changelog
 
+## 0.9.27 - beta
+
+- No folder is read before it is classified, and a tool's hidden folder (`.crowd`, `.tokenguard`, `.vscode`, any name
+  starting with a dot) is never indexed, whatever the scope says:
+  - An empty or missing `include` used to mean "the whole project" (the saved scopes of `sistema` and a Claude memory
+    folder had `include: []`), and a search of a folder with no index and no scope read every file of it from disk (one
+    registered folder had 715 427). An empty `include` now indexes only the loose files at the root, and that first
+    keyword search reads only those.
+  - The `scope` action without a saved scope defaulted to `include: ["."]`; it now requires `include`, and refuses an
+    include inside a hidden folder.
+  - A model that answered with `docs/**`, `*` or `**/*` (the cause of the `error` status of two registered folders)
+    now has the entry reduced to the folder it names; an entry that is absolute, climbs out with `..`, sits in a hidden
+    folder or keeps a wildcard in the middle is dropped and kept in `dropped_include` of the saved scope.
+  - Hidden files at the root and inside any included folder are no longer listed (they were listed and then skipped
+    when read).
+  - The mechanical exclusion grows: `site-packages`, `bower_components`, `jspm_packages` everywhere, and `target`
+    (Cargo.toml or pom.xml at the root) and `Pods` (Podfile). These are never offered to the scope model.
+- Tests for each rule: `tests/test_scope_rules.py`.
+
+## 0.9.26 - beta
+
+- The daemon no longer re-reads and re-hashes every watched folder in a loop. Measured on the owner's machine: one
+  daemon process read 181 GB and wrote 29.7 GB in 15 h (3.6 MB/s read, 2.2 MB/s write while idle), which kept the disk
+  busy and made Defender scan the files again and again. Causes, all fixed:
+  - A file-event buffer overflow, or more than 1 000 changed paths, raised a full check that opened and hashed every
+    file of the scope; the flag was cleared only when no event arrived during the job, so under constant events (the
+    `~/.claude` session logs, a state folder) every job hashed everything again, and the stuck scope flag also asked the
+    scope model to decide the scope again on every job. An overflow now asks for a stat comparison (mtime and size) of
+    the scope, and a finished job clears the full-check and scope flags raised before it started.
+  - An event under a folder the scope leaves out (`plugins/cache`, `projects/*.jsonl` of `~/.claude`) marked the project
+    dirty and queued an index job. Events are now filtered by the saved scope, which is reloaded only when its file
+    changes.
+  - A folder growing inside an excluded one counted as a structure change and called the scope model again. The
+    structure kept in the scope and the one compared against it skip excluded folders.
+  - The runtime folders of a `.claude` home (`plugins`, `projects`, `sessions`, `shell-snapshots`, `file-history`,
+    `cache`, `debug`, `telemetry`, ...) are excluded mechanically, whatever the scope says.
+  - The monitor decrypted up to 50 job payloads per watched folder every second just to learn whether a job was
+    pending; it now asks the `status` column once per tick. A Git folder is identified again (four git processes)
+    only when `HEAD`, the index or the refs moved, not every 5 s. With a live watcher the stat reconciliation runs
+    every 15 min instead of 3; the tick sleeps 5 s instead of 1 s while nothing is pending.
+  - A burst of events cost two reads and two writes of the project registry per batch; it is one of each, and the
+    registry schema is created once per process instead of per connection.
+  - Every save of a job (several per index job) ordered the whole `jobs` table, payloads included, to prune the history:
+    1.7 MB read per save. The pruning now takes its threshold from an `updated_at` index (50 saves: 85.7 MB read before,
+    5.0 MB after, `tests/save_job_io_probe.py`).
+  - The pending-job query of the monitor tick scanned the whole `jobs` table (2.7 MB) every second; it now reads the
+    covering index `jobs_status`.
+  - A folder that changes all the time (the orchestrator's state folder queued an index job every 8 s) is indexed by the
+    monitor at most once a minute; the changes accumulate and one job covers them.
+  - Measured with `tests/monitor_io_probe.py` on a copy of the real registry (14 watched folders, 100 event batches
+    under excluded folders with a buffer overflow every 10th): 149 MB read and 45.7 MB written before, 28 MB read and
+    5.4 MB written after; folders left with a pending full check 14 before, 0 after.
 ## 0.9.25 - beta
 
 - A redirected search made while the first index of a folder is still being built no longer waits in the folder's queue

package/README.md

@@ -15,7 +15,7 @@ registered as the `playwright` server where this machine lets a browser be drive
 An optional hook routes the agent's native Grep/Glob/Read/Bash/WebSearch/WebFetch calls to these tools when they
 are cheaper, or just tells the agent what Smart Tool would do.
 
-**Status:** 0.9.25 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
+**Status:** 0.9.27 beta. Windows only. What changed in each version: [CHANGELOG.md](CHANGELOG.md).
 
 ## How it works
 

package/index_scope.py

@@ -28,8 +28,18 @@ SCOPE_DIR = paths.DATA_DIR
 _ALWAYS_EXCLUDE = {
     ".git", "node_modules", "__pycache__", ".venv", "venv", "env",
     ".tox", ".mypy_cache", ".pytest_cache", "*.egg-info",
+    "site-packages", "bower_components", "jspm_packages",
 }
 
+# Folders that are generated output only when the root proves the project type: a root marker file maps to the names
+# it makes unambiguous (Cargo's and Maven's `target`, CocoaPods' `Pods`). Excluded mechanically, before any model sees
+# the folder list.
+_PROJECT_TYPE_EXCLUDES = (
+    (("Cargo.toml",), ("target",)),
+    (("pom.xml",), ("target",)),
+    (("Podfile",), ("Pods",)),
+)
+
 # Mesmo critério do `_ALWAYS_EXCLUDE`, para arquivo: o significado não varia por
 # projeto. Lockfile é grafo de dependência resolvido por ferramenta — nunca é fonte de
 # resposta numa busca semântica, e é grande o bastante pra dominar o índice.
@@ -41,6 +51,64 @@ _ALWAYS_EXCLUDE_FILES = {
 
 _ROOT_DOC_NAMES = ("AGENTS.md", "CLAUDE.md", "README.md", "README")
 
+# Runtime folders a tool keeps inside its own config home (conversation logs, plugin caches, snapshots): never project
+# source, written all day by the tool itself. Keyed by the root's folder name; excluded mechanically, whatever the
+# scope decision says, so a registered `~/.claude` never indexes or re-scans them.
+_TOOL_STATE_DIRS = {
+    ".claude": ("backups", "cache", "daemon", "debug", "downloads", "file-history", "ide", "paste-cache", "plugins",
+                "projects", "session-env", "sessions", "shell-snapshots", "state", "statsig", "telemetry", "todos"),
+}
+
+
+def tool_state_excludes(root):
+    """Names of the tool-runtime folders under `root` (empty unless `root` is a known tool config home)."""
+    return _TOOL_STATE_DIRS.get(os.path.basename(os.path.normpath(root)).casefold(), ())
+
+
+def project_type_excludes(root):
+    """Names of the generated-output folders `root`'s marker files make unambiguous (empty for an unknown type)."""
+    found = []
+    for markers, names in _PROJECT_TYPE_EXCLUDES:
+        if any(os.path.isfile(os.path.join(root, marker)) for marker in markers):
+            found.extend(names)
+    return tuple(found)
+
+
+def mechanical_excludes(root):
+    """Everything excluded without asking a model: the tool-runtime folders of a config home and the generated output
+    of the project type. The names of `_ALWAYS_EXCLUDE` and every hidden folder are excluded on top of these."""
+    return (*tool_state_excludes(root), *project_type_excludes(root))
+
+
+def is_hidden(rel_path):
+    """Whether any part of the relative path starts with a dot (a tool's folder: .git, .vscode, .crowd, .tokenguard)."""
+    return any(part.startswith(".") for part in str(rel_path).replace("\\", "/").split("/") if part not in ("", ".", ".."))
+
+
+def normalize_include(include):
+    """`(bases, dropped)` from the include entries of a scope. A base is a folder or file path relative to the root, with
+    the shapes a model emits for "everything under it" (`docs/**`, `docs/*`, `docs/`) reduced to the folder; `*`, `**`
+    and `**/*` mean the whole project and become ``. An entry that is absolute, climbs out with `..`, sits inside a hidden
+    folder or keeps a wildcard in the middle is dropped: it can never name something indexable, and it is returned so the
+    caller can report it. No entries means no folder: only the loose files at the root are indexed."""
+    bases, dropped = [], []
+    for raw in include or []:
+        base = str(raw).replace("\\", "/").strip()
+        while base.startswith("./"):
+            base = base[2:]
+        for tail in ("/**/*", "/**", "/*"):
+            if base.endswith(tail):
+                base = base[:-len(tail)]
+        base = base.strip("/")
+        if base in ("*", "**", "**/*", "."):
+            base = ""
+        if (os.path.isabs(str(raw)) or str(raw).strip().startswith(("/", "\\")) or ".." in base.split("/") or is_hidden(base)
+                or any(ch in base for ch in "*?[")):
+            dropped.append(raw)
+        elif base not in bases:
+            bases.append(base)
+    return bases, dropped
+
 
 def always_excluded(name):
     """Directory name in _ALWAYS_EXCLUDE, entries may be globs (e.g. *.egg-info)."""
@@ -109,10 +177,12 @@ def _top_level_dirs(root):
         names = os.listdir(root)
     except OSError:
         return []
+    skipped = mechanical_excludes(root)
     return sorted(
         name for name in names
         if not name.startswith(".")
         and not always_excluded(name)
+        and name not in skipped
         and os.path.isdir(os.path.join(root, name))
         and _within_root(root, os.path.join(root, name))
     )
@@ -173,11 +243,14 @@ def save_scope(root, scope):
         {**scope, "root": project_identity.canonical_root(root)}, indent=2, ensure_ascii=False))
 
 
-def _structure(root, max_dirs=1500):
+def _structure(root, max_dirs=1500, keep=None):
+    """Relative folders of `root` (at most `max_dirs`). `keep(rel)` prunes the walk: a folder it rejects is neither
+    listed nor entered."""
     found = []
     for current, dirs, _files in os.walk(root):
         dirs[:] = sorted(d for d in dirs if not d.startswith(".") and not always_excluded(d)
-                         and _within_root(root, os.path.join(current, d)))
+                         and _within_root(root, os.path.join(current, d))
+                         and (keep is None or keep(os.path.relpath(os.path.join(current, d), root).replace(os.sep, "/"))))
         for name in dirs:
             found.append(os.path.relpath(os.path.join(current, name), root).replace(os.sep, "/"))
             if len(found) >= max_dirs:
@@ -200,8 +273,10 @@ def needs_rescan(root, cached_scope):
         return True
     known = set(cached_scope.get("top_level_dirs", []))
     current = set(_top_level_dirs(root))
+    # A folder the scope excludes (a cache that grows all day) is not a structure change: the walk skips it.
     return (not current.issubset(known) or
-            not set(_structure(root)).issubset(set(cached_scope.get("structure", []))))
+            not set(_structure(root, keep=scope_filter(cached_scope, root))).issubset(
+                set(cached_scope.get("structure", []))))
 
 
 def _read_root_docs(root, max_chars=4000):
@@ -275,7 +350,8 @@ def _build_scope_prompt(root, ambiguous_dirs, root_files, doc_context):
         "You are deciding what to include in a semantic code-search index for the "
         f"repository at {root}.\n\n"
         "These directories are already excluded automatically (dependency/tooling caches "
-        f"whose meaning does not vary by project): {', '.join(sorted(_ALWAYS_EXCLUDE))}. "
+        f"whose meaning does not vary by project): {', '.join(sorted({*_ALWAYS_EXCLUDE, *mechanical_excludes(root)}))}. "
+        "Every hidden folder (name starting with a dot: .git, .vscode, .crowd, .tokenguard...) is also never indexed. "
         "Do not decide on them.\n\n"
         f"Directories that still need a decision: {', '.join(ambiguous_dirs) or '(none)'}\n\n"
         "Loose files at the repository ROOT are indexed automatically, because in a flat "
@@ -299,6 +375,8 @@ def _build_scope_prompt(root, ambiguous_dirs, root_files, doc_context):
         "source, even if it also contains a few generated/log files. Only call submit_scope "
         "once every non-obvious directory is grounded in inspection data or in the "
         "documentation.\n\n"
+        "`include` lists folder paths relative to the root, without wildcards; a folder you leave out of `include` is "
+        "not indexed, and an empty `include` indexes only the loose root files. "
         "Include sources useful for search even if they are gitignored (e.g. docs/, README, "
         "ADRs). Java, Angular, HTML/CSS, plain text and Markdown are supported sources. "
         "DOCX documents are supported through bounded local text extraction; do not exclude "
@@ -315,7 +393,7 @@ def decide_scope(root, router_model, force=False, deadline=None, cancel_check=No
         return cached
 
     top_level_dirs = _top_level_dirs(root)
-    ambiguous_dirs = [d for d in top_level_dirs if not always_excluded(d)]
+    ambiguous_dirs = list(top_level_dirs)
     root_files = _root_level_files(root, _exclude_matcher(_ALWAYS_EXCLUDE_FILES))
     doc_context = _read_root_docs(root)
 
@@ -374,14 +452,19 @@ def decide_scope(root, router_model, force=False, deadline=None, cancel_check=No
 
     if decision is None:
         raise RuntimeError("The model did not define a valid scope. Check the model or set the scope manually.")
+    include, dropped = normalize_include(decision.get("include"))
+    final = {"include": include,
+             "exclude": [*(decision.get("exclude") or []), *mechanical_excludes(root)],
+             "user_exclude": (cached or {}).get("user_exclude", [])}
     scope = {
-        "include": (decision or {}).get("include") or [],
-        "exclude": (decision or {}).get("exclude") or [],
+        "include": final["include"],
+        "dropped_include": [str(item) for item in dropped],
+        "exclude": final["exclude"],
         "top_level_dirs": top_level_dirs,
         "scope_version": SCOPE_VERSION,
-        "structure": _structure(root),
+        "structure": _structure(root, keep=scope_filter(final, root)),
         "docs_digest": _docs_digest(root),
-        "user_exclude": (cached or {}).get("user_exclude", []),
+        "user_exclude": final["user_exclude"],
         "decided_at": time.time(),
     }
     save_scope(root, scope)
@@ -401,7 +484,7 @@ def _root_level_files(root, matcher):
         raise OSError("Could not list the root; the existing index was kept.") from exc
     out = []
     for name in names:
-        if matcher(name):
+        if matcher(name) or name.startswith("."):
             continue
         full = os.path.join(root, name)
         # `isfile` segue link: sem a checagem de contenção, um link na raiz apontando
@@ -441,35 +524,46 @@ def _exclude_matcher(exclude):
     return matches
 
 
-def in_scope(scope, rel_path):
+def scope_filter(scope, root=None):
+    """Predicate `keep(rel_path)`: whether resolve_included_files would list rel_path (a file) or walk it (a folder) —
+    under an include base, outside the exclusions, outside hidden folders. Built once so a burst of file events does not
+    rebuild the matcher per path; `root` adds the tool-runtime folders of a known config home."""
+    bases, _dropped = normalize_include(scope.get("include"))
+    matcher = _exclude_matcher(set(scope.get("exclude") or []) | set(scope.get("user_exclude") or []) |
+                               set(mechanical_excludes(root) if root else ()) | _ALWAYS_EXCLUDE | _ALWAYS_EXCLUDE_FILES)
+    whole = "" in bases
+
+    def keep(rel_path):
+        rel_path = rel_path.replace("\\", "/")
+        parts = rel_path.split("/")
+        if matcher(rel_path) or is_hidden(rel_path):
+            return False
+        if whole or len(parts) == 1:
+            return True
+        return any(rel_path == base or rel_path.startswith(base + "/") for base in bases)
+
+    return keep
+
+
+def in_scope(scope, rel_path, root=None):
     """Whether resolve_included_files would list rel_path: under an include base, outside the exclusions and outside
     hidden folders."""
-    rel_path = rel_path.replace("\\", "/")
-    include = scope.get("include") or [""]
-    matcher = _exclude_matcher(set(scope.get("exclude") or []) | set(scope.get("user_exclude") or []) |
-                               _ALWAYS_EXCLUDE | _ALWAYS_EXCLUDE_FILES)
-    parts = rel_path.split("/")
-    if matcher(rel_path) or any(part.startswith(".") for part in parts[:-1]):
-        return False
-    if any(base in ("", ".") for base in include):
-        return True
-    return len(parts) == 1 or any(rel_path == base.strip("/") or rel_path.startswith(base.strip("/") + "/")
-                                  for base in include)
+    return scope_filter(scope, root)(rel_path)
 
 
 def resolve_included_files(root, scope, cancel_check=None):
     if not valid_scope(scope):
         raise ValueError("Invalid scope; request a new project analysis.")
-    include = scope.get("include") or [""]
+    include, _dropped = normalize_include(scope.get("include"))
     matcher = _exclude_matcher(set(scope.get("exclude") or []) | set(scope.get("user_exclude") or []) |
-                               _ALWAYS_EXCLUDE | _ALWAYS_EXCLUDE_FILES)
+                               set(mechanical_excludes(root)) | _ALWAYS_EXCLUDE | _ALWAYS_EXCLUDE_FILES)
     files = []
     def walk_error(error):
         raise OSError("Could not list a folder in the scope; try again when it is accessible.") from error
-    if not any(base in ("", ".") for base in include):
+    if "" not in include:
         files.extend(_root_level_files(root, matcher))
     for base in include:
-        base_full = os.path.join(root, base) if base not in ("", ".") else root
+        base_full = os.path.join(root, base) if base else root
         # `include` vem de tool call de um LLM cujo prompt embute documentação do
         # repositório analisado: um valor absoluto ou com `..` faria a varredura sair do
         # projeto e mandar conteúdo de fora pro endpoint de embeddings.
@@ -477,7 +571,7 @@ def resolve_included_files(root, scope, cancel_check=None):
             continue
         if os.path.isfile(base_full):
             rel = os.path.relpath(base_full, root).replace(os.sep, "/")
-            if not matcher(rel):
+            if not matcher(rel) and not is_hidden(rel):
                 files.append(rel)
             continue
         for dirpath, dirnames, filenames in os.walk(base_full, onerror=walk_error):
@@ -495,6 +589,6 @@ def resolve_included_files(root, scope, cancel_check=None):
             ]
             for name in filenames:
                 rel_path = f"{rel_dir}/{name}" if rel_dir else name
-                if not matcher(rel_path) and _within_root(root, os.path.join(root, rel_path)):
+                if not name.startswith(".") and not matcher(rel_path) and _within_root(root, os.path.join(root, rel_path)):
                     files.append(rel_path)
     return sorted(set(files))

package/index_views.py

@@ -97,6 +97,11 @@ def storage_key(root):
     return key + '.v-' + view['view_id'] if view['git'] else key
 
 
+def stamp_changed(view):
+    """Whether the Git files that define `view` moved since it was described (a stat of five files, no process)."""
+    return bool(view.get('git')) and _stamp(view) != view.get('stamp')
+
+
 def assert_current(view):
     if view.get('git') and _stamp(view) != view.get('stamp'):
         _CACHE.pop(view['root'], None)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@allansantos-dev/smart-tool",
-  "version": "0.9.25",
+  "version": "0.9.27",
   "description": "Local MCP server that gives coding agents (Claude Code, Codex) cheaper, sharper tools than their built-in search. Windows.",
   "license": "Apache-2.0",
   "author": "Allan Santos",

package/project_monitor.py

@@ -13,6 +13,13 @@ import index_views
 
 DEBOUNCE_S = 5
 RECONCILE_S = 180
+# With a live watcher the events carry the changes; the stat reconciliation is only the safety net for what the buffer
+# lost, so it runs less often (a buffer overflow already forces a stat comparison of the whole scope).
+RECONCILE_WATCHED_S = 900
+IDLE_TICK_S = 5
+# A folder that changes all the time (a tool's state folder) is indexed at most this often by the monitor; the changes
+# keep accumulating and one job covers them, and a search meanwhile reads the stale files straight from disk.
+MIN_JOB_GAP_S = 60
 FULL_CHECK_S = 86400
 MAX_EVENT_PATHS = 1000
 
@@ -50,6 +57,9 @@ class ProjectMonitor:
         self._git_seen = {}
         self._git_checked = {}
         self._pending = {}
+        self._next_job = {}
+        self._filters = {}
+        self._views = {}
         self._lock = threading.RLock()
         self._stop = threading.Event()
         self._thread = None
@@ -71,16 +81,33 @@ class ProjectMonitor:
     def refresh(self):
         self._checked.clear()
 
+    def _scope_for(self, project):
+        """`(scope, keep)` for the project's current view; reloaded only when the saved scope file changed, so a burst
+        of events does not read the scope and rebuild its matcher once per batch."""
+        path = index_scope.scope_path(project["root"])
+        try:
+            stamp = (path, os.stat(path).st_mtime_ns)
+        except OSError:
+            stamp = (path, None)
+        cached = self._filters.get(project["id"])
+        if cached and cached[0] == stamp:
+            return cached[1], cached[2]
+        scope = index_scope.load_scope(project["root"]) or {}
+        keep = index_scope.scope_filter(scope, project["root"]) if scope else (lambda rel: True)
+        self._filters[project["id"]] = (stamp, scope, keep)
+        return scope, keep
+
     def events(self, project_id, rows, overflow=False):
         project = project_store.get(project_id)
         if not project or project.get("paused") or not project.get("watch") or not project.get("enabled"):
             return
-        paths = {row["path"] for row in rows if relevant_path(row.get("path", ""))}
+        # An event under a folder the scope leaves out (plugin cache, session logs) changes nothing in the index.
+        scope, keep = self._scope_for(project)
+        paths = {row["path"] for row in rows if relevant_path(row.get("path", "")) and keep(row["path"])}
         if not paths and not overflow:
             return
         # Um diretório existente também recebe MODIFIED ao mudar um arquivo filho.
         # Só uma estrutura nova ou documentação da raiz exige reavaliar o escopo.
-        scope = index_scope.load_scope(project["root"]) or {}
         known_dirs = {p.casefold() for p in scope.get("structure", [])}
         docs = {name.casefold() for name in index_scope._ROOT_DOC_NAMES}
         force_scope = any(path.casefold() in docs for path in paths) or any(
@@ -94,12 +121,12 @@ class ProjectMonitor:
             state["revision"] = state.get("revision", 0) + 1
             state["paths"].update(paths)
             state["last"] = time.monotonic()
-            state["full"] |= overflow or len(state["paths"]) > MAX_EVENT_PATHS
             state["force_scope"] |= force_scope
-            if state["full"]:
+            # Lost or too many events: the update compares stat of the whole scope (mtime and size), which finds what
+            # changed without reading and hashing every file the way a full check does.
+            if len(state["paths"]) > MAX_EVENT_PATHS:
                 state["paths"].clear()
-        project_store.mark_dirty(project_id, "Changes detected by the monitor.")
-        project_store.record_changes(project_id, paths, full=overflow, scope=force_scope)
+        project_store.note_changes(project_id, "Changes detected by the monitor.", paths, scope=force_scope)
 
     def _watch_error(self, project_id, message):
         project_store.update(project_id, watch_backend="reconciliation", watch_error=message[:300])
@@ -132,6 +159,7 @@ class ProjectMonitor:
         now = time.monotonic()
         projects = project_store.all_projects()
         self._sync_watchers(projects, now)
+        busy = project_store.projects_with_pending_job()
         for project in projects:
             key = project["id"]
             if not project.get("watch") or not project.get("enabled") or project.get("paused"):
@@ -139,16 +167,21 @@ class ProjectMonitor:
             if not os.path.isdir(project["root"]):
                 project_store.update(key, status="unavailable", last_error="The folder is not accessible.")
                 continue
-            if any(job.get("status") == "pending" for job in project_store.jobs(key, limit=50)):
+            if key in busy:
                 continue  # Não planejar outra geração com metadados de uma ainda em curso.
             if now - self._git_checked.get(key, -5) >= 5:
-                view = index_views.describe(project['root'], fresh=True)
+                # A Git view is re-identified (four git processes) only when the files that define it moved.
+                view = self._views.get(key)
+                if view is None or not view.get('git') or index_views.stamp_changed(view):
+                    view = self._views[key] = index_views.describe(project['root'], fresh=True)
                 signature = (view['view_id'], view.get('commit'))
                 if key in self._git_seen and signature != self._git_seen[key]:
                     self.events(key, [], overflow=True)
                 self._git_seen[key] = signature
                 self._git_checked[key] = now
-            if now - self._checked.get(key, -RECONCILE_S) >= RECONCILE_S:
+            watcher = self._watchers.get(key)
+            interval = RECONCILE_WATCHED_S if watcher is not None and watcher.alive else RECONCILE_S
+            if now - self._checked.get(key, -interval) >= interval:
                 self._checked[key] = now
                 scope = index_scope.load_scope(project["root"])
                 meta = indexer.info(project["root"])
@@ -177,7 +210,7 @@ class ProjectMonitor:
                 continue
             with self._lock:
                 pending = self._pending.get(key)
-                if not pending or now - pending["last"] < DEBOUNCE_S:
+                if not pending or now - pending["last"] < DEBOUNCE_S or now < self._next_job.get(key, 0):
                     continue
                 options = {"changed_paths": sorted(pending["paths"]), "full_check": pending["full"],
                            "force_scope": pending["force_scope"]}
@@ -190,6 +223,7 @@ class ProjectMonitor:
             except Exception as exc:
                 project_store.update(key, status="error", last_error=str(exc)[:300], next_retry_at=time.time() + 60)
                 continue
+            self._next_job[key] = now + MIN_JOB_GAP_S
             with self._lock:
                 # Um evento recebido durante o submit continua pendente para a próxima geração.
                 if self._pending.get(key) is pending and pending.get("revision", 0) == revision:
@@ -201,4 +235,4 @@ class ProjectMonitor:
                 self.tick()
             except Exception:
                 pass  # Falha opcional do monitor não derruba o daemon MCP.
-            self._stop.wait(1)
+            self._stop.wait(1 if self._pending else IDLE_TICK_S)

package/project_store.py

@@ -13,18 +13,25 @@ import secret_store
 
 DB_PATH = os.path.join(paths.DATA_DIR, "projects.sqlite3")
 _LOCK = threading.RLock()
+_SCHEMA_READY = set()
 MAX_JOBS = 500
 
 
 @contextmanager
 def _db():
     os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
+    if not os.path.isfile(DB_PATH):
+        _SCHEMA_READY.discard(DB_PATH)
     db = sqlite3.connect(DB_PATH, timeout=10)
     try:
         db.execute("PRAGMA secure_delete=ON")
-        db.execute("CREATE TABLE IF NOT EXISTS projects(id TEXT PRIMARY KEY, root TEXT NOT NULL, data TEXT NOT NULL, updated_at REAL NOT NULL)")
-        db.execute("CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, project_id TEXT, status TEXT NOT NULL, updated_at REAL NOT NULL, payload TEXT NOT NULL)")
-        db.execute("CREATE INDEX IF NOT EXISTS jobs_project ON jobs(project_id,updated_at)")
+        if DB_PATH not in _SCHEMA_READY:
+            db.execute("CREATE TABLE IF NOT EXISTS projects(id TEXT PRIMARY KEY, root TEXT NOT NULL, data TEXT NOT NULL, updated_at REAL NOT NULL)")
+            db.execute("CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, project_id TEXT, status TEXT NOT NULL, updated_at REAL NOT NULL, payload TEXT NOT NULL)")
+            db.execute("CREATE INDEX IF NOT EXISTS jobs_project ON jobs(project_id,updated_at)")
+            db.execute("CREATE INDEX IF NOT EXISTS jobs_updated ON jobs(updated_at)")
+            db.execute("CREATE INDEX IF NOT EXISTS jobs_status ON jobs(status,project_id)")
+            _SCHEMA_READY.add(DB_PATH)
         with db:
             yield db
     finally:
@@ -102,17 +109,62 @@ def record_changes(project_id, paths=(), full=False, scope=False):
         project = get(project_id)
         if not project:
             return
-        merged = set(project.get('dirty_paths', [])) | set(paths)
-        full = full or project.get('needs_full_check', False) or len(merged) > 1000
-        update(project_id, dirty_paths=[] if full else sorted(merged), needs_full_check=full,
-               scope_dirty=scope or project.get('scope_dirty', False))
+        update(project_id, **_changes_fields(project, paths, full, scope))
+
+
+def _changes_fields(project, paths, full, scope):
+    """`needs_full_check` / `scope_dirty` remember the `dirty_seq` at which they were raised (`*_seq`), so a job that ran
+    after that sequence can clear them even while events keep arriving."""
+    merged = set(project.get('dirty_paths', [])) | set(paths)
+    full = full or project.get('needs_full_check', False)
+    scope = scope or project.get('scope_dirty', False)
+    seq = project.get('dirty_seq', 0)
+    # Past 1000 paths the list stops helping: the update compares stat of the whole scope instead.
+    return {"dirty_paths": [] if len(merged) > 1000 else sorted(merged), "needs_full_check": full,
+            "scope_dirty": scope,
+            "full_check_seq": seq if full and not project.get('needs_full_check') else project.get('full_check_seq', 0),
+            "scope_dirty_seq": seq if scope and not project.get('scope_dirty') else project.get('scope_dirty_seq', 0)}
+
+
+def note_changes(project_id, reason, paths=(), full=False, scope=False):
+    """mark_dirty + record_changes in one read and one write of the registry; a burst of file events costs one commit per
+    batch instead of two reads and two commits."""
+    with _LOCK, _db() as db:
+        row = db.execute("SELECT data FROM projects WHERE id=?", (project_id,)).fetchone()
+        if not row:
+            return None
+        data = json.loads(row[0])
+        data["dirty_seq"] = data.get("dirty_seq", 0) + 1
+        data.update(status="dirty" if not data.get("paused") else "paused", dirty_reason=reason,
+                    last_event=time.time(), **_changes_fields(data, paths, full, scope))
+        db.execute("UPDATE projects SET data=?,updated_at=? WHERE id=?",
+                   (json.dumps(data, ensure_ascii=False), time.time(), project_id))
+    return data
 
 
 def confirm_changes(project_id, sequence):
+    """Clears what the finished job covered: the path list only if nothing changed since the job captured `sequence`;
+    the full-check and scope flags if they were raised at or before it (the job did both from that state)."""
     with _LOCK:
         project = get(project_id)
-        if project and project.get('dirty_seq', 0) == sequence:
-            update(project_id, dirty_paths=[], needs_full_check=False, scope_dirty=False)
+        if not project:
+            return
+        fields = {}
+        if project.get('dirty_seq', 0) == sequence:
+            fields.update(dirty_paths=[], needs_full_check=False, scope_dirty=False)
+        else:
+            if project.get('needs_full_check') and project.get('full_check_seq', 0) <= sequence:
+                fields["needs_full_check"] = False
+            if project.get('scope_dirty') and project.get('scope_dirty_seq', 0) <= sequence:
+                fields["scope_dirty"] = False
+        if fields:
+            update(project_id, **fields)
+
+
+def projects_with_pending_job():
+    """Ids of the projects with a job still pending, from the `status` column: no job payload is decrypted."""
+    with _LOCK, _db() as db:
+        return {row[0] for row in db.execute("SELECT DISTINCT project_id FROM jobs WHERE status='pending'")}
 
 
 def save_job(job_id, payload):
@@ -123,8 +175,10 @@ def save_job(job_id, payload):
     with _LOCK, _db() as db:
         db.execute("INSERT OR REPLACE INTO jobs VALUES(?,?,?,?,?)",
                    (job_id, payload.get("project_id"), payload.get("status", "pending"), time.time(), protected))
-        db.execute("DELETE FROM jobs WHERE status!='pending' AND id NOT IN "
-                   "(SELECT id FROM jobs ORDER BY updated_at DESC LIMIT ?)", (MAX_JOBS,))
+        # The threshold comes from the updated_at index alone: ordering the payload rows read the whole jobs table
+        # (1.7 MB) on every save, several saves per job.
+        db.execute("DELETE FROM jobs WHERE status!='pending' AND updated_at < "
+                   "(SELECT updated_at FROM jobs ORDER BY updated_at DESC LIMIT 1 OFFSET ?)", (MAX_JOBS - 1,))
 
 
 def _decode(payload):

package/smart_tool_daemon.py

@@ -892,11 +892,11 @@ def _cold_index(root, cfg, problem):
 
 
 def _cold_overlay(root, queries):
-    """`(overlay, warning)` for a project with no index: the in-scope files (the saved scope, else the whole folder minus
-    the always-excluded names) holding a query term, read from disk and ranked by keyword, plus the notice that the
-    index is being built. The scan stops after `COLD_SCAN_BUDGET_S`."""
+    """`(overlay, warning)` for a project with no index: the in-scope files (the saved scope, else only the loose files
+    at the root: no folder is read before it is classified) holding a query term, read from disk and ranked by
+    keyword, plus the notice that the index is being built. The scan stops after `COLD_SCAN_BUDGET_S`."""
     scope = index_scope.load_scope(root)
-    files = index_scope.resolve_included_files(root, scope if index_scope.valid_scope(scope) else {"include": [""]})
+    files = index_scope.resolve_included_files(root, scope if index_scope.valid_scope(scope) else {"include": []})
     chunks, scanned = indexer.keyword_chunks(root, files, queries, COLD_SCAN_BUDGET_S)
     reached = f"{scanned} of {len(files)} files" if scanned < len(files) else f"all {len(files)} files"
     warning = ("Indexing in progress: this folder has no index yet, so the results are keyword matches read straight "
@@ -1873,7 +1873,8 @@ def _project_action_locked(arguments):
                 "storage_dir": indexer.INDEX_DIR,
                 "gateway": dict(_GATEWAY_STATUS),
                 "monitor": {"debounce_s": project_monitor.DEBOUNCE_S,
-                            "reconcile_s": project_monitor.RECONCILE_S}}
+                            "reconcile_s": project_monitor.RECONCILE_S,
+                            "reconcile_watched_s": project_monitor.RECONCILE_WATCHED_S}}
     if action == "register":
         root = arguments.get("project_root")
         if not isinstance(root, str) or not os.path.isabs(root):
@@ -1957,7 +1958,10 @@ def _project_action_locked(arguments):
         manual = arguments.get("manual", True)
         if type(manual) is not bool:
             raise ValueError("manual must be true or false.")
-        scope = dict(index_scope.load_scope(project["root"]) or {"include": ["."], "exclude": []})
+        scope = dict(index_scope.load_scope(project["root"]) or {"include": [], "exclude": []})
+        if "include" not in arguments and not scope.get("include"):
+            raise ValueError("include is required: list the folders with the project's code, tests and docs. "
+                             "Nothing is indexed from a folder that is not listed.")
         for field in ("include", "exclude", "user_exclude"):
             values = arguments.get(field, scope.get(field, []))
             if not isinstance(values, list) or len(values) > 200 or any(
@@ -1968,6 +1972,10 @@ def _project_action_locked(arguments):
                     raise ValueError("The scope must stay inside the project folder.")
                 if field == "include" and not project_identity.within_root(project["root"], os.path.join(project["root"], value)):
                     raise ValueError("Include points outside the folder.")
+                if field == "include" and index_scope.is_hidden(value):
+                    raise ValueError(f"Include '{value}' is inside a hidden folder: hidden folders are never indexed.")
+            if field == "include":
+                values = index_scope.normalize_include(values)[0]
             scope[field] = values
         scope.update(manual=manual, scope_version=index_scope.SCOPE_VERSION if manual else 0)
         scope.pop("profile", None)

package/version.py

@@ -1,7 +1,7 @@
 """Single source of the Smart Tool version and of the User-Agent sent to public APIs."""
 import re
 
-VERSION = "0.9.25"
+VERSION = "0.9.27"
 _CONTACT_RE = re.compile(r"^(?:[^\s@()<>;]+@[^\s@()<>;]+\.[^\s@()<>;]+|https?://[^\s()<>;]+)$")