symgrep-codesearch 0.6.4__tar.gz → 0.7.0__tar.gz

{symgrep_codesearch-0.6.4/src/symgrep_codesearch.egg-info → symgrep_codesearch-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: symgrep-codesearch
-Version: 0.6.4
+Version: 0.7.0
 Summary: Code search with honest recall — structure-aware grep that tells you what it does not know, plus search by meaning
 License-Expression: MIT
 Project-URL: Repository, https://github.com/ashdamle/symgrep
@@ -140,11 +140,20 @@ that needed "every function with its scope-qualified name and span" wrote its ow
 ## Use
 
 ```bash
-symgrep PATTERN [ROOT ...] [-i] [-F] [-U] [--glob G] [--max-sites N] [--within PAT2] [--usage] [--resolve] [--json]
+symgrep PATTERN [ROOT ...] [-i] [-F] [-U] [--glob G] [--lang L] [--max-sites N] [--within PAT2] [--usage] [--resolve] [--json]
 symgrep --index src/                                   # build the semantic index (incremental)
 symgrep --semantic "what the function would do" src/   # search by meaning
+symgrep --semantic "…" --glob '*.py' --lang python src/ # meaning, scoped to a file set (ranked WITHIN it)
 ```
 
+`--glob` and `--lang` scope **every** mode, `--semantic` included: they are literal file bounds, and under
+`--semantic` meaning is ranked only *within* the matching files — never used to *select* files (selecting by
+meaning would silently drop relevant files, the exact blindness symgrep exists to remove). Because a glob is
+a guess about *where* code lives, a **regex** search also reports what the glob **hid** — files that match the
+pattern but fall outside the glob — so a too-narrow `--glob` is visible, not a silent false-negative. And
+archives (`.zip`/`.tar.gz`/`.jar`/…) are **opaque**: never indexed or searched, but **named** in the output
+when they sit under the scope, so their contents can't be mistaken for *not there*.
+
 The index stores one sentence per function describing its **job**, keyed by body hash, so a rebuild only
 costs the functions that changed. Embeddings are local (`fastembed` / bge-small): no API key, no network,
 nothing about the code leaves the machine.
@@ -384,7 +393,9 @@ wrong engine): `4900` ccwatch · `4910` 7thsense · `4911` symgrep. The SessionS
   (fastmcp requires both media types in Accept even though the reply is plain JSON).
 - Call: `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search","arguments":{…}}}`
   with the same arguments as the stdio tool: `pattern`, `roots` (required), `case_insensitive`,
-  `fixed_string`, `multiline`, `globs`, `include_ignored`, `max_sites`, `semantic`, `k`.
+  `fixed_string`, `multiline`, `globs`, `include_ignored`, `max_sites`, `semantic`, `k`, `lang`.
+  `globs` and `lang` scope **every** mode including `semantic` (literal file bounds; under `semantic`,
+  meaning is ranked only within them, never used to select files).
 - Reply: `result.content[0].text` = the same rendered report the stdio tool returns (honest zero-hits,
   FRESHNESS line, scope line included). Tool errors arrive as JSON-RPC errors, not empty results.
 - **Structured output**: pass `"structured": true` (or the alias `"json_output": true`) and
@@ -394,6 +405,12 @@ wrong engine): `4900` ccwatch · `4910` 7thsense · `4911` symgrep. The SessionS
     basis, score?, desc?, match_lines?}], …}` — name hits carry `match_lines`; semantic hits carry
     `score` + `desc` and resolve the span from the file's AST. `scope` (semantic) carries the
     staleness/coverage counters.
+  - Two result-level **honesty** keys expose a silent false-negative to a program, not just a human:
+    `glob_excluded` (name mode) lists files that match the pattern but `--glob` excluded — the dropped
+    matches of a too-narrow glob, empty without a glob; `archives` lists opaque containers
+    (`.zip`/`.tar.gz`/…) under the scope whose contents are never indexed/searched — present on a semantic
+    search always, and on a name search only at zero hits (mirroring the prose; no walk on the hit path),
+    with `archives_capped` flagging a truncated walk rather than a silent undercount.
   - `{status: "no_index", …}` when the semantic index isn't built (fixable by `symgrep --index`) vs
     `{status: "unavailable", reason: …, message: …}` when it cannot run (no embedder / corrupt). These
     are distinguishable **without** reading the prose — the honesty feature a program needs.

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/README.md

@@ -109,11 +109,20 @@ that needed "every function with its scope-qualified name and span" wrote its ow
 ## Use
 
 ```bash
-symgrep PATTERN [ROOT ...] [-i] [-F] [-U] [--glob G] [--max-sites N] [--within PAT2] [--usage] [--resolve] [--json]
+symgrep PATTERN [ROOT ...] [-i] [-F] [-U] [--glob G] [--lang L] [--max-sites N] [--within PAT2] [--usage] [--resolve] [--json]
 symgrep --index src/                                   # build the semantic index (incremental)
 symgrep --semantic "what the function would do" src/   # search by meaning
+symgrep --semantic "…" --glob '*.py' --lang python src/ # meaning, scoped to a file set (ranked WITHIN it)
 ```
 
+`--glob` and `--lang` scope **every** mode, `--semantic` included: they are literal file bounds, and under
+`--semantic` meaning is ranked only *within* the matching files — never used to *select* files (selecting by
+meaning would silently drop relevant files, the exact blindness symgrep exists to remove). Because a glob is
+a guess about *where* code lives, a **regex** search also reports what the glob **hid** — files that match the
+pattern but fall outside the glob — so a too-narrow `--glob` is visible, not a silent false-negative. And
+archives (`.zip`/`.tar.gz`/`.jar`/…) are **opaque**: never indexed or searched, but **named** in the output
+when they sit under the scope, so their contents can't be mistaken for *not there*.
+
 The index stores one sentence per function describing its **job**, keyed by body hash, so a rebuild only
 costs the functions that changed. Embeddings are local (`fastembed` / bge-small): no API key, no network,
 nothing about the code leaves the machine.
@@ -353,7 +362,9 @@ wrong engine): `4900` ccwatch · `4910` 7thsense · `4911` symgrep. The SessionS
   (fastmcp requires both media types in Accept even though the reply is plain JSON).
 - Call: `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search","arguments":{…}}}`
   with the same arguments as the stdio tool: `pattern`, `roots` (required), `case_insensitive`,
-  `fixed_string`, `multiline`, `globs`, `include_ignored`, `max_sites`, `semantic`, `k`.
+  `fixed_string`, `multiline`, `globs`, `include_ignored`, `max_sites`, `semantic`, `k`, `lang`.
+  `globs` and `lang` scope **every** mode including `semantic` (literal file bounds; under `semantic`,
+  meaning is ranked only within them, never used to select files).
 - Reply: `result.content[0].text` = the same rendered report the stdio tool returns (honest zero-hits,
   FRESHNESS line, scope line included). Tool errors arrive as JSON-RPC errors, not empty results.
 - **Structured output**: pass `"structured": true` (or the alias `"json_output": true`) and
@@ -363,6 +374,12 @@ wrong engine): `4900` ccwatch · `4910` 7thsense · `4911` symgrep. The SessionS
     basis, score?, desc?, match_lines?}], …}` — name hits carry `match_lines`; semantic hits carry
     `score` + `desc` and resolve the span from the file's AST. `scope` (semantic) carries the
     staleness/coverage counters.
+  - Two result-level **honesty** keys expose a silent false-negative to a program, not just a human:
+    `glob_excluded` (name mode) lists files that match the pattern but `--glob` excluded — the dropped
+    matches of a too-narrow glob, empty without a glob; `archives` lists opaque containers
+    (`.zip`/`.tar.gz`/…) under the scope whose contents are never indexed/searched — present on a semantic
+    search always, and on a name search only at zero hits (mirroring the prose; no walk on the hit path),
+    with `archives_capped` flagging a truncated walk rather than a silent undercount.
   - `{status: "no_index", …}` when the semantic index isn't built (fixable by `symgrep --index`) vs
     `{status: "unavailable", reason: …, message: …}` when it cannot run (no embedder / corrupt). These
     are distinguishable **without** reading the prose — the honesty feature a program needs.

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/pyproject.toml

@@ -26,7 +26,7 @@ name = "symgrep-codesearch"
 # semantic search. Minor bump (dependency surface changed; no breaking import/CLI/MCP change).
 # 0.5.0: added the public `file_structure` library API (+ its MCP tool and `symgrep structure` CLI), import
 # module-name normalization, and the FileStructure `connects` list (structure schema v2).
-version = "0.6.4"
+version = "0.7.0"
 description = "Code search with honest recall — structure-aware grep that tells you what it does not know, plus search by meaning"
 readme = "README.md"
 # 3.11+ : the maintained grammar pack (tree-sitter-language-pack, a CORE dep below) ships abi3 wheels for

symgrep_codesearch-0.7.0/src/symgrep/archives.py

@@ -0,0 +1,86 @@
+"""Archive/container files are OPAQUE to symgrep — the honest skip, named so it is never a silent one.
+
+symgrep never extracts or descends into a .zip / .tar.gz / .jar / … : it imports no zipfile/tarfile/gzip and
+shells out to no unzip/tar, and the index/symbol/data pipelines gate on an extension ALLOWLIST that contains no
+archive extension. So an archive's inner files and bytes are never indexed, never embedded, and never
+content-searched — a `.tar.gz` that IS a data bundle is invisible to both name search and search-by-meaning.
+
+That is the right default (auto-extraction would resurrect the whole-tree describe runaway this tool is built to
+avoid), but invisible-by-policy must not read as "nothing there". This module enumerates the archives present
+under a search scope so the search/semantic renderers can SAY the contents were not covered — the same
+"honest zero-hit expansion" discipline the name search already applies, extended to containers.
+
+Detection is a BOUNDED filesystem walk (no subprocess, so no ripgrep dependency and nothing to fail closed on).
+When the walk hits its file-count cap it STOPS and says so (`capped=True`) rather than silently returning an
+undercount — a count that looks complete but is not is exactly the truncation this estate forbids.
+"""
+import os
+import pathlib
+from typing import List, Optional, Tuple
+
+# Matched on the full lowercased filename so the COMPOUND forms (.tar.gz) are recognised, and a bare .gz / .bz2
+# (a single compressed file, not a container of subfiles) is deliberately NOT treated as an archive.
+ARCHIVE_SUFFIXES = (".zip", ".jar", ".war", ".whl", ".egg", ".tar",
+                    ".tgz", ".tbz2", ".txz",
+                    ".tar.gz", ".tar.bz2", ".tar.xz", ".tar.zst")
+
+# Directories never worth walking for this advisory scan — VCS metadata and dependency dumps. A leading-dot dir
+# is pruned separately (keeps the walk off .git, .venv, caches) in the spirit of symgrep's other bounded walks.
+_SKIP_DIRS = {"node_modules", "__pycache__", ".git", ".hg", ".svn"}
+
+_DEFAULT_CAP = 50000             # files examined before the scan stops; a stop is REPORTED (capped=True), never silent
+
+
+def is_archive(name: str) -> bool:
+    """True if `name` is an opaque container by extension (compound .tar.* included). Pure format test."""
+    n = (name or "").lower()
+    return any(n.endswith(s) for s in ARCHIVE_SUFFIXES)
+
+
+def archive_files(roots, cap: int = _DEFAULT_CAP) -> Tuple[List[str], bool]:
+    """(archive_paths, capped) under `roots` — a bounded walk with dot-dirs and dependency dumps pruned.
+    `capped` is True when the walk stopped at `cap` files BEFORE exhausting the tree, so the path list is a
+    floor the caller must report as such (never present a capped count as complete). Never raises on a bad
+    root; a missing or unreadable path is skipped."""
+    out: List[str] = []
+    seen = 0
+    for root in (roots or []):
+        try:
+            rp = pathlib.Path(root)
+            if rp.is_file():
+                if is_archive(rp.name):
+                    out.append(str(rp))
+                continue
+            if not rp.is_dir():
+                continue
+            for dirpath, dirnames, filenames in os.walk(rp):
+                dirnames[:] = [d for d in dirnames if not d.startswith(".") and d not in _SKIP_DIRS]
+                for fn in filenames:
+                    seen += 1
+                    if seen > cap:
+                        return out, True          # bounded AND reported — the count is a floor, say so
+                    if is_archive(fn):
+                        out.append(os.path.join(dirpath, fn))
+        except OSError:
+            continue
+    return out, False
+
+
+def archive_note(roots, prefix: str = "  ") -> Optional[str]:
+    """One advisory line about opaque archives under `roots`, or None when there are none AND the scan was
+    complete — so a caller appends it unconditionally and it vanishes when irrelevant. Reports an exact count
+    normally, and 'at least N (scan stopped at the cap)' when the walk was capped, so an undercount can never
+    read as the full picture."""
+    found, capped = archive_files(roots)
+    if not found and not capped:
+        return None
+    count = (f"at least {len(found)}" if capped else f"{len(found)}")
+    tail = (" The archive scan stopped at its file cap, so more may be present." if capped else "")
+    if not found and capped:
+        # Scope too large to finish scanning and no archive seen yet — honest 'unknown', not a false 'none'.
+        return (f"{prefix}⚠ archive scan did not finish under this large scope (stopped at the file cap); "
+                f"any .zip/.tar.gz/… present would be OPAQUE — symgrep does not index or search archive "
+                f"contents. Narrow the scope to check, or extract archives you care about.")
+    return (f"{prefix}⚠ {count} archive(s) under scope (.zip/.tar.gz/.jar/…) are OPAQUE: symgrep never "
+            f"extracts or indexes archive contents, so files INSIDE them are not covered by this result. Extract "
+            f"an archive first (e.g. `tar xf <a>` / `unzip <a>`) to search or index what it contains.{tail}")

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/src/symgrep/cli.py

@@ -84,6 +84,19 @@ BULK_BACKOFF_S_DEFAULT = 20            # $SYMGREP_BULK_BACKOFF_S — sleep betwe
 def build_report(pattern, roots, case_insensitive=False, fixed_string=False,
                  multiline=False, globs=None, include_ignored=False,
                  max_sites=DEFAULT_MAX_SITES):
+    """Run the name/regex search and return a grouped, annotated Report (the core search seam).
+
+    pattern: ripgrep regex (literal if fixed_string). roots: dirs/files to search; an EMPTY roots falls back
+    to the host cwd and the resolved scope is recorded on Report.roots (so a result is never attributed to an
+    unrecorded ambient cwd). case_insensitive/fixed_string/multiline/globs/include_ignored map to the rg
+    flags. max_sites bounds how many enclosing-symbol sites are SHOWN (Report.shown_sites); counts
+    (total_line_matches/total_files) are over everything, never the cap. On zero hits the Report carries
+    mechanical `expansions`; whenever a glob excluded files that also match, it carries `glob_excluded` +
+    `glob_excluded_note`. Coverage gaps ride `skipped_errors`. Propagates ripgrep.RipgrepNotFound (and other
+    rg errors) from the underlying passes rather than masking them as an empty result."""
+    # Scope resolution: an omitted `roots` falls back to the host cwd. The RESOLVED scope is recorded on the
+    # returned Report (`roots=roots` below) as provenance, so the result names exactly which directories were
+    # searched rather than leaving it to an unrecorded ambient value.
     roots = [os.path.abspath(r) for r in (roots or [os.getcwd()])]
     rg = run_rg(pattern, roots, case_insensitive=case_insensitive,
                 fixed_string=fixed_string, multiline=multiline, globs=globs,
@@ -92,13 +105,22 @@ def build_report(pattern, roots, case_insensitive=False, fixed_string=False,
     expansions = []
     if not sites:
         expansions = zero_hit_expansions(pattern, roots, case_insensitive, include_ignored)
+    # Honest recall for --glob: name the files that match the pattern but the glob EXCLUDED, so a too-narrow
+    # glob is never a silent false-negative (glob_recall reuses rg's own passes; no-op without globs).
+    from .glob_recall import glob_excluded_matches
+    glob_excluded, glob_note = glob_excluded_matches(pattern, roots, globs,
+                                                     case_insensitive=case_insensitive,
+                                                     fixed_string=fixed_string,
+                                                     include_ignored=include_ignored)
     return Report(pattern=pattern, roots=roots, sites=sites,
                   total_line_matches=rg.total_line_matches,
                   total_files=len({m.path for m in rg.matches}),
                   truncated_events=rg.truncated_events,
                   shown_sites=min(max_sites, len(sites)),   # what is SHOWN, not the cap asked for
                   expansions=expansions,
-                  skipped_errors=rg.skipped_errors)
+                  skipped_errors=rg.skipped_errors,
+                  glob_excluded=glob_excluded,
+                  glob_excluded_note=glob_note)
 
 
 def _describer():
@@ -483,7 +505,10 @@ def main(argv=None) -> int:
     ap.add_argument("-U", "--multiline", action="store_true",
                     help="let PATTERN span lines (rg -U)")
     ap.add_argument("--glob", action="append", dest="globs",
-                    help="restrict files by glob, repeatable, e.g. --glob '*.py' (rg --glob)")
+                    help="restrict files by glob, repeatable, e.g. --glob '*.py' (rg --glob). Also scopes "
+                         "--semantic (meaning is ranked only WITHIN the matching files). On a regex search a "
+                         "too-narrow glob is made VISIBLE: files that match the pattern but fall outside the "
+                         "glob are reported as excluded-but-matching (drop/widen it, or use --semantic)")
     ap.add_argument("--no-ignore", action="store_true", dest="include_ignored",
                     help="search gitignored + hidden files too (rg --no-ignore --hidden)")
     ap.add_argument("--max-sites", type=int, default=DEFAULT_MAX_SITES,
@@ -513,7 +538,8 @@ def main(argv=None) -> int:
                     help="treat PATTERN as an AST SHAPE (pattern-by-example: $NAME=one node, $$$=variadic) "
                          "and match it structurally via tree-sitter — a third question beside name/meaning")
     ap.add_argument("--lang", default=None,
-                    help="--structural: force the pattern's language (else inferred per file by extension)")
+                    help="--structural: force the pattern's language (else inferred per file by extension). "
+                         "--semantic: restrict meaning-search to files of this language (by extension)")
     ap.add_argument("--reindex", action="store_true",
                     help="with --index: FORCE re-describe of every function under ROOTS, ignoring the "
                          "body-hash cache (use after changing the describer or its instruction)")
@@ -523,7 +549,14 @@ def main(argv=None) -> int:
                          "block), data (schema/data/config symbol), both, or all")
     ap.add_argument("--symbol-kind", dest="kind", default=None,
                     help="--semantic: filter within a unit, e.g. data_schema, sql_table, proto_message")
-    args = ap.parse_args(argv)
+    # parse_intermixed_args, NOT parse_args: `pattern` (nargs="?") and `roots` (nargs="*") are two
+    # variadic positionals, and on argparse < 3.12 a value-taking option BETWEEN them — the ordinary
+    # `symgrep PAT --glob '*.py' ROOT` — makes parse_args drop the trailing ROOT as "unrecognized
+    # arguments" (silently wrong scope on 3.11, the version floor). parse_intermixed_args parses the
+    # optionals first and distributes the positionals after, so the same command behaves identically on
+    # every supported Python. Equivalent on 3.12+, so this only widens recall; it never changes a parse
+    # that already worked.
+    args = ap.parse_intermixed_args(argv)
 
     if args.index:
         # `symgrep --index src` used to parse `src` as the PATTERN and index the cwd instead — a silent
@@ -553,7 +586,8 @@ def main(argv=None) -> int:
                                    progress=lambda i, n: print(f"  {i}/{n} described", flush=True))
                 print(f"symgrep index: {r}")
                 return 0
-            hits = semantic.search_by_meaning(args.pattern, roots, k=args.k, unit=args.unit, kind=args.kind)
+            hits = semantic.search_by_meaning(args.pattern, roots, k=args.k, unit=args.unit, kind=args.kind,
+                                              globs=args.globs, lang=args.lang)
             print(semantic.render_meaning_hits(args.pattern, hits))
             return 0 if hits else 1
         except semantic.Unavailable as e:

symgrep_codesearch-0.7.0/src/symgrep/glob_recall.py

@@ -0,0 +1,54 @@
+"""Honest recall for --glob: a literal glob is a GUESS about WHERE code lives, and a wrong guess hides
+relevant files SILENTLY — the exact false-negative-with-no-signal this tool exists to refuse, reintroduced
+through a filename filter. An agent whose real intent is semantic ("the auth code", "the config loader")
+reaches for a glob that encodes that intent as a path pattern; when the pattern is narrower than the intent,
+the matching files outside it vanish from the result with nothing to say they were dropped.
+
+This reports the files that MATCH the pattern but the glob EXCLUDED, so a too-narrow --glob is VISIBLE. It is
+not a "semantic glob" (selecting files by meaning would itself silently drop relevant files, the same failure
+one level up) — the glob stays a mechanical bound; this only makes what it dropped honest. Finding files by
+MEANING regardless of filename is --semantic's job, and the note points there.
+
+Mechanism (no new spawn here — reuses ripgrep's own passes): the EXCLUDED-but-matching set is the files that
+match the pattern with NO glob, minus the files the glob admits (glob membership, ignore-neutral so the glob
+alone decides). Best-effort and advisory: it never changes the result, only annotates it, and any rg failure
+yields no note rather than a false "nothing excluded" (the main search already reports its own coverage).
+"""
+import os
+from typing import List, Optional, Tuple
+
+_SAMPLE = 5                      # how many excluded paths to name inline before summarising the remainder
+
+
+def glob_excluded_matches(pattern, roots, globs, *, case_insensitive=False, fixed_string=False,
+                          include_ignored=False) -> Tuple[List[str], Optional[str]]:
+    """(excluded_paths, note) — files under `roots` that MATCH `pattern` but the `globs` filter EXCLUDED.
+
+    Returns ([], None) when there is no glob, nothing was excluded, or an rg pass could not run — never a
+    false "nothing excluded". The flags mirror the main search so the counterfactual is the same question
+    minus the glob. Paths are reported exactly as ripgrep prints them (no host-dependent rewriting)."""
+    if not globs:
+        return [], None
+    from . import ripgrep
+    try:
+        # Files matching the pattern under the SAME ignore rules, WITHOUT the glob — the honest denominator.
+        matching = ripgrep.count_files_with_match(pattern, roots, case_insensitive=case_insensitive,
+                                                  fixed_string=fixed_string, include_ignored=include_ignored)
+        # Files the glob ADMITS (ignore-neutral: the glob alone decides membership, so an indexed/tracked
+        # file is judged by the pattern, never hidden by an ignore rule here).
+        members = {os.path.realpath(p) for p in ripgrep.list_files(roots, include_ignored=True, globs=globs)}
+    except (ripgrep.ExpansionFailed, ripgrep.RipgrepNotFound, OSError):
+        # The extra passes are advisory; if they cannot run, emit no note (the main search reports coverage).
+        # Narrow catch so a genuine deliberate-stop elsewhere is never swallowed here.
+        return [], None
+    excluded = [p for p in matching if os.path.realpath(p) not in members]
+    if not excluded:
+        return [], None
+    shown = sorted(excluded)[:_SAMPLE]
+    sample = ", ".join(shown)
+    more = f", +{len(excluded) - _SAMPLE} more" if len(excluded) > _SAMPLE else ""
+    note = (f"⚠ --glob EXCLUDED {len(excluded)} file(s) that ALSO match this pattern ({sample}{more}) — "
+            f"a glob is a guess about WHERE the code is, and these matches fall outside it, so they are NOT in "
+            f"the results above. If your intent was broader than the filename pattern, drop or widen --glob, or "
+            f"search by MEANING with symgrep --semantic.")
+    return excluded, note

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/src/symgrep/mcp_server.py

@@ -78,7 +78,18 @@ TOOL_DESCRIPTION = (
     "'python' | 'go' | 'javascript' | 'typescript' | 'tsx' | 'rust' | 'java' | 'c' | 'cpp' | 'ruby' | 'php' |\n"
     "'c_sharp' | 'kotlin' | 'scala' | 'bash'); with no usable grammar it reports UNAVAILABLE rather than\n"
     "silently falling back to text. Use it for structural queries a name/regex search cannot express; use\n"
-    "plain regex or semantic=true for everything else."
+    "plain regex or semantic=true for everything else.\n\n"
+    "globs= and lang= SCOPE EVERY MODE, semantic=true included: they are literal file bounds (ripgrep glob\n"
+    "engine; lang by extension), and under --semantic meaning is ranked only WITHIN that file set, never\n"
+    "used to SELECT files. A glob is a guess about WHERE code lives, so a REGEX search also reports what the\n"
+    "glob HID: structured=true returns `glob_excluded` (files that match the pattern but the glob excluded —\n"
+    "the silent false-negatives of a too-narrow glob; empty without a glob), and the prose render warns the\n"
+    "same. If the glob was narrower than your intent, drop/widen it or switch to semantic=true.\n\n"
+    "ARCHIVES ARE OPAQUE: .zip/.tar.gz/.jar/etc. are NEVER indexed or searched, so their contents cannot\n"
+    "appear in any result. When archives sit under the searched scope the prose render NAMES them (so a miss\n"
+    "is not misread as 'nothing anywhere'); structured=true returns them under `archives` — on a semantic\n"
+    "search always, and on a name search when there were zero hits (mirroring the prose, no extra walk on the\n"
+    "hit path)."
 )
 
 
@@ -100,7 +111,8 @@ def _search(pattern, roots, case_insensitive=False, fixed_string=False,
         # remember at the moment remembering fails.
         from . import semantic as sem
         try:
-            hits = sem.search_by_meaning(pattern, roots or [os.getcwd()], k=k, unit=unit, kind=kind)
+            hits = sem.search_by_meaning(pattern, roots or [os.getcwd()], k=k, unit=unit, kind=kind,
+                                         globs=globs, lang=lang)
             return sem.render_meaning_hits(pattern, hits)
         except sem.Unavailable as e:
             # An unavailable search is NOT an empty result, and saying "no matches" here would recreate the
@@ -157,7 +169,11 @@ def _run_fastmcp(http=None) -> bool:
         structural=True: match by AST SHAPE, not regex — `pattern` is a code snippet with metavariables
         ($NAME matches one node, $$$ is variadic). Requires lang= (the tree-sitter grammar the pattern parses
         in, e.g. "python"/"go"/"rust"); reports UNAVAILABLE when no grammar is usable rather than falling back
-        to text. lang is used ONLY by structural search.
+        to text.
+
+        globs / lang also SCOPE semantic=True: globs restricts meaning-search to files matching an rg glob
+        (e.g. ["*.py"]) — needs ripgrep, else UNAVAILABLE rather than a silently unfiltered result — and lang
+        restricts it to one language by file extension. They select the file set; meaning only ranks within it.
 
         unit: --semantic granularity — "symbol" (default; a whole function/method, the only pre-v6 behaviour,
         byte-stable) | "block" (a sub-function AST block, Type-4 clone retrieval) | "data" (a schema/data/config
@@ -358,7 +374,23 @@ def main() -> None:
             sys.exit(2)
         host = os.environ.get("SYMGREP_MCP_HOST", HTTP_HOST_DEFAULT)
         port = int(os.environ.get("SYMGREP_MCP_PORT", HTTP_PORT_DEFAULT))
-        _run_fastmcp(http=(host, port))
+        # WARM SHARED DAEMON = SINGLETON (one per port): every session CONNECTS to this warm HTTP server via urllib
+        # instead of spawning a stdio copy. A second start detects the incumbent (flock via spendguard's single-instance
+        # primitive) and exits 0 — never races FastMCP into a bind crash — so a supervisor can (re)start it idempotently.
+        try:
+            from spendguard import config as _sgc
+            _lock_cm = _sgc.single_instance_lock(f"symgrep-mcp-http-{port}")
+        except Exception:
+            _lock_cm = None
+        if _lock_cm is not None:
+            with _lock_cm as _got:
+                if not _got:
+                    sys.stderr.write(f"symgrep-mcp: a warm HTTP daemon is already serving on :{port} — not starting "
+                                     f"a second (the warm daemon is a singleton; connect to the running one).\n")
+                    return
+                _run_fastmcp(http=(host, port))
+        else:
+            _run_fastmcp(http=(host, port))
         return
     if not _run_fastmcp():
         _run_fallback()

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/src/symgrep/mcp_structured.py

@@ -15,6 +15,15 @@ ONE hit shape for BOTH search modes, so a consumer writes one parser:
   {symbol, path, line, start_line, end_line, kind, basis, score?, desc?, match_lines?}
     - name search : score/desc absent; match_lines = the matched line numbers in that symbol
     - semantic    : score + desc present; match_lines absent; span resolved from the file's AST
+
+Two result-level HONESTY keys make a silent false-negative visible to a PROGRAM, not just a human reader:
+  - glob_excluded : (name mode) files that MATCH the pattern but --glob EXCLUDED — the dropped matches of a
+                    too-narrow glob; empty without a glob. The machine twin of the prose warning.
+  - archives      : opaque containers (.zip/.tar.gz/.jar/…) under the searched scope, which are NEVER
+                    indexed or searched, so their contents can never appear in `hits`. Present on a
+                    semantic search always, and on a name search only when there were zero hits (mirroring
+                    the prose render; no filesystem walk on the hit path). archives_capped flags a truncated
+                    walk, never a silent undercount.
 """
 import os
 from typing import List, Optional
@@ -39,13 +48,20 @@ def structured_search(pattern, roots, *, case_insensitive=False, fixed_string=Fa
                       structural=False, lang=None, usage=False, resolve=False, unit="symbol", kind=None) -> dict:
     """Return a dict: {status, mode, hits, ...}. status ∈ ok | no_index | unavailable — mechanical, so a
     consumer never parses prose to learn availability."""
+    # Resolve the search scope ONCE and RECORD it on every result below as `roots` provenance: an empty
+    # `roots` falls back to the host cwd, and an ambient cwd that silently decides which files are scanned —
+    # and therefore the findings — is not reproducible unless the output names it. A caller that passed roots
+    # sees them echoed back unchanged; a caller that passed none sees exactly which directory was searched.
+    # Every return below carries "roots": eff_roots so no result's scope is left to an unrecorded ambient value.
+    eff_roots = list(roots) if roots else [os.getcwd()]
     if resolve:
         # R0 cross-file candidate resolution: definitions (cross-file symbol table) + references, with
         # AMBIGUITY surfaced (n_def>1). Candidates are by name (basis=candidate-name), never binding-resolved.
         from .resolve import resolve_name
-        r = resolve_name(pattern, roots or [os.getcwd()], case_insensitive=case_insensitive,
+        r = resolve_name(pattern, eff_roots, case_insensitive=case_insensitive,
                          fixed_string=fixed_string, include_ignored=include_ignored, globs=globs)
-        return {"status": "ok", "mode": "resolve", "verdict": r.verdict, "ambiguous": r.ambiguous,
+        return {"status": "ok", "mode": "resolve", "roots": eff_roots, "verdict": r.verdict,
+                "ambiguous": r.ambiguous,
                 "n_def": r.n_def, "n_ref": r.n_ref, "files_scanned": r.files_scanned,
                 # R1: how many references were narrowed to a single binding vs left as R0 candidates. Each
                 # reference carries its own basis (resolved-local/-import/-external/candidate-name) + resolved_to.
@@ -58,36 +74,38 @@ def structured_search(pattern, roots, *, case_insensitive=False, fixed_string=Fa
         # Count ALL occurrences (not the display cap) — a truncated count could make a used symbol look
         # unused, the silent undercount the tool refuses. references[] is still trimmed to max_sites for the
         # payload, but n_ref/verdict are computed over everything.
-        rep = build_report(pattern, roots or None, case_insensitive=case_insensitive,
+        rep = build_report(pattern, eff_roots, case_insensitive=case_insensitive,
                            fixed_string=fixed_string, multiline=multiline, globs=globs,
                            include_ignored=include_ignored, max_sites=10_000_000)
         mark_definitions(rep, pattern, fixed_string=fixed_string, case_insensitive=case_insensitive)
         s = usage_summary(rep)
-        return {"status": "ok", "mode": "usage", "verdict": s.verdict, "n_def": s.n_def, "n_ref": s.n_ref,
+        return {"status": "ok", "mode": "usage", "roots": eff_roots, "verdict": s.verdict, "n_def": s.n_def,
+                "n_ref": s.n_ref,
                 "ref_files": s.ref_files, "definitions": s.definitions, "references": s.references[:max_sites],
                 "lexical": True}
     if structural:
         from . import structural as st
         try:
-            report = st.structural_report(pattern, roots or [os.getcwd()], lang=lang,
+            report = st.structural_report(pattern, eff_roots, lang=lang,
                                           include_ignored=include_ignored, max_sites=max_sites)
         except st.StructuralUnavailable as e:
-            return {"status": "unavailable", "mode": "structural", "message": str(e)}
+            return {"status": "unavailable", "mode": "structural", "roots": eff_roots, "message": str(e)}
         hits = [{"symbol": site.symbol.name, "path": site.path,
                  "line": site.matches[0].line_number if site.matches else site.symbol.start,
                  "start_line": site.symbol.start, "end_line": site.symbol.end, "kind": site.symbol.kind,
                  "basis": "ast-pattern", "match_lines": [m.line_number for m in site.matches]}
                 for site in report.sites[:max_sites]]
-        return {"status": "ok", "mode": "structural", "hits": hits,
+        return {"status": "ok", "mode": "structural", "roots": eff_roots, "hits": hits,
                 "total_sites": len(report.sites), "skipped": report.skipped_errors}
     if semantic:
         from . import semantic as sem
         try:
-            hits = sem.search_by_meaning(pattern, roots or [os.getcwd()], k=k, unit=unit, kind=kind)
+            hits = sem.search_by_meaning(pattern, eff_roots, k=k, unit=unit, kind=kind,
+                                         globs=globs, lang=lang)
         except sem.Unavailable as e:
             code = getattr(e, "code", "unavailable")
             return {"status": "no_index" if code == "no_index" else "unavailable",
-                    "mode": "semantic", "reason": code, "message": str(e)}
+                    "mode": "semantic", "roots": eff_roots, "reason": code, "message": str(e)}
         # Payloads aligned to `hits` (search_by_meaning.last_hits) carry the unit + span, so a BLOCK or DATA
         # hit surfaces its OWN span + kind + enclosing while a SYMBOL hit keeps the exact pre-v6 key set (the
         # AST-resolved enclosing span). A consumer parses ONE shape; block/data-only keys are purely additive.
@@ -110,10 +128,16 @@ def structured_search(pattern, roots, *, case_insensitive=False, fixed_string=Fa
                 out.append({"symbol": qual, "path": path, "line": lineno,
                             "start_line": start, "end_line": end, "kind": skind, "basis": basis,
                             "score": score, "desc": what})
-        return {"status": "ok", "mode": "semantic", "hits": out,
-                "scope": getattr(sem.search_by_meaning, "last_scope", None)}
+        # Opaque archives under scope, so a machine consumer sees what the prose render already NAMES — a
+        # .zip/.tar.gz is never indexed, so its contents can never appear here, and that absence must be
+        # visible, not inferred. Always on a semantic search (the semantic render always notes them).
+        from . import archives
+        _apaths, _acapped = archives.archive_files(eff_roots)
+        return {"status": "ok", "mode": "semantic", "roots": eff_roots, "hits": out,
+                "scope": getattr(sem.search_by_meaning, "last_scope", None),
+                "archives": _apaths, "archives_capped": _acapped}
 
-    report = build_report(pattern, roots or None, case_insensitive=case_insensitive,
+    report = build_report(pattern, eff_roots, case_insensitive=case_insensitive,
                           fixed_string=fixed_string, multiline=multiline, globs=globs,
                           include_ignored=include_ignored, max_sites=max_sites)
     # Kythe/OpenGrok anchor: tag which of a hit's occurrences DECLARE a matching symbol (definition)
@@ -132,10 +156,25 @@ def structured_search(pattern, roots, *, case_insensitive=False, fixed_string=Fa
                     "match_lines": match_lines,
                     # occurrences on this symbol's declaration line (a subset of match_lines); [] = all refs
                     "def_lines": defs, "role": "definition" if defs else "reference"})
-    return {"status": "ok", "mode": "name", "hits": out,
+    # Opaque archives under scope — ONLY on a zero-hit name search, mirroring the prose render (report.py
+    # appends the archive note only when there are no sites) and avoiding a filesystem walk on the common
+    # hit path. A .zip/.tar.gz is never indexed/searched, so on a miss a consumer must be able to tell
+    # "nothing matched, and there were also these unopened containers" from "nothing anywhere".
+    if report.sites:
+        _apaths, _acapped = [], False
+    else:
+        from . import archives
+        _apaths, _acapped = archives.archive_files(eff_roots)
+    return {"status": "ok", "mode": "name", "roots": eff_roots, "hits": out,
             "total_sites": len(report.sites), "shown_sites": min(max_sites, len(report.sites)),
             "total_line_matches": report.total_line_matches, "total_files": report.total_files,
             "truncated_events": report.truncated_events,
+            # Files matching the pattern that --glob excluded (honest recall for a too-narrow glob); empty
+            # without a glob. A structured consumer sees the silent false-negatives, not just what matched.
+            "glob_excluded": report.glob_excluded,
+            # Opaque archives under scope on a zero-hit search (empty when there were hits); contents are
+            # never indexed/searched, so this is how a program learns they were skipped, not absent.
+            "archives": _apaths, "archives_capped": _acapped,
             "expansions": report.expansions, "skipped_errors": report.skipped_errors}
 
 

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/src/symgrep/report.py

@@ -16,6 +16,7 @@ from collections import OrderedDict
 from dataclasses import dataclass, field
 from typing import Dict, List, Optional
 
+from .archives import archive_note
 from .enclosing import Symbol, file_symbols
 from .ripgrep import RgResult, RawMatch, count_files_with_match, ExpansionFailed
 
@@ -75,6 +76,11 @@ class Report:
     # line where that symbol's own name matches the query). Everything else is a reference. Lists (not sets)
     # so dataclasses.asdict → json.dumps stays serializable. Populated by mark_definitions(), empty until then.
     def_lines: Dict[str, List[int]] = field(default_factory=dict)
+    # Files that MATCH the pattern but --glob EXCLUDED (honest recall for a too-narrow glob), plus the
+    # rendered advisory note. Empty/None when no glob was given or nothing matching was excluded. Populated by
+    # build_report; surfaced by render_report whether or not there were hits.
+    glob_excluded: List[str] = field(default_factory=list)
+    glob_excluded_note: Optional[str] = None
 
 
 def build_sites(rg: RgResult) -> List[Site]:
@@ -364,6 +370,13 @@ def render_report(report: Report, max_lines_per_site: int = DEFAULT_MAX_LINES_PE
         if len(report.skipped_errors) > 5:
             lines.append(f"    … {len(report.skipped_errors) - 5} more")
 
+    # GLOB RECALL HONESTY (whether or not there were hits): a --glob is a GUESS about where code lives, and a
+    # wrong guess hides relevant files SILENTLY. If the glob excluded files that ALSO match the pattern, say
+    # so here so a too-narrow glob is visible, never a silent false-negative. (Finding files by MEANING is
+    # --semantic's job; this only makes the glob's exclusions honest.)
+    if getattr(report, "glob_excluded_note", None):
+        lines.append(report.glob_excluded_note)
+
     if not report.sites and report.mode == "structural":
         # A structural zero-hit is "no code of this SHAPE" — NOT the name-search story below, and NOT a
         # reason to reach for --semantic. Coverage (skipped_errors above) is already stated.
@@ -392,6 +405,12 @@ def render_report(report: Report, max_lines_per_site: int = DEFAULT_MAX_LINES_PE
                      "nothing DOES this.")
         lines.append(f"  before writing something new, ask by meaning: symgrep --semantic '{report.pattern}' "
                      f"(or describe the job: --semantic \"what the function would do\")")
+        # An absent NAME is one way a result reads falsely as "nothing here"; an OPAQUE ARCHIVE is another —
+        # rg does not search inside .zip/.tar.gz and symgrep does not index them, so a hit living inside one is
+        # invisible with no signal. Say so when archives sit under the searched scope (advisory; no extraction).
+        _arch = archive_note(report.roots)
+        if _arch:
+            lines.append(_arch)
         return "\n".join(lines)
 
     shown = report.sites[:report.shown_sites]

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/src/symgrep/semantic.py

@@ -568,9 +568,18 @@ def _module_name(f: pathlib.Path) -> str:
 
 
 def search_by_meaning(query: str, roots: List[str], k: int = 8,
-                      unit: str = "symbol", kind: Optional[str] = None) -> List[Tuple[float, str, str, int, str]]:
+                      unit: str = "symbol", kind: Optional[str] = None,
+                      globs: Optional[List[str]] = None,
+                      lang: Optional[str] = None) -> List[Tuple[float, str, str, int, str]]:
     """[(score, qual, path, lineno, what)] — the closest capabilities BY MEANING. Retrieval only.
 
+    `globs`/`lang` SCOPE the result to files matching an rg glob (e.g. ["*.py"]) and/or a tree-sitter
+    language (by extension). They select the FILE SET mechanically — exactly as the literal name search does
+    with --glob — and meaning still only RANKS within it; there is no meaning-based file selection. The filter
+    is applied BEFORE the top-k cut so it never under-fills. A glob uses ripgrep's OWN glob engine (so the
+    scope matches the name search's, byte for byte); if rg is unavailable the call raises Unavailable rather
+    than silently returning results that ignore the requested glob.
+
     Raises Unavailable when the index or the embedder is missing, so a caller can never mistake "this could
     not run" for "nothing in the codebase does that". The ranking is veccore's (dot == cosine over the
     L2-normalized description vectors); the roots scope + staleness signal + liveness filter are symgrep's.
@@ -637,6 +646,32 @@ def search_by_meaning(query: str, roots: List[str], k: int = 8,
     # function deleted since the last build that covered it); the next such build reclaims the vector via
     # veccore's delete seam (_embed_live), so orphans do not accumulate.
     kept = [h for h in hits if h.id in live and in_scope((h.payload or {}).get("path", ""))]
+
+    # GLOB / LANG SCOPING (honest: the literal path/extension decides membership — meaning only RANKS within
+    # it, it never SELECTS files). Applied BEFORE the top-k cut below so filtering never under-fills. GLOB
+    # reuses ripgrep's OWN glob engine (ignore-neutral via --no-ignore, so an indexed-but-gitignored file
+    # still matches its glob and the scope equals what the name search would show); if rg is absent we RAISE
+    # rather than return a result that quietly ignores the glob. LANG filters by the same extension→grammar
+    # map symbol extraction uses. Both are mechanical set membership (parsing), not a meaning judgement.
+    def _hit_path(h):
+        return os.path.realpath((h.payload or {}).get("path", "") or "")
+    if globs:
+        from . import ripgrep
+        try:
+            ripgrep.rg_binary()                       # RipgrepNotFound is a deliberate stop — surfaced as Unavailable, not swallowed
+        except ripgrep.RipgrepNotFound as e:
+            raise Unavailable(
+                "--glob scoping needs ripgrep, which is not available — refusing to return a semantic result "
+                "that silently ignores the requested glob. Install rg, or drop --glob.",
+                code="glob_needs_rg") from e
+        glob_roots = root_abs or [os.path.realpath(r) for r in indexed_roots()]
+        allowed = {os.path.realpath(p) for p in ripgrep.list_files(glob_roots, include_ignored=True, globs=globs)}
+        kept = [h for h in kept if _hit_path(h) in allowed]
+    if lang:
+        from .enclosing import TS_LANG_BY_EXT
+        kept = [h for h in kept
+                if TS_LANG_BY_EXT.get(os.path.splitext(_hit_path(h))[1].lower()) == lang]
+
     scored = [(h.score, (h.payload or {}).get("qual"), (h.payload or {}).get("path"),
                (h.payload or {}).get("lineno"), (h.payload or {}).get("what")) for h in kept[:k]]
     # Full payloads aligned to `scored`, for the structured consumer to read block span + unit/parent WITHOUT
@@ -670,7 +705,16 @@ def search_by_meaning(query: str, roots: List[str], k: int = 8,
     # (rglob + INDEXABLE_EXTS + not _is_generated — see functions()) and count those absent from the index. A
     # DISPLAY signal (how far behind the semantic view is), never a decision; scanned over the roots asked
     # about, or the index's own recorded roots when the search is unscoped.
-    indexed_paths = {os.path.realpath(p) for p in file_stamp}
+    # CONTROL for the not_indexed count below (so the number matches its claim). The claim is: "indexable
+    # files on disk under the scanned roots that are in NO index record". The generating constraint is set
+    # membership against the FULL bookkeeping set `book` — NOT the in-scope-only `file_stamp` — so an
+    # already-indexed file that happens to sit outside the in-scope subset can never be miscounted as a fresh,
+    # never-described one. (For a scoped search scan_roots IS the in-scope domain, so this only hardens the
+    # invariant; the count is unchanged there. For an unscoped search it closes the one path by which the
+    # measured count could exceed the claimed one.) The denominator for the freshness rate above is
+    # file_stamp (asserted non-empty by `in_index`); this count is a raw count, no rate, so it needs only its
+    # named membership constraint, which is `indexed_paths`.
+    indexed_paths = {os.path.realpath(p) for _k, p, _s in book}
     scan_roots = root_abs or [os.path.realpath(r) for r in indexed_roots()]
     not_indexed = 0
     for _root in scan_roots:
@@ -685,7 +729,8 @@ def search_by_meaning(query: str, roots: List[str], k: int = 8,
                                     "bad_dim": 0, "roots": root_abs,
                                     "files_checked": len(file_stamp), "stale_files": stale,
                                     "unknown_age": unknown_age, "gone_files": gone_files,
-                                    "not_indexed_files": not_indexed}
+                                    "not_indexed_files": not_indexed,
+                                    "glob": list(globs) if globs else None, "lang": lang}
     return scored
 
 
@@ -699,6 +744,15 @@ def render_meaning_hits(query: str, hits, index=None) -> str:
         lines.append(f"  scope: {scope.get('in_index', 0)} indexed function(s); "
                      f"{scope.get('out_of_scope', 0)} outside the roots asked about (excluded, say so); "
                      f"{scope.get('bad_dim', 0)} with incompatible vectors (excluded)")
+        if scope.get("glob"):
+            lines.append(f"  scope: restricted to files matching --glob {scope['glob']} (ripgrep glob "
+                         f"engine) — meaning was ranked only WITHIN that file set")
+        if scope.get("lang"):
+            lines.append(f"  scope: restricted to --lang {scope['lang']} (by file extension)")
+        from . import archives
+        arch_note = archives.archive_note(scope.get("roots") or [])
+        if arch_note:
+            lines.append(arch_note)
         stale = scope.get("stale_files", 0); unk = scope.get("unknown_age", 0); gone = scope.get("gone_files", 0)
         new = scope.get("not_indexed_files", 0)
         if stale or unk or gone:

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0/src/symgrep_codesearch.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: symgrep-codesearch
-Version: 0.6.4
+Version: 0.7.0
 Summary: Code search with honest recall — structure-aware grep that tells you what it does not know, plus search by meaning
 License-Expression: MIT
 Project-URL: Repository, https://github.com/ashdamle/symgrep
@@ -140,11 +140,20 @@ that needed "every function with its scope-qualified name and span" wrote its ow
 ## Use
 
 ```bash
-symgrep PATTERN [ROOT ...] [-i] [-F] [-U] [--glob G] [--max-sites N] [--within PAT2] [--usage] [--resolve] [--json]
+symgrep PATTERN [ROOT ...] [-i] [-F] [-U] [--glob G] [--lang L] [--max-sites N] [--within PAT2] [--usage] [--resolve] [--json]
 symgrep --index src/                                   # build the semantic index (incremental)
 symgrep --semantic "what the function would do" src/   # search by meaning
+symgrep --semantic "…" --glob '*.py' --lang python src/ # meaning, scoped to a file set (ranked WITHIN it)
 ```
 
+`--glob` and `--lang` scope **every** mode, `--semantic` included: they are literal file bounds, and under
+`--semantic` meaning is ranked only *within* the matching files — never used to *select* files (selecting by
+meaning would silently drop relevant files, the exact blindness symgrep exists to remove). Because a glob is
+a guess about *where* code lives, a **regex** search also reports what the glob **hid** — files that match the
+pattern but fall outside the glob — so a too-narrow `--glob` is visible, not a silent false-negative. And
+archives (`.zip`/`.tar.gz`/`.jar`/…) are **opaque**: never indexed or searched, but **named** in the output
+when they sit under the scope, so their contents can't be mistaken for *not there*.
+
 The index stores one sentence per function describing its **job**, keyed by body hash, so a rebuild only
 costs the functions that changed. Embeddings are local (`fastembed` / bge-small): no API key, no network,
 nothing about the code leaves the machine.
@@ -384,7 +393,9 @@ wrong engine): `4900` ccwatch · `4910` 7thsense · `4911` symgrep. The SessionS
   (fastmcp requires both media types in Accept even though the reply is plain JSON).
 - Call: `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search","arguments":{…}}}`
   with the same arguments as the stdio tool: `pattern`, `roots` (required), `case_insensitive`,
-  `fixed_string`, `multiline`, `globs`, `include_ignored`, `max_sites`, `semantic`, `k`.
+  `fixed_string`, `multiline`, `globs`, `include_ignored`, `max_sites`, `semantic`, `k`, `lang`.
+  `globs` and `lang` scope **every** mode including `semantic` (literal file bounds; under `semantic`,
+  meaning is ranked only within them, never used to select files).
 - Reply: `result.content[0].text` = the same rendered report the stdio tool returns (honest zero-hits,
   FRESHNESS line, scope line included). Tool errors arrive as JSON-RPC errors, not empty results.
 - **Structured output**: pass `"structured": true` (or the alias `"json_output": true`) and
@@ -394,6 +405,12 @@ wrong engine): `4900` ccwatch · `4910` 7thsense · `4911` symgrep. The SessionS
     basis, score?, desc?, match_lines?}], …}` — name hits carry `match_lines`; semantic hits carry
     `score` + `desc` and resolve the span from the file's AST. `scope` (semantic) carries the
     staleness/coverage counters.
+  - Two result-level **honesty** keys expose a silent false-negative to a program, not just a human:
+    `glob_excluded` (name mode) lists files that match the pattern but `--glob` excluded — the dropped
+    matches of a too-narrow glob, empty without a glob; `archives` lists opaque containers
+    (`.zip`/`.tar.gz`/…) under the scope whose contents are never indexed/searched — present on a semantic
+    search always, and on a name search only at zero hits (mirroring the prose; no walk on the hit path),
+    with `archives_capped` flagging a truncated walk rather than a silent undercount.
   - `{status: "no_index", …}` when the semantic index isn't built (fixable by `symgrep --index`) vs
     `{status: "unavailable", reason: …, message: …}` when it cannot run (no embedder / corrupt). These
     are distinguishable **without** reading the prose — the honesty feature a program needs.

{symgrep_codesearch-0.6.4 → symgrep_codesearch-0.7.0}/src/symgrep_codesearch.egg-info/SOURCES.txt

@@ -2,12 +2,14 @@ LICENSE
 README.md
 pyproject.toml
 src/symgrep/__init__.py
+src/symgrep/archives.py
 src/symgrep/blocks.py
 src/symgrep/classfile.py
 src/symgrep/cli.py
 src/symgrep/data_symbols.py
 src/symgrep/docsymbols.py
 src/symgrep/enclosing.py
+src/symgrep/glob_recall.py
 src/symgrep/mcp_server.py
 src/symgrep/mcp_structured.py
 src/symgrep/report.py
@@ -49,6 +51,7 @@ tests/test_resolve_python.py
 tests/test_resolve_r2.py
 tests/test_review_regressions.py
 tests/test_semantic_freshness_signal.py
+tests/test_semantic_glob_lang.py
 tests/test_sites_and_symbols.py
 tests/test_structural.py
 tests/test_symbol_vocabulary.py

symgrep_codesearch-0.7.0/tests/test_semantic_glob_lang.py

@@ -0,0 +1,239 @@
+"""Glob/lang scoping under --semantic, and the opaque-archive honest-skip note.
+
+Two fixes, both guarding a SILENT non-handling that read as "nothing there":
+  1. `--glob`/`--lang` were PARSED but never passed to search_by_meaning, so a semantic search silently
+     ignored the requested scope. These pin the plumbing (the exact drop) AND the mechanics (rg glob
+     membership, the extension→grammar map) AND the honest render + the Unavailable-when-rg-absent contract.
+  2. Archives (.zip/.tar.gz/…) are opaque — never indexed or searched. These pin the detector and that both
+     the literal zero-hit report and the semantic render SAY SO instead of letting archive contents vanish.
+
+Also pins the ambient-cwd provenance fix: structured_search records `roots` on every result.
+
+Run: python tests/test_semantic_glob_lang.py
+"""
+import os
+import sys
+import tempfile
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(os.path.dirname(HERE), "src"))
+sys.path.insert(0, HERE)
+from checklib import Checks                                                            # noqa: E402
+_t = Checks("test_semantic_glob_lang")
+check = _t.check
+
+from symgrep import archives                                                           # noqa: E402
+from symgrep import semantic                                                           # noqa: E402
+from symgrep import mcp_structured                                                     # noqa: E402
+from symgrep import mcp_server                                                         # noqa: E402
+from symgrep.enclosing import TS_LANG_BY_EXT                                           # noqa: E402
+from symgrep.report import Report, render_report                                       # noqa: E402
+
+
+def _fixture_tree():
+    d = tempfile.mkdtemp(prefix="symgrep-glob-test-")
+    open(os.path.join(d, "a.py"), "w").close()
+    open(os.path.join(d, "b.js"), "w").close()
+    open(os.path.join(d, "bundle.zip"), "w").close()
+    open(os.path.join(d, "data.tar.gz"), "w").close()
+    open(os.path.join(d, "notes.txt.gz"), "w").close()        # bare .gz — a single compressed file, NOT a container
+    sub = os.path.join(d, "sub")
+    os.makedirs(sub)
+    open(os.path.join(sub, "lib.jar"), "w").close()
+    return d
+
+
+# ---- fix 2: archive detection (the OPAQUE contract made visible) --------------------------------------
+check("is_archive: .zip / .tar.gz / .jar are archives", all(archives.is_archive(n)
+      for n in ("x.zip", "x.tar.gz", "x.tgz", "lib.jar")), "a known archive suffix was not recognised")
+check("is_archive: source + bare .gz are NOT archives (a .gz is one file, not a container)",
+      not any(archives.is_archive(n) for n in ("a.py", "b.js", "notes.txt.gz", "r.txt")),
+      "a non-container was misread as an archive")
+
+_fx = _fixture_tree()
+found, capped = archives.archive_files([_fx])
+_names = sorted(os.path.basename(p) for p in found)
+check("archive_files finds the containers (recursively), not the source/.gz",
+      _names == ["bundle.zip", "data.tar.gz", "lib.jar"], f"got {_names}")
+check("archive_files reports capped=False on a small complete scan", capped is False, f"capped={capped}")
+check("archive_note names the opaque archives and says contents are not covered",
+      (lambda n: n is not None and "archive(s)" in n and "OPAQUE" in n)(archives.archive_note([_fx])),
+      "archive_note did not produce the honest-skip line")
+check("archive_note is None when there are no archives (vanishes when irrelevant)",
+      archives.archive_note([tempfile.mkdtemp(prefix="empty-")]) is None, "note fired with no archives")
+
+# capped scan must SIGNAL, never silently undercount (the truncation doctrine)
+_f2, _cap2 = archives.archive_files([_fx], cap=1)
+check("archive_files signals capped=True when the file cap is hit (no silent undercount)",
+      _cap2 is True, f"capped={_cap2} with cap=1")
+
+
+# ---- fix 2 wiring: the literal zero-hit report emits the archive note ----------------------------------
+_rep = Report(pattern="zzz_no_such_symbol_qpz", roots=[_fx], sites=[], total_line_matches=0,
+              total_files=0, truncated_events=False, shown_sites=0)
+_rendered = render_report(_rep)
+check("literal zero-hit report appends the opaque-archive note when archives are under scope",
+      "archive(s)" in _rendered and "OPAQUE" in _rendered, _rendered[-200:])
+
+
+# ---- fix 1 plumbing: --glob/--lang actually REACH search_by_meaning (the exact drop that was the bug) --
+class _Recorder:
+    """Stands in for search_by_meaning; records the kwargs it was called with and returns no hits."""
+    def __init__(self):
+        self.calls = []
+        self.last_hits = []
+        self.last_scope = {}
+
+    def __call__(self, pattern, roots, k=8, unit="symbol", kind=None, globs=None, lang=None):
+        self.calls.append({"globs": globs, "lang": lang, "roots": roots, "k": k, "unit": unit})
+        self.last_scope = {"roots": roots, "glob": globs, "lang": lang, "in_index": 0, "out_of_scope": 0,
+                           "bad_dim": 0, "files_checked": 0, "stale_files": 0, "unknown_age": 0,
+                           "gone_files": 0, "not_indexed_files": 0}
+        return []
+
+
+_orig = semantic.search_by_meaning
+
+rec = _Recorder()
+semantic.search_by_meaning = rec
+try:
+    # structured path (structured=True)
+    out = mcp_structured.structured_search("find the thing", [_fx], semantic=True,
+                                           globs=["*.py"], lang="python")
+    check("structured_search(semantic) forwards globs to search_by_meaning",
+          rec.calls and rec.calls[-1]["globs"] == ["*.py"], f"calls={rec.calls}")
+    check("structured_search(semantic) forwards lang to search_by_meaning",
+          rec.calls[-1]["lang"] == "python", f"calls={rec.calls}")
+    check("structured_search records roots provenance on the result (ambient-cwd fix)",
+          out.get("roots") == [_fx], f"roots={out.get('roots')}")
+
+    # MCP non-structured semantic path (_search)
+    rec.calls.clear()
+    mcp_server._search("find the thing", [_fx], semantic=True, globs=["*.js"], lang="javascript")
+    check("mcp _search(semantic) forwards globs+lang to search_by_meaning",
+          rec.calls and rec.calls[-1]["globs"] == ["*.js"] and rec.calls[-1]["lang"] == "javascript",
+          f"calls={rec.calls}")
+
+    # CLI path (symgrep --semantic --glob --lang)
+    from symgrep import cli
+    rec.calls.clear()
+    _orig_render = semantic.render_meaning_hits
+    semantic.render_meaning_hits = lambda *a, **k: ""
+    try:
+        cli.main(["myquery", "--semantic", "--glob", "*.py", "--lang", "python", _fx])
+    finally:
+        semantic.render_meaning_hits = _orig_render
+    check("CLI --semantic forwards --glob and --lang to search_by_meaning (the parsed-but-dropped bug)",
+          rec.calls and rec.calls[-1]["globs"] == ["*.py"] and rec.calls[-1]["lang"] == "python",
+          f"calls={rec.calls}")
+finally:
+    semantic.search_by_meaning = _orig
+
+
+# ---- fix 1 mechanics: the building blocks the filter composes --------------------------------------
+check("lang map: .py→python, .js→javascript, .go→go (the extension→grammar filter basis)",
+      TS_LANG_BY_EXT.get(".py") == "python" and TS_LANG_BY_EXT.get(".js") == "javascript"
+      and TS_LANG_BY_EXT.get(".go") == "go", "TS_LANG_BY_EXT is missing an expected mapping")
+
+# rg glob membership is what the glob filter intersects against — pin it if rg is present.
+from symgrep import ripgrep                                                            # noqa: E402
+try:
+    _rg = ripgrep.rg_binary()
+except Exception:
+    _rg = None
+if _rg:
+    py_only = {os.path.realpath(p) for p in ripgrep.list_files([_fx], include_ignored=True, globs=["*.py"])}
+    check("rg glob membership (the filter's engine): *.py selects a.py, excludes b.js",
+          os.path.realpath(os.path.join(_fx, "a.py")) in py_only
+          and os.path.realpath(os.path.join(_fx, "b.js")) not in py_only, f"py_only={py_only}")
+else:
+    _t.skip("rg absent — glob-membership mechanics not exercised here")
+
+
+# ---- fix 1 render: the semantic output SAYS what was scoped and what is opaque ------------------------
+semantic.search_by_meaning.last_scope = {"roots": [_fx], "glob": ["*.py"], "lang": "python",
+                                         "in_index": 3, "out_of_scope": 0, "bad_dim": 0,
+                                         "files_checked": 3, "stale_files": 0, "unknown_age": 0,
+                                         "gone_files": 0, "not_indexed_files": 0}
+_r = semantic.render_meaning_hits("find the thing", [])
+check("semantic render states the --glob scope restriction", "--glob" in _r and "*.py" in _r, _r)
+check("semantic render states the --lang scope restriction", "--lang" in _r and "python" in _r, _r)
+check("semantic render appends the opaque-archive note (archives under the searched scope)",
+      "archive(s)" in _r and "OPAQUE" in _r, _r)
+
+
+# ---- the glob-exclusion honesty signal: a too-narrow --glob on a REGEX search is never silent ----
+# The concern: an agent whose real intent is semantic reaches for a --glob to encode WHERE the code is;
+# when that guess is narrower than the intent, matching files outside it vanish with no signal. This set
+# pins that glob_excluded_matches names those dropped-but-matching files, that the note vanishes when the
+# glob excludes nothing, and that build_report + render SURFACE the exclusion ALONGSIDE the in-glob hits.
+from symgrep import glob_recall                                                        # noqa: E402
+from symgrep.cli import build_report                                                   # noqa: E402
+
+_gx = tempfile.mkdtemp(prefix="symgrep-glob-excl-")
+open(os.path.join(_gx, "widget.py"), "w").write("def handle_WIDGETZ(): pass\n")
+open(os.path.join(_gx, "widget.js"), "w").write("function handle_WIDGETZ(){}\n")
+open(os.path.join(_gx, "readme.md"), "w").write("WIDGETZ documentation\n")
+
+if _rg:  # glob_excluded_matches + build_report both need ripgrep
+    _excl, _note = glob_recall.glob_excluded_matches("WIDGETZ", [_gx], ["*.py"])
+    _exn = sorted(os.path.basename(p) for p in _excl)
+    check("glob_excluded_matches names the matching files a *.py glob EXCLUDED (widget.js, readme.md)",
+          _exn == ["readme.md", "widget.js"], f"excluded={_exn}")
+    check("glob_excluded_matches: the note fires and points at --semantic as the meaning-based alternative",
+          _note is not None and "--glob EXCLUDED" in _note and "--semantic" in _note, f"note={_note!r}")
+    check("glob_excluded_matches: no glob → ([], None), never a false 'nothing excluded'",
+          glob_recall.glob_excluded_matches("WIDGETZ", [_gx], None) == ([], None), "fired without a glob")
+    check("glob_excluded_matches: a glob that drops no MATCHING file → ([], None) (nothing to warn about)",
+          glob_recall.glob_excluded_matches("WIDGETZ", [_gx], ["*.py", "*.js", "*.md"]) == ([], None),
+          "warned when the glob excluded no matching file")
+
+    # end-to-end: there ARE hits (in widget.py), yet the report still surfaces what the glob hid.
+    _repx = build_report("WIDGETZ", [_gx], globs=["*.py"])
+    check("build_report populates glob_excluded with the matching files the glob hid",
+          sorted(os.path.basename(p) for p in _repx.glob_excluded) == ["readme.md", "widget.js"],
+          f"glob_excluded={_repx.glob_excluded}")
+    check("build_report sets glob_excluded_note (the honest-recall warning) when a glob hid matches",
+          _repx.glob_excluded_note is not None and "--glob EXCLUDED" in _repx.glob_excluded_note,
+          f"note={_repx.glob_excluded_note!r}")
+    _renderx = render_report(_repx)
+    check("rendered report SURFACES the glob-exclusion warning next to the in-glob hit (not silent)",
+          "--glob EXCLUDED" in _renderx and "--semantic" in _renderx, _renderx[-400:])
+
+    # a search WITHOUT a glob carries no exclusion note (the signal is glob-specific, not always-on).
+    _repn = build_report("WIDGETZ", [_gx])
+    check("build_report without a glob leaves glob_excluded empty and the note unset",
+          _repn.glob_excluded == [] and _repn.glob_excluded_note is None,
+          f"glob_excluded={_repn.glob_excluded} note={_repn.glob_excluded_note!r}")
+
+    # the structured (machine) twin exposes the same dropped set, so a programmatic consumer sees it too.
+    _structx = mcp_structured.structured_search("WIDGETZ", [_gx], globs=["*.py"])
+    check("structured_search(name) exposes glob_excluded so a program sees the silent false-negatives",
+          sorted(os.path.basename(p) for p in _structx.get("glob_excluded", [])) == ["readme.md", "widget.js"],
+          f"glob_excluded={_structx.get('glob_excluded')}")
+else:
+    _t.skip("rg absent — glob-exclusion signal not exercised here")
+
+
+# ---- archive honesty reaches the STRUCTURED payload too (machine parity with the prose render) -------
+# A .zip/.tar.gz is never indexed/searched, so a programmatic consumer must be able to tell "searched,
+# nothing matched, and there were these unopened containers" from "nothing anywhere". The prose render
+# names archives on a name-search ONLY at zero hits; the structured name payload mirrors that exactly.
+_ax = _fixture_tree()  # has bundle.zip, data.tar.gz, sub/lib.jar + a.py/b.js (empty)
+if _rg:
+    _zero = mcp_structured.structured_search("zzz_no_such_symbol_qpz", [_ax])
+    _azn = sorted(os.path.basename(p) for p in _zero.get("archives", []))
+    check("structured_search(name) surfaces opaque archives on a ZERO-hit search (machine parity)",
+          _azn == ["bundle.zip", "data.tar.gz", "lib.jar"], f"archives={_azn}")
+    check("structured_search(name) reports archives_capped=False on a small complete scan",
+          _zero.get("archives_capped") is False, f"archives_capped={_zero.get('archives_capped')}")
+
+    # a name search WITH hits does NOT walk for archives (empty), matching the prose (no note when hits) and
+    # keeping the hit path free of an extra filesystem walk — the cost discipline, pinned.
+    open(os.path.join(_ax, "hit.py"), "w").write("def zzz_no_such_symbol_qpz(): pass\n")
+    _hit = mcp_structured.structured_search("zzz_no_such_symbol_qpz", [_ax])
+    check("structured_search(name) with HITS omits the archive walk (archives empty on the hit path)",
+          _hit.get("total_sites", 0) >= 1 and _hit.get("archives") == [],
+          f"total_sites={_hit.get('total_sites')} archives={_hit.get('archives')}")
+
+_t.finish()