cocoindex-code 0.2.36__tar.gz → 0.2.37__tar.gz

{cocoindex_code-0.2.36 → cocoindex_code-0.2.37}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex-code
-Version: 0.2.36
+Version: 0.2.37
 Summary: MCP server for indexing and querying codebases using CocoIndex
 Project-URL: Homepage, https://github.com/cocoindex-io/cocoindex-code
 Project-URL: Repository, https://github.com/cocoindex-io/cocoindex-code
@@ -17,7 +17,7 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.11
-Requires-Dist: cocoindex[litellm]<1.1.0,>=1.0.6
+Requires-Dist: cocoindex[litellm]<1.1.0,>=1.0.13
 Requires-Dist: einops>=0.8.2
 Requires-Dist: mcp>=1.0.0
 Requires-Dist: msgspec>=0.19.0
@@ -29,7 +29,7 @@ Requires-Dist: questionary>=2.0.0
 Requires-Dist: sqlite-vec>=0.1.0
 Requires-Dist: typer>=0.9.0
 Provides-Extra: dev
-Requires-Dist: cocoindex[sentence-transformers]<1.1.0,>=1.0.6; extra == 'dev'
+Requires-Dist: cocoindex[sentence-transformers]<1.1.0,>=1.0.13; extra == 'dev'
 Requires-Dist: mypy>=1.0.0; extra == 'dev'
 Requires-Dist: prek>=0.1.0; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
@@ -37,9 +37,9 @@ Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
 Requires-Dist: pytest>=7.0.0; extra == 'dev'
 Requires-Dist: ruff>=0.1.0; extra == 'dev'
 Provides-Extra: embeddings-local
-Requires-Dist: cocoindex[sentence-transformers]<1.1.0,>=1.0.6; extra == 'embeddings-local'
+Requires-Dist: cocoindex[sentence-transformers]<1.1.0,>=1.0.13; extra == 'embeddings-local'
 Provides-Extra: full
-Requires-Dist: cocoindex[sentence-transformers]<1.1.0,>=1.0.6; extra == 'full'
+Requires-Dist: cocoindex[sentence-transformers]<1.1.0,>=1.0.13; extra == 'full'
 Description-Content-Type: text/markdown
 
 <p align="center">
@@ -224,6 +224,7 @@ The background daemon starts automatically on first use.
 | `ccc init` | Initialize a project — creates settings files, adds `.cocoindex_code/` to `.gitignore` |
 | `ccc index` | Build or update the index (auto-inits if needed). Shows streaming progress. |
 | `ccc search <query>` | Semantic search across the codebase |
+| `ccc grep <pattern> [path]` | Structural code search by example (no index needed) |
 | `ccc status` | Show index stats (chunk count, file count, language breakdown) |
 | `ccc mcp` | Run as MCP server in stdio mode |
 | `ccc doctor` | Run diagnostics — checks settings, daemon, model, file matching, and index health |
@@ -244,6 +245,36 @@ ccc search --refresh database schema                 # update index first, then
 
 By default, `ccc search` scopes results to your current working directory (relative to the project root). Use `--path` to override.
 
+### Structural Search (`ccc grep`)
+
+`ccc grep` finds code by **structure**, not text — you write a by-example pattern
+and it matches the syntax tree (via cocoindex's `code_match`), so formatting,
+whitespace, and intervening tokens don't matter. It runs entirely locally: no
+index, daemon, or embeddings required.
+
+```bash
+ccc grep 'def \NAME(\(ARGS*\)):'                      # every Python function def under the cwd
+ccc grep 'foo(\(ARGS*\))' src/                        # calls to foo(...) anywhere under src/
+ccc grep 'fn \NAME(\(A*\))' --lang rust               # restrict to one language
+ccc grep 'class \NAME:' --path 'tests/**'            # restrict to a path glob
+ccc grep 'TODO(\(A*\))' path/to/file.py               # a single file
+```
+
+Metavariables use the `\` sigil: `\NAME` captures one node, `\(NAME*\)` a run of
+siblings, `\_`/`\*` match anonymously. The pattern is matched per language, so a
+single invocation scans every supported source file (others are skipped). Inside
+an initialized project, `ccc grep` honors the project's include/exclude patterns
+and `.gitignore`; otherwise it scans all supported source files under the path.
+
+Results stream to the terminal file-by-file as each match is found (in completion
+order, since files are matched in parallel) rather than all at once at the end.
+Each matching file shows its matched line range; under a TTY the path is colored,
+line numbers are dimmed, and the unmatched context around a match is dimmed so the
+match stands out.
+
+> **Note:** `ccc grep` relies on cocoindex's structural `code_match` feature.
+> Until it ships in a released cocoindex, run against a local cocoindex build.
+
 ## Docker
 
 A Docker image is available for teams who want a reproducible, dependency-free

{cocoindex_code-0.2.36 → cocoindex_code-0.2.37}/README.md

@@ -180,6 +180,7 @@ The background daemon starts automatically on first use.
 | `ccc init` | Initialize a project — creates settings files, adds `.cocoindex_code/` to `.gitignore` |
 | `ccc index` | Build or update the index (auto-inits if needed). Shows streaming progress. |
 | `ccc search <query>` | Semantic search across the codebase |
+| `ccc grep <pattern> [path]` | Structural code search by example (no index needed) |
 | `ccc status` | Show index stats (chunk count, file count, language breakdown) |
 | `ccc mcp` | Run as MCP server in stdio mode |
 | `ccc doctor` | Run diagnostics — checks settings, daemon, model, file matching, and index health |
@@ -200,6 +201,36 @@ ccc search --refresh database schema                 # update index first, then
 
 By default, `ccc search` scopes results to your current working directory (relative to the project root). Use `--path` to override.
 
+### Structural Search (`ccc grep`)
+
+`ccc grep` finds code by **structure**, not text — you write a by-example pattern
+and it matches the syntax tree (via cocoindex's `code_match`), so formatting,
+whitespace, and intervening tokens don't matter. It runs entirely locally: no
+index, daemon, or embeddings required.
+
+```bash
+ccc grep 'def \NAME(\(ARGS*\)):'                      # every Python function def under the cwd
+ccc grep 'foo(\(ARGS*\))' src/                        # calls to foo(...) anywhere under src/
+ccc grep 'fn \NAME(\(A*\))' --lang rust               # restrict to one language
+ccc grep 'class \NAME:' --path 'tests/**'            # restrict to a path glob
+ccc grep 'TODO(\(A*\))' path/to/file.py               # a single file
+```
+
+Metavariables use the `\` sigil: `\NAME` captures one node, `\(NAME*\)` a run of
+siblings, `\_`/`\*` match anonymously. The pattern is matched per language, so a
+single invocation scans every supported source file (others are skipped). Inside
+an initialized project, `ccc grep` honors the project's include/exclude patterns
+and `.gitignore`; otherwise it scans all supported source files under the path.
+
+Results stream to the terminal file-by-file as each match is found (in completion
+order, since files are matched in parallel) rather than all at once at the end.
+Each matching file shows its matched line range; under a TTY the path is colored,
+line numbers are dimmed, and the unmatched context around a match is dimmed so the
+match stands out.
+
+> **Note:** `ccc grep` relies on cocoindex's structural `code_match` feature.
+> Until it ships in a released cocoindex, run against a local cocoindex build.
+
 ## Docker
 
 A Docker image is available for teams who want a reproducible, dependency-free

{cocoindex_code-0.2.36 → cocoindex_code-0.2.37}/pyproject.toml

@@ -23,7 +23,7 @@ classifiers = [
 
 dependencies = [
     "mcp>=1.0.0",
-    "cocoindex[litellm]>=1.0.6,<1.1.0",
+    "cocoindex[litellm]>=1.0.13,<1.1.0",
     "sqlite-vec>=0.1.0",
     "pydantic>=2.0.0",
     "numpy>=1.24.0",
@@ -39,7 +39,7 @@ dependencies = [
 # `embeddings-local` is the primary feature extra: it pulls in
 # `sentence-transformers` (via cocoindex) so local embeddings work without
 # an API key.
-embeddings-local = ["cocoindex[sentence-transformers]>=1.0.6,<1.1.0"]
+embeddings-local = ["cocoindex[sentence-transformers]>=1.0.13,<1.1.0"]
 # `full` is the umbrella "batteries-included" alias. Today it's just
 # `embeddings-local`, but we expect to bundle more optional niceties under
 # it over time — users who want everything can keep using `[full]` and pick
@@ -47,7 +47,7 @@ embeddings-local = ["cocoindex[sentence-transformers]>=1.0.6,<1.1.0"]
 # `:full` image variant for consistency across install paths. Contents are
 # inlined rather than self-referencing `cocoindex-code[embeddings-local]`
 # to avoid resolver edge cases with older pip.
-full = ["cocoindex[sentence-transformers]>=1.0.6,<1.1.0"]
+full = ["cocoindex[sentence-transformers]>=1.0.13,<1.1.0"]
 dev = [
     "pytest>=7.0.0",
     "pytest-asyncio>=0.21.0",
@@ -55,7 +55,7 @@ dev = [
     "ruff>=0.1.0",
     "mypy>=1.0.0",
     "prek>=0.1.0",
-    "cocoindex[sentence-transformers]>=1.0.6,<1.1.0",
+    "cocoindex[sentence-transformers]>=1.0.13,<1.1.0",
 ]
 
 [project.scripts]
@@ -89,7 +89,7 @@ dev = [
     "mypy>=1.0.0",
     "prek>=0.1.0",
     "types-pyyaml>=6.0.12.20250915",
-    "cocoindex[sentence-transformers]>=1.0.7,<1.1.0",
+    "cocoindex[sentence-transformers]>=1.0.13,<1.1.0",
 ]
 
 [tool.ruff]

{cocoindex_code-0.2.36 → cocoindex_code-0.2.37}/src/cocoindex_code/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.36'
-__version_tuple__ = version_tuple = (0, 2, 36)
+__version__ = version = '0.2.37'
+__version_tuple__ = version_tuple = (0, 2, 37)
 
 __commit_id__ = commit_id = None

{cocoindex_code-0.2.36 → cocoindex_code-0.2.37}/src/cocoindex_code/cli.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import functools
 import os
 import sys
+import threading
 from collections.abc import Callable
 from pathlib import Path
 from typing import TYPE_CHECKING, TypeVar
@@ -12,6 +13,7 @@ from typing import TYPE_CHECKING, TypeVar
 import typer as _typer
 
 if TYPE_CHECKING:
+    from .grep import FileMatches, GrepWarning
     from .protocol import (
         DoctorCheckResult,
         IndexingProgress,
@@ -657,6 +659,79 @@ def search(
     print_search_results(resp)
 
 
+@app.command()
+def grep(
+    pattern: str = _typer.Argument(
+        ...,
+        help=r"By-example structural pattern; use \ for metavariables, "
+        r"e.g. 'def \NAME(\(ARGS*\)):' or 'foo(\(ARGS*\))'.",
+    ),
+    path: str = _typer.Argument(
+        ".", help="File or directory to search (default: current directory)."
+    ),
+    lang: list[str] = _typer.Option(
+        [], "--lang", help="Only match files of these languages (e.g. python, rust, cpp)."
+    ),
+    path_glob: str | None = _typer.Option(
+        None, "--path", help="Only match files whose path matches this glob (globset syntax)."
+    ),
+    no_color: bool = _typer.Option(False, "--no-color", help="Disable colored output."),
+) -> None:
+    r"""Structurally grep code by example (no index or daemon required).
+
+    Compiles the pattern per language and matches every supported source file
+    under PATH in parallel. Inside an initialized project it honors the project's
+    include/exclude and .gitignore rules; otherwise it scans all supported source
+    files.
+    """
+    from . import grep as _grep
+
+    target = Path(path)
+    if not target.exists():
+        _typer.echo(f"Error: path not found: {path}", err=True)
+        raise _typer.Exit(code=1)
+
+    req = _grep.GrepRequest(
+        pattern=pattern,
+        root=target,
+        languages=frozenset(lang_name.lower() for lang_name in lang) or None,
+        path_glob=path_glob,
+    )
+    use_color = not no_color and sys.stdout.isatty() and not os.environ.get("NO_COLOR")
+    grep_run = _grep.Grep(req)
+    matched = 0
+    # `run` calls `_emit` from several worker threads at once; the lock keeps one
+    # file's output (and the `matched` bookkeeping) from interleaving with another's.
+    output_lock = threading.Lock()
+
+    def _emit(item: FileMatches | GrepWarning) -> None:
+        nonlocal matched
+        if isinstance(item, _grep.GrepWarning):
+            with output_lock:
+                _typer.echo(f"warning: {item.message}", err=True)
+            return
+        block = _grep.render_file(item, color=use_color)  # render outside the lock
+        with output_lock:
+            if matched:
+                _typer.echo()  # blank line between files
+            _typer.echo(block)
+            matched += 1
+
+    grep_run.run(_emit)
+
+    # The "unusable everywhere" verdict needs the whole walk, so it's known only
+    # once the run is done — report it before exiting.
+    if grep_run.unusable:
+        langs = ", ".join(grep_run.failed_languages)
+        _typer.echo(
+            f"Error: the pattern did not compile for any of the languages found ({langs}).",
+            err=True,
+        )
+        raise _typer.Exit(code=1)
+    if matched == 0:
+        _typer.echo("No matches found.")
+
+
 @app.command()
 @_catch_daemon_start_error
 def status() -> None:

{cocoindex_code-0.2.36 → cocoindex_code-0.2.37}/src/cocoindex_code/daemon.py

@@ -372,10 +372,8 @@ async def _check_file_walk(project_root_str: str) -> DoctorCheckResult:
     """Walk project files and report counts + gitignore paths."""
     from pathlib import PurePath
 
-    from cocoindex.resources.file import PatternFilePathMatcher
-
-    from .indexer import GitignoreAwareMatcher
-    from .settings import load_gitignore_spec, load_project_settings
+    from .file_walk import build_matcher
+    from .settings import load_project_settings
 
     project_root = Path(project_root_str)
     try:
@@ -383,12 +381,7 @@ async def _check_file_walk(project_root_str: str) -> DoctorCheckResult:
     except FileNotFoundError as e:
         return DoctorCheckResult(name="File Walk", ok=False, details=[], errors=[str(e)])
 
-    gitignore_spec = load_gitignore_spec(project_root)
-    base_matcher = PatternFilePathMatcher(
-        included_patterns=ps.include_patterns,
-        excluded_patterns=ps.exclude_patterns,
-    )
-    matcher = GitignoreAwareMatcher(base_matcher, gitignore_spec, project_root)
+    matcher = build_matcher(project_root, ps.include_patterns, ps.exclude_patterns)
 
     counts_by_ext: dict[str, int] = {}
     gitignore_dirs: list[str] = []

cocoindex_code-0.2.36/src/cocoindex_code/indexer.py → cocoindex_code-0.2.37/src/cocoindex_code/file_walk.py

@@ -1,36 +1,22 @@
-"""CocoIndex app for indexing codebases."""
+"""Shared source-file walking: pattern + .gitignore matching, reused by the
+indexer, the daemon's doctor file-walk, and ``ccc grep``.
+
+The matcher (include/exclude globs + nested ``.gitignore`` awareness) is the
+single source of truth for "which files count as part of the project". The
+indexer feeds it to CocoIndex's incremental file source; the daemon and ``ccc
+grep`` drive a plain :func:`os.walk` over it via :func:`iter_included_files`.
+"""
 
 from __future__ import annotations
 
-from collections.abc import Iterable
+import os
+from collections.abc import Iterable, Iterator
 from pathlib import Path, PurePath
 
-import cocoindex as coco
-from cocoindex.connectors import localfs, sqlite
-from cocoindex.connectors.sqlite import Vec0TableDef
-from cocoindex.ops.text import RecursiveSplitter, detect_code_language
-from cocoindex.resources.chunk import Chunk
 from cocoindex.resources.file import FilePathMatcher, PatternFilePathMatcher
-from cocoindex.resources.id import IdGenerator
 from pathspec import GitIgnoreSpec
 
-from .chunking import CHUNKER_REGISTRY
-from .settings import load_gitignore_spec, load_project_settings
-from .shared import (
-    CODEBASE_DIR,
-    EMBEDDER,
-    INDEXING_EMBED_PARAMS,
-    SQLITE_DB,
-    CodeChunk,
-)
-
-# Chunking configuration
-CHUNK_SIZE = 1000
-MIN_CHUNK_SIZE = 250
-CHUNK_OVERLAP = 150
-
-# Chunking splitter (stateless, can be module-level)
-splitter = RecursiveSplitter()
+from .settings import load_gitignore_spec
 
 
 def _normalize_gitignore_lines(lines: Iterable[str], directory: PurePath) -> list[str]:
@@ -134,98 +120,57 @@ class GitignoreAwareMatcher(FilePathMatcher):
         return self._delegate.is_file_included(path)
 
 
-@coco.fn(memo=True)
-async def process_file(
-    file: localfs.File,
-    table: sqlite.TableTarget[CodeChunk],
-) -> None:
-    """Process a single file: chunk, embed, and store."""
-    embedder = coco.use_context(EMBEDDER)
-    indexing_params = coco.use_context(INDEXING_EMBED_PARAMS)
-
-    try:
-        content = await file.read_text()
-    except UnicodeDecodeError:
-        return
-
-    if not content.strip():
-        return
-
-    suffix = file.file_path.path.suffix
-    project_root = coco.use_context(CODEBASE_DIR)
-    ps = load_project_settings(project_root)
-    ext_lang_map = {f".{lo.ext}": lo.lang for lo in ps.language_overrides}
-    language = (
-        ext_lang_map.get(suffix)
-        or detect_code_language(filename=file.file_path.path.name)
-        or "text"
-    )
-
-    chunker_registry = coco.use_context(CHUNKER_REGISTRY)
-    chunker = chunker_registry.get(suffix)
-    if chunker is not None:
-        language_override, chunks = chunker(Path(file.file_path.path), content)
-        if language_override is not None:
-            language = language_override
-    else:
-        chunks = splitter.split(
-            content,
-            chunk_size=CHUNK_SIZE,
-            min_chunk_size=MIN_CHUNK_SIZE,
-            chunk_overlap=CHUNK_OVERLAP,
-            language=language,
-        )
-
-    id_gen = IdGenerator()
-
-    async def process(chunk: Chunk) -> None:
-        table.declare_row(
-            row=CodeChunk(
-                id=await id_gen.next_id(chunk.text),
-                file_path=file.file_path.path.as_posix(),
-                language=language,
-                content=chunk.text,
-                start_line=chunk.start.line,
-                end_line=chunk.end.line,
-                embedding=await embedder.embed(chunk.text, **indexing_params),
-            )
-        )
-
-    await coco.map(process, chunks)
-
-
-@coco.fn
-async def indexer_main() -> None:
-    """Main indexing function - walks files and processes each."""
-    project_root = coco.use_context(CODEBASE_DIR)
-    ps = load_project_settings(project_root)
-    gitignore_spec = load_gitignore_spec(project_root)
-
-    table = await sqlite.mount_table_target(
-        db=SQLITE_DB,
-        table_name="code_chunks_vec",
-        table_schema=await sqlite.TableSchema.from_class(
-            CodeChunk,
-            primary_key=["id"],
-        ),
-        virtual_table_def=Vec0TableDef(
-            partition_key_columns=["language"],
-            auxiliary_columns=["file_path", "content", "start_line", "end_line"],
-        ),
-    )
-
+def find_git_root(start: Path) -> Path | None:
+    """Walk up from ``start`` to the nearest directory holding a ``.git`` entry — a
+    directory for a normal repo, or a *file* for a submodule or linked worktree.
+    Returns that directory, or ``None`` if ``start`` is not inside a git repo.
+
+    Used to anchor ``.gitignore`` resolution at the real repo root when grepping a
+    subdirectory that isn't inside an initialized cocoindex project."""
+    current = start.resolve()
+    while True:
+        if (current / ".git").exists():
+            return current
+        if current.parent == current:
+            return None
+        current = current.parent
+
+
+def build_matcher(
+    project_root: Path,
+    included_patterns: list[str],
+    excluded_patterns: list[str],
+) -> FilePathMatcher:
+    """Build the project's file matcher: include/exclude globs plus nested
+    ``.gitignore`` awareness anchored at ``project_root``."""
     base_matcher = PatternFilePathMatcher(
-        included_patterns=ps.include_patterns,
-        excluded_patterns=ps.exclude_patterns,
-    )
-    matcher: FilePathMatcher = GitignoreAwareMatcher(base_matcher, gitignore_spec, project_root)
-
-    files = localfs.walk_dir(
-        CODEBASE_DIR,
-        recursive=True,
-        path_matcher=matcher,
-    )
-
-    await coco.mount_each(
-        coco.component_subpath(coco.Symbol("process_file")), process_file, files.items(), table
+        included_patterns=included_patterns,
+        excluded_patterns=excluded_patterns,
     )
+    return GitignoreAwareMatcher(base_matcher, load_gitignore_spec(project_root), project_root)
+
+
+def iter_included_files(
+    start: Path,
+    base: Path,
+    matcher: FilePathMatcher,
+) -> Iterator[tuple[Path, PurePath]]:
+    """Walk ``start`` recursively, yielding ``(absolute_path, path_relative_to_base)``
+    for every file ``matcher`` includes, pruning excluded directories.
+
+    ``base`` anchors the relative paths the matcher sees (the project root, so
+    its patterns line up); ``start`` is where traversal begins and may be a
+    subdirectory of ``base``. Both must be absolute. Traversal is deterministic
+    (directories and files are visited in sorted order).
+    """
+    for dirpath_str, dirnames, filenames in os.walk(start):
+        dirpath = Path(dirpath_str)
+        rel_dir = PurePath(dirpath.relative_to(base))
+        if rel_dir != PurePath(".") and not matcher.is_dir_included(rel_dir):
+            dirnames.clear()
+            continue
+        dirnames.sort()
+        for fname in sorted(filenames):
+            rel_path = rel_dir / fname if rel_dir != PurePath(".") else PurePath(fname)
+            if matcher.is_file_included(rel_path):
+                yield dirpath / fname, rel_path

cocoindex_code-0.2.37/src/cocoindex_code/grep.py

@@ -0,0 +1,411 @@
+r"""``ccc grep`` — by-example structural code search over files.
+
+Unlike ``ccc search`` (semantic, needs the index + daemon + embeddings), ``grep``
+runs entirely locally: it compiles a structural pattern (cocoindex ``code_match``)
+once per language, walks the matching source files, and matches them in parallel.
+No index or daemon is required.
+
+Patterns use the ``\`` sigil for metavariables, e.g. ``def \NAME(\(ARGS*\)):`` or
+``foo(\(ARGS*\))``. See the cocoindex code_match design for the full syntax.
+"""
+
+from __future__ import annotations
+
+import functools
+from collections.abc import Callable, Iterator
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass
+from pathlib import Path
+
+import click
+from cocoindex.ops.code import CodeMatch, CodePattern
+from cocoindex.ops.text import detect_code_language
+from cocoindex.resources.file import PatternFilePathMatcher
+
+from .file_walk import build_matcher, find_git_root, iter_included_files
+from .settings import (
+    DEFAULT_EXCLUDED_PATTERNS,
+    DEFAULT_INCLUDED_PATTERNS,
+    find_project_root,
+    load_project_settings,
+)
+
+# A trivial, always-valid pattern (a bare identifier) used to probe whether a
+# language is structurally matchable, independent of the user's pattern.
+_PROBE_PATTERN = "x"
+
+
+@dataclass(frozen=True, slots=True)
+class GrepWarning:
+    """A non-fatal problem surfaced during a grep — a file that couldn't be read,
+    or a supported language the pattern failed to compile for. grep keeps going;
+    the CLI prints these to stderr."""
+
+    message: str
+
+    language: str | None = None
+    """Set when this is a per-language pattern-compile failure (so :class:`Grep` can
+    tell whether the pattern is unusable for *every* language it tried)."""
+
+
+@dataclass(frozen=True, slots=True)
+class FileMatches:
+    """Every match found in one file."""
+
+    path: str
+    """Display path, mirroring the root passed to grep (e.g. ``src/a.py`` or
+    ``/tmp/x/a.py``)."""
+
+    source: str
+    """Full file content — kept so the renderer can show context around each match."""
+
+    matches: list[CodeMatch]
+    """Matches in source order (at least one)."""
+
+
+@dataclass(frozen=True, slots=True)
+class GrepRequest:
+    """A grep invocation."""
+
+    pattern: str
+    root: Path
+    """File or directory to search."""
+
+    languages: frozenset[str] | None = None
+    """Restrict to these languages (lowercased canonical names); ``None`` = all."""
+
+    path_glob: str | None = None
+    """Extra include glob (globset syntax) on the project-relative path; ``None`` = all."""
+
+
+@dataclass(frozen=True, slots=True)
+class _Target:
+    path: Path
+    display: str
+    pattern: CodePattern
+
+
+# ---------------------------------------------------------------------------
+# Per-language pattern compilation
+# ---------------------------------------------------------------------------
+
+
+@functools.cache
+def _is_match_supported(language: str) -> bool:
+    """Whether code_match can structurally match ``language``.
+
+    Probes with a trivial always-valid pattern so the answer doesn't depend on
+    the user's pattern. Cached across the process.
+    """
+    try:
+        CodePattern(_PROBE_PATTERN, language=language)
+        return True
+    except ValueError:
+        return False
+
+
+class _PatternCompiler:
+    r"""Compiles one pattern per language on demand, caching the result.
+
+    code_match patterns are language-bound (compiled against a grammar's token
+    table), so we keep one compiled :class:`CodePattern` per language and reuse it
+    across every file of that language. A language maps to ``None`` (its files are
+    skipped) when either code_match can't match it at all, or the pattern won't
+    compile for it — the latter records a :class:`GrepWarning` once for that
+    language, so the user learns why those files were skipped instead of the whole
+    run aborting.
+    """
+
+    def __init__(self, pattern: str) -> None:
+        self._pattern = pattern
+        self._cache: dict[str, CodePattern | None] = {}
+        self.warnings: list[GrepWarning] = []
+
+    def for_language(self, language: str) -> CodePattern | None:
+        if language not in self._cache:
+            self._cache[language] = self._compile(language)
+        return self._cache[language]
+
+    def _compile(self, language: str) -> CodePattern | None:
+        try:
+            return CodePattern(self._pattern, language=language)
+        except ValueError as e:
+            # A *supported* language that won't compile the pattern is a real
+            # problem to surface (once per language); an *unsupported* language is
+            # an expected silent skip. `_is_match_supported` tells them apart,
+            # independent of the user's pattern.
+            if _is_match_supported(language):
+                self.warnings.append(
+                    GrepWarning(f"pattern invalid for {language}: {e}", language=language)
+                )
+            return None
+
+
+# ---------------------------------------------------------------------------
+# Target collection (which files to match, with which compiled pattern)
+# ---------------------------------------------------------------------------
+
+
+def _detect_language(path: Path, ext_overrides: dict[str, str]) -> str | None:
+    """Language for ``path``: project extension override first, then auto-detect."""
+    return ext_overrides.get(path.suffix) or detect_code_language(filename=path.name)
+
+
+def _ext_overrides(project_root: Path | None) -> dict[str, str]:
+    if project_root is None:
+        return {}
+    ps = load_project_settings(project_root)
+    return {f".{lo.ext}": lo.lang for lo in ps.language_overrides}
+
+
+def _target_for_file(
+    abs_path: Path,
+    display: str,
+    ext_overrides: dict[str, str],
+    req: GrepRequest,
+    compiler: _PatternCompiler,
+) -> _Target | None:
+    language = _detect_language(abs_path, ext_overrides)
+    if language is None:
+        return None
+    if req.languages is not None and language.lower() not in req.languages:
+        return None
+    cp = compiler.for_language(language)
+    if cp is None:
+        return None
+    return _Target(path=abs_path, display=display, pattern=cp)
+
+
+def _resolve_file(
+    abs_path: Path,
+    display: str,
+    ext_overrides: dict[str, str],
+    req: GrepRequest,
+    compiler: _PatternCompiler,
+) -> Iterator[_Target | GrepWarning]:
+    """Yield any compile warning newly raised for this file's language, then the
+    file's target if the pattern compiled for that language."""
+    before = len(compiler.warnings)
+    target = _target_for_file(abs_path, display, ext_overrides, req, compiler)
+    yield from compiler.warnings[before:]
+    if target is not None:
+        yield target
+
+
+def _iter_targets(req: GrepRequest, compiler: _PatternCompiler) -> Iterator[_Target | GrepWarning]:
+    """Lazily resolve a request into the files to match, yielded as they're
+    discovered (so matching can begin before the walk finishes), interleaved with
+    each new per-language compile warning.
+
+    Single source of truth with the indexer: the same include/exclude patterns and
+    ``.gitignore`` rules decide which files belong to the project. Outside a project
+    we fall back to the default source-file patterns.
+    """
+    root = req.root.resolve()
+
+    if root.is_file():
+        project_root = find_project_root(root.parent)
+        yield from _resolve_file(
+            root, req.root.as_posix(), _ext_overrides(project_root), req, compiler
+        )
+        return
+
+    project_root = find_project_root(root)
+    if project_root is not None:
+        ps = load_project_settings(project_root)
+        included, excluded = ps.include_patterns, ps.exclude_patterns
+        ext_overrides = {f".{lo.ext}": lo.lang for lo in ps.language_overrides}
+        base = project_root
+    else:
+        included = list(DEFAULT_INCLUDED_PATTERNS)
+        excluded = list(DEFAULT_EXCLUDED_PATTERNS)
+        ext_overrides = {}
+        # Anchor at the enclosing git repo so grepping a subdirectory still honors the
+        # repo-root (and intervening) .gitignore; fall back to the target dir itself.
+        base = find_git_root(root) or root
+
+    matcher = build_matcher(base, included, excluded)
+    path_filter = (
+        PatternFilePathMatcher(included_patterns=[req.path_glob]) if req.path_glob else None
+    )
+
+    for abs_path, rel in iter_included_files(root, base, matcher):
+        if path_filter is not None and not path_filter.is_file_included(rel):
+            continue
+        # Display paths mirror the root the user gave (e.g. "src/a.py", "/tmp/x/a.py"),
+        # rather than always being cwd-relative.
+        display = (req.root / abs_path.relative_to(root)).as_posix()
+        yield from _resolve_file(abs_path, display, ext_overrides, req, compiler)
+
+
+# ---------------------------------------------------------------------------
+# Matching
+# ---------------------------------------------------------------------------
+
+
+def _match_file(target: _Target) -> FileMatches | GrepWarning | None:
+    """Match one file via code_match's ``match_file``: it reads the file and runs
+    the prefilter in Rust — skipping the parse (and the Python-side read) for files
+    that can't match — all with the GIL released, so a worker-thread pool scans many
+    files truly in parallel.
+
+    Returns ``None`` for a file that's binary, prefilter-rejected, or has no matches
+    (a silent skip, like ``grep -I``); a :class:`GrepWarning` for an unreadable file.
+    """
+    try:
+        fm = target.pattern.match_file(str(target.path))
+    except OSError as e:
+        return GrepWarning(f"cannot read {target.display}: {e}")
+    if fm is None:
+        return None  # binary, prefilter-rejected, or no matches
+    matches = sorted(fm.matches, key=lambda m: m.chunks[0].start.byte_offset if m.chunks else 0)
+    return FileMatches(path=target.display, source=fm.ast.source, matches=matches)
+
+
+class Grep:
+    """A single grep run. :meth:`run` matches each file as it's listed and streams
+    results as they complete. After ``run`` is exhausted, :attr:`unusable` /
+    :attr:`failed_languages` report the compile verdict."""
+
+    def __init__(self, req: GrepRequest) -> None:
+        self._req = req
+        self._compiler = _PatternCompiler(req.pattern)
+        self._target_count = 0
+
+    def run(self, emit: Callable[[FileMatches | GrepWarning], object]) -> None:
+        """Match the request, calling ``emit`` with each file's matches and each
+        compile warning the moment it's ready — *while the walk is still running*.
+
+        Fully synchronous, no event loop: the walk runs on the calling thread and
+        submits each file's match to a thread pool. ``match_file`` reads + prefilters
+        + parses + matches in Rust with the GIL released, so the pool threads run
+        truly in parallel — with the walk and with each other — and hand each result
+        straight to ``emit`` as it finishes. ``run`` returns once the walk and every
+        match are done; afterwards :attr:`unusable` / :attr:`failed_languages` hold the
+        compile verdict.
+
+        ``emit`` is called concurrently from the pool's worker threads (plus this
+        thread, for warnings), so a consumer that does I/O must serialize it itself.
+        """
+
+        def _match(item: _Target) -> None:
+            result = _match_file(item)
+            if result is not None:  # None = binary / prefiltered / no match (skip)
+                emit(result)
+
+        with ThreadPoolExecutor() as pool:
+            for item in _iter_targets(self._req, self._compiler):
+                if isinstance(item, GrepWarning):
+                    emit(item)
+                else:
+                    self._target_count += 1
+                    pool.submit(_match, item)
+            # ThreadPoolExecutor.__exit__ waits for every submitted match to finish.
+
+    @property
+    def failed_languages(self) -> list[str]:
+        """Supported languages the pattern would not compile for (valid once
+        :meth:`run` is exhausted)."""
+        return [w.language for w in self._compiler.warnings if w.language is not None]
+
+    @property
+    def unusable(self) -> bool:
+        """The pattern compiled for *none* of the languages actually encountered — a
+        supported language was found but every one rejected the pattern, so there was
+        nothing to match. A target exists only for a file whose language compiled, so
+        zero targets plus ≥1 failed language means unusable everywhere it was tried —
+        distinct from "no matchable files found" (no failures). Valid once
+        :meth:`run` is exhausted."""
+        return bool(self.failed_languages) and self._target_count == 0
+
+
+# ---------------------------------------------------------------------------
+# Rendering
+# ---------------------------------------------------------------------------
+
+
+def _line_char_offsets(source: str) -> list[int]:
+    """Char offset of the start of each line (index ``line - 1``)."""
+    offsets = [0]
+    for i, ch in enumerate(source):
+        if ch == "\n":
+            offsets.append(i + 1)
+    return offsets
+
+
+def _paint(text: str, color: bool, **style: object) -> str:
+    if not color or not text:
+        return text
+    return click.style(text, **style)  # type: ignore[arg-type]
+
+
+def _render_code_line(
+    line_no: int,
+    text: str,
+    dim_pre_end: int,
+    dim_post_start: int,
+    width: int,
+    color: bool,
+) -> str:
+    """One code line: dimmed ``line_no| `` gutter, then the text with the leading
+    (before ``dim_pre_end``) and trailing (from ``dim_post_start``) context dimmed
+    and the matched span shown normally."""
+    pre = max(0, min(dim_pre_end, len(text)))
+    post = max(pre, min(dim_post_start, len(text)))
+    gutter = _paint(f"{line_no:>{width}}| ", color, fg="bright_black")
+    if not color:
+        return f"{gutter}{text}"
+    before = _paint(text[:pre], color, dim=True)
+    matched = text[pre:post]
+    after = _paint(text[post:], color, dim=True)
+    return f"{gutter}{before}{matched}{after}"
+
+
+def _render_match(
+    src_lines: list[str],
+    offsets: list[int],
+    match: CodeMatch,
+    width: int,
+    color: bool,
+) -> list[str]:
+    chunk = match.chunks[0]
+    s_off, e_off = chunk.start.char_offset, chunk.end.char_offset
+    s_line, e_line = chunk.start.line, chunk.end.line
+    out: list[str] = []
+    for line_no in range(s_line, e_line + 1):
+        idx = line_no - 1
+        text = src_lines[idx] if 0 <= idx < len(src_lines) else ""
+        line_start = offsets[idx] if 0 <= idx < len(offsets) else 0
+        dim_pre_end = (s_off - line_start) if line_no == s_line else 0
+        dim_post_start = (e_off - line_start) if line_no == e_line else len(text)
+        out.append(_render_code_line(line_no, text, dim_pre_end, dim_post_start, width, color))
+    return out
+
+
+def render_file(fm: FileMatches, *, color: bool) -> str:
+    """Render one file's matches: the path, then each match's line range, with
+    matches separated by a ``---`` line."""
+    # Split on "\n" to keep line numbers aligned with the offsets below (which
+    # count "\n"), then drop the trailing "\r" that CRLF files leave on each line.
+    src_lines = [line.rstrip("\r") for line in fm.source.split("\n")]
+    offsets = _line_char_offsets(fm.source)
+    max_line = max((m.chunks[0].end.line for m in fm.matches if m.chunks), default=1)
+    width = len(str(max_line))
+
+    parts = [_paint(fm.path, color, fg="magenta", bold=True)]
+    emitted = False
+    for match in fm.matches:
+        if not match.chunks:
+            continue
+        if emitted:
+            parts.append(_paint("---", color, fg="bright_black"))
+        parts.extend(_render_match(src_lines, offsets, match, width, color))
+        emitted = True
+    return "\n".join(parts)
+
+
+def render_results(results: list[FileMatches], *, color: bool) -> str:
+    """Render a list of per-file matches in the ``ccc grep`` output format, files
+    separated by a blank line. The CLI streams with :func:`render_file` instead;
+    this is the batch form (used in tests)."""
+    return "\n\n".join(render_file(fm, color=color) for fm in results)

cocoindex_code-0.2.37/src/cocoindex_code/indexer.py

@@ -0,0 +1,123 @@
+"""CocoIndex app for indexing codebases."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import cocoindex as coco
+from cocoindex.connectors import localfs, sqlite
+from cocoindex.connectors.sqlite import Vec0TableDef
+from cocoindex.ops.text import RecursiveSplitter, detect_code_language
+from cocoindex.resources.chunk import Chunk
+from cocoindex.resources.id import IdGenerator
+
+from .chunking import CHUNKER_REGISTRY
+from .file_walk import build_matcher
+from .settings import load_project_settings
+from .shared import (
+    CODEBASE_DIR,
+    EMBEDDER,
+    INDEXING_EMBED_PARAMS,
+    SQLITE_DB,
+    CodeChunk,
+)
+
+# Chunking configuration
+CHUNK_SIZE = 1000
+MIN_CHUNK_SIZE = 250
+CHUNK_OVERLAP = 150
+
+# Chunking splitter (stateless, can be module-level)
+splitter = RecursiveSplitter()
+
+
+@coco.fn(memo=True)
+async def process_file(
+    file: localfs.File,
+    table: sqlite.TableTarget[CodeChunk],
+) -> None:
+    """Process a single file: chunk, embed, and store."""
+    embedder = coco.use_context(EMBEDDER)
+    indexing_params = coco.use_context(INDEXING_EMBED_PARAMS)
+
+    try:
+        content = await file.read_text()
+    except UnicodeDecodeError:
+        return
+
+    if not content.strip():
+        return
+
+    suffix = file.file_path.path.suffix
+    project_root = coco.use_context(CODEBASE_DIR)
+    ps = load_project_settings(project_root)
+    ext_lang_map = {f".{lo.ext}": lo.lang for lo in ps.language_overrides}
+    language = (
+        ext_lang_map.get(suffix)
+        or detect_code_language(filename=file.file_path.path.name)
+        or "text"
+    )
+
+    chunker_registry = coco.use_context(CHUNKER_REGISTRY)
+    chunker = chunker_registry.get(suffix)
+    if chunker is not None:
+        language_override, chunks = chunker(Path(file.file_path.path), content)
+        if language_override is not None:
+            language = language_override
+    else:
+        chunks = splitter.split(
+            content,
+            chunk_size=CHUNK_SIZE,
+            min_chunk_size=MIN_CHUNK_SIZE,
+            chunk_overlap=CHUNK_OVERLAP,
+            language=language,
+        )
+
+    id_gen = IdGenerator()
+
+    async def process(chunk: Chunk) -> None:
+        table.declare_row(
+            row=CodeChunk(
+                id=await id_gen.next_id(chunk.text),
+                file_path=file.file_path.path.as_posix(),
+                language=language,
+                content=chunk.text,
+                start_line=chunk.start.line,
+                end_line=chunk.end.line,
+                embedding=await embedder.embed(chunk.text, **indexing_params),
+            )
+        )
+
+    await coco.map(process, chunks)
+
+
+@coco.fn
+async def indexer_main() -> None:
+    """Main indexing function - walks files and processes each."""
+    project_root = coco.use_context(CODEBASE_DIR)
+    ps = load_project_settings(project_root)
+
+    table = await sqlite.mount_table_target(
+        db=SQLITE_DB,
+        table_name="code_chunks_vec",
+        table_schema=await sqlite.TableSchema.from_class(
+            CodeChunk,
+            primary_key=["id"],
+        ),
+        virtual_table_def=Vec0TableDef(
+            partition_key_columns=["language"],
+            auxiliary_columns=["file_path", "content", "start_line", "end_line"],
+        ),
+    )
+
+    matcher = build_matcher(project_root, ps.include_patterns, ps.exclude_patterns)
+
+    files = localfs.walk_dir(
+        CODEBASE_DIR,
+        recursive=True,
+        path_matcher=matcher,
+    )
+
+    await coco.mount_each(
+        coco.component_subpath(coco.Symbol("process_file")), process_file, files.items(), table
+    )