agent-coderag 1.4.1__tar.gz → 1.5.0__tar.gz

{agent_coderag-1.4.1/agent_coderag.egg-info → agent_coderag-1.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agent-coderag
-Version: 1.4.1
+Version: 1.5.0
 Summary: Lightweight semantic code search and distillation utility for AI coding agents. It solves the API knowledge gap via real-time local signature extraction and intent analysis without PyTorch. Optimized for token efficiency, it compresses codebase context into compact semantic summaries stored in a local DuckDB vector similarity index.
 Author-email: Igor Boloban <naranor@gmail.com>
 License: MIT
@@ -190,6 +190,7 @@ async def main():
 - **Sidecars:** DuckDB may write WAL sidecars (e.g. `.coderag.db.wal`) beside the index during writes; locks should not persist after an operation finishes.
 - **Connect timeout:** `connect_timeout_seconds=5` (CLI `--connect-timeout`) waits on file locks, then raises `StorageBusyError` (`ErrorCode.STORAGE_BUSY`). Pass `0` for a single attempt.
 - **Read-only search:** `search` (and `api` when storage is needed) opens read-only. A missing **index file** is an error — use `sync`/`rebuild` to create it. An index file that exists but has no embeddings table (e.g. opened/written without a completed vector sync) raises `StorageError` with `ErrorCode.EMBEDDINGS_MISSING` — run `sync` (library: `CodeRAG.sync`) before search. With `--json`, success is a hit array; errors are `{"status":"error","message":...}` and include `"code"` when the exception carries an `ErrorCode`.
+- **Paths:** `sync` stores paths relative to `root` (`src/a.py`). The first sync rewrites an older absolute index when the file is still under `root` or its unit hashes match a file in the tree. Search returns an absolute path under the current root. `search --relative-paths`, config `relative_paths: true`, or `CodeRAG(relative_paths=True)` returns the stored relative path. Until that sync runs, search returns the absolute path stored in the index. Ignore rules use the same project-relative path, so a `.worktrees/<name>` checkout is indexed when that directory is the root.
 - **Lifetime:** embedder/parser/distiller stay warm; DuckDB opens per operation and closes afterward. One `CodeRAG` instance serializes overlapping ops. `config()` with embedding flags / `--clear-embedding` closes the process embedder so the next op rebuilds it; distill-only `config` refreshes Distiller and keeps the embedder.
 - **`api()` without DB:** providers that do not need the index (e.g. Python) skip DuckDB entirely; Java uses a short read-only open for JAR cache lookup.
 - **Errors:** catch `CodeRAGError` and inspect `.code` — `STORAGE_BUSY`, `STORAGE_CORRUPT`, `EMBEDDING_MISMATCH`, `EMBEDDINGS_MISSING` (`from code_rag import ErrorCode`).

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/README.md

@@ -144,6 +144,7 @@ async def main():
 - **Sidecars:** DuckDB may write WAL sidecars (e.g. `.coderag.db.wal`) beside the index during writes; locks should not persist after an operation finishes.
 - **Connect timeout:** `connect_timeout_seconds=5` (CLI `--connect-timeout`) waits on file locks, then raises `StorageBusyError` (`ErrorCode.STORAGE_BUSY`). Pass `0` for a single attempt.
 - **Read-only search:** `search` (and `api` when storage is needed) opens read-only. A missing **index file** is an error — use `sync`/`rebuild` to create it. An index file that exists but has no embeddings table (e.g. opened/written without a completed vector sync) raises `StorageError` with `ErrorCode.EMBEDDINGS_MISSING` — run `sync` (library: `CodeRAG.sync`) before search. With `--json`, success is a hit array; errors are `{"status":"error","message":...}` and include `"code"` when the exception carries an `ErrorCode`.
+- **Paths:** `sync` stores paths relative to `root` (`src/a.py`). The first sync rewrites an older absolute index when the file is still under `root` or its unit hashes match a file in the tree. Search returns an absolute path under the current root. `search --relative-paths`, config `relative_paths: true`, or `CodeRAG(relative_paths=True)` returns the stored relative path. Until that sync runs, search returns the absolute path stored in the index. Ignore rules use the same project-relative path, so a `.worktrees/<name>` checkout is indexed when that directory is the root.
 - **Lifetime:** embedder/parser/distiller stay warm; DuckDB opens per operation and closes afterward. One `CodeRAG` instance serializes overlapping ops. `config()` with embedding flags / `--clear-embedding` closes the process embedder so the next op rebuilds it; distill-only `config` refreshes Distiller and keeps the embedder.
 - **`api()` without DB:** providers that do not need the index (e.g. Python) skip DuckDB entirely; Java uses a short read-only open for JAR cache lookup.
 - **Errors:** catch `CodeRAGError` and inspect `.code` — `STORAGE_BUSY`, `STORAGE_CORRUPT`, `EMBEDDING_MISMATCH`, `EMBEDDINGS_MISSING` (`from code_rag import ErrorCode`).

{agent_coderag-1.4.1 → agent_coderag-1.5.0/agent_coderag.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agent-coderag
-Version: 1.4.1
+Version: 1.5.0
 Summary: Lightweight semantic code search and distillation utility for AI coding agents. It solves the API knowledge gap via real-time local signature extraction and intent analysis without PyTorch. Optimized for token efficiency, it compresses codebase context into compact semantic summaries stored in a local DuckDB vector similarity index.
 Author-email: Igor Boloban <naranor@gmail.com>
 License: MIT
@@ -190,6 +190,7 @@ async def main():
 - **Sidecars:** DuckDB may write WAL sidecars (e.g. `.coderag.db.wal`) beside the index during writes; locks should not persist after an operation finishes.
 - **Connect timeout:** `connect_timeout_seconds=5` (CLI `--connect-timeout`) waits on file locks, then raises `StorageBusyError` (`ErrorCode.STORAGE_BUSY`). Pass `0` for a single attempt.
 - **Read-only search:** `search` (and `api` when storage is needed) opens read-only. A missing **index file** is an error — use `sync`/`rebuild` to create it. An index file that exists but has no embeddings table (e.g. opened/written without a completed vector sync) raises `StorageError` with `ErrorCode.EMBEDDINGS_MISSING` — run `sync` (library: `CodeRAG.sync`) before search. With `--json`, success is a hit array; errors are `{"status":"error","message":...}` and include `"code"` when the exception carries an `ErrorCode`.
+- **Paths:** `sync` stores paths relative to `root` (`src/a.py`). The first sync rewrites an older absolute index when the file is still under `root` or its unit hashes match a file in the tree. Search returns an absolute path under the current root. `search --relative-paths`, config `relative_paths: true`, or `CodeRAG(relative_paths=True)` returns the stored relative path. Until that sync runs, search returns the absolute path stored in the index. Ignore rules use the same project-relative path, so a `.worktrees/<name>` checkout is indexed when that directory is the root.
 - **Lifetime:** embedder/parser/distiller stay warm; DuckDB opens per operation and closes afterward. One `CodeRAG` instance serializes overlapping ops. `config()` with embedding flags / `--clear-embedding` closes the process embedder so the next op rebuilds it; distill-only `config` refreshes Distiller and keeps the embedder.
 - **`api()` without DB:** providers that do not need the index (e.g. Python) skip DuckDB entirely; Java uses a short read-only open for JAR cache lookup.
 - **Errors:** catch `CodeRAGError` and inspect `.code` — `STORAGE_BUSY`, `STORAGE_CORRUPT`, `EMBEDDING_MISMATCH`, `EMBEDDINGS_MISSING` (`from code_rag import ErrorCode`).

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/agent_coderag.egg-info/SOURCES.txt

@@ -50,6 +50,7 @@ code_rag/services/dependencies.py
 code_rag/services/discovery_api.py
 code_rag/services/factory.py
 code_rag/services/indexing.py
+code_rag/services/path_migration.py
 code_rag/services/search.py
 code_rag/services/setup.py
 code_rag/services/sync.py
@@ -84,6 +85,7 @@ tests/test_embedder_interface.py
 tests/test_embedder_stubs.py
 tests/test_error_codes.py
 tests/test_factory_async.py
+tests/test_gitignore_root.py
 tests/test_indexing_detailed.py
 tests/test_indexing_embeddings.py
 tests/test_indexing_extra.py
@@ -91,13 +93,18 @@ tests/test_interfaces.py
 tests/test_js_discovery_detailed.py
 tests/test_languages.py
 tests/test_local_onnx_embedder.py
+tests/test_migration_cli_e2e.py
+tests/test_migration_coderag.py
 tests/test_models.py
 tests/test_multi_parser.py
 tests/test_openai_embedder.py
+tests/test_path_migration.py
 tests/test_readme_library_usage.py
 tests/test_readme_lifetime_storage.py
 tests/test_readme_offline_embeddings.py
+tests/test_relative_sync_paths.py
 tests/test_rust_discovery_detailed.py
+tests/test_search_relative_paths.py
 tests/test_storage_detailed.py
 tests/test_storage_embeddings.py
 tests/test_sync_worker_errors.py

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/api/client.py

@@ -19,7 +19,7 @@ from code_rag.services.config import (
 )
 from code_rag.services.discovery_api import run_api
 from code_rag.services.factory import create_stack
-from code_rag.services.search import run_search
+from code_rag.services.search import format_unit_path, run_search
 from code_rag.services.setup import run_setup
 from code_rag.services.indexing import IndexStack
 from code_rag.services.sync import SyncOptions, run_rebuild, run_sync
@@ -40,12 +40,17 @@ class CodeRAG:  # pylint: disable=too-many-instance-attributes
         root: Optional[Union[str, Path]] = None,
         allow_build_execution: bool = False,
         connect_timeout_seconds: float = DEFAULT_CONNECT_TIMEOUT_SECONDS,
+        relative_paths: bool | None = None,
     ):
         self._onnx = onnx
         self._root = Path(root) if root is not None else Path.cwd()
         self._allow_build_execution = allow_build_execution
         self._connect_timeout_seconds = connect_timeout_seconds
         self._db_path = resolve_db_path(db, root=self._root)
+        if relative_paths is None:
+            self._relative_paths = bool(DistillerConfig.load().relative_paths)
+        else:
+            self._relative_paths = relative_paths
         self._embedder: Optional[IEmbedder] = None
         self._parser: Optional[IParser] = None
         self._distiller: Optional[IIntelligence] = None
@@ -199,7 +204,12 @@ class CodeRAG:  # pylint: disable=too-many-instance-attributes
 
     async def search(self, query: str, *, limit: int = 5) -> list[KnowledgeUnit]:
         async with self._with_storage(AccessMode.READ_ONLY) as storage:
-            return await run_search(storage, query, limit=limit)
+            units = await run_search(storage, query, limit=limit)
+        for unit in units:
+            unit.path = format_unit_path(
+                unit.path, root=self._root, relative_paths=self._relative_paths
+            )
+        return units
 
     async def _api_java(
         self,

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/core/interfaces.py

@@ -7,7 +7,13 @@ class IParser(ABC):
     """Interface for extracting structure from code."""
 
     @abstractmethod
-    async def distill_file(self, file_path: str) -> List[KnowledgeUnit]:
+    async def distill_file(
+        self,
+        file_path: str,
+        *,
+        stored_path: str | None = None,
+        raise_on_failure: bool = False,
+    ) -> List[KnowledgeUnit]:
         pass  # pragma: no cover
 
 

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/entry/args.py

@@ -50,6 +50,13 @@ def build_parser() -> argparse.ArgumentParser:
     search = subparsers.add_parser("search", help="Semantic search.")
     search.add_argument("query", help="Natural language query.")
     search.add_argument("--limit", type=int, default=5, help="Result limit.")
+    search.add_argument(
+        "--relative-paths",
+        dest="relative_paths",
+        action="store_true",
+        default=None,
+        help="Print paths relative to the project root. Default: absolute paths under the current root.",
+    )
 
     api = subparsers.add_parser("api", help="Discover library API.")
     api.add_argument("library", help="Library name (e.g., pydantic).")

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/entry/cli.py

@@ -46,6 +46,7 @@ __all__ = [
 
 
 def _coderag_from_args(args) -> CodeRAG:
+    relative_paths = getattr(args, "relative_paths", None)
     return CodeRAG(
         db=args.db,
         onnx=getattr(args, "onnx", None),
@@ -54,6 +55,7 @@ def _coderag_from_args(args) -> CodeRAG:
             getattr(args, "connect_timeout", DEFAULT_CONNECT_TIMEOUT_SECONDS)
         ),
         allow_build_execution=bool(getattr(args, "allow_build_execution", False)),
+        relative_paths=True if relative_paths else None,
     )
 
 
@@ -70,9 +72,14 @@ def load_ignore_patterns() -> pathspec.PathSpec:
     return sync_service.load_ignore_patterns(Path.cwd())
 
 
-def should_index(path: Path, ignore_spec: Optional[pathspec.PathSpec] = None) -> bool:
+def should_index(
+    path: Path,
+    ignore_spec: Optional[pathspec.PathSpec] = None,
+    *,
+    root: Optional[Path] = None,
+) -> bool:
     """Filters files that should NOT be indexed."""
-    return sync_service.should_index(path, ignore_spec)
+    return sync_service.should_index(path, ignore_spec, root=root)
 
 
 def _emit_sync_outcome(result: SyncResult, *, json_mode: bool, label: str) -> None:

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/intelligence/distiller.py

@@ -23,6 +23,7 @@ class DistillerConfig(BaseModel):
     embedding_key: Optional[str] = None
     embedding_model: Optional[str] = None
     embedding_provider: Optional[str] = None
+    relative_paths: bool = False
 
     @field_validator(
         "model",

agent_coderag-1.5.0/code_rag/parsers/multi_parser.py

@@ -0,0 +1,46 @@
+import os
+import logging
+from typing import List
+from ..core.interfaces import IParser
+from ..core.models import KnowledgeUnit
+from .tree_sitter import TreeSitterParser
+from .languages import EXTENSION_TO_LANGUAGE
+
+logger = logging.getLogger(__name__)
+
+
+class MultiParser(IParser):
+    """
+    Delegates parsing to TreeSitterParser for all supported languages.
+    """
+
+    def __init__(self) -> None:
+        self.tree_sitter_parser = TreeSitterParser()
+
+    async def distill_file(
+        self,
+        file_path: str,
+        *,
+        stored_path: str | None = None,
+        raise_on_failure: bool = False,
+    ) -> List[KnowledgeUnit]:
+        _, ext = os.path.splitext(file_path)
+        ext = ext.lower()
+
+        if ext in EXTENSION_TO_LANGUAGE:
+            if stored_path is None and not raise_on_failure:
+                return await self.tree_sitter_parser.distill_file(file_path)
+            if stored_path is not None and raise_on_failure:
+                return await self.tree_sitter_parser.distill_file(
+                    file_path, stored_path=stored_path, raise_on_failure=True
+                )
+            if stored_path is not None:
+                return await self.tree_sitter_parser.distill_file(
+                    file_path, stored_path=stored_path
+                )
+            return await self.tree_sitter_parser.distill_file(
+                file_path, raise_on_failure=True
+            )
+
+        logger.debug("No parser for extension %s", ext)
+        return []

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/parsers/tree_sitter.py

@@ -45,10 +45,14 @@ class TreeSitterParser(IParser):
                 f"[MISSING DEPENDENCY] Please install: pip install {lang_config.package}"
             ) from e
 
-    async def distill_file(self, file_path: str) -> List[KnowledgeUnit]:
-        """
-        Parses a file and extracts high-level units (classes, functions).
-        """
+    async def distill_file(
+        self,
+        file_path: str,
+        *,
+        stored_path: str | None = None,
+        raise_on_failure: bool = False,
+    ) -> List[KnowledgeUnit]:
+        identity = file_path if stored_path is None else stored_path
         ext = Path(file_path).suffix.lower()
         lang_config = next((cfg for cfg in LANGUAGES if ext in cfg.extensions), None)
 
@@ -67,17 +71,18 @@ class TreeSitterParser(IParser):
             ctx: Dict[str, Any] = {
                 "source": source,
                 "config": lang_config,
-                "file_path": file_path,
+                "file_path": identity,
                 "units": units,
                 "scope": None,
             }
             self._recursive_distill(tree.root_node, ctx)
             return units
         except GrammarNotFoundError:
-            # Re-raise grammar errors so callers can handle them (e.g. CLI setup suggestion)
             raise
         except Exception as e:
             logger.error("Failed to parse %s: %s", file_path, e)
+            if raise_on_failure:
+                raise
             return []
 
     def _recursive_distill(self, node: Node, ctx: Dict[str, Any]) -> None:

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/paths.py

@@ -13,6 +13,11 @@ def default_db_path(root: Path | None = None) -> Path:
     return root_path / ".coderag.db"
 
 
+def project_relative_posix(path: Path, root: Path) -> str:
+    """Posix path of ``path`` relative to ``root``. Raises ValueError when outside ``root``."""
+    return path.resolve().relative_to(root.resolve()).as_posix()
+
+
 def resolve_db_path(db: str | Path | None, *, root: Path | None = None) -> Path:
     if db is not None:
         path = Path(db)

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/services/indexing.py

@@ -7,12 +7,14 @@ import inspect
 import logging
 import sys
 from dataclasses import dataclass
+from pathlib import Path
 from typing import List, Optional
 
 from code_rag.core.constants import EMBEDDING_BATCH_SIZE, MAX_CONCURRENT_TASKS
 from code_rag.core.exceptions import IntelligenceError, StorageError
 from code_rag.core.interfaces import IIntelligence, IParser, IStorage
 from code_rag.core.models import KnowledgeUnit
+from code_rag.paths import project_relative_posix
 
 logger = logging.getLogger(__name__)
 
@@ -24,6 +26,7 @@ class IndexStack:
     storage: IStorage
     parser: IParser
     intelligence: IIntelligence
+    root: Optional[Path] = None
 
 
 def _report_worker_failure(path: str, exc: BaseException) -> tuple[str, str]:
@@ -146,11 +149,19 @@ async def sync_file(
     *,
     force_distill: bool = False,
     max_concurrency: int = MAX_CONCURRENT_TASKS,
+    stored_path: str | None = None,
 ) -> None:
     """Parse, distill, embed, and upsert one file into ``stack.storage``."""
     await _reject_dirty_incremental(stack.storage)
     semaphore = asyncio.Semaphore(max_concurrency)
-    current_units = await stack.parser.distill_file(file_path)
+    if stored_path is None:
+        current_units = await stack.parser.distill_file(file_path)
+        identity = file_path
+    else:
+        current_units = await stack.parser.distill_file(
+            file_path, stored_path=stored_path
+        )
+        identity = stored_path
     pending: list[KnowledgeUnit] = []
     for unit in current_units:
         await _process_unit(
@@ -162,7 +173,7 @@ async def sync_file(
         )
     await _embed_and_upsert(stack.storage, pending)
     await stack.storage.delete_stale_units(
-        file_path, [unit.id for unit in current_units]
+        identity, [unit.id for unit in current_units]
     )
 
 
@@ -194,11 +205,17 @@ async def sync_project(
             except asyncio.QueueEmpty:
                 return failures
             try:
+                stored_path = (
+                    None
+                    if stack.root is None
+                    else project_relative_posix(Path(path), stack.root)
+                )
                 await sync_file(
                     stack,
                     path,
                     force_distill=force_distill,
                     max_concurrency=max_concurrency,
+                    stored_path=stored_path,
                 )
             except Exception as exc:
                 failures.append(_report_worker_failure(path, exc))

agent_coderag-1.5.0/code_rag/services/path_migration.py

@@ -0,0 +1,121 @@
+"""One-time rewrite of absolute index paths onto project-relative posix paths."""
+
+from __future__ import annotations
+
+import logging
+from collections import Counter
+from pathlib import Path
+
+from code_rag.core.models import KnowledgeUnit
+from code_rag.parsers.tree_sitter import GrammarNotFoundError
+from code_rag.paths import project_relative_posix
+from code_rag.storage.duckdb_impl import DuckDBStorage
+
+logger = logging.getLogger(__name__)
+
+
+def _groups(units: list[KnowledgeUnit]) -> dict[str, Counter[str]]:
+    groups: dict[str, Counter[str]] = {}
+    for unit in units:
+        if not Path(unit.path).is_absolute():
+            continue
+        groups.setdefault(unit.path, Counter())[unit.code_hash] += 1
+    return groups
+
+
+def claim_relative_paths(
+    groups: dict[str, Counter[str]],
+    root: Path,
+    files: list[tuple[Path, Counter[str]]],
+) -> dict[str, str]:
+    mapping: dict[str, str] = {}
+    taken: set[str] = set()
+    pending: list[tuple[str, Counter[str]]] = []
+    for old, hashes in groups.items():
+        path = Path(old)
+        if path.is_absolute():
+            try:
+                rel = project_relative_posix(path, root)
+            except ValueError:
+                pending.append((old, hashes))
+                continue
+            if rel in taken:
+                continue
+            mapping[old] = rel
+            taken.add(rel)
+            continue
+        pending.append((old, hashes))
+
+    claimed: set[str] = set()
+    for file_path, hashes in files:
+        try:
+            rel = project_relative_posix(file_path, root)
+        except ValueError:
+            continue
+        if rel in taken:
+            continue
+        for old, group_hashes in pending:
+            if old in claimed or old in mapping:
+                continue
+            if group_hashes != hashes:
+                continue
+            mapping[old] = rel
+            taken.add(rel)
+            claimed.add(old)
+            break
+    return mapping
+
+
+def _suffix(path: str) -> str:
+    return Path(path).suffix.lower()
+
+
+async def migrate_absolute_paths(
+    storage, parser, root: Path, files: list[Path]
+) -> bool:
+    if not isinstance(storage, DuckDBStorage):
+        return False
+    if await storage.paths_migration_done():
+        return False
+    groups = _groups(await storage.list_units())
+    if not groups:
+        await storage.commit_path_migration({})
+        return False
+    incomplete = False
+    skipped_suffixes: set[str] = set()
+    parsed: list[tuple[Path, Counter[str]]] = []
+    for path in files:
+        try:
+            units = await parser.distill_file(str(path), raise_on_failure=True)
+        except GrammarNotFoundError:
+            logger.warning("Skipping unparsed file during path migration: %s", path)
+            skipped_suffixes.add(_suffix(str(path)))
+            incomplete = True
+            continue
+        except Exception:
+            logger.exception("Path migration could not parse %s", path)
+            skipped_suffixes.add(_suffix(str(path)))
+            incomplete = True
+            continue
+        parsed.append((path, Counter(unit.code_hash for unit in units)))
+    mapping = claim_relative_paths(groups, root, parsed)
+    failed_commits: set[str] = set()
+    for old, new in mapping.items():
+        try:
+            await storage.commit_rewritten_path(old, new)
+        except Exception:
+            logger.exception("Path migration failed for %s", old)
+            failed_commits.add(old)
+            incomplete = True
+    orphans = [
+        old
+        for old in groups
+        if old not in mapping
+        and old not in failed_commits
+        and _suffix(old) not in skipped_suffixes
+    ]
+    if orphans:
+        await storage.delete_absolute_paths(orphans)
+    if not incomplete:
+        await storage.mark_paths_migrated()
+    return True

agent_coderag-1.5.0/code_rag/services/search.py

@@ -0,0 +1,20 @@
+from pathlib import Path
+
+from code_rag.core.interfaces import IStorage
+from code_rag.core.models import KnowledgeUnit
+
+
+def format_unit_path(stored: str, *, root: Path, relative_paths: bool) -> str:
+    """Format a stored index path for search output. Does not read or write storage."""
+    if Path(stored).is_absolute():
+        return stored
+    if relative_paths:
+        return stored
+    return str((root / stored).resolve())
+
+
+async def run_search(
+    storage: IStorage, query: str, *, limit: int = 5
+) -> list[KnowledgeUnit]:
+    """Performs semantic search across indexed units."""
+    return await storage.search_units(query, limit=limit)

{agent_coderag-1.4.1 → agent_coderag-1.5.0}/code_rag/services/sync.py

@@ -1,5 +1,6 @@
 import logging
 import os
+import sys
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Optional
@@ -10,9 +11,12 @@ from code_rag.api.models import SyncFileError, SyncResult
 from code_rag.core.constants import MAX_CONCURRENT_TASKS
 from code_rag.core.exceptions import DiscoveryError
 from code_rag.core.utils import validate_path
+from code_rag.paths import project_relative_posix
 from code_rag.parsers.languages import EXTENSION_TO_LANGUAGE
 from code_rag.services.dependencies import sync_dependencies
 from code_rag.services.indexing import IndexStack, sync_project
+from code_rag.services.path_migration import migrate_absolute_paths
+from code_rag.storage.duckdb_impl import DuckDBStorage
 
 logger = logging.getLogger(__name__)
 
@@ -52,11 +56,25 @@ def load_ignore_patterns(root: Path) -> pathspec.PathSpec:
     return pathspec.PathSpec.from_lines("gitignore", lines + common_excludes)
 
 
-def should_index(path: Path, ignore_spec: Optional[pathspec.PathSpec] = None) -> bool:
-    """Filters files that should NOT be indexed."""
-    path_str = str(path).replace(os.sep, "/")
-
-    if ignore_spec and ignore_spec.match_file(path_str):
+def should_index(
+    path: Path,
+    ignore_spec: Optional[pathspec.PathSpec] = None,
+    *,
+    root: Optional[Path] = None,
+) -> bool:
+    """Filters files that should NOT be indexed.
+
+    When ``root`` is set, gitignore sees only the posix path relative to ``root``.
+    """
+    if root is not None:
+        try:
+            match_target = project_relative_posix(path, root)
+        except ValueError:
+            return False
+    else:
+        match_target = str(path).replace(os.sep, "/")
+
+    if ignore_spec and ignore_spec.match_file(match_target):
         return False
 
     return path.suffix.lower() in EXTENSION_TO_LANGUAGE
@@ -70,9 +88,16 @@ def _result_from_failures(
     return SyncResult(status=status, indexed_files=indexed_files, errors=errors)
 
 
-def _indexable_under(root: Path, ignore_spec: pathspec.PathSpec) -> list[str]:
+def _indexable_under(
+    walk_root: Path,
+    ignore_spec: pathspec.PathSpec,
+    *,
+    project_root: Path,
+) -> list[str]:
     return [
-        str(p) for p in root.rglob("*") if p.is_file() and should_index(p, ignore_spec)
+        str(p)
+        for p in walk_root.rglob("*")
+        if p.is_file() and should_index(p, ignore_spec, root=project_root)
     ]
 
 
@@ -81,8 +106,14 @@ async def _run_project_sync(
     paths: list[str],
     options: SyncOptions,
 ) -> list[tuple[str, str]]:
+    rooted = IndexStack(
+        stack.storage,
+        stack.parser,
+        stack.intelligence,
+        root=options.root,
+    )
     return await sync_project(
-        stack,
+        rooted,
         paths,
         force_distill=options.force,
         index_all=options.index_all,
@@ -98,12 +129,12 @@ async def _sync_validated_path(
 ) -> SyncResult:
     target_path = Path(validated_path)
     if target_path.is_file():
-        if not should_index(target_path, ignore_spec):
+        if not should_index(target_path, ignore_spec, root=options.root):
             return _result_from_failures(0, [])
         failures = await _run_project_sync(stack, [str(target_path)], options)
         return _result_from_failures(1, failures)
 
-    paths = _indexable_under(target_path, ignore_spec)
+    paths = _indexable_under(target_path, ignore_spec, project_root=options.root)
     if not paths and not options.index_all:
         return _result_from_failures(len(paths), [])
     failures = await _run_project_sync(stack, paths, options)
@@ -129,11 +160,34 @@ async def run_sync(stack: IndexStack, options: SyncOptions) -> SyncResult:
         logger.warning("Dependency discovery failed: %s", de)
 
     ignore_spec = load_ignore_patterns(options.root)
+    migration_ran = False
+    if (
+        isinstance(stack.storage, DuckDBStorage)
+        and not await stack.storage.paths_migration_done()
+    ):
+        candidates = [
+            Path(item)
+            for item in _indexable_under(
+                options.root, ignore_spec, project_root=options.root
+            )
+        ]
+        try:
+            migration_ran = await migrate_absolute_paths(
+                stack.storage, stack.parser, options.root, candidates
+            )
+        except Exception:
+            logger.exception("Path migration failed; continuing sync")
+            migration_ran = True
+    if migration_ran and not options.index_all:
+        print(
+            "Run sync --all to reindex the project after path migration.",
+            file=sys.stderr,
+        )
 
     if validated_path:
         return await _sync_validated_path(stack, validated_path, options, ignore_spec)
 
-    paths = _indexable_under(options.root, ignore_spec)
+    paths = _indexable_under(options.root, ignore_spec, project_root=options.root)
     failures = await _run_project_sync(stack, paths, options)
     return _result_from_failures(len(paths), failures)