ragx-cli 0.10.0__tar.gz → 0.11.0__tar.gz

{ragx_cli-0.10.0 → ragx_cli-0.11.0}/CHANGELOG.md

@@ -4,6 +4,18 @@ All notable changes to `ragx-cli` are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow
 [Semantic Versioning](https://semver.org/).
 
+## [0.11.0] — 2026-10-28
+
+### Changed
+
+- Discovery now honors every `.gitignore` in the corpus tree, not only the root one. Git
+  precedence applies: the deepest `.gitignore` with a matching pattern decides, so a subdirectory
+  can re-include (`!pattern`) what the root ignores; anchored patterns (`/foo.md`) resolve
+  relative to the directory holding the `.gitignore`; ignored directories are pruned, so a
+  negation can't re-include a file under them. Patterns are parsed with
+  `pathspec.GitIgnoreSpec`. An incremental `ragx-cli index` drops already-indexed files that a
+  nested `.gitignore` now excludes.
+
 ## [0.10.0] — 2026-10-21
 
 ### Added

{ragx_cli-0.10.0 → ragx_cli-0.11.0}/CLAUDE.md

@@ -55,7 +55,7 @@ src/ragx/
     models.py      #   Chunk, ChunkDraft, FileRecord, Edge, ScoredChunk, QueryOutput
     store.py       #   SQLite: files/chunks/edges/meta; replace_edges normalizes src<dst
     vectors.py     #   hnswlib wrapper; soft-delete via JSON sidecar; get_vectors for graph
-    discovery.py   #   file walk, root-level .gitignore, binary/size/junk-dir filters, xxh64
+    discovery.py   #   file walk, nested .gitignore (git precedence), binary/size/junk-dir filters, xxh64
     chunking.py    #   markdown/code/recursive splitters; byte-exact slices; tiny-fragment merge;
                    #   subchunk_texts (sentence-aligned windows for subchunk edge mode)
     graph.py       #   knn_edges + subchunk_knn_edges (max over sub-chunk pairs, near-dup guard)
@@ -90,13 +90,13 @@ src/ragx/
                    # handlers.py (routing), query_runner.py (lazy providers, lock-serialized
                    # queries), static_files.py (path-traversal-safe static serving);
                    # static/ (frontend-owned: index.html, JS/CSS, vendored sigma.js)
-tests/             # 301 tests; mocked HTTP (respx), FakeEmbedder integration tests, no live network
+tests/             # 305 tests; mocked HTTP (respx), FakeEmbedder integration tests, no live network
 ```
 
 ## Flows
 
 **Index** (`ragx-cli index`): discover files (include/exclude globs — defaults in
-`core/corpus_globs.py`: source + text docs, minus lockfiles/requirements/build output — root .gitignore,
+`core/corpus_globs.py`: source + text docs, minus lockfiles/requirements/build output — every .gitignore in the tree (deeper overrides shallower, ignored dirs pruned),
 skip node_modules/.venv/hidden/binaries/>2MB) → xxhash diff (incremental by default since 0.8.0;
 `--full` rebuilds everything; `--changed` is a deprecated no-op alias for the default) → chunk
 (~800 tok target, chunks are exact byte slices, sub-100-char fragments
@@ -205,7 +205,7 @@ mostly on Dutch/multilingual queries. Details: `research/bge-m3-dense-q8-vs-nomi
 
 ## Dev loop
 
-`uv sync --group dev --extra rerank` · `uv run pytest -q` (300 pass + 1 skipped, ~9 s) ·
+`uv sync --group dev --extra rerank` · `uv run pytest -q` (304 pass + 1 skipped, ~13 s) ·
 `uv run ruff check src tests` · file soft cap ~150 lines · expected failures raise `RagxError`
 (CLI maps to exit 2). Live smoke: LM Studio must be running with the configured embedding model.
 Changes that impact usage (CLI flags, config keys/precedence, output schemas, install steps)

{ragx_cli-0.10.0 → ragx_cli-0.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ragx-cli
-Version: 0.10.0
+Version: 0.11.0
 Summary: Similarity-graph RAG CLI: index a corpus into a chunk-level embedding graph and query it
 Project-URL: Repository, https://github.com/alber70g/ragx-cli
 Project-URL: Issues, https://github.com/alber70g/ragx-cli/issues
@@ -706,7 +706,7 @@ Key defaults:
 
 | section | defaults |
 |---|---|
-| `[corpus]` | `include` = source code (`*.py`, `*.ts`, `*.go`, `*.rs`, … ~70 extensions, plus `Dockerfile`/`Makefile`) and text docs (`*.md`, `*.mdx`, `*.rst`, `*.txt`, `*.adoc`, `*.org`); `exclude` = lockfiles (`*.lock`, `package-lock.json`, `pnpm-lock.yaml`, `go.sum`, …), `requirements*.txt`/`constraints*.txt`, `LICENSE*`/`COPYING*`, `*.min.js`/`*.min.css`/`*.map`, and `dist/`, `build/`, `target/`, `coverage/`, `vendor/`; `respect_gitignore=true`. Full lists: `src/ragx/core/corpus_globs.py`. Existing `ragx.toml` files keep their own lists |
+| `[corpus]` | `include` = source code (`*.py`, `*.ts`, `*.go`, `*.rs`, … ~70 extensions, plus `Dockerfile`/`Makefile`) and text docs (`*.md`, `*.mdx`, `*.rst`, `*.txt`, `*.adoc`, `*.org`); `exclude` = lockfiles (`*.lock`, `package-lock.json`, `pnpm-lock.yaml`, `go.sum`, …), `requirements*.txt`/`constraints*.txt`, `LICENSE*`/`COPYING*`, `*.min.js`/`*.min.css`/`*.map`, and `dist/`, `build/`, `target/`, `coverage/`, `vendor/`; `respect_gitignore=true` (every `.gitignore` in the tree applies to its own subtree, with git precedence: deeper files override shallower ones, ignored directories are skipped whole). Full lists: `src/ragx/core/corpus_globs.py`. Existing `ragx.toml` files keep their own lists |
 | `[chunking]` | `size_tokens=800`, `overlap=0.15` |
 | `[graph]` | `k=8`, `min_edge_sim=0.55`, `edge_source="subchunk"`, `subchunk_size_tokens=256`, `near_dup_sim=0.9` |
 | `[traversal]` | `hops=2`, `decay=0.5`, `query_floor=0.35`, `max_frontier=150` |

{ragx_cli-0.10.0 → ragx_cli-0.11.0}/README.md

@@ -678,7 +678,7 @@ Key defaults:
 
 | section | defaults |
 |---|---|
-| `[corpus]` | `include` = source code (`*.py`, `*.ts`, `*.go`, `*.rs`, … ~70 extensions, plus `Dockerfile`/`Makefile`) and text docs (`*.md`, `*.mdx`, `*.rst`, `*.txt`, `*.adoc`, `*.org`); `exclude` = lockfiles (`*.lock`, `package-lock.json`, `pnpm-lock.yaml`, `go.sum`, …), `requirements*.txt`/`constraints*.txt`, `LICENSE*`/`COPYING*`, `*.min.js`/`*.min.css`/`*.map`, and `dist/`, `build/`, `target/`, `coverage/`, `vendor/`; `respect_gitignore=true`. Full lists: `src/ragx/core/corpus_globs.py`. Existing `ragx.toml` files keep their own lists |
+| `[corpus]` | `include` = source code (`*.py`, `*.ts`, `*.go`, `*.rs`, … ~70 extensions, plus `Dockerfile`/`Makefile`) and text docs (`*.md`, `*.mdx`, `*.rst`, `*.txt`, `*.adoc`, `*.org`); `exclude` = lockfiles (`*.lock`, `package-lock.json`, `pnpm-lock.yaml`, `go.sum`, …), `requirements*.txt`/`constraints*.txt`, `LICENSE*`/`COPYING*`, `*.min.js`/`*.min.css`/`*.map`, and `dist/`, `build/`, `target/`, `coverage/`, `vendor/`; `respect_gitignore=true` (every `.gitignore` in the tree applies to its own subtree, with git precedence: deeper files override shallower ones, ignored directories are skipped whole). Full lists: `src/ragx/core/corpus_globs.py`. Existing `ragx.toml` files keep their own lists |
 | `[chunking]` | `size_tokens=800`, `overlap=0.15` |
 | `[graph]` | `k=8`, `min_edge_sim=0.55`, `edge_source="subchunk"`, `subchunk_size_tokens=256`, `near_dup_sim=0.9` |
 | `[traversal]` | `hops=2`, `decay=0.5`, `query_floor=0.35`, `max_frontier=150` |

{ragx_cli-0.10.0 → ragx_cli-0.11.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ragx-cli"
-version = "0.10.0"
+version = "0.11.0"
 description = "Similarity-graph RAG CLI: index a corpus into a chunk-level embedding graph and query it"
 readme = "README.md"
 license = "MIT"

{ragx_cli-0.10.0 → ragx_cli-0.11.0}/src/ragx/core/discovery.py

@@ -20,12 +20,29 @@ def _is_binary(path: Path) -> bool:
     return b"\x00" in chunk
 
 
-def _load_gitignore_spec(root: Path) -> pathspec.PathSpec | None:
-    gitignore = root / ".gitignore"
+def _load_gitignore_spec(directory: Path) -> pathspec.GitIgnoreSpec | None:
+    gitignore = directory / ".gitignore"
     if not gitignore.is_file():
         return None
     lines = gitignore.read_text(encoding="utf-8", errors="ignore").splitlines()
-    return pathspec.PathSpec.from_lines("gitwildmatch", lines)
+    return pathspec.GitIgnoreSpec.from_lines(lines)
+
+
+def _is_gitignored(rel: str, specs: dict[str, pathspec.GitIgnoreSpec]) -> bool:
+    """Git precedence: the deepest .gitignore with a matching pattern decides.
+
+    `specs` maps a directory's root-relative POSIX path ("" = root) to its .gitignore.
+    Directory paths carry a trailing "/" so dir-only patterns (`build/`) match them.
+    """
+    parts = rel.rstrip("/").split("/")
+    for depth in range(len(parts) - 1, -1, -1):
+        spec = specs.get("/".join(parts[:depth]))
+        if spec is None:
+            continue
+        verdict = spec.check_file("/".join(parts[depth:]) + ("/" if rel.endswith("/") else ""))
+        if verdict.include is not None:
+            return verdict.include
+    return False
 
 
 def discover_files(
@@ -37,17 +54,27 @@ def discover_files(
     """Return sorted relative POSIX paths of files matching include/exclude under root.
 
     Always skips ALWAYS_SKIP_DIRS, hidden directories, the corpus's own root-level
-    ragx.toml, binaries (NUL byte in first 8KB), and files larger than 2MB. Only a
-    root-level .gitignore is honored (MVP limitation).
+    ragx.toml, binaries (NUL byte in first 8KB), and files larger than 2MB. With
+    respect_gitignore, every .gitignore in the tree applies to its own subtree (git
+    semantics: deeper files override shallower ones; ignored directories are pruned,
+    so a negation can't re-include a file under them).
     """
     include_spec = pathspec.PathSpec.from_lines("gitwildmatch", include)
     exclude_spec = pathspec.PathSpec.from_lines("gitwildmatch", exclude) if exclude else None
-    gitignore_spec = _load_gitignore_spec(root) if respect_gitignore else None
+    gitignore_specs: dict[str, pathspec.GitIgnoreSpec] = {}
 
     results: list[str] = []
     for dirpath, dirnames, filenames in root.walk():
+        dir_rel = dirpath.relative_to(root).as_posix()
+        dir_rel = "" if dir_rel == "." else dir_rel
+        if respect_gitignore and (spec := _load_gitignore_spec(dirpath)) is not None:
+            gitignore_specs[dir_rel] = spec
         dirnames[:] = [
-            d for d in dirnames if d not in ALWAYS_SKIP_DIRS and not d.startswith(".")
+            d
+            for d in dirnames
+            if d not in ALWAYS_SKIP_DIRS
+            and not d.startswith(".")
+            and not _is_gitignored(f"{dir_rel}/{d}/".lstrip("/"), gitignore_specs)
         ]
         for name in filenames:
             path = dirpath / name
@@ -59,7 +86,7 @@ def discover_files(
                 continue
             if exclude_spec is not None and exclude_spec.match_file(rel):
                 continue
-            if gitignore_spec is not None and gitignore_spec.match_file(rel):
+            if _is_gitignored(rel, gitignore_specs):
                 continue
 
             try:

{ragx_cli-0.10.0 → ragx_cli-0.11.0}/tests/test_discovery.py

@@ -101,6 +101,58 @@ def test_discover_respects_root_gitignore(tmp_path: Path):
     assert files == [".gitignore", "kept.txt"]
 
 
+def test_discover_respects_nested_gitignore_scoped_to_its_subtree(tmp_path: Path):
+    # A subproject's .gitignore (e.g. a vendored repo's generated docs) must keep its
+    # junk out of the index, but must not leak onto same-named files elsewhere.
+    _write(tmp_path, "sub/.gitignore", "generated.md\n/anchored.md\n")
+    _write(tmp_path, "sub/generated.md")
+    _write(tmp_path, "sub/deep/generated.md")
+    _write(tmp_path, "sub/anchored.md")
+    _write(tmp_path, "sub/deep/anchored.md")
+    _write(tmp_path, "generated.md")
+    _write(tmp_path, "anchored.md")
+
+    files = discover_files(tmp_path, include=["**/*.md"], exclude=[], respect_gitignore=True)
+
+    # "/anchored.md" anchors to sub/, so only sub/anchored.md is dropped.
+    assert files == ["anchored.md", "generated.md", "sub/deep/anchored.md"]
+
+
+def test_discover_deeper_gitignore_overrides_root(tmp_path: Path):
+    # Git precedence: the deeper file wins, so a subdir can re-include what root ignores.
+    _write(tmp_path, ".gitignore", "*.log.md\n")
+    _write(tmp_path, "notes/.gitignore", "!keep.log.md\n")
+    _write(tmp_path, "notes/keep.log.md")
+    _write(tmp_path, "notes/drop.log.md")
+    _write(tmp_path, "keep.log.md")
+
+    files = discover_files(tmp_path, include=["**/*.md"], exclude=[], respect_gitignore=True)
+
+    assert files == ["notes/keep.log.md"]
+
+
+def test_discover_prunes_gitignored_directories(tmp_path: Path):
+    # Git can't re-include a file whose parent dir is ignored; ragx must not index it
+    # either, or the index would contain files git itself never tracks.
+    _write(tmp_path, ".gitignore", "out/\n")
+    _write(tmp_path, "pkg/out/.gitignore", "!kept.md\n")
+    _write(tmp_path, "pkg/out/kept.md")
+    _write(tmp_path, "pkg/src.md")
+
+    files = discover_files(tmp_path, include=["**/*.md"], exclude=[], respect_gitignore=True)
+
+    assert files == ["pkg/src.md"]
+
+
+def test_discover_ignores_nested_gitignore_when_disabled(tmp_path: Path):
+    _write(tmp_path, "sub/.gitignore", "hidden.md\n")
+    _write(tmp_path, "sub/hidden.md")
+
+    files = discover_files(tmp_path, include=["**/*.md"], exclude=[], respect_gitignore=False)
+
+    assert files == ["sub/hidden.md"]
+
+
 def test_discover_ignores_gitignore_when_disabled(tmp_path: Path):
     _write(tmp_path, ".gitignore", "notignored.txt\n")
     _write(tmp_path, "notignored.txt")