rag-your-code 0.7.0__tar.gz → 0.8.0__tar.gz

{rag_your_code-0.7.0/src/rag_your_code.egg-info → rag_your_code-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.7.0
+Version: 0.8.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -407,14 +407,66 @@ it stopped.
 Nothing authored lives under `.rag-your-code/` — that directory is what people
 delete to clear the cache.
 
-## Not here yet
+## Bringing your own model
 
-Provider-backed embeddings, Tree-sitter parsing, and a SQLite/ANN storage layer
-for repositories past the measured JSON envelope. Agent-authored descriptions
-are deliberately the cheaper answer to the same problem provider embeddings
-solve: they keep the zero-dependency, offline, reproducible-index properties,
-and produce text a human can read and correct rather than opaque floats. See
-[docs/ROADMAP.md](docs/ROADMAP.md).
+Everything above works with no model at all. If you would rather have real
+semantics, point the index at any OpenAI-compatible embeddings endpoint —
+which includes a model server on your own machine:
+
+```toml
+# rag-your-code.toml
+[embedding]
+provider   = "openai-compatible"
+endpoint   = "http://localhost:11434/v1/embeddings"   # ollama, LM Studio, vLLM…
+model      = "nomic-embed-text"
+dimensions = 768                                      # must match the model
+```
+
+A hosted service is the same three lines with an `https://` endpoint, plus the
+name of the environment variable holding your key:
+
+```toml
+api_key_env = "OPENAI_API_KEY"    # the NAME of the variable, never the key
+```
+
+**The key is never a setting.** `rag-your-code.toml` is meant to be committed
+so everyone who clones can see what shaped the index; a credential is the one
+value with the opposite requirement, so the file only ever names the variable
+it lives in. Sending a key over plain `http://` to anything but your own
+machine is refused rather than warned about.
+
+Three things follow from turning this on, and it is worth knowing all three
+before you do:
+
+- **Your source leaves the machine**, unless the endpoint is local. That is
+  the whole reason the local case is written first here.
+- **Similarity may now find things, not just order them.** With the local
+  hash a cosine shortlist is measurably noise, so it is confined to
+  re-ranking. A real model earns the right to add candidates the words never
+  reached, which is the one gap no amount of ranking closes:
+  `search.vector_recall` sets how many. Lexical evidence still dominates — a
+  unit found by similarity alone scores at most `search.vector_weight`.
+- **A failure stops the build.** Falling back to the local hash would leave an
+  index whose vectors come from two incompatible spaces, and ranking would act
+  on the meaningless cosine between them with full confidence.
+
+Switching provider, model or width discards the old vectors and rebuilds, so
+an index can never be a mixture. An incremental run over unchanged files makes
+no request at all.
+
+**What is not measured:** whether this helps *your* repository, and by how
+much. No number here is from a real model — this project has no key, and a
+figure produced by a stub would be fiction. The instrument ships instead:
+point `benchmarks/repo_queries.py --index` at your own index and grade it. You
+will probably also want a higher `search.vector_weight` than the 0.15 tuned
+for a hash that carries no meaning.
+
+## Still not here
+
+Tree-sitter parsing, and a SQLite/ANN storage layer for repositories past the
+measured JSON envelope. Note also that `search.vector_recall` scans every
+unit's vector on every query, which is fine at the measured envelope and is
+the thing an ANN index would replace. See [docs/ROADMAP.md](docs/ROADMAP.md).
 
 ## Development
 

{rag_your_code-0.7.0 → rag_your_code-0.8.0}/README.md

@@ -380,14 +380,66 @@ it stopped.
 Nothing authored lives under `.rag-your-code/` — that directory is what people
 delete to clear the cache.
 
-## Not here yet
-
-Provider-backed embeddings, Tree-sitter parsing, and a SQLite/ANN storage layer
-for repositories past the measured JSON envelope. Agent-authored descriptions
-are deliberately the cheaper answer to the same problem provider embeddings
-solve: they keep the zero-dependency, offline, reproducible-index properties,
-and produce text a human can read and correct rather than opaque floats. See
-[docs/ROADMAP.md](docs/ROADMAP.md).
+## Bringing your own model
+
+Everything above works with no model at all. If you would rather have real
+semantics, point the index at any OpenAI-compatible embeddings endpoint —
+which includes a model server on your own machine:
+
+```toml
+# rag-your-code.toml
+[embedding]
+provider   = "openai-compatible"
+endpoint   = "http://localhost:11434/v1/embeddings"   # ollama, LM Studio, vLLM…
+model      = "nomic-embed-text"
+dimensions = 768                                      # must match the model
+```
+
+A hosted service is the same three lines with an `https://` endpoint, plus the
+name of the environment variable holding your key:
+
+```toml
+api_key_env = "OPENAI_API_KEY"    # the NAME of the variable, never the key
+```
+
+**The key is never a setting.** `rag-your-code.toml` is meant to be committed
+so everyone who clones can see what shaped the index; a credential is the one
+value with the opposite requirement, so the file only ever names the variable
+it lives in. Sending a key over plain `http://` to anything but your own
+machine is refused rather than warned about.
+
+Three things follow from turning this on, and it is worth knowing all three
+before you do:
+
+- **Your source leaves the machine**, unless the endpoint is local. That is
+  the whole reason the local case is written first here.
+- **Similarity may now find things, not just order them.** With the local
+  hash a cosine shortlist is measurably noise, so it is confined to
+  re-ranking. A real model earns the right to add candidates the words never
+  reached, which is the one gap no amount of ranking closes:
+  `search.vector_recall` sets how many. Lexical evidence still dominates — a
+  unit found by similarity alone scores at most `search.vector_weight`.
+- **A failure stops the build.** Falling back to the local hash would leave an
+  index whose vectors come from two incompatible spaces, and ranking would act
+  on the meaningless cosine between them with full confidence.
+
+Switching provider, model or width discards the old vectors and rebuilds, so
+an index can never be a mixture. An incremental run over unchanged files makes
+no request at all.
+
+**What is not measured:** whether this helps *your* repository, and by how
+much. No number here is from a real model — this project has no key, and a
+figure produced by a stub would be fiction. The instrument ships instead:
+point `benchmarks/repo_queries.py --index` at your own index and grade it. You
+will probably also want a higher `search.vector_weight` than the 0.15 tuned
+for a hash that carries no meaning.
+
+## Still not here
+
+Tree-sitter parsing, and a SQLite/ANN storage layer for repositories past the
+measured JSON envelope. Note also that `search.vector_recall` scans every
+unit's vector on every query, which is fine at the measured envelope and is
+the thing an ANN index would replace. See [docs/ROADMAP.md](docs/ROADMAP.md).
 
 ## Development
 

{rag_your_code-0.7.0 → rag_your_code-0.8.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rag-your-code"
-version = "0.7.0"
+version = "0.8.0"
 description = "A local, explainable RAG index for codebases and coding agents"
 readme = "README.md"
 requires-python = ">=3.10"

{rag_your_code-0.7.0 → rag_your_code-0.8.0/src/rag_your_code.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rag-your-code
-Version: 0.7.0
+Version: 0.8.0
 Summary: A local, explainable RAG index for codebases and coding agents
 Author: rag-your-code contributors
 License-Expression: MIT
@@ -407,14 +407,66 @@ it stopped.
 Nothing authored lives under `.rag-your-code/` — that directory is what people
 delete to clear the cache.
 
-## Not here yet
+## Bringing your own model
 
-Provider-backed embeddings, Tree-sitter parsing, and a SQLite/ANN storage layer
-for repositories past the measured JSON envelope. Agent-authored descriptions
-are deliberately the cheaper answer to the same problem provider embeddings
-solve: they keep the zero-dependency, offline, reproducible-index properties,
-and produce text a human can read and correct rather than opaque floats. See
-[docs/ROADMAP.md](docs/ROADMAP.md).
+Everything above works with no model at all. If you would rather have real
+semantics, point the index at any OpenAI-compatible embeddings endpoint —
+which includes a model server on your own machine:
+
+```toml
+# rag-your-code.toml
+[embedding]
+provider   = "openai-compatible"
+endpoint   = "http://localhost:11434/v1/embeddings"   # ollama, LM Studio, vLLM…
+model      = "nomic-embed-text"
+dimensions = 768                                      # must match the model
+```
+
+A hosted service is the same three lines with an `https://` endpoint, plus the
+name of the environment variable holding your key:
+
+```toml
+api_key_env = "OPENAI_API_KEY"    # the NAME of the variable, never the key
+```
+
+**The key is never a setting.** `rag-your-code.toml` is meant to be committed
+so everyone who clones can see what shaped the index; a credential is the one
+value with the opposite requirement, so the file only ever names the variable
+it lives in. Sending a key over plain `http://` to anything but your own
+machine is refused rather than warned about.
+
+Three things follow from turning this on, and it is worth knowing all three
+before you do:
+
+- **Your source leaves the machine**, unless the endpoint is local. That is
+  the whole reason the local case is written first here.
+- **Similarity may now find things, not just order them.** With the local
+  hash a cosine shortlist is measurably noise, so it is confined to
+  re-ranking. A real model earns the right to add candidates the words never
+  reached, which is the one gap no amount of ranking closes:
+  `search.vector_recall` sets how many. Lexical evidence still dominates — a
+  unit found by similarity alone scores at most `search.vector_weight`.
+- **A failure stops the build.** Falling back to the local hash would leave an
+  index whose vectors come from two incompatible spaces, and ranking would act
+  on the meaningless cosine between them with full confidence.
+
+Switching provider, model or width discards the old vectors and rebuilds, so
+an index can never be a mixture. An incremental run over unchanged files makes
+no request at all.
+
+**What is not measured:** whether this helps *your* repository, and by how
+much. No number here is from a real model — this project has no key, and a
+figure produced by a stub would be fiction. The instrument ships instead:
+point `benchmarks/repo_queries.py --index` at your own index and grade it. You
+will probably also want a higher `search.vector_weight` than the 0.15 tuned
+for a hash that carries no meaning.
+
+## Still not here
+
+Tree-sitter parsing, and a SQLite/ANN storage layer for repositories past the
+measured JSON envelope. Note also that `search.vector_recall` scans every
+unit's vector on every query, which is fine at the measured envelope and is
+the thing an ANN index would replace. See [docs/ROADMAP.md](docs/ROADMAP.md).
 
 ## Development
 

{rag_your_code-0.7.0 → rag_your_code-0.8.0}/src/rag_your_code.egg-info/SOURCES.txt

@@ -19,6 +19,7 @@ src/ragyourcode/graph.py
 src/ragyourcode/indexer.py
 src/ragyourcode/models.py
 src/ragyourcode/parser.py
+src/ragyourcode/providers.py
 src/ragyourcode/py.typed
 src/ragyourcode/search.py
 src/ragyourcode/workflow.py
@@ -36,6 +37,7 @@ tests/test_large_repo.py
 tests/test_metadata.py
 tests/test_multilanguage.py
 tests/test_parser_edges.py
+tests/test_providers.py
 tests/test_ragyourcode.py
 tests/test_ranking.py
 tests/test_repo_queries.py

{rag_your_code-0.7.0 → rag_your_code-0.8.0}/src/ragyourcode/__init__.py

@@ -3,4 +3,4 @@
 from .models import CodeUnit, SearchResult
 
 __all__ = ["CodeUnit", "SearchResult"]
-__version__ = "0.7.0"
+__version__ = "0.8.0"

{rag_your_code-0.7.0 → rag_your_code-0.8.0}/src/ragyourcode/agentic.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 from .graph import CodeGraph, graph_search
 from .models import CodeUnit, SearchResult
-from .search import DEFAULT_VECTOR_WEIGHT, SearchIndex, context, search, within_budget
+from .search import DEFAULT_VECTOR_RECALL, DEFAULT_VECTOR_WEIGHT, SearchIndex, context, search, within_budget
 
 
 def _result_ids(results: list[SearchResult]) -> set[str]:
@@ -80,6 +80,7 @@ def research(
     graph: CodeGraph | None = None,
     search_index: SearchIndex | None = None,
     vector_weight: float = DEFAULT_VECTOR_WEIGHT,
+    vector_recall: int = DEFAULT_VECTOR_RECALL,
     max_chars: int = 12000,
 ) -> dict:
     """Run at most two deterministic retrieval steps and explain the stop.
@@ -96,7 +97,7 @@ def research(
     """
     max_steps = min(2, max(1, max_steps))
     steps: list[dict] = []
-    initial = search(units, query, max(limit, 1), search_index=search_index, vector_weight=vector_weight)
+    initial = search(units, query, max(limit, 1), search_index=search_index, vector_weight=vector_weight, vector_recall=vector_recall)
     steps.append({"action": "search", "query": query, "results": _trace(initial)})
     if not initial:
         return {"query": query, "results": [], "steps": steps, "stop_reason": "no_results", "context": ""}
@@ -105,7 +106,7 @@ def research(
         kept = initial[:limit]
         return {"query": query, "results": _serialize(kept), "steps": steps, "stop_reason": "high_confidence", "context": context(within_budget(kept, max_chars), max_chars)}
 
-    expanded = graph_search(units, query, limit=max(limit * 2, 8), hops=hops, graph=graph, search_index=search_index, vector_weight=vector_weight)
+    expanded = graph_search(units, query, limit=max(limit * 2, 8), hops=hops, graph=graph, search_index=search_index, vector_weight=vector_weight, vector_recall=vector_recall)
     steps.append({"action": "graph_expand", "hops": hops, "results": _trace(expanded[:limit])})
     merged = {result.unit.id: result for result in initial}
     for result in expanded:

{rag_your_code-0.7.0 → rag_your_code-0.8.0}/src/ragyourcode/cli.py

@@ -15,10 +15,11 @@ from .agentic import DEFAULT_DOMINANCE, research
 from .config import BY_PATH, SETTINGS, Config, ConfigError
 from .descriptions import index_descriptions_fingerprint
 from .document import plan as plan_documentation, render_patch, summarise as summarise_documentation
-from .embeddings import embed, embedding_metadata
+from .embeddings import embedder
 from .graph import build_graph, graph_from_dict, graph_search
 from .indexer import StaleMonitor, build_fingerprint, build_units, fingerprint, index_build_fingerprint, read_index, snapshot_repository, write_index
 from .models import SearchResult
+from .providers import ProviderError
 from .search import build_search_index, context, search, within_budget
 from .workflow import apply_descriptions, bootstrap, describe_batch, store_descriptions
 
@@ -48,6 +49,10 @@ def _refresh_index(root: Path, output: Path, full: bool = False, compact: bool |
     units still have none.
     """
     cfg = cfg if cfg is not None else config_module.load(root)
+    # Built once and handed to every stage of this run, so the vectors stored,
+    # the metadata published and the queries later asked all come from the
+    # same scheme by construction rather than by three call sites agreeing.
+    embed_with = embedder(cfg)
     previous_payload: dict = {}
     previous_units = []
     if output.exists() and not full:
@@ -55,7 +60,7 @@ def _refresh_index(root: Path, output: Path, full: bool = False, compact: bool |
             previous_payload, previous_units = read_index(output)
         except (OSError, TypeError, ValueError, json.JSONDecodeError):
             previous_payload, previous_units = {}, []
-    if previous_payload.get("embedding") != embedding_metadata(cfg["embedding.dimensions"]):
+    if previous_payload.get("embedding") != embed_with.metadata:
         for unit in previous_units:
             unit.vector = []
     if compact is None:
@@ -83,9 +88,10 @@ def _refresh_index(root: Path, output: Path, full: bool = False, compact: bool |
         cfg=cfg,
         previous_build=None if inputs_changed else previous_build,
         descriptions=store,
+        embed_with=embed_with,
     )
     graph = build_graph(units)
-    write_index(output, root, units, graph.to_dict(), compact=compact, diagnostics=diagnostics, snapshot=snapshot, cfg=cfg, descriptions_fingerprint=store.fingerprint)
+    write_index(output, root, units, graph.to_dict(), compact=compact, diagnostics=diagnostics, snapshot=snapshot, cfg=cfg, descriptions_fingerprint=store.fingerprint, embed_with=embed_with)
     groups = store.classify(units)
     return {
         "indexed_units": len(units),
@@ -175,11 +181,12 @@ def _cmd_search(args: argparse.Namespace) -> int:
     limit = args.limit if args.limit is not None else cfg["search.limit"]
     max_chars = args.max_chars if args.max_chars is not None else cfg["search.max_chars"]
     weight = cfg["search.vector_weight"]
-    search_index = build_search_index(units)
+    search_index = build_search_index(units, embedder(cfg))
+    recall = cfg["search.vector_recall"]
     results = (
-        graph_search(units, args.query, limit, args.hops, graph, search_index, vector_weight=weight)
+        graph_search(units, args.query, limit, args.hops, graph, search_index, vector_weight=weight, vector_recall=recall)
         if args.graph
-        else search(units, args.query, limit, search_index=search_index, vector_weight=weight)
+        else search(units, args.query, limit, search_index=search_index, vector_weight=weight, vector_recall=recall)
     )
     if args.json:
         # Results are navigation and cost almost nothing, so every one that was
@@ -390,11 +397,12 @@ def _request_int(request: dict, key: str, default: int, minimum: int, maximum: i
 def _cmd_agent(args: argparse.Namespace) -> int:
     """Serve one JSON request per line, suitable for a plugin subprocess."""
     payload, units, graph, cfg, store = _load(args)
-    search_index = build_search_index(units)
+    search_index = build_search_index(units, embedder(cfg))
     root = Path(args.root).resolve()
     weight = cfg["search.vector_weight"]
     default_limit = cfg["search.limit"]
     default_chars = cfg["search.max_chars"]
+    recall = cfg["search.vector_recall"]
     stale_monitor = StaleMonitor(root, payload, assume_checked=True, cfg=cfg, descriptions_fingerprint=store.fingerprint)
     # Descriptions stored this session reach the live units immediately but not
     # the published index, which is a different thing from the index being
@@ -420,9 +428,9 @@ def _cmd_agent(args: argparse.Namespace) -> int:
                 hops = _request_int(request, "hops", 1, 0, 3)
                 use_graph = bool(request.get("graph", False))
                 results = (
-                    graph_search(units, query, limit, hops, graph, search_index, vector_weight=weight)
+                    graph_search(units, query, limit, hops, graph, search_index, vector_weight=weight, vector_recall=recall)
                     if use_graph
-                    else search(units, query, limit, search_index=search_index, vector_weight=weight)
+                    else search(units, query, limit, search_index=search_index, vector_weight=weight, vector_recall=recall)
                 )
                 budget = _request_int(request, "max_chars", default_chars, 0, 100000)
                 shown = within_budget(results, budget)
@@ -438,6 +446,7 @@ def _cmd_agent(args: argparse.Namespace) -> int:
                     graph,
                     search_index,
                     vector_weight=weight,
+                    vector_recall=recall,
                     max_chars=_request_int(request, "max_chars", default_chars, 0, 100000),
                 )
                 response["stale"] = payload.get("stale", True)
@@ -479,13 +488,13 @@ def _cmd_agent(args: argparse.Namespace) -> int:
                 # waiting for a refresh the agent has no reason to expect.
                 response["applied"] = apply_descriptions(units, store, cfg)
                 if response["applied"]:
-                    search_index = build_search_index(units)
+                    search_index = build_search_index(units, embedder(cfg))
                 index_behind = index_behind or response["reindex_required"]
             elif action == "refresh":
                 output = Path(args.index) if args.index else _default_index(root)
                 response = _refresh_index(root, output, cfg=cfg)
                 payload, units, graph, cfg, store = _load(args)
-                search_index = build_search_index(units)
+                search_index = build_search_index(units, embedder(cfg))
                 stale_monitor = StaleMonitor(root, payload, assume_checked=True, cfg=cfg, descriptions_fingerprint=store.fingerprint)
                 index_behind = False
             elif action == "stats":
@@ -609,6 +618,12 @@ def main(argv: list[str] | None = None) -> int:
     args = build_parser().parse_args(argv)
     try:
         return args.func(args)
+    except ProviderError as exc:
+        # A provider that cannot be reached or cannot be trusted stops the run
+        # rather than falling back: an index whose vectors come from two
+        # spaces would rank confidently on a number that means nothing.
+        print(f"error: embedding provider: {exc}", file=sys.stderr)
+        return 2
     except ConfigError as exc:
         # Surfaced separately from the generic handler because the fix is
         # always in one named file: say which, so the message is actionable.

{rag_your_code-0.7.0 → rag_your_code-0.8.0}/src/ragyourcode/config.py

@@ -117,7 +117,74 @@ SETTINGS: tuple[Setting, ...] = (
         affects_build=True,
         minimum=32,
         maximum=4096,
-        help="feature-hash vector width; changing it invalidates every vector",
+        help="vector width; changing it invalidates every vector",
+    ),
+    # The default keeps the whole pipeline offline and dependency-free. The
+    # other value sends each unit's text to an OpenAI-compatible embeddings
+    # endpoint, which may be a vendor or a model server on localhost -- one
+    # request shape covers both, and the local one keeps source on the machine.
+    Setting(
+        "embedding.provider",
+        "str",
+        "signed-feature-hash",
+        affects_build=True,
+        members=frozenset({"signed-feature-hash", "openai-compatible"}),
+        help="who computes vectors; the default never opens a socket",
+    ),
+    Setting(
+        "embedding.endpoint",
+        "str",
+        "",
+        affects_build=True,
+        help="OpenAI-compatible embeddings URL, e.g. http://localhost:11434/v1/embeddings",
+    ),
+    Setting(
+        "embedding.model",
+        "str",
+        "",
+        affects_build=True,
+        help="model name the endpoint expects; part of what an index records",
+    ),
+    # The NAME of an environment variable, never a key. Every other setting
+    # here is meant to be committed so everyone who clones sees what shaped the
+    # index; a credential is the one value with the opposite requirement, so it
+    # is the one value this file only points at. `config list` prints whether
+    # the variable is set, never what it holds.
+    Setting(
+        "embedding.api_key_env",
+        "str",
+        "RAG_YOUR_CODE_API_KEY",
+        help="environment variable holding the endpoint's key; empty means no auth header",
+    ),
+    Setting(
+        "embedding.batch",
+        "int",
+        64,
+        minimum=1,
+        maximum=512,
+        help="units per embeddings request; one request per unit is unusable at scale",
+    ),
+    Setting("embedding.timeout", "int", 60, minimum=1, maximum=600, help="seconds to wait for one embeddings request"),
+    Setting(
+        "embedding.retries",
+        "int",
+        3,
+        minimum=0,
+        maximum=10,
+        help="attempts per request before the build aborts rather than mixing schemes",
+    ),
+    # Only consulted when the vectors carry real semantics. Under the feature
+    # hash a cosine shortlist is noise, and letting it add candidates would
+    # dilute a ranking that measured better without it; with a trained model
+    # it is the one thing that can make a unit retrievable that shares no word
+    # with the query.
+    Setting(
+        "search.vector_recall",
+        "int",
+        50,
+        minimum=0,
+        maximum=500,
+        help="units a semantic provider may add to the candidate set by similarity alone",
     ),
     Setting(
         "search.vector_weight",
@@ -186,6 +253,15 @@ def _coerce(setting: Setting, value: Any) -> Any:
                     f"Adding a language means adding a rule table entry in parser.py."
                 )
         return items
+    if setting.kind == "str":
+        if not isinstance(value, str):
+            raise ConfigError(f"{setting.path} must be a string")
+        text = value.strip()
+        if setting.members is not None and text not in setting.members:
+            raise ConfigError(
+                f"{setting.path}: {text!r} is not one of {', '.join(sorted(setting.members))}"
+            )
+        return text
     if setting.kind == "int":
         if isinstance(value, bool) or not isinstance(value, int):
             raise ConfigError(f"{setting.path} must be an integer")