ragx-cli 0.3.3__tar.gz → 0.5.0__tar.gz

{ragx_cli-0.3.3 → ragx_cli-0.5.0}/CHANGELOG.md

@@ -4,6 +4,52 @@ All notable changes to `ragx-cli` are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow
 [Semantic Versioning](https://semver.org/).
 
+## [0.5.0] — 2026-07-15
+
+### Added
+
+- **`ragx-cli models`**: recommends an embedding + reranker combo (quality tier,
+  RAM/macOS detection), downloads models through LM Studio (`lms get --yes`), and writes
+  the config to `ragx.toml` (with an `index --full` warning when the embedding model
+  changed). Scriptable: `--quality fast|balanced|best|jina-nano`, `--embed-engine`,
+  `--rerank-engine`, `--yes`, `--dry-run`, `--json` (schema `ragx.models.v1`).
+  Interactive `init` offers the flow as its final step. Embedding catalog:
+  EmbeddingGemma-300M / BGE-M3 / Qwen3-Embedding-0.6B / Jina-v5-nano (CC-BY-NC,
+  llama-server engine only); reranker: bge-reranker-v2-m3 (multilingual).
+- **`llama-server` rerank engine** (`rerank.provider = "llama-server"`): reranking via
+  llama.cpp's `/v1/rerank` on a GGUF (`rerank.gguf`), auto-spawning `llama-server`
+  (`rerank.server_bin`, `rerank.base_url`) when nothing is listening and terminating it
+  at exit. Reranker GGUFs download through LM Studio (`ragx-cli models --rerank-engine
+  llama-server`), so reranking needs **no direct huggingface.co access** and no torch/
+  sentence-transformers install. Q8_0 GGUFs validated against the safetensors originals
+  (scores within ~0.5 logit, identical ordering).
+- **`llama-server` embedding engine** (`embeddings.provider = "llama-server"`): embeddings
+  served by an auto-spawned `llama-server --embedding` on `embeddings.gguf` (same managed
+  lifecycle as the rerank engine; default port 9813). `ragx-cli models --embed-engine
+  llama-server` wires it up. With both engines on llama-server, LM Studio is purely a
+  model downloader — nothing but ragx-managed processes run at query time. Also unblocks
+  EuroBERT models (e.g. `jina-embeddings-v5-text-nano-retrieval-GGUF`) that LM Studio
+  downloads but cannot serve.
+
+## [0.4.0] — 2026-07-15
+
+### Changed (BREAKING)
+
+- **`ragx-cli index` is now incremental by default** (previously a full rebuild).
+  It hash-diffs the corpus and re-processes only new/modified/deleted files. The old
+  behavior moved to `ragx-cli index --full`; `--changed` remains as a deprecated,
+  hidden alias for the (now default) incremental mode.
+- **Chunking params joined the index manifest**: changing `chunking.size_tokens` or
+  `chunking.overlap` now fails loud on an incremental run (previously it silently left
+  old files chunked under the old params). Fix is `ragx-cli index --full`.
+
+### Added
+
+- **`ragx-cli status` reports corpus drift**: a `drift` object
+  (`{"new": N, "changed": N, "deleted": N}`, same in text output) diffing the corpus
+  on disk against the stored file hashes, so agents and humans can see when a
+  re-index is due without running one.
+
 ## [0.3.3] — 2026-07-15
 
 ### Added

{ragx_cli-0.3.3 → ragx_cli-0.5.0}/CLAUDE.md

@@ -22,6 +22,9 @@ API contracts for every core module. Deferred feature specs live in `docs/`.
   hashing), pathspec (gitignore globs).
 - Providers speak **OpenAI-compatible HTTP**. Local default: LM Studio at
   `http://localhost:1234/v1`. `provider="ollama"` auto-switches to `:11434/v1`.
+  `embeddings.provider="llama-server"` instead auto-spawns `llama-server --embedding` on
+  `embeddings.gguf` (port 9813; shared lifecycle code in `providers/llama_process.py`) —
+  with the llama-server rerank engine, LM Studio is downloader-only at query time.
   Env fallbacks: `OPENAI_BASE_URL` (only while base_url is still the default) and
   `OPENAI_API_KEY` (only when `api_key_env` is unset); explicit config always wins.
 - Machine-level provider settings live in `~/.ragxrc` (TOML; ONLY the embeddings/
@@ -29,8 +32,11 @@ API contracts for every core module. Deferred feature specs live in `docs/`.
   ragx.toml < ~/.ragxrc — the rc OVERRIDES corpus values and logs a stderr warning
   per overridden key. Write via `ragx-cli config set --global <key> <value>`.
   `Config.save` never bakes rc values into the corpus file.
-- Reranker is always local: sentence-transformers CrossEncoder `BAAI/bge-reranker-v2-m3`
-  (optional extra `ragx-cli[rerank]`). LM Studio has NO /v1/rerank endpoint — verified; don't try.
+- Reranker is always local, two engines: sentence-transformers CrossEncoder (default,
+  `BAAI/bge-reranker-v2-m3`, optional extra `ragx-cli[rerank]`) or `rerank.provider=
+  "llama-server"` (llama.cpp `/v1/rerank` on a GGUF, auto-spawned from `rerank.gguf`;
+  GGUFs download via LM Studio — validated ≈ safetensors scores). LM Studio itself has
+  NO /v1/rerank endpoint — verified; don't try.
 
 ## Layout (core/CLI split is the one architectural rule — MCP server later rides on it)
 
@@ -57,10 +63,16 @@ src/ragx/
     indexer.py     #   run_index: discover->hash->chunk->embed->HNSW+store->kNN edges
     query.py       #   run_query: expansion->fan-out->RRF->traversal->rerank->combine; JSON serializers
     eval.py        #   recall@5/@10 + MRR over file-level ranking, injected query_fn
+    catalog.py     #   curated embedding/reranker catalog (tiers, prefixes) + spec detect
+    lmstudio.py    #   `lms` CLI wrapper: find/download (`lms get --yes`)/`lms ls --json`;
+                   #   LM Studio can't download plain safetensors — rerankers stay on HF
   providers/       # base.py protocols (Embedder/Generator/Reranker); openai_compat.py;
-                   # st_reranker.py; registry.py factories (env-var + api_key_env resolution)
+                   # st_reranker.py; llama_process.py (managed llama-server lifecycle) +
+                   # llama_server.py (rerank) + llama_embedder.py (embeddings);
+                   # registry.py factories (env-var + api_key_env resolution)
   cli/             # thin shells only: app.py (init/status/config + registration),
-                   # pipeline.py (index/query), inspect_cmd.py, eval_cmd.py, output.py
+                   # pipeline.py (index/query), inspect_cmd.py, eval_cmd.py, models_cmd.py
+                   # (recommend/download/configure models), output.py
 tests/             # 177 tests; mocked HTTP (respx), FakeEmbedder integration tests, no live network
 ```
 

{ragx_cli-0.3.3 → ragx_cli-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ragx-cli
-Version: 0.3.3
+Version: 0.5.0
 Summary: Similarity-graph RAG CLI: index a corpus into a chunk-level embedding graph and query it
 Project-URL: Repository, https://github.com/alber70g/ragx-cli
 Project-URL: Issues, https://github.com/alber70g/ragx-cli/issues
@@ -49,9 +49,10 @@ that can justify every result via the exact graph path that produced it.
 ```bash
 uv tool install ragx-cli --with ragx-cli[rerank]  # install and use `ragx-cli`
 ragx-cli init                  # create ragx.toml next to your corpus (interactive on a TTY; --yes for defaults)
-ragx-cli index                 # chunk -> embed -> HNSW + kNN similarity graph
+ragx-cli index                 # chunk -> embed -> HNSW + kNN similarity graph (incremental after the first run)
 ragx-cli query "why did we switch build tools?" --json --files-only
-ragx-cli index --changed       # incremental: only new/modified/deleted files
+ragx-cli index --full          # full rebuild: re-chunk + re-embed everything
+ragx-cli models                # recommend + download an embedding/reranker combo via LM Studio
 ```
 
 Runs against any OpenAI-compatible embedding endpoint (LM Studio, Ollama, OpenAI). Reranking
@@ -93,7 +94,7 @@ mixing several concepts gets one sharp edge per concept instead of a diluted ave
 between near-duplicate chunks (whole-chunk cosine ≥ `near_dup_sim`) are dropped — those are the
 measured precision-killers. Costs roughly 5–10× more embedding calls at index time; retrieval
 units, traversal, and query flow are unchanged. Switching `edge_source` (or the sub-chunk size)
-requires a full re-index; `--changed` fails loud on the mismatch. Literature grounding:
+requires `index --full`; incremental runs fail loud on the mismatch. Literature grounding:
 `research/fine-grained-sub-chunk-edges-with-coarse-chunk-nodes-multi-granularity-graph-rag-literature-validation.md`.
 In this mode `k` counts links per *sub-chunk* with no per-chunk cap, so the graph is denser and
 shallower traversal suffices. Measured on the tuned eval corpus (2026-07): at `hops=2` it ties
@@ -116,8 +117,13 @@ flowchart TD
     G --> F
 ```
 
-Incremental runs (`--changed`) re-embed only changed files and repair only the edge lists that
-those chunks touch. Content hashes make `touch`-ed but unchanged files free.
+`ragx-cli index` is **incremental by default** (0.4.0; `--changed` is a deprecated alias): it
+re-embeds only new/modified files, drops deleted ones, and repairs only the edge lists those
+chunks touch. Content hashes make `touch`-ed but unchanged files free, and include/exclude glob
+changes converge to the same file set a rebuild would produce. `--full` rebuilds from scratch;
+changing `embeddings.model`, `chunking.*`, or `graph.edge_source` requires it — incremental runs
+fail loud on the manifest mismatch. `ragx-cli status` reports **drift** (files new/changed/deleted
+since the last index) so you can see when a re-index is due without running one.
 
 ### Querying
 
@@ -314,6 +320,53 @@ Full write-ups: `research/bge-m3-dense-q8-vs-nomic-q4-benchmark-2026-07-12-workt
 
 ---
 
+## Picking models: `ragx-cli models`
+
+`ragx-cli models` recommends an embedding + reranker combo, downloads the embedding model
+**through LM Studio** (`lms get` — one download path, plays well with restricted networks),
+pre-fetches the reranker from Hugging Face, and writes both to `ragx.toml`. Interactive on a
+TTY (quality tier + engine questions, RAM auto-detected); scriptable via flags:
+
+```bash
+ragx-cli models                                        # interactive
+ragx-cli models --quality balanced --yes --json        # agents
+ragx-cli models --quality fast --dry-run               # recommend only, change nothing
+```
+
+The curated catalog (every entry verified working end-to-end):
+
+| tier | embedding model | notes |
+|---|---|---|
+| `fast` | EmbeddingGemma 300M | 308M, 100+ languages, 2048-token context |
+| `balanced` | BGE-M3 | ragx's benchmarked production model (~100 languages) |
+| `best` | Qwen3-Embedding-0.6B | 100+ languages, 32K context |
+| `jina-nano` | Jina embeddings v5 nano | 239M, best-in-class sub-500M; **CC-BY-NC (non-commercial)**; llama-server engine only — LM Studio downloads it but can't serve EuroBERT |
+
+**Serving engines.** Embeddings run through `--embed-engine`: **`llama-server`** (default when
+llama.cpp is installed — ragx auto-spawns `llama-server --embedding` on the downloaded GGUF and
+terminates it at exit; LM Studio is only the downloader) or **`lm-studio`** (LM Studio serves
+`/v1/embeddings`; zero extra processes, LM Studio must be running). The reranker —
+`BAAI/bge-reranker-v2-m3` (multilingual) — runs through `--rerank-engine`:
+
+- **`llama-server`** (default when llama.cpp is installed): the reranker's Q8_0 GGUF is also
+  downloaded via LM Studio, and ragx auto-spawns/terminates a `llama-server --rerank` process
+  at query time (`rerank.provider="llama-server"`, `rerank.gguf`, `rerank.base_url`,
+  `rerank.server_bin`). **No huggingface.co access needed anywhere.** GGUF scores are validated
+  within ~0.5 logit of the safetensors originals.
+- **`sentence-transformers`**: the classic CrossEncoder path; model pre-fetched from Hugging
+  Face (needs the `ragx-cli[rerank]` extra).
+
+With both engines on `llama-server` the whole stack is ragx-managed: LM Studio downloads the
+GGUFs, ragx spawns and reaps the servers, and neither LM Studio nor huggingface.co is needed
+at query time. Under 8 GB RAM the embedding tier drops to `fast`; on macOS
+`lms get --yes` picks MLX variants where LM Studio's catalog offers them. Switching the
+embedding model invalidates the index — the command tells you to run `ragx-cli index --full`.
+`init` offers this flow as its final step; rerun `ragx-cli models` anytime to switch.
+
+Why not `jina-embeddings-v5-text-nano`? Excellent model, but its EuroBERT architecture needs
+Jina's llama.cpp fork — stock LM Studio can't load it (and the license is CC-BY-NC). See
+`research/lm-studio-model-download-api-and-curated-embedding-reranker-catalog-for-ragx-models-command.md`.
+
 ## Agent-first conventions
 
 - `--json` emits exactly one JSON document on stdout (versioned schemas: `ragx.query.v1`,
@@ -322,6 +375,8 @@ Full write-ups: `research/bge-m3-dense-q8-vs-nomic-q4-benchmark-2026-07-12-workt
 - Every chunk carries `file`, `line_start/line_end`, `byte_start/byte_end` — agents jump to the
   exact source location and read the full text themselves (JSON chunk text is truncated).
 - `--files-only` aggregates chunk scores per file (sum of top-3) — the mode coding agents use most.
+- `ragx-cli status` includes a `drift` object (`new`/`changed`/`deleted` file counts vs the index) —
+  agents check it to decide whether to run `ragx-cli index` before querying.
 - `ragx-cli query -` reads the query from stdin; `ragx-cli inspect chunk|file|neighbors|communities|community`
   debugs the graph.
 
@@ -341,7 +396,8 @@ questions; fall back to grep for exact identifiers.
   `file` + `line_start/line_end`; the JSON `text` is truncated, so read the file yourself
   for full context.
 - Fast mode (no LLM call, no cross-encoder): add `--no-expand --no-rerank`.
-- After adding or editing files: `ragx-cli index --changed` (cheap, hash-based).
+- After adding or editing files: `ragx-cli index` (incremental and cheap — hash-based).
+  `ragx-cli status --json` shows `drift` counts if you want to check staleness first.
 - stdout is exactly one JSON document; logs are on stderr.
   Exit codes: 0 = results, 1 = no results (not an error), 2 = error.
 - Why did this result appear? `ragx-cli query "..." --explain`.
@@ -401,9 +457,19 @@ detects the mismatch and asks you to run a full `ragx-cli index`.
 
 ### Reranker on restricted networks (no huggingface.co)
 
-The reranker model (`BAAI/bge-reranker-v2-m3`, ~2.3 GB) is downloaded from huggingface.co on
-first use. If your network blocks huggingface.co, ragx-cli degrades gracefully — queries run
-without reranking and a warning explains why — but you have three ways to get reranking working:
+With the default `sentence-transformers` engine, the reranker model
+(`BAAI/bge-reranker-v2-m3`, ~2.3 GB) is downloaded from huggingface.co on first use. If your
+network blocks huggingface.co, ragx-cli degrades gracefully — queries run without reranking
+and a warning explains why — and the easiest fix is to skip huggingface.co entirely:
+
+**Option 0 — the `llama-server` engine (recommended).** `ragx-cli models --rerank-engine
+llama-server` downloads a validated Q8_0 GGUF of the reranker **through LM Studio** (same
+download path as the embedding model) and reranks via llama.cpp's `llama-server --rerank`,
+which ragx starts and stops automatically. Requires a recent llama.cpp
+(`brew install llama.cpp`); no torch, no sentence-transformers, no huggingface.co. The GGUFs
+score within ~0.5 logit of the safetensors originals with identical ordering.
+
+Prefer the sentence-transformers engine? Three more ways:
 
 **Option A — copy the model and point `rerank.model` at the directory** (recommended).
 On any machine with access:
@@ -499,7 +565,7 @@ Checked features are built and validated per [the implementation plan](ragx-cli-
 unchecked ones are next up. Release history: [CHANGELOG.md](CHANGELOG.md).
 
 - [x] **CLI & storage**: typer CLI, SQLite schema, provider abstraction (Embedder/Generator/Reranker)
-- [x] **Baseline vector RAG**: discovery, chunking, embeddings, HNSW search, incremental `--changed`
+- [x] **Baseline vector RAG**: discovery, chunking, embeddings, HNSW search, incremental indexing
 - [x] **Similarity graph**: kNN edge construction, heat-propagation traversal, `inspect`, `--explain`
 - [x] **Quality & measurement**: multi-query/HyDE expansion, RRF fusion, cross-encoder rerank, `eval` harness
 - [x] **Communities**: Leiden detection over the edge list (index-time, read-only via status/inspect)
@@ -507,11 +573,13 @@ unchecked ones are next up. Release history: [CHANGELOG.md](CHANGELOG.md).
 - [x] **Committable config**: `ragx.toml` at the corpus root (0.3.0; was `.ragx/config.toml`), index data stays gitignored in `.ragx/`
 - [x] **Offline-friendly rerank**: graceful degradation + pre-seeding docs when huggingface.co is unreachable
 - [x] **Sub-chunk edges** (opt-in `graph.edge_source="subchunk"`): edge weight = max cosine over sentence-aligned sub-chunk pairs, for corpora with long multi-concept chunks
+- [x] **Incremental by default** (0.4.0): `index` hash-diffs, `--full` rebuilds, `status` reports corpus drift, chunking params are manifest-guarded
+- [x] **`ragx-cli models`**: curated embedding/reranker recommendation (quality tier + RAM detection), downloads via LM Studio's `lms get`, serving via llama-server or LM Studio
 - [ ] **Community labels**: name the Leiden communities so they're browsable without reading member chunks
 - [ ] **query --global** for corpus-level questions (answer from community summaries, not individual chunks)
 - [ ] **MCP server**: a second thin shell over `ragx.core` (the core/CLI split it needs is already enforced)
 - [ ] **[Temporal weighting](docs/feature-temporal-weighting.md)**: opt-in `--since`/`--until`/`--temporal recent|oldest`, date cascade filename/frontmatter → git → mtime
 - [x] **Release**: publish to PyPI as `ragx-cli` (plain `ragx` is name-blocked, too similar to an existing project) so `uvx ragx-cli` works out of the box
 
-Development: `uv sync --group dev && uv run pytest`. 177 tests; module contracts live in
+Development: `uv sync --group dev && uv run pytest`. 183 tests; module contracts live in
 `CONTRACTS.md` / `CONTRACTS-PHASE23.md`.

{ragx_cli-0.3.3 → ragx_cli-0.5.0}/README.md

@@ -21,9 +21,10 @@ that can justify every result via the exact graph path that produced it.
 ```bash
 uv tool install ragx-cli --with ragx-cli[rerank]  # install and use `ragx-cli`
 ragx-cli init                  # create ragx.toml next to your corpus (interactive on a TTY; --yes for defaults)
-ragx-cli index                 # chunk -> embed -> HNSW + kNN similarity graph
+ragx-cli index                 # chunk -> embed -> HNSW + kNN similarity graph (incremental after the first run)
 ragx-cli query "why did we switch build tools?" --json --files-only
-ragx-cli index --changed       # incremental: only new/modified/deleted files
+ragx-cli index --full          # full rebuild: re-chunk + re-embed everything
+ragx-cli models                # recommend + download an embedding/reranker combo via LM Studio
 ```
 
 Runs against any OpenAI-compatible embedding endpoint (LM Studio, Ollama, OpenAI). Reranking
@@ -65,7 +66,7 @@ mixing several concepts gets one sharp edge per concept instead of a diluted ave
 between near-duplicate chunks (whole-chunk cosine ≥ `near_dup_sim`) are dropped — those are the
 measured precision-killers. Costs roughly 5–10× more embedding calls at index time; retrieval
 units, traversal, and query flow are unchanged. Switching `edge_source` (or the sub-chunk size)
-requires a full re-index; `--changed` fails loud on the mismatch. Literature grounding:
+requires `index --full`; incremental runs fail loud on the mismatch. Literature grounding:
 `research/fine-grained-sub-chunk-edges-with-coarse-chunk-nodes-multi-granularity-graph-rag-literature-validation.md`.
 In this mode `k` counts links per *sub-chunk* with no per-chunk cap, so the graph is denser and
 shallower traversal suffices. Measured on the tuned eval corpus (2026-07): at `hops=2` it ties
@@ -88,8 +89,13 @@ flowchart TD
     G --> F
 ```
 
-Incremental runs (`--changed`) re-embed only changed files and repair only the edge lists that
-those chunks touch. Content hashes make `touch`-ed but unchanged files free.
+`ragx-cli index` is **incremental by default** (0.4.0; `--changed` is a deprecated alias): it
+re-embeds only new/modified files, drops deleted ones, and repairs only the edge lists those
+chunks touch. Content hashes make `touch`-ed but unchanged files free, and include/exclude glob
+changes converge to the same file set a rebuild would produce. `--full` rebuilds from scratch;
+changing `embeddings.model`, `chunking.*`, or `graph.edge_source` requires it — incremental runs
+fail loud on the manifest mismatch. `ragx-cli status` reports **drift** (files new/changed/deleted
+since the last index) so you can see when a re-index is due without running one.
 
 ### Querying
 
@@ -286,6 +292,53 @@ Full write-ups: `research/bge-m3-dense-q8-vs-nomic-q4-benchmark-2026-07-12-workt
 
 ---
 
+## Picking models: `ragx-cli models`
+
+`ragx-cli models` recommends an embedding + reranker combo, downloads the embedding model
+**through LM Studio** (`lms get` — one download path, plays well with restricted networks),
+pre-fetches the reranker from Hugging Face, and writes both to `ragx.toml`. Interactive on a
+TTY (quality tier + engine questions, RAM auto-detected); scriptable via flags:
+
+```bash
+ragx-cli models                                        # interactive
+ragx-cli models --quality balanced --yes --json        # agents
+ragx-cli models --quality fast --dry-run               # recommend only, change nothing
+```
+
+The curated catalog (every entry verified working end-to-end):
+
+| tier | embedding model | notes |
+|---|---|---|
+| `fast` | EmbeddingGemma 300M | 308M, 100+ languages, 2048-token context |
+| `balanced` | BGE-M3 | ragx's benchmarked production model (~100 languages) |
+| `best` | Qwen3-Embedding-0.6B | 100+ languages, 32K context |
+| `jina-nano` | Jina embeddings v5 nano | 239M, best-in-class sub-500M; **CC-BY-NC (non-commercial)**; llama-server engine only — LM Studio downloads it but can't serve EuroBERT |
+
+**Serving engines.** Embeddings run through `--embed-engine`: **`llama-server`** (default when
+llama.cpp is installed — ragx auto-spawns `llama-server --embedding` on the downloaded GGUF and
+terminates it at exit; LM Studio is only the downloader) or **`lm-studio`** (LM Studio serves
+`/v1/embeddings`; zero extra processes, LM Studio must be running). The reranker —
+`BAAI/bge-reranker-v2-m3` (multilingual) — runs through `--rerank-engine`:
+
+- **`llama-server`** (default when llama.cpp is installed): the reranker's Q8_0 GGUF is also
+  downloaded via LM Studio, and ragx auto-spawns/terminates a `llama-server --rerank` process
+  at query time (`rerank.provider="llama-server"`, `rerank.gguf`, `rerank.base_url`,
+  `rerank.server_bin`). **No huggingface.co access needed anywhere.** GGUF scores are validated
+  within ~0.5 logit of the safetensors originals.
+- **`sentence-transformers`**: the classic CrossEncoder path; model pre-fetched from Hugging
+  Face (needs the `ragx-cli[rerank]` extra).
+
+With both engines on `llama-server` the whole stack is ragx-managed: LM Studio downloads the
+GGUFs, ragx spawns and reaps the servers, and neither LM Studio nor huggingface.co is needed
+at query time. Under 8 GB RAM the embedding tier drops to `fast`; on macOS
+`lms get --yes` picks MLX variants where LM Studio's catalog offers them. Switching the
+embedding model invalidates the index — the command tells you to run `ragx-cli index --full`.
+`init` offers this flow as its final step; rerun `ragx-cli models` anytime to switch.
+
+Why not `jina-embeddings-v5-text-nano`? Excellent model, but its EuroBERT architecture needs
+Jina's llama.cpp fork — stock LM Studio can't load it (and the license is CC-BY-NC). See
+`research/lm-studio-model-download-api-and-curated-embedding-reranker-catalog-for-ragx-models-command.md`.
+
 ## Agent-first conventions
 
 - `--json` emits exactly one JSON document on stdout (versioned schemas: `ragx.query.v1`,
@@ -294,6 +347,8 @@ Full write-ups: `research/bge-m3-dense-q8-vs-nomic-q4-benchmark-2026-07-12-workt
 - Every chunk carries `file`, `line_start/line_end`, `byte_start/byte_end` — agents jump to the
   exact source location and read the full text themselves (JSON chunk text is truncated).
 - `--files-only` aggregates chunk scores per file (sum of top-3) — the mode coding agents use most.
+- `ragx-cli status` includes a `drift` object (`new`/`changed`/`deleted` file counts vs the index) —
+  agents check it to decide whether to run `ragx-cli index` before querying.
 - `ragx-cli query -` reads the query from stdin; `ragx-cli inspect chunk|file|neighbors|communities|community`
   debugs the graph.
 
@@ -313,7 +368,8 @@ questions; fall back to grep for exact identifiers.
   `file` + `line_start/line_end`; the JSON `text` is truncated, so read the file yourself
   for full context.
 - Fast mode (no LLM call, no cross-encoder): add `--no-expand --no-rerank`.
-- After adding or editing files: `ragx-cli index --changed` (cheap, hash-based).
+- After adding or editing files: `ragx-cli index` (incremental and cheap — hash-based).
+  `ragx-cli status --json` shows `drift` counts if you want to check staleness first.
 - stdout is exactly one JSON document; logs are on stderr.
   Exit codes: 0 = results, 1 = no results (not an error), 2 = error.
 - Why did this result appear? `ragx-cli query "..." --explain`.
@@ -373,9 +429,19 @@ detects the mismatch and asks you to run a full `ragx-cli index`.
 
 ### Reranker on restricted networks (no huggingface.co)
 
-The reranker model (`BAAI/bge-reranker-v2-m3`, ~2.3 GB) is downloaded from huggingface.co on
-first use. If your network blocks huggingface.co, ragx-cli degrades gracefully — queries run
-without reranking and a warning explains why — but you have three ways to get reranking working:
+With the default `sentence-transformers` engine, the reranker model
+(`BAAI/bge-reranker-v2-m3`, ~2.3 GB) is downloaded from huggingface.co on first use. If your
+network blocks huggingface.co, ragx-cli degrades gracefully — queries run without reranking
+and a warning explains why — and the easiest fix is to skip huggingface.co entirely:
+
+**Option 0 — the `llama-server` engine (recommended).** `ragx-cli models --rerank-engine
+llama-server` downloads a validated Q8_0 GGUF of the reranker **through LM Studio** (same
+download path as the embedding model) and reranks via llama.cpp's `llama-server --rerank`,
+which ragx starts and stops automatically. Requires a recent llama.cpp
+(`brew install llama.cpp`); no torch, no sentence-transformers, no huggingface.co. The GGUFs
+score within ~0.5 logit of the safetensors originals with identical ordering.
+
+Prefer the sentence-transformers engine? Three more ways:
 
 **Option A — copy the model and point `rerank.model` at the directory** (recommended).
 On any machine with access:
@@ -471,7 +537,7 @@ Checked features are built and validated per [the implementation plan](ragx-cli-
 unchecked ones are next up. Release history: [CHANGELOG.md](CHANGELOG.md).
 
 - [x] **CLI & storage**: typer CLI, SQLite schema, provider abstraction (Embedder/Generator/Reranker)
-- [x] **Baseline vector RAG**: discovery, chunking, embeddings, HNSW search, incremental `--changed`
+- [x] **Baseline vector RAG**: discovery, chunking, embeddings, HNSW search, incremental indexing
 - [x] **Similarity graph**: kNN edge construction, heat-propagation traversal, `inspect`, `--explain`
 - [x] **Quality & measurement**: multi-query/HyDE expansion, RRF fusion, cross-encoder rerank, `eval` harness
 - [x] **Communities**: Leiden detection over the edge list (index-time, read-only via status/inspect)
@@ -479,11 +545,13 @@ unchecked ones are next up. Release history: [CHANGELOG.md](CHANGELOG.md).
 - [x] **Committable config**: `ragx.toml` at the corpus root (0.3.0; was `.ragx/config.toml`), index data stays gitignored in `.ragx/`
 - [x] **Offline-friendly rerank**: graceful degradation + pre-seeding docs when huggingface.co is unreachable
 - [x] **Sub-chunk edges** (opt-in `graph.edge_source="subchunk"`): edge weight = max cosine over sentence-aligned sub-chunk pairs, for corpora with long multi-concept chunks
+- [x] **Incremental by default** (0.4.0): `index` hash-diffs, `--full` rebuilds, `status` reports corpus drift, chunking params are manifest-guarded
+- [x] **`ragx-cli models`**: curated embedding/reranker recommendation (quality tier + RAM detection), downloads via LM Studio's `lms get`, serving via llama-server or LM Studio
 - [ ] **Community labels**: name the Leiden communities so they're browsable without reading member chunks
 - [ ] **query --global** for corpus-level questions (answer from community summaries, not individual chunks)
 - [ ] **MCP server**: a second thin shell over `ragx.core` (the core/CLI split it needs is already enforced)
 - [ ] **[Temporal weighting](docs/feature-temporal-weighting.md)**: opt-in `--since`/`--until`/`--temporal recent|oldest`, date cascade filename/frontmatter → git → mtime
 - [x] **Release**: publish to PyPI as `ragx-cli` (plain `ragx` is name-blocked, too similar to an existing project) so `uvx ragx-cli` works out of the box
 
-Development: `uv sync --group dev && uv run pytest`. 177 tests; module contracts live in
+Development: `uv sync --group dev && uv run pytest`. 183 tests; module contracts live in
 `CONTRACTS.md` / `CONTRACTS-PHASE23.md`.

{ragx_cli-0.3.3 → ragx_cli-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ragx-cli"
-version = "0.3.3"
+version = "0.5.0"
 description = "Similarity-graph RAG CLI: index a corpus into a chunk-level embedding graph and query it"
 readme = "README.md"
 license = "MIT"

ragx_cli-0.5.0/research/lm-studio-model-download-api-and-curated-embedding-reranker-catalog-for-ragx-models-command.md

@@ -0,0 +1,119 @@
+# LM Studio model downloads + curated embedding/reranker catalog for `ragx-cli models`
+
+Date: 2026-07-15. Sources: lmstudio.ai official docs, lmstudio-ai/lms + lmstudio-python source,
+Hugging Face model cards, live tests on LM Studio 0.4.12 (this machine). Full agent reports
+summarized here; doc URLs at the bottom.
+
+## 1. How to trigger LM Studio downloads programmatically
+
+Three mechanisms exist; capabilities verified live on 0.4.12:
+
+| Mechanism | Unattended | Needs server up | Notes |
+|---|---|---|---|
+| `lms get <ref> --yes` | yes (exit 0/1, ANSI noise on stdout) | **no** | `<ref>` = catalog id (`google/embedding-gemma-300m`), search term, or full HF URL; `@quant` suffix; `--gguf`/`--mlx` filters. `--yes` auto-picks the hardware-preferred variant. |
+| REST `POST /api/v1/models/download` + poll `GET /api/v1/models/download/status/:job_id` | yes (clean JSON) | yes, ≥0.4.0 | structured errors (`model_not_found`); optional bearer auth off by default. |
+| Python SDK `lmstudio` (`client.repository.search_models → get_download_options → download`) | yes (blocking + progress callback) | yes | extra dependency; nicest API. |
+
+- `lms` ships with the app (linked at `~/.lmstudio/bin/lms` after first app run); detect via
+  `shutil.which("lms")` + that fallback path. `lms version` prints no semver (use the app plist
+  on macOS if ever needed).
+- Verify success + resolve the served model key via `lms ls --json`: entries carry `modelKey`,
+  `type` (`"embedding"`/`"llm"`), `format` (`gguf`/`mlx`), `path` (relative to models root),
+  `sizeBytes`, `architecture`.
+- Downloads land in `<lmstudio-home>/models/{publisher}/{repo}/…`; home found via
+  `~/.lmstudio-home-pointer` (internal, not contractual).
+- REST has **no** catalog-search endpoint; SDK does.
+
+**Hard limitation (verified live, twice):** LM Studio only downloads GGUF or MLX-converted
+artifacts. Plain transformers safetensors repos are rejected at resolution time —
+`lms get https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base --yes` → exit 1,
+"No download options available". A GGUF conversion of that reranker exists
+(`jolleyboy/gte-reranker-modernbert-base-GGUF`) and downloads fine, but sentence-transformers
+cannot load GGUF, and LM Studio has no /v1/rerank endpoint to serve it. **Conclusion: the
+reranker cannot ride the LM Studio download path in any usable form; it stays on the
+HF/sentence-transformers path** (huggingface_hub `snapshot_download` for pre-fetching).
+
+## 2. Requested models — verdicts
+
+### jina-embeddings-v5-text-nano (`jinaai/jina-embeddings-v5-text-nano`)
+Real (released 2026-02-18): 239M EuroBERT-based, 768 dims (Matryoshka 32–768), 8192 ctx,
+71.0 MTEB-en — best-in-class sub-500M. BUT:
+- **EuroBERT is not supported by stock llama.cpp** — Jina's cards say to use *their fork*;
+  LM Studio ships stock llama.cpp, so the GGUF repos (`…-nano-retrieval[-GGUF]`) won't load.
+- The MLX ports (`…-nano-mlx`) are custom-code (own `model.py`), not LM Studio-loadable.
+- License **CC-BY-NC-4.0** (non-commercial) — bad default for a recommendation catalog.
+→ Excluded from the catalog until upstream llama.cpp support lands. Revisit: watch
+ggml-org/llama.cpp for EuroBERT and ollama/ollama#14641.
+
+### gte-reranker-modernbert-base (`Alibaba-NLP/gte-reranker-modernbert-base`)
+149M ModernBERT cross-encoder, 8192 ctx, Apache-2.0, loads in `CrossEncoder(...)` natively
+(transformers ≥4.48, no trust_remote_code). BEIR 56.73; third-party Feb-2026 benchmark: tied
+best Hit@1 at 8× smaller than a 1.2B competitor, ahead of bge-reranker-v2-m3 — on **English**.
+It is English-only; ragx's eval corpus is EN+NL where bge-reranker-v2-m3's multilingual
+training is the reason the Dutch recall win exists. → Catalog **English tier** reranker;
+bge-reranker-v2-m3 stays the multilingual tier/default.
+
+## 3. Curated catalog (all embedding entries verified LM Studio-loadable)
+
+| Tier | Model | Why | Prefixes |
+|---|---|---|---|
+| fast | `lmstudio-community/embeddinggemma-300m-qat-GGUF` (curated catalog id `google/embedding-gemma-300m`) | 308M, 100+ langs, in LM Studio's own catalog; 2048 ctx / 768 dims cap quality | query `"task: search result \| query: "`, doc `"title: none \| text: "` |
+| balanced | BGE-M3 GGUF (`gaianet/bge-m3-GGUF`) | current ragx production model; ~100 langs, 1024 dims, 8192 ctx, MIT; benchmarked best on the ragx eval corpus | none |
+| best | `Qwen/Qwen3-Embedding-0.6B-GGUF` | Apache-2.0, 100+ langs, 32K ctx, MRL 1024 dims; top multilingual MTEB family; runs on upstream llama.cpp (early 0.3.16/0.3.17 LM Studio bugs since fixed) | query instruct prefix, doc none |
+| reranker (multilingual, default) | `BAAI/bge-reranker-v2-m3` | current default; only clean multilingual CrossEncoder without trust_remote_code | – |
+| reranker (English) | `Alibaba-NLP/gte-reranker-modernbert-base` | 149M, faster, beats bge-v2-m3 on EN benchmarks | – |
+
+Rejected alternatives: `gte-multilingual-reranker-base` (306M, good numbers, but needs
+trust_remote_code — security/UX friction), `jina-reranker-v2-base-multilingual` (NC license,
+trust_remote_code, 1024-tok window), `Qwen3-Reranker-0.6B` (causal-LM yes/no scoring, doesn't
+fit the CrossEncoder wrapper).
+
+All three embedding tiers are multilingual, so the "multilingual?" question only steers the
+reranker (EN → gte, otherwise bge-v2-m3). Spec detection: total RAM (<8 GB → force fast tier);
+macOS → let `lms get --yes` hardware-preselect MLX where LM Studio's catalog offers it
+(pinned GGUF URLs stay GGUF — they run on Metal fine; that's how bge-m3 Q8 runs today).
+
+## 4. Design consequences for `ragx-cli models`
+
+- Prefer `lms` CLI subprocess over REST/SDK: works with the server down, zero new deps.
+  Absent `lms` → graceful message pointing at lmstudio.ai.
+- After `lms get` exit 0: re-run `lms ls --json`, match the new entry by repo fragment, use its
+  `modelKey` as `embeddings.model`, and write the tier's doc/query prefixes to config.
+- Reranker pre-fetch: `huggingface_hub.snapshot_download` (ships with the `rerank` extra);
+  degrade to a printed hint when the extra isn't installed.
+- Changing `embeddings.model` invalidates the index (manifest guard) → command must end with a
+  loud `ragx-cli index --full` pointer, never auto-reindex.
+
+## Decision outcomes (2026-07-15, reviewed by Albert — folded from the session's DECISIONS.md)
+
+1. **jina-embeddings-v5-text-nano**: first excluded (stock LM Studio couldn't load EuroBERT),
+   then EuroBERT landed in upstream llama.cpp and the retrieval GGUF proved out end-to-end via
+   `llama-server --embedding` → **included as the 4th catalog option** (`--quality jina-nano`),
+   llama-server engine only, CC-BY-NC warning attached. LM Studio still downloads-but-misclassifies
+   it (`type: "llm"` → its /v1/embeddings refuses; candidate LM Studio bug report).
+2. **Reranker delivery**: HF-only at first (LM Studio rejects safetensors; ST can't read GGUF);
+   superseded the same day by `rerank.provider="llama-server"` — GGUF via LM Studio, served by
+   an auto-spawned `llama-server --rerank`. Q8_0 GGUFs validated ≈ safetensors (within ~0.5
+   logit, identical ordering — the all-negative bge logits on toy Dutch pairs match safetensors
+   exactly; model behavior, not conversion loss). Verified on brew llama.cpp 9960.
+3. **gte-reranker-modernbert-base (English-only) dropped entirely** per Albert — one multilingual
+   reranker (bge-reranker-v2-m3) keeps the measured Dutch-recall win; the multilingual question
+   left the CLI with it.
+4. **Downloads**: `lms get --yes` subprocess (works with the server down, zero deps) over REST/SDK.
+5. **MLX**: no hard pinning; `lms get --yes` hardware-preselects; explicit `gguf_ref`s exist per
+   entry because the llama-server engine can't load MLX.
+6. **Embedding tiers**: embeddinggemma-300m / bge-m3 / qwen3-embedding-0.6b confirmed by Albert.
+7. **Single-engine mode** (Albert's direction): `embeddings.provider="llama-server"` mirrors the
+   rerank engine (shared `providers/llama_process.py`, ports 9813/9814, atexit reaping); both
+   engine prompts default to llama-server when the binary exists — LM Studio becomes
+   downloader-only at query time. Trade-off: ~1–2 s cold-spawn per engine per process lifetime.
+8. **No auto-reindex on model switch** — loud `ragx-cli index --full` pointer instead; rebuild
+   spend stays explicit.
+
+## Doc URLs
+
+lmstudio.ai/docs/cli · /docs/cli/get · /docs/developer/rest · /docs/developer/rest/download ·
+/docs/developer/rest/download-status · /docs/python · github.com/lmstudio-ai/lms ·
+github.com/lmstudio-ai/lmstudio-python · huggingface.co/jinaai/jina-embeddings-v5-text-nano ·
+huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base · huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF ·
+lmstudio.ai/models/google/embedding-gemma-300m · huggingface.co/BAAI/bge-reranker-v2-m3

ragx_cli-0.5.0/skills/ragx-cli/SKILL.md

@@ -0,0 +1,90 @@
+---
+name: ragx-cli
+description: |
+  Query a ragx-cli semantic index instead of grepping. Use when the user asks a conceptual or
+  natural-language question about a corpus ("where is X discussed/decided?", "find notes about Y"),
+  when you are about to grep/rg for a concept (not an exact identifier) in a directory that contains
+  a `.ragx/` folder or `ragx.toml`, or when the user says `/ragx`, "search my notes", or "semantic
+  search". Covers running queries, interpreting the JSON output, and keeping the index fresh.
+---
+
+# ragx-cli — semantic search over an indexed corpus
+
+Indexes files into a chunk-level embedding similarity graph; answers queries via vector search +
+graph traversal + cross-encoder rerank. Agent-first: one JSON doc on stdout, logs on stderr,
+deterministic exit codes, byte-exact source locations.
+
+## When
+
+A corpus is ragx-indexed if its root has `ragx.toml` and `.ragx/`.
+
+- Use ragx-cli for conceptual, natural-language questions ("where do we discuss auth tradeoffs?",
+  cross-language queries). Full questions beat keywords.
+- Use grep/rg for exact identifiers and literal strings. ragx does not replace it.
+- Not on PATH: `uvx ragx-cli …` works (PyPI name is `ragx-cli`). Run from anywhere inside the
+  corpus — root is found upward.
+
+## Commands
+
+```bash
+ragx-cli status --json          # root, model, counts, and drift (new/changed/deleted vs index)
+ragx-cli query "question" --json --files-only   # which files are relevant — most-used mode
+ragx-cli query "..." --json --top 8             # ranked chunks with exact locations
+ragx-cli query "..." --json --no-expand --no-rerank   # fast: skip LLM call + cross-encoder
+echo "..." | ragx-cli query - --json            # long/multiline query from stdin
+ragx-cli index                  # incremental reindex (hash-based, cheap) — the default
+ragx-cli index --full           # full rebuild: re-chunks + re-embeds everything
+```
+
+Other query flags: `--no-graph`, `--hops N`, `--explain`.
+
+## Output
+
+- stdout is exactly one JSON doc (`ragx.query.v1` / `ragx.files.v1` / `ragx.status.v1`); logs go
+  to stderr. Never parse stderr.
+- Exit codes: `0` results, `1` success-but-empty (report "nothing found", don't retry as failure),
+  `2` error.
+- Chunk results carry `file`, `line_start/end`, `byte_start/end`, `score`, `breakdown`. JSON
+  `text` is truncated — read the file at the line range for full context.
+- Typical flow: `--files-only` to pick files → `--top 8` for locations → read files at ranges.
+
+## Freshness
+
+- `status` includes `drift` counts; nonzero means the index is stale → run `ragx-cli index`
+  (incremental, safe, converges even after include/exclude glob changes).
+- Changing `embeddings.model`, `chunking.*`, or `graph.edge_source` invalidates the index:
+  commands fail loud on the manifest mismatch, and the fix is `ragx-cli index --full` —
+  re-embeds everything, so confirm before running on a large corpus.
+- `ragx-cli models` picks + downloads an embedding/reranker combo. All files download via
+  LM Studio; serving is per engine — `--embed-engine llama-server|lm-studio` and
+  `--rerank-engine llama-server|sentence-transformers`. With both on `llama-server`
+  (the default when llama.cpp is installed), ragx auto-spawns/reaps the servers and
+  needs neither LM Studio nor huggingface.co at query time. Other flags: `--quality
+  fast|balanced|best|jina-nano`, `--yes`, `--dry-run`, `--json`.
+
+## Why did/didn't a result appear?
+
+- `ragx-cli query "..." --explain` — per-result trace: seed, graph edge, hop.
+- `ragx-cli inspect neighbors <chunk_id>` / `inspect chunk <id>` / `inspect file <path>` /
+  `inspect communities` / `inspect community <id>`.
+
+## Gotchas
+
+- The embedding provider must be reachable (default: LM Studio at `localhost:1234`; Ollama and
+  any OpenAI-compatible endpoint via config). On connection error, report it and suggest starting
+  the provider — never silently fall back to grep.
+- Query expansion degrades gracefully (LLM down → no-op); embeddings do not.
+- `--changed` is a deprecated alias — incremental is the default now.
+- The corpus may be someone's personal notes repo: never `git add`/commit there. Querying and
+  `ragx-cli index` are safe.
+
+## New corpus (only when asked)
+
+```bash
+ragx-cli init
+ragx-cli config set embeddings.model <model-id>   # match what the provider serves
+ragx-cli index
+```
+
+Provider recipes are in the ragx-cli README. Check `ragx-cli status` and
+`curl -s http://localhost:1234/v1/models` before guessing model ids.