cfunklabs-rag-react-docs 0.1.2__py3-none-any.whl

cfunklabs_rag_react_docs-0.1.2.dist-info/METADATA

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: cfunklabs-rag-react-docs
+Version: 0.1.2
+Summary: Retrieval-only MCP server over the indexed React documentation, with a prebuilt index downloaded on first run.
+Project-URL: Homepage, https://github.com/cfunklabs/rag-react-docs
+Project-URL: Repository, https://github.com/cfunklabs/rag-react-docs
+Author: Corey Funk
+License: MIT
+Keywords: chromadb,documentation,mcp,rag,react,retrieval
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Documentation
+Requires-Python: >=3.14
+Requires-Dist: chromadb>=1.5.9
+Requires-Dist: mcp>=1.28.1
+Requires-Dist: platformdirs>=4.0.0
+Description-Content-Type: text/markdown
+
+# cfunklabs-rag-react-docs
+
+Backend for the RAG demo, built with LangChain, LangGraph, Anthropic Claude, and ChromaDB.
+It also ships a retrieval-only MCP server, published to PyPI as `cfunklabs-rag-react-docs`,
+that serves grounding context from the indexed React documentation (see [MCP server](#mcp-server)).
+
+## Prerequisites
+
+- [Python 3.14+](https://www.python.org/downloads/)
+- [uv](https://docs.astral.sh/uv/getting-started/installation/) — used for dependency management and running the project
+- An [Anthropic API key](https://console.anthropic.com/)
+
+## Setup
+
+All commands should be run from the `backend` directory.
+
+### 1. Install dependencies
+
+```bash
+uv sync
+```
+
+### 2. Configure environment variables
+
+Copy the sample env file and add your Anthropic API key:
+
+```bash
+cp .env.sample .env
+```
+
+Open `.env` and set your key:
+
+```
+ANTHROPIC_API_KEY=your_api_key_here
+```
+
+### 3. Initialize the vector database
+
+Create the ChromaDB collection used to store document embeddings:
+
+```bash
+uv run src/utils/init_db.py
+```
+
+This creates a persistent ChromaDB store under `rag_datastore/` and a collection
+named after `tool.rag_db.rag_doc_collection_name` in [pyproject.toml](pyproject.toml).
+The `rag_datastore/` directory is gitignored.
+
+### 4. Fetch the React docs dataset
+
+The corpus is a local mirror of the [React documentation](https://react.dev/) markdown files. Download it with:
+
+```bash
+uv run src/utils/fetch_react_docs.py
+```
+
+This script:
+
+1. Fetches the index at [react.dev/llms.txt](https://react.dev/llms.txt)
+2. Follows every linked `https://react.dev/*.md` URL
+3. Saves each page under `docs/` at the **repository root**, using the index heading structure as directories and each file's frontmatter `title` as the filename
+
+Example output path:
+
+```
+docs/API Reference/React/Components/Built-in React Components.md
+```
+
+The `docs/` directory is gitignored — run the script locally after cloning and re-run it anytime to refresh the dataset.
+
+## Running
+
+### Startup check
+
+Run `main.py` to verify your environment is configured correctly. It prints installed package versions and performs a live health check against the Claude API:
+
+```bash
+uv run main.py
+```
+
+A passing run looks like:
+
+```
+Versions:
+  - langchain_core_version: x.x.x
+  - langgraph_version: x.x.x
+  - langchain_anthropic_version: x.x.x
+
+Checking LLM service health... PASSED
+```
+
+If the health check fails, confirm that `ANTHROPIC_API_KEY` is set correctly in your `.env` file.
+
+### Ingesting documents
+
+Chunk every Markdown file under `docs/`, embed the chunks, and upsert them into the vector store:
+
+```bash
+uv run main.py
+```
+
+To process a single file (useful while iterating on chunking), pass `--md_file_path`. Add
+`--evaluate_chunking` to write LLM-as-judge quality reports to `evals/results`, or
+`--print_chunks` to dump each chunk to stdout.
+
+### Querying
+
+Ask a question against the ingested docs. The query pipeline is a LangGraph graph
+(`retrieve` -> `generate`) that embeds your question with the same model used at ingestion,
+retrieves the most similar chunks from ChromaDB, and has Claude answer using only that
+context. The answer streams token-by-token and is followed by the cited sources:
+
+```bash
+uv run query.py "How does memo work?"
+```
+
+Options:
+
+- `--k N` — number of chunks to retrieve (default `top_k` in [pyproject.toml](pyproject.toml)).
+- `--no-stream` — wait for the full answer instead of streaming tokens.
+- `--show-scores` — show the retrieval distance for each cited source.
+- `-v`, `--verbose` — show diagnostic output (the preflight datastore and LLM health checks). Hidden by default.
+
+Retrieval and generation settings live under `[tool.rag_query]` in [pyproject.toml](pyproject.toml)
+(`top_k` and `generation_model`). Run `uv run main.py` first — querying requires a populated
+collection.
+
+### MCP server
+
+In addition to the CLI, the retrieval pipeline is exposed as an [MCP](https://modelcontextprotocol.io/)
+server over stdio, so MCP clients (Cursor, Claude Desktop, etc.) can pull grounding context
+directly. It exposes a single **retrieval-only** tool:
+
+- `search_docs(question, k?)` — embeds the question with the same model used at ingestion,
+  retrieves the most similar chunks from ChromaDB, and returns each chunk's `source` label,
+  `content`, and retrieval `distance`. The client LLM generates the answer from those chunks,
+  so no Anthropic key is needed to run the server.
+
+#### For end users (published package)
+
+The server is published to PyPI as **`cfunklabs-rag-react-docs`**. End users don't clone the
+repo or run the ingestion pipeline — the prebuilt index (~34 MB) is downloaded from a GitHub
+Release and cached on first run. Just register it with your MCP client:
+
+```json
+{
+  "mcpServers": {
+    "rag-react-docs": {
+      "command": "uvx",
+      "args": ["cfunklabs-rag-react-docs"]
+    }
+  }
+}
+```
+
+The first launch needs network access to fetch the index; subsequent runs read from the local
+cache (`platformdirs` cache dir) and work offline. Optional environment overrides: `RAG_TOP_K`,
+`RAG_COLLECTION_NAME`, `RAG_INDEX_URL`, and `RAG_DATASTORE_DIR`.
+
+#### Local development
+
+Run the dev server from the `backend` directory (so `pyproject.toml` resolves) against the
+locally-built `rag_datastore`:
+
+```bash
+uv run mcp_server.py
+```
+
+Run standalone this way, the server prints a short startup banner to stderr and then blocks
+silently by design — the stdio transport reserves stdout for the JSON-RPC protocol, so it
+waits for a client to connect rather than logging. Running it directly is mainly a smoke test;
+press Ctrl+C to stop.
+
+Run `uv run main.py` first — the dev server needs a populated collection. For interactive
+testing, launch the MCP Inspector with `uv run mcp dev mcp_server.py`.
+
+### Publishing to PyPI
+
+The published package (`cfunklabs-rag-react-docs`) contains only the retrieval + MCP server
+(the import package `rag_react_docs` under `src/`). Ingestion/query tooling and `src/utils/*`
+are dev-only and excluded from the wheel.
+
+Two artifacts get published: the Python package (to PyPI) and the prebuilt index (to a GitHub
+Release). They version independently — the index version is pinned as `INDEX_VERSION` in
+[src/rag_react_docs/config.py](src/rag_react_docs/config.py).
+
+1. Build and upload the index archive (after `uv run main.py` has populated `rag_datastore`):
+
+```bash
+uv run scripts/build_index_archive.py
+gh release create index-19-2-v1 dist/rag-index-19-2-v1.tar.gz dist/rag-index-19-2-v1.tar.gz.sha256
+```
+
+2. Build and publish the package (test on TestPyPI first):
+
+```bash
+uv build                                   # -> dist/ wheel + sdist (only rag_react_docs)
+uv publish --publish-url https://test.pypi.org/legacy/   # TestPyPI dry run
+uv publish                                 # PyPI
+```
+
+The index version follows the standard `index-<react-version>-v<incremental>` (e.g.
+`index-19-2-v1`), composed in [src/rag_react_docs/config.py](src/rag_react_docs/config.py) from
+`REACT_VERSION` and `INDEX_REVISION`. Bump `REACT_VERSION` when re-fetching the docs for a new
+React release, and bump `INDEX_REVISION` for re-chunk or embedding-model changes within the same
+React version. Either bump changes the release tag/asset name and cache path, so clients pull a
+fresh, compatible index instead of reusing a stale cache — re-release the archive under the new
+`index-<react-version>-v<incremental>` tag.

cfunklabs_rag_react_docs-0.1.2.dist-info/RECORD

@@ -0,0 +1,10 @@
+rag_react_docs/__init__.py,sha256=nupgbeHjZH1ctdSVHnuX6mk7daPKoHGlLov9jDd0mwc,351
+rag_react_docs/config.py,sha256=d8nR65kfBkpkbrdroAYz5qQFfqGSCAXiXBSp39bwduw,2289
+rag_react_docs/datastore.py,sha256=lGB7EvctHa-cVayY3-YSIPaeqfHaQjtVleHNylNYnrA,4540
+rag_react_docs/retrieval.py,sha256=wtxZ7SqbpTApKeTT_5A5eToij3Qw9LvAsC8XGB89JA8,1879
+rag_react_docs/server.py,sha256=Z0xK7KVy06nVOy-w715DSykenMULpBYbc1gPfW8dSSg,3146
+rag_react_docs/source_label.py,sha256=-4q3yknGmI8eT_CBZaaOABPXSPVXdwG3WiyOI3Vm1jA,910
+cfunklabs_rag_react_docs-0.1.2.dist-info/METADATA,sha256=4vNJ61N18foC6ZgwB0DhUQ3A620qlZfpIsoVvEnFE68,8239
+cfunklabs_rag_react_docs-0.1.2.dist-info/WHEEL,sha256=mffPy8wBnZQn2VnJUU5jE99KsxaSfiyMHV9Yt0aLVxs,87
+cfunklabs_rag_react_docs-0.1.2.dist-info/entry_points.txt,sha256=gANoXAUJMNFOzlDJH_S93cJXYkUzYJI6ldkRtf8LXk8,72
+cfunklabs_rag_react_docs-0.1.2.dist-info/RECORD,,

cfunklabs_rag_react_docs-0.1.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.30.1
+Root-Is-Purelib: true
+Tag: py3-none-any

cfunklabs_rag_react_docs-0.1.2.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+cfunklabs-rag-react-docs = rag_react_docs.server:main

rag_react_docs/__init__.py

@@ -0,0 +1,8 @@
+"""rag-react-docs: a retrieval-only MCP server over the indexed React documentation.
+
+Distributed on PyPI as `cfunklabs-rag-react-docs`; imported as `rag_react_docs`. The prebuilt
+ChromaDB index is downloaded from a GitHub Release on first run (see `datastore.py`), so end
+users never run the ingestion pipeline themselves.
+"""
+
+__version__ = "0.1.2"

rag_react_docs/config.py

@@ -0,0 +1,49 @@
+"""Runtime configuration for the published package.
+
+Unlike the dev tooling (which reads `pyproject.toml`), the installed package has no repo
+checkout to read from, so defaults are baked in here and overridable via environment variables.
+This keeps the wheel self-contained while still letting power users retarget the collection,
+retrieval depth, index location, or download URL from their MCP client config.
+"""
+
+import os
+from pathlib import Path
+
+import platformdirs
+
+
+# The all-MiniLM-L6-v2 vectors were ingested into this collection; the name must match the
+# collection stored inside the downloaded index archive.
+COLLECTION_NAME = os.environ.get("RAG_COLLECTION_NAME", "rag_doc_collection")
+
+DEFAULT_TOP_K = int(os.environ.get("RAG_TOP_K", "5"))
+
+# Index version standard: index-<react-version>-v<incremental> (e.g. index-19-2-v1).
+# REACT_VERSION is the React docs version with dots as dashes; bump it when the corpus is
+# re-fetched for a new React release. INDEX_REVISION bumps for re-chunk/embedding changes
+# within the same React version. INDEX_VERSION is part of both the cache path and the release
+# asset name, so any bump forces a fresh download rather than reusing a stale cached index.
+REACT_VERSION = "19-2"
+INDEX_REVISION = "v1"
+INDEX_VERSION = f"{REACT_VERSION}-{INDEX_REVISION}"
+
+# The prebuilt index is published as a GitHub Release asset. A sibling `<archive>.sha256` file
+# is fetched alongside it to verify the download before extraction.
+_DEFAULT_INDEX_URL = (
+    "https://github.com/cfunklabs/rag-react-docs/releases/download/"
+    f"index-{INDEX_VERSION}/rag-index-{INDEX_VERSION}.tar.gz"
+)
+INDEX_URL = os.environ.get("RAG_INDEX_URL", _DEFAULT_INDEX_URL)
+
+
+def datastore_dir() -> Path:
+    """Return the directory that holds (or will hold) the extracted ChromaDB index.
+
+    Defaults to a per-user cache directory namespaced by index version so multiple versions
+    can coexist and a version bump never collides with an older cached index. Overridable via
+    `RAG_DATASTORE_DIR` (e.g. to point at a repo-local store during development).
+    """
+    override = os.environ.get("RAG_DATASTORE_DIR")
+    if override:
+        return Path(override).expanduser()
+    return Path(platformdirs.user_cache_dir("cfunklabs-rag-react-docs")) / "index" / INDEX_VERSION

rag_react_docs/datastore.py

@@ -0,0 +1,125 @@
+"""Download-on-first-run access to the prebuilt ChromaDB index.
+
+The published package ships no vectors: the ~34 MB index lives as a GitHub Release asset and is
+fetched + cached the first time the server needs it. Subsequent runs read straight from the
+cache and work offline. Only the standard library is used for the download so the wheel stays
+dependency-light (no httpx/requests).
+"""
+
+import hashlib
+import os
+import shutil
+import tarfile
+import tempfile
+import urllib.request
+from pathlib import Path
+
+import chromadb
+
+from .config import COLLECTION_NAME, INDEX_URL, datastore_dir
+
+
+# Sentinel file that marks a fully-extracted index. We only treat the cache as populated when
+# this exists, so an interrupted download/extraction never leaves a half-written index that
+# looks valid on the next run.
+_MARKER = "chroma.sqlite3"
+
+
+def _download(url: str, dest: Path) -> None:
+    with urllib.request.urlopen(url) as response, open(dest, "wb") as out:
+        shutil.copyfileobj(response, out)
+
+
+def _sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with open(path, "rb") as f:
+        for block in iter(lambda: f.read(1024 * 1024), b""):
+            digest.update(block)
+    return digest.hexdigest()
+
+
+def _verify_checksum(archive: Path, url: str) -> None:
+    """Verify `archive` against its published `<url>.sha256`, if one is available.
+
+    The checksum file contents may be a bare hex digest or the common `"<digest>  <filename>"`
+    format; we take the first whitespace-delimited token either way. A missing checksum file is
+    tolerated (some releases may not publish one) but a present-and-mismatched one is fatal.
+    """
+    try:
+        with urllib.request.urlopen(url + ".sha256") as response:
+            expected = response.read().decode().strip().split()[0]
+    except Exception:
+        return
+
+    actual = _sha256(archive)
+    if actual != expected:
+        raise RuntimeError(
+            f"Index checksum mismatch: expected {expected}, got {actual}. "
+            f"Refusing to use a corrupted download from {url}."
+        )
+
+
+def ensure_index() -> Path:
+    """Return the local index directory, downloading + extracting it on first use.
+
+    Extraction is staged in a sibling temp directory and atomically renamed into place so a
+    concurrent or interrupted run can't expose a partially-written index.
+    """
+    dest = datastore_dir()
+    if (dest / _MARKER).exists():
+        return dest
+
+    dest.parent.mkdir(parents=True, exist_ok=True)
+
+    with tempfile.TemporaryDirectory(dir=str(dest.parent)) as tmp:
+        tmp_path = Path(tmp)
+        archive = tmp_path / "index.tar.gz"
+
+        _download(INDEX_URL, archive)
+        _verify_checksum(archive, INDEX_URL)
+
+        extract_dir = tmp_path / "extracted"
+        extract_dir.mkdir()
+        with tarfile.open(archive) as tar:
+            _safe_extractall(tar, extract_dir)
+
+        # Support archives that either contain the store files at the top level or nest them
+        # under a single wrapping directory.
+        root = extract_dir
+        if not (root / _MARKER).exists():
+            subdirs = [p for p in root.iterdir() if p.is_dir()]
+            if len(subdirs) == 1 and (subdirs[0] / _MARKER).exists():
+                root = subdirs[0]
+        if not (root / _MARKER).exists():
+            raise RuntimeError(
+                f"Downloaded index archive from {INDEX_URL} did not contain '{_MARKER}'."
+            )
+
+        # Another process may have won the race and populated dest while we were downloading.
+        if (dest / _MARKER).exists():
+            return dest
+        if dest.exists():
+            shutil.rmtree(dest)
+        os.replace(root, dest)
+
+    return dest
+
+
+def _safe_extractall(tar: tarfile.TarFile, dest: Path) -> None:
+    """Extract `tar` into `dest`, rejecting members that would escape the target directory.
+
+    Guards against path-traversal ("tar slip") in a downloaded archive by resolving each
+    member's destination and confirming it stays within `dest`.
+    """
+    dest_root = dest.resolve()
+    for member in tar.getmembers():
+        target = (dest / member.name).resolve()
+        if not (target == dest_root or dest_root in target.parents):
+            raise RuntimeError(f"Unsafe path in index archive: {member.name!r}")
+    tar.extractall(dest)
+
+
+def get_rag_collection():
+    """Return the persistent ChromaDB collection, ensuring the index is present first."""
+    client = chromadb.PersistentClient(path=str(ensure_index()))
+    return client.get_collection(name=COLLECTION_NAME)

rag_react_docs/retrieval.py

@@ -0,0 +1,48 @@
+"""Retrieval side of the RAG pipeline for the published package.
+
+Embeds the question with the *same* embedding function used at ingestion (Chroma's
+DefaultEmbeddingFunction, all-MiniLM-L6-v2) and runs a nearest-neighbour query against the
+downloaded ChromaDB collection. Using an identical embedding model is critical: the stored
+vectors were produced by that same model, so a different embedder would make cosine distances
+meaningless. Results are returned as plain dicts so the package needs no langchain-core.
+"""
+
+from chromadb.utils import embedding_functions
+
+from .datastore import get_rag_collection
+from .source_label import format_source_label
+
+
+# Constructing the embedding function loads the all-MiniLM-L6-v2 model, so build it once at
+# import time rather than per query.
+_embedding_fn = embedding_functions.DefaultEmbeddingFunction()
+
+
+def retrieve_chunks(question: str, k: int) -> list[dict]:
+    """Return the `k` chunks most semantically similar to `question`.
+
+    Each result is a dict with `source` (provenance label), `content` (raw chunk text), and
+    `distance` (retrieval distance; lower is more similar).
+    """
+    collection = get_rag_collection()
+
+    query_embedding = _embedding_fn([question])
+    results = collection.query(
+        query_embeddings=query_embedding,
+        n_results=k,
+        include=["documents", "metadatas", "distances"],
+    )
+
+    # Chroma returns a list-of-lists keyed by query; we issued a single query so index [0].
+    documents = results.get("documents", [[]])[0]
+    metadatas = results.get("metadatas", [[]])[0]
+    distances = results.get("distances", [[]])[0]
+
+    return [
+        {
+            "source": format_source_label(dict(metadata or {})),
+            "content": content,
+            "distance": distance,
+        }
+        for content, metadata, distance in zip(documents, metadatas, distances)
+    ]

rag_react_docs/server.py

@@ -0,0 +1,83 @@
+"""MCP server exposing the RAG retrieval pipeline over stdio.
+
+Published as the `cfunklabs-rag-react-docs` console script (`uvx cfunklabs-rag-react-docs`). It
+exposes a single `search_docs` tool that performs *retrieval only* against the downloaded
+ChromaDB collection and returns the top-k chunks with their source/heading labels. The
+consuming LLM (Cursor, Claude Desktop, etc.) ingests those chunks and generates its own
+grounded answer, so no Anthropic API key or generation stack is needed server-side.
+"""
+
+import sys
+
+from mcp.server.fastmcp import FastMCP
+
+from .config import DEFAULT_TOP_K
+from .datastore import get_rag_collection
+from .retrieval import retrieve_chunks
+
+
+mcp = FastMCP("rag-react-docs")
+
+
+def _collection_is_empty() -> bool:
+    try:
+        return get_rag_collection().count() == 0
+    except Exception:
+        # Treat a missing/uninitialized/failed-download collection the same as an empty one.
+        return True
+
+
+@mcp.tool()
+def search_docs(question: str, k: int = DEFAULT_TOP_K) -> list[dict]:
+    """Search the indexed React documentation and return the most relevant chunks.
+
+    Args:
+        question: A natural-language question to search the React docs for.
+        k: How many chunks to return (defaults to `DEFAULT_TOP_K`).
+
+    Returns a list of results ordered by relevance. Each item has:
+        - source:   a human-readable provenance label (file path > heading path)
+        - content:  the raw chunk text to ground an answer on
+        - distance: the retrieval distance (lower is more similar)
+    """
+    if _collection_is_empty():
+        return [
+            {
+                "source": "rag-react-docs",
+                "content": (
+                    "The documentation index is empty or could not be loaded. "
+                    "Check network access on first run so the index can be downloaded."
+                ),
+                "distance": None,
+            }
+        ]
+
+    return retrieve_chunks(question, k)
+
+
+def main() -> None:
+    """Console-script entry point: start the MCP server on stdio."""
+    # Human-facing messages must go to stderr: the stdio transport reserves stdout for the
+    # JSON-RPC protocol, so anything printed there would corrupt the stream.
+    print(f"[rag-react-docs] MCP server starting on stdio (top_k={DEFAULT_TOP_K}).", file=sys.stderr)
+    print("[rag-react-docs] Ensuring documentation index is available...", file=sys.stderr)
+    try:
+        if _collection_is_empty():
+            print(
+                "[rag-react-docs] Warning: index empty or unavailable -- check network access.",
+                file=sys.stderr,
+            )
+        else:
+            print("[rag-react-docs] Index ready.", file=sys.stderr)
+    except Exception as exc:  # pragma: no cover - defensive; _collection_is_empty swallows most
+        print(f"[rag-react-docs] Warning: could not verify index: {exc}", file=sys.stderr)
+
+    print("[rag-react-docs] Ready. Press Ctrl+C to stop.", file=sys.stderr)
+    try:
+        mcp.run()
+    except KeyboardInterrupt:
+        print("\n[rag-react-docs] Shutting down.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

rag_react_docs/source_label.py

@@ -0,0 +1,26 @@
+"""Build human-readable provenance labels from a chunk's stored metadata.
+
+Ported from the dev tooling's `format_source_label`. The installed package has no repo `docs/`
+directory, so the relative-path shortening will not match; in that case we fall back to the
+source file's basename, still prefixed onto the stored heading hierarchy.
+"""
+
+from pathlib import Path
+
+
+def format_source_label(metadata: dict) -> str:
+    """Return a label like "memo.md > Reference > memo(Component, arePropsEqual?)".
+
+    Combines the source file (shortened to its basename) with the stored heading hierarchy.
+    """
+    source = metadata.get("source", "unknown source")
+    source = Path(source).name
+
+    headings = [
+        metadata[field]
+        for field in ("Header 1", "Header 2", "Header 3")
+        if metadata.get(field)
+    ]
+    if headings:
+        return f"{source} > {' > '.join(headings)}"
+    return source