rag-ladder 0.1.0__tar.gz

rag_ladder-0.1.0/PKG-INFO

@@ -0,0 +1,9 @@
+Metadata-Version: 2.4
+Name: rag-ladder
+Version: 0.1.0
+Summary: 6 RAG recipes, ladder-first: BM25 full-text, LLM query rewriting, hybrid retrieval on SQLite FTS5
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.26
+Provides-Extra: docs
+Requires-Dist: pypdf>=4; extra == "docs"
+Requires-Dist: python-docx>=1; extra == "docs"

rag_ladder-0.1.0/README.md

@@ -0,0 +1,217 @@
+# rag-ladder — the 6 RAG recipe book, ladder-first
+
+The article's thesis: most stacks jump to embeddings + vector DB + rerankers while
+users just want the doc that says "how to reset my password". So this code
+climbs the ladder and stops where the problem is already solved.
+
+**Inspiration.** This code implements the recipe book from Rafael Pierre's
+article ["RAG Is Simpler Than You Think"](https://www.lighthousenewsletter.com/p/rag-is-simpler-than-you-think)
+(Lighthouse Newsletter, Jun 10, 2026). The six recipes — BM25, agentic query
+rewriting, hybrid, on-the-fly, hot/cold, and full pre-embedding — and the
+decision tree that picks between them come directly from that article. The
+guiding principle: *don't build the 5% solution for a 60% problem.*
+
+**Start here. Four commands do the whole job:**
+
+    rag-ladder init          # choose your model backend — writes rag.json
+    rag-ladder add <folder>  # index PDFs, DOCX, markdown, text, images (vision), video
+    rag-ladder serve         # open the browser UI, ask questions, get cited answers
+    rag-ladder doctor        # when something doesn't work
+
+`rag-ladder ask "..."` answers from your documents with citations, and says
+`NOT_FOUND_IN_CORPUS` when your documents don't contain the answer. It never
+invents one — that is the whole point for legal and contract work.
+
+## Install
+
+    uv sync                    # core: numpy only, stdlib search
+    uv sync --extra docs       # adds pypdf + python-docx, for PDF/DOCX folders
+    uv tool install -e .       # then `rag-ladder` is a standalone command
+
+No new pip dependency for images: each photo is read through your LLM's vision
+capability. Video needs the `ffmpeg` system binary (no pip) — see "Image and video".
+
+Without `--extra docs`, `rag-ladder add` still works for `.txt .md .csv .json .html`
+and tells you the exact install command when it meets a PDF.
+
+## Image and video
+
+Photos in a listing folder are usually the actual content. `rag-ladder add` sends each
+image through `llm()` as an OpenAI `image_url` content block and stores the
+transcription plus a scene description for search — the citation is the file path,
+so an answer quoting a photo traces back to it. No extra pip dependency; your
+`llm_model` just needs vision.
+
+- **Images** (`.jpg .jpeg .png .webp .gif .bmp .tif .tiff`) — need a **vision-capable**
+  `llm_model` (e.g. `gpt-4o-mini`). A text-only model is skipped with a clear
+  message and never filled with guessed text.
+- **Video** (`.mp4 .mov .avi .mkv .webm .flv .m4v`) — needs `ffmpeg`. Up to
+  `MAX_VIDEO_FRAMES` (12) evenly spaced frames are extracted, each described via
+  vision and joined. This is the most expensive part of `rag-ladder add`: one vision
+  call per frame.
+- **Size** — a 20 MB per-image cap (`MAX_IMAGE_BYTES`); resize oversized files
+  before indexing.
+
+`rag-ladder doctor` reports whether `ffmpeg` is present and which `llm_model` is resolved
+— confirm that model has vision before adding photo folders.
+
+## Which model to use
+
+`rag-ladder init` offers four backends:
+
+| preset | endpoint | model |
+|---|---|---|
+| `local` | `http://127.0.0.1:8888` | whatever the endpoint serves |
+| `ollama` | `http://127.0.0.1:11434` | `llama3.1:8b` |
+| `openrouter` | `https://openrouter.ai/api` | `openai/gpt-4o-mini` |
+| `openai` | `https://api.openai.com` | `gpt-4o-mini` |
+
+Anything OpenAI-compatible works: set `llm_base` and `llm_model` in `rag.json`.
+Cloud providers need a key — `rag-ladder init --preset openrouter --key sk-or-...`
+writes it into `rag.json` with mode `0600`, or export `RAG_API_KEY`.
+
+`rag-ladder doctor` tells you what your endpoint actually serves. If it reports a
+reasoning model, recipe 2 (query rewriting) costs far more than the article's
+$0.001/query — point `llm_model` at a small non-reasoning model for the real
+price. `rag-ladder ask` works with no model at all: it returns matching passages and
+says so instead of guessing.
+
+## CLI reference
+
+    rag-ladder add docs/ contracts.pdf       # index files or folders (recursive)
+    rag-ladder add docs/                     # re-run: unchanged files are skipped
+    rag-ladder ask "how long do refunds take?"
+    rag-ladder ask --mode legal "what is the notice period?"   # quotes clauses, never infers
+    rag-ladder search "invoice #12345"       # recipe 1, raw BM25 hits
+    rag-ladder hybrid "alternatives to X"    # recipe 3/4
+    rag-ladder multi "read csv, clean, plot"
+    rag-ladder hot "invoice paid"            # recipe 5
+    rag-ladder pre                           # recipe 6: build vectors
+    rag-ladder pre "reset password"          # recipe 6: query them
+    rag-ladder stats                         # corpus and index sizes
+    rag-ladder glossary                      # write the glossary template
+    rag-ladder recommend --qpd 500 --churn 2 --complaint "can't find docs"
+
+Add `--json` to `search ask hybrid multi hot pre stats` for scripting.
+Exit codes: `0` ok, `1` runtime failure, `2` bad input or missing extractor,
+`3` empty corpus (the message tells you to run `rag-ladder add`).
+
+## Decision tree
+
+`rag-ladder recommend` picks the next recipe for you. It needs two numbers and a
+complaint:
+
+    rag-ladder recommend --qpd 500 --churn 2 --complaint "can't find docs"
+
+| Flag | Meaning | How to estimate |
+|---|---|---|
+| `--qpd` | **Queries per day** — how many searches your users make | Count `rag-ladder ask`/`rag-ladder search` calls in a day, or estimate from traffic |
+| `--churn` | **Corpus churn %/day** — how many documents change daily | Adding 50 docs to a 500-doc corpus = `--churn 10` |
+| `--complaint` | What's wrong with the BM25 results | "can't find", "not great", "okay" |
+
+The tree's answer tells you the next rung to build:
+
+| Complaint | Condition | Next recipe |
+|---|---|---|
+| "can't find" | vocabulary mismatch | **2** — LLM query rewriting |
+| "not great" / "okay" | low latency OK, low churn | **3** — hybrid BM25 + embedding rerank |
+| "not great" | churn >10%/day | **4** — embed candidates on the fly |
+| "not great" | Pareto access (hot 20%) | **5** — hot/cold tiers |
+| "not great" | >100K docs, >10K qpd, ML team | **6** — full pre-embedding |
+| (satisfied) | users happy | **stop** — ship features |
+
+Run it after `rag-ladder add`, before you write any code. It returns `None` when
+recipe 1 + 2 are enough — that's the target for ~60% of systems.
+
+## Config
+
+`rag-ladder init` writes `rag.json` (see `rag.example.json`). Every key is optional.
+
+| key | default | what it does |
+|---|---|---|
+| `llm_base` | `http://127.0.0.1:8888` | OpenAI-compatible endpoint |
+| `llm_model` | endpoint's own | model name |
+| `llm_api_key` | — | sent as `Authorization: Bearer` |
+| `llm_reasoning` | `auto` | `none` sends `reasoning_effort=none` — for endpoints running a reasoning model, where a trivial rewrite otherwise burns ~20s thinking |
+| `embed_base` | — | `/embeddings` endpoint; blank = hashed fallback |
+| `embed_model` | `hash-bow-512` | embedding model name |
+| `embed_max_chars` | `4000` | truncation cap for long documents |
+| `db` | `rag.db` | SQLite file |
+| `corpus` | `.` | root folder document ids are relative to |
+| `glossary` | `glossary.txt` | domain terms never rewritten |
+| `answer_mode` | `general` | `general` or `legal` (stricter, quotes clauses) |
+| `serve_host` / `serve_port` | `127.0.0.1` / `8765` | browser UI bind |
+
+Resolution order everywhere: **CLI flag > env var > `rag.json` > default.**
+Env vars: `RAG_CONFIG`, `RAG_LLM_BASE`, `RAG_LLM_MODEL`, `RAG_API_KEY`,
+`RAG_LLM_REASONING`, `RAG_EMBED_BASE`, `RAG_EMBED_MODEL`, `RAG_EMBED_MAX_CHARS`,
+`RAG_DB`, `RAG_CORPUS`, `RAG_GLOSSARY`, `RAG_SERVE_HOST`, `RAG_SERVE_PORT`.
+`--no-llm` forces deterministic rewriting — useful when there's no model around.
+
+## Browser UI
+
+`rag-ladder serve` binds `127.0.0.1:8765` and serves a read-only page: ask box, answer
+pane, citations, corpus count. Routes: `GET /`, `GET /api/ask?q=&k=`,
+`GET /api/search?q=&k=`, `GET /api/stats`. No write endpoint, so there's no CSRF
+surface. Binding a non-local `--host` prints a warning — that exposes the corpus
+to your network.
+
+## Architecture
+
+Two modules split at the engine/app seam:
+
+- `rag.py` — engine: BM25 retrieval, LLM/embed clients, whole-document storage,
+  answer synthesis, `recommend()`.
+- `ragcli.py` — app layer: config, ingestion, CLI, browser UI.
+- `test_rag.py` — 26 runnable checks, no pytest, no network. `uv run test_rag.py`.
+
+One SQLite file. Documents are stored **whole** — no chunking, no chunk-size or
+overlap decisions, no eval harness for chunks (recipe 1's whole point). A `meta`
+table holds the content hash, so re-indexing skips unchanged files.
+
+| Piece | Rung | Why |
+|---|---|---|
+| BM25 index | SQLite FTS5 (`bm25()`, `ORDER BY rank`) | stdlib, native, zero ops |
+| LLM client | `urllib` → OpenAI-compatible `/chat/completions` | no SDK dependency |
+| Embeddings | `urllib` → `/embeddings`, else hashed bag-of-words | recipes 3-6 runnable with zero setup |
+| Vector math | numpy>=1.26 (declared dep) | rerank + brute-force cosine |
+| ANN | brute-force scan over BLOBs | ponytail: O(n); faiss/hnsw past ~100K docs |
+| UI | `http.server` + one inline page | stdlib, no framework, no build step |
+
+Tables: `docs` (FTS5, id + full text), `vectors` (id, vec BLOB, model),
+`access` (id, hit count — feeds the hot tier), `meta` (id, sha, mtime, chars).
+
+## Recipe → code
+
+| Recipe | Entry point | When |
+|---|---|---|
+| 1 BM25 | `Rag.search` | start here, always |
+| 2 query rewriting | `Rag.rewrite`, `Rag.agentic_search` | vocabulary mismatch; results bad → edit the prompt, not the corpus |
+| 3 hybrid | `Rag.hybrid` | "okay but not great"; 100-500ms is acceptable |
+| 4 on-the-fly | `Rag.hybrid` (docs embedded per query) | churn >10%/day, or you're swapping embedding models |
+| 5 hot/cold | `Rag.search_hot_cold`, `Rag.refresh_hot` | Pareto access pattern, 100K+ docs |
+| 6 pre-embedding | `Rag.preembed`, `Rag.search_preembedded` | >10K q/day, <5% churn/month, ML team |
+| multi-intent | `Rag.decompose`, `Rag.search_multi_intent` | one query carrying several intents; parallel → latency is max, not sum |
+| cited answers | `Rag.answer` | every user-facing question; `answer_mode: legal` for contracts |
+
+The decision tree from the article is `recommend()`; the CLI runs it.
+
+## Deliberate corners
+
+- Fallback embeddings are lexical (hashed BoW), **not semantic**. Point
+  `embed_base` at a real model for recipes 3-6 to mean what the article says.
+- Documents are embedded whole but truncated to `embed_max_chars` — a 2048-token
+  model can't see a long document's tail. That's the chunking cost recipe 1
+  avoids; pay it only if truncation measurably hurts ranking.
+- `agentic_search`'s quality signal is lexical coverage, not an LLM judge.
+- ANN is a linear scan; hot tier is recomputed on read, not on a cron.
+- `decompose`'s fallback splits on "and"/commas and invents no dependencies.
+- `answer()` cites the whole document it retrieved, not a page number — page
+  offsets aren't tracked, because documents are stored whole.
+- A reasoning model spends its budget on `reasoning_content` before emitting the
+  rewrite, so recipe 2 costs far more than the article's $0.001/query. `llm()`
+  retries such truncations with `reasoning_effort="none"` (llama.cpp-style
+  servers); `rag-ladder doctor` warns, and presets supply a cheap model.
+
+Bottom line from the article, encoded as the default: don't build the 5%
+solution for a 60% problem. 60% of systems stop at recipe 1 + recipe 2.

rag_ladder-0.1.0/pyproject.toml

@@ -0,0 +1,21 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rag-ladder"
+version = "0.1.0"
+description = "6 RAG recipes, ladder-first: BM25 full-text, LLM query rewriting, hybrid retrieval on SQLite FTS5"
+requires-python = ">=3.10"
+dependencies = [
+    "numpy>=1.26",
+]
+
+[project.optional-dependencies]
+docs = ["pypdf>=4", "python-docx>=1"]
+
+[tool.setuptools]
+py-modules = ["rag", "ragcli"]
+
+[project.scripts]
+rag-ladder = "ragcli:main"