raglite-chromadb 1.0.1__tar.gz

raglite_chromadb-1.0.1/PKG-INFO

@@ -0,0 +1,167 @@
+Metadata-Version: 2.4
+Name: raglite-chromadb
+Version: 1.0.1
+Summary: Local-first RAG-lite CLI: condense docs into structured Markdown, then index/query with Chroma + hybrid search
+Author: Viraj Sanghvi
+License: MIT
+Keywords: rag,docs,chroma,ollama,openclaw,summarization,local-first
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: beautifulsoup4==4.12.3
+Requires-Dist: lxml==5.3.0
+Requires-Dist: pypdf==5.2.0
+
+# RAGLite
+
+<p align="center">
+  <img src="assets/hero.svg" alt="RAGLite: Compress first. Index second." width="900" />
+</p>
+
+RAGLite is a local-first CLI that turns a folder of docs (PDF/HTML/TXT/MD) into **structured, low-fluff Markdown** — and then makes it searchable with **Chroma** 🧠 + **ripgrep** 🔎.
+
+Core idea: **compression-before-embeddings** ✂️➡️🧠
+
+<p align="center">
+  <img src="assets/diagram.svg" alt="RAGLite workflow: condense, index, query" width="900" />
+</p>
+
+## What you get
+
+For each input file:
+- `*.execution-notes.md` — practical run/operate notes (checks, failure modes, commands)
+- `*.tool-summary.md` — compact index entry (purpose, capabilities, entrypoints, footguns)
+
+Optionally:
+- `raglite index` stores embeddings in **Chroma** 🧠 (one DB, many collections)
+- `raglite query` runs **hybrid search** 🔎 (vector + keyword)
+
+## Why local + open-source?
+
+If you want a private, local setup (no managed “fancy vector DB” required), RAGLite keeps everything on your machine:
+- Distilled Markdown artifacts are plain files you can audit + version control
+- Indexing uses **Chroma** (open-source, local) and keyword search uses **ripgrep**
+- You can still swap in a hosted vector DB later if you outgrow local
+
+## Engines
+
+RAGLite supports two backends:
+
+- **OpenClaw (recommended):** uses your local OpenClaw Gateway `/v1/responses` endpoint for higher-quality, format-following condensation.
+- **Ollama:** uses `POST /api/generate` for fully local inference (often less reliable at strict templates).
+
+## Prereqs
+
+- **Python 3.11+**
+- An LLM engine:
+  - **OpenClaw** (recommended) 🦞, or
+  - **Ollama** 🦙
+- For search:
+  - **Chroma** (open-source, local) 🧠 at `http://127.0.0.1:8100`
+
+## Install
+
+```bash
+# from repo root
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+## Quickstart (60s)
+
+```bash
+# 0) Setup
+cd ~/Projects/raglite
+source .venv/bin/activate
+
+# 1) Condense → Index (one command)
+raglite run /path/to/docs \
+  --out ./raglite_out \
+  --engine ollama --ollama-model llama3.2:3b \
+  --collection my-docs \
+  --chroma-url http://127.0.0.1:8100 \
+  --skip-indexed
+
+# 2) Query
+raglite query ./raglite_out \
+  --collection my-docs \
+  "rollback procedure"
+```
+
+## Usage
+
+### 1) Distill docs ✍️
+
+```bash
+raglite condense /path/to/docs \
+  --out ./raglite_out \
+  --engine openclaw
+```
+
+(Or fully local: `--engine ollama --ollama-model llama3.2:3b`.)
+
+### 2) Index distilled output (Chroma)
+
+```bash
+raglite index ./raglite_out \
+  --collection my-docs \
+  --chroma-url http://127.0.0.1:8100
+```
+
+### 3) Query (hybrid)
+
+```bash
+raglite query ./raglite_out \
+  --collection my-docs \
+  --top-k 5 \
+  --keyword-top-k 5 \
+  "rollback procedure"
+```
+
+### Useful flags
+
+- `--skip-existing` : don’t redo files that already have both outputs
+- `--skip-indexed` : don’t re-embed chunks that are already indexed
+- `--nodes` : write per-section nodes + per-doc/root indices
+- `--node-max-chars 1200` : keep nodes embed-friendly
+- `--sleep-ms 200` : throttle between files (helps avoid timeouts)
+- `--max-chars 180000` : cap extracted text per file before summarizing
+
+## Output layout
+
+RAGLite preserves folder structure under your `--out` dir:
+
+```text
+<out>/
+  some/subdir/file.execution-notes.md
+  some/subdir/file.tool-summary.md
+
+(Default output folder is `./raglite_out`.)
+```
+
+## Notes / gotchas
+
+- PDF extraction is best-effort: scanned PDFs without embedded text won’t be great.
+- If you use `--engine openclaw`, pass `--gateway-token` or set `OPENCLAW_GATEWAY_TOKEN`.
+- Indexing defaults to high-signal artifacts (nodes/summaries/notes) and skips `*.outline.md` unless you opt in.
+
+## Roadmap
+
+### Current (implemented)
+- `condense` — condense/summarize documents into Markdown artifacts
+- `index` — chunk + embed + store in **Chroma** collections
+- `query` — retrieve relevant chunks (vector + keyword)
+- `run` — one-command pipeline (condense → index)
+- Outline + nodes + indices: `--outline`, `--nodes`, root `index.md` + per-doc `*.index.md`
+
+### Next (near-term)
+- Detect deletions (prune removed chunks from Chroma)
+- Batch upserts to Chroma for speed
+- Better query output formatting (snippets + anchors)
+- `raglite doctor` (dependency checks)
+
+(Full: [ROADMAP.md](ROADMAP.md))
+
+---
+
+Built to turn “docs” into **usable, searchable tool knowledge**.

raglite_chromadb-1.0.1/README.md

@@ -0,0 +1,154 @@
+# RAGLite
+
+<p align="center">
+  <img src="assets/hero.svg" alt="RAGLite: Compress first. Index second." width="900" />
+</p>
+
+RAGLite is a local-first CLI that turns a folder of docs (PDF/HTML/TXT/MD) into **structured, low-fluff Markdown** — and then makes it searchable with **Chroma** 🧠 + **ripgrep** 🔎.
+
+Core idea: **compression-before-embeddings** ✂️➡️🧠
+
+<p align="center">
+  <img src="assets/diagram.svg" alt="RAGLite workflow: condense, index, query" width="900" />
+</p>
+
+## What you get
+
+For each input file:
+- `*.execution-notes.md` — practical run/operate notes (checks, failure modes, commands)
+- `*.tool-summary.md` — compact index entry (purpose, capabilities, entrypoints, footguns)
+
+Optionally:
+- `raglite index` stores embeddings in **Chroma** 🧠 (one DB, many collections)
+- `raglite query` runs **hybrid search** 🔎 (vector + keyword)
+
+## Why local + open-source?
+
+If you want a private, local setup (no managed “fancy vector DB” required), RAGLite keeps everything on your machine:
+- Distilled Markdown artifacts are plain files you can audit + version control
+- Indexing uses **Chroma** (open-source, local) and keyword search uses **ripgrep**
+- You can still swap in a hosted vector DB later if you outgrow local
+
+## Engines
+
+RAGLite supports two backends:
+
+- **OpenClaw (recommended):** uses your local OpenClaw Gateway `/v1/responses` endpoint for higher-quality, format-following condensation.
+- **Ollama:** uses `POST /api/generate` for fully local inference (often less reliable at strict templates).
+
+## Prereqs
+
+- **Python 3.11+**
+- An LLM engine:
+  - **OpenClaw** (recommended) 🦞, or
+  - **Ollama** 🦙
+- For search:
+  - **Chroma** (open-source, local) 🧠 at `http://127.0.0.1:8100`
+
+## Install
+
+```bash
+# from repo root
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
+```
+
+## Quickstart (60s)
+
+```bash
+# 0) Setup
+cd ~/Projects/raglite
+source .venv/bin/activate
+
+# 1) Condense → Index (one command)
+raglite run /path/to/docs \
+  --out ./raglite_out \
+  --engine ollama --ollama-model llama3.2:3b \
+  --collection my-docs \
+  --chroma-url http://127.0.0.1:8100 \
+  --skip-indexed
+
+# 2) Query
+raglite query ./raglite_out \
+  --collection my-docs \
+  "rollback procedure"
+```
+
+## Usage
+
+### 1) Distill docs ✍️
+
+```bash
+raglite condense /path/to/docs \
+  --out ./raglite_out \
+  --engine openclaw
+```
+
+(Or fully local: `--engine ollama --ollama-model llama3.2:3b`.)
+
+### 2) Index distilled output (Chroma)
+
+```bash
+raglite index ./raglite_out \
+  --collection my-docs \
+  --chroma-url http://127.0.0.1:8100
+```
+
+### 3) Query (hybrid)
+
+```bash
+raglite query ./raglite_out \
+  --collection my-docs \
+  --top-k 5 \
+  --keyword-top-k 5 \
+  "rollback procedure"
+```
+
+### Useful flags
+
+- `--skip-existing` : don’t redo files that already have both outputs
+- `--skip-indexed` : don’t re-embed chunks that are already indexed
+- `--nodes` : write per-section nodes + per-doc/root indices
+- `--node-max-chars 1200` : keep nodes embed-friendly
+- `--sleep-ms 200` : throttle between files (helps avoid timeouts)
+- `--max-chars 180000` : cap extracted text per file before summarizing
+
+## Output layout
+
+RAGLite preserves folder structure under your `--out` dir:
+
+```text
+<out>/
+  some/subdir/file.execution-notes.md
+  some/subdir/file.tool-summary.md
+
+(Default output folder is `./raglite_out`.)
+```
+
+## Notes / gotchas
+
+- PDF extraction is best-effort: scanned PDFs without embedded text won’t be great.
+- If you use `--engine openclaw`, pass `--gateway-token` or set `OPENCLAW_GATEWAY_TOKEN`.
+- Indexing defaults to high-signal artifacts (nodes/summaries/notes) and skips `*.outline.md` unless you opt in.
+
+## Roadmap
+
+### Current (implemented)
+- `condense` — condense/summarize documents into Markdown artifacts
+- `index` — chunk + embed + store in **Chroma** collections
+- `query` — retrieve relevant chunks (vector + keyword)
+- `run` — one-command pipeline (condense → index)
+- Outline + nodes + indices: `--outline`, `--nodes`, root `index.md` + per-doc `*.index.md`
+
+### Next (near-term)
+- Detect deletions (prune removed chunks from Chroma)
+- Batch upserts to Chroma for speed
+- Better query output formatting (snippets + anchors)
+- `raglite doctor` (dependency checks)
+
+(Full: [ROADMAP.md](ROADMAP.md))
+
+---
+
+Built to turn “docs” into **usable, searchable tool knowledge**.

raglite_chromadb-1.0.1/pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "raglite-chromadb"
+version = "1.0.1"
+description = "Local-first RAG-lite CLI: condense docs into structured Markdown, then index/query with Chroma + hybrid search"
+readme = "README.md"
+requires-python = ">=3.11"
+authors = [{ name = "Viraj Sanghvi" }]
+license = { text = "MIT" }
+keywords = ["rag", "docs", "chroma", "ollama", "openclaw", "summarization", "local-first"]
+dependencies = [
+  "beautifulsoup4==4.12.3",
+  "lxml==5.3.0",
+  "pypdf==5.2.0",
+]
+
+[project.scripts]
+raglite = "raglite.raglite_cli:cli"
+
+[tool.setuptools]
+package-dir = {"" = "."}
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["raglite*"]

raglite_chromadb-1.0.1/raglite/__init__.py

@@ -0,0 +1 @@
+__all__ = []

raglite_chromadb-1.0.1/raglite/chroma_rest.py

@@ -0,0 +1,111 @@
+from __future__ import annotations
+
+import json
+import urllib.error
+import urllib.request
+from dataclasses import dataclass
+
+
+@dataclass
+class ChromaLoc:
+    base_url: str = "http://127.0.0.1:8100"
+    tenant: str = "default_tenant"
+    database: str = "default_database"
+
+    def collections_url(self) -> str:
+        return f"{self.base_url}/api/v2/tenants/{self.tenant}/databases/{self.database}/collections"
+
+
+def _req_json(method: str, url: str, body: dict | None = None, timeout: int = 120) -> dict | list:
+    data = None if body is None else json.dumps(body).encode("utf-8")
+    req = urllib.request.Request(url, data=data, method=method)
+    req.add_header("Content-Type", "application/json")
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            raw = resp.read().decode("utf-8")
+            if not raw:
+                return {}
+            return json.loads(raw)
+    except urllib.error.HTTPError as e:  # type: ignore[attr-defined]
+        detail = ""
+        try:
+            detail = e.read().decode("utf-8", errors="ignore")
+        except Exception:
+            detail = ""
+        raise RuntimeError(f"Chroma HTTP {e.code} calling {url}: {detail[:500]}")
+
+
+def list_collections(loc: ChromaLoc) -> list[dict]:
+    res = _req_json("GET", loc.collections_url())
+    if not isinstance(res, list):
+        raise RuntimeError(f"Expected list from Chroma list_collections, got {type(res).__name__}")
+    return res
+
+
+def get_or_create_collection(loc: ChromaLoc, name: str, *, space: str = "cosine") -> dict:
+    cols = list_collections(loc)
+    for c in cols:
+        if c.get("name") == name:
+            return c
+
+    created = _req_json(
+        "POST",
+        loc.collections_url(),
+        {"name": name, "metadata": {"hnsw:space": space}},
+    )
+    if not isinstance(created, dict):
+        raise RuntimeError(f"Expected dict from Chroma create_collection, got {type(created).__name__}")
+    return created
+
+
+def add(
+    loc: ChromaLoc,
+    collection_id: str,
+    *,
+    ids: list[str],
+    documents: list[str],
+    embeddings: list[list[float]],
+    metadatas: list[dict] | None = None,
+) -> None:
+    url = f"{loc.collections_url()}/{collection_id}/add"
+    body: dict = {"ids": ids, "documents": documents, "embeddings": embeddings}
+    if metadatas is not None:
+        body["metadatas"] = metadatas
+    _req_json("POST", url, body, timeout=600)
+
+
+def upsert(
+    loc: ChromaLoc,
+    collection_id: str,
+    *,
+    ids: list[str],
+    documents: list[str],
+    embeddings: list[list[float]],
+    metadatas: list[dict] | None = None,
+) -> None:
+    """Upsert records into a collection.
+
+    Chroma's /upsert updates existing ids and inserts new ones.
+    """
+    url = f"{loc.collections_url()}/{collection_id}/upsert"
+    body: dict = {"ids": ids, "documents": documents, "embeddings": embeddings}
+    if metadatas is not None:
+        body["metadatas"] = metadatas
+    _req_json("POST", url, body, timeout=600)
+
+
+def query(
+    loc: ChromaLoc,
+    collection_id: str,
+    *,
+    query_embeddings: list[list[float]],
+    n_results: int = 10,
+    include: list[str] | None = None,
+) -> dict:
+    url = f"{loc.collections_url()}/{collection_id}/query"
+    body: dict = {"query_embeddings": query_embeddings, "n_results": n_results}
+    if include is not None:
+        body["include"] = include
+    res = _req_json("POST", url, body, timeout=600)
+    assert isinstance(res, dict)
+    return res

raglite_chromadb-1.0.1/raglite/extract.py

@@ -0,0 +1,63 @@
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal
+
+from bs4 import BeautifulSoup
+from pypdf import PdfReader
+
+
+FileKind = Literal["pdf", "html", "htm", "txt"]
+
+
+@dataclass
+class ExtractResult:
+    kind: FileKind
+    text: str
+
+
+def _clean_text(s: str) -> str:
+    s = s.replace("\r\n", "\n").replace("\r", "\n")
+    s = re.sub(r"[ \t]+", " ", s)
+    s = re.sub(r"\n{3,}", "\n\n", s)
+    return s.strip()
+
+
+def extract_pdf(path: Path) -> ExtractResult:
+    reader = PdfReader(str(path))
+    parts = []
+    for i, page in enumerate(reader.pages):
+        try:
+            txt = page.extract_text() or ""
+        except Exception:
+            txt = ""
+        if txt.strip():
+            parts.append(f"\n\n--- Page {i+1} ---\n\n{txt}")
+    return ExtractResult(kind="pdf", text=_clean_text("\n".join(parts)))
+
+
+def extract_html(path: Path) -> ExtractResult:
+    html = path.read_text(encoding="utf-8", errors="ignore")
+    soup = BeautifulSoup(html, "lxml")
+
+    # Remove scripts/styles
+    for tag in soup(["script", "style", "noscript"]):
+        tag.decompose()
+
+    text = soup.get_text("\n")
+    return ExtractResult(kind="html", text=_clean_text(text))
+
+
+def extract_txt(path: Path) -> ExtractResult:
+    return ExtractResult(kind="txt", text=_clean_text(path.read_text(encoding="utf-8", errors="ignore")))
+
+
+def extract_file(path: Path) -> ExtractResult:
+    suffix = path.suffix.lower().lstrip(".")
+    if suffix == "pdf":
+        return extract_pdf(path)
+    if suffix in ("html", "htm"):
+        return extract_html(path)
+    return extract_txt(path)

raglite_chromadb-1.0.1/raglite/prompts.py

@@ -0,0 +1,56 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass
+class PromptPair:
+    execution_notes_prompt: str
+    tool_summary_prompt: str
+
+
+def build_prompts(*, token_cap_hint: str = "~1200 tokens max") -> PromptPair:
+    # These prompts are designed to be copy/pasted into Cosmo.
+    execution_notes = f"""You are an expert at converting documentation into EXECUTION-RELEVANT notes for an AI agent that can run tools (CLI commands, HTTP calls, scripts, and functions).
+
+OUTPUT FORMAT (Markdown):
+- Title
+- What this tool/service is
+- When to use
+- Inputs (required/optional)
+- Outputs
+- Preconditions / assumptions
+- Step-by-step 'golden path' (numbered)
+- Verification checks (how to confirm success)
+- Common errors + fixes
+- Safety/rollback notes (what not to do / how to undo)
+
+RULES:
+- Be concise and operational; no marketing.
+- Prefer concrete commands, flags, endpoints, and example payloads.
+- If the doc is long, extract only what is needed to execute.
+- Keep the final output within {token_cap_hint}.
+
+SOURCE DOCUMENT (extracted text) is below. Use it as the only source of truth.
+---
+"""
+
+    tool_summary = f"""You are an expert at writing ultra-condensed TOOL INDEX summaries for an agent tool library.
+
+Write a short Markdown file with:
+- Tool name
+- 1-sentence purpose
+- Capabilities (3-7 bullets)
+- Required environment/dependencies
+- Primary entrypoints (commands/endpoints)
+- Key limitations / footguns (1-3 bullets)
+
+RULES:
+- No fluff. Assume the reader is an executor agent.
+- Keep within ~250-400 tokens.
+
+SOURCE DOCUMENT (extracted text) is below. Use it as the only source of truth.
+---
+"""
+
+    return PromptPair(execution_notes_prompt=execution_notes, tool_summary_prompt=tool_summary)