getbased-rag 0.6.1__py3-none-any.whl

getbased_rag-0.6.1.dist-info/METADATA

@@ -0,0 +1,243 @@
+Metadata-Version: 2.4
+Name: getbased-rag
+Version: 0.6.1
+Summary: getbased-rag — standalone RAG knowledge server (formerly the Electron-bundled Lens)
+License-Expression: GPL-3.0-only
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.110
+Requires-Dist: uvicorn[standard]>=0.29
+Requires-Dist: qdrant-client>=1.9
+Requires-Dist: sentence-transformers>=2.6
+Requires-Dist: typer>=0.12
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: python-multipart>=0.0.6
+Provides-Extra: pdf
+Requires-Dist: pypdf>=3.9; extra == "pdf"
+Provides-Extra: docx
+Requires-Dist: python-docx>=1.0; extra == "docx"
+Provides-Extra: onnx
+Requires-Dist: onnxruntime>=1.17; extra == "onnx"
+Requires-Dist: optimum[onnxruntime]>=1.17; extra == "onnx"
+Requires-Dist: transformers>=5.0; extra == "onnx"
+Provides-Extra: full
+Requires-Dist: pypdf>=3.9; extra == "full"
+Requires-Dist: python-docx>=1.0; extra == "full"
+Requires-Dist: onnxruntime>=1.17; extra == "full"
+Requires-Dist: optimum[onnxruntime]>=1.17; extra == "full"
+Requires-Dist: transformers>=5.0; extra == "full"
+Provides-Extra: test
+Requires-Dist: pytest>=8.0; extra == "test"
+Requires-Dist: httpx>=0.27; extra == "test"
+Dynamic: license-file
+
+# getbased-rag
+
+> **Installing for the first time?** The [getbased-agent-stack](https://github.com/elkimek/getbased-agents/tree/main/packages/stack) meta-package bundles this server with the MCP that Claude Code / Hermes / OpenClaw talk to, plus [getbased-dashboard](https://github.com/elkimek/getbased-agents/tree/main/packages/dashboard) for a browser UI. One command and you're up.
+
+A standalone RAG knowledge server — the backend that used to ship inside the getbased Electron desktop app, now just Python. Point any client (the getbased PWA's *External server* lens backend, the dashboard, or your own) at it.
+
+- **Stack**: FastAPI + Uvicorn · Qdrant (embedded local mode) · sentence-transformers / ONNX Runtime
+- **Default port**: 8322, loopback only
+- **Auth**: Bearer token, auto-generated on first start
+- **Stores**: every library is its own Qdrant collection, pinned to its own embedding model at creation
+
+---
+
+## Install
+
+Requires Python ≥ 3.10.
+
+```bash
+pipx install "getbased-rag[full]"
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/elkimek/getbased-agents.git
+cd getbased-agents
+uv sync --all-packages --all-extras
+```
+
+---
+
+## Run
+
+```bash
+lens serve
+```
+
+First start auto-generates an API key at the data dir (see below), prints the bind address, and lazy-loads the embedding model on the first query (~90 MB download for MiniLM).
+
+Copy the API key out when you need to configure a client:
+
+```bash
+lens key
+```
+
+Smoke test:
+
+```bash
+curl -s http://127.0.0.1:8322/health
+curl -s -H "Authorization: Bearer $(lens key)" http://127.0.0.1:8322/info | jq
+```
+
+Ingest a file or directory from the CLI:
+
+```bash
+lens ingest ~/Documents/research
+lens stats
+```
+
+Or over HTTP (what the dashboard + PWA use):
+
+```bash
+curl -H "Authorization: Bearer $(lens key)" \
+  -F "files=@paper.pdf" -F "files=@notes.md" \
+  http://127.0.0.1:8322/ingest
+```
+
+---
+
+## Per-library embedding models
+
+Every library is pinned to one embedding model at creation time — Qdrant collections are dimension-locked, so you can't swap models on an existing library without re-ingesting. Call `GET /models` for the curated list (MiniLM-L6-v2 · BGE-small/base/large-en · BGE-M3) with dims and download sizes, then pass `embedding_model` on create:
+
+```bash
+curl -H "Authorization: Bearer $(lens key)" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"Research","embedding_model":"BAAI/bge-m3"}' \
+  http://127.0.0.1:8322/libraries
+```
+
+Libraries on the same model share one embedder instance in memory. Two libraries both on BGE-M3 use ~2 GB total, not 4.
+
+---
+
+## Streaming ingest progress
+
+The HTTP `POST /ingest` endpoint speaks two content types:
+
+- Default (no `Accept`): single-shot JSON summary after the run completes
+- `Accept: application/x-ndjson`: newline-delimited JSON progress stream — one `start` event (with total chunks), per-batch `embed` events every ~5 chunks (with current source + index), terminal `result` or `error` event
+
+The dashboard uses the streaming path for its bottom-right pill (chunks/sec rate, cancel button, per-filename status). Cancellation works by client disconnect: aborting the fetch causes the server's worker thread to exit at the next batch boundary with `cancelled: true` in the result. Partial-commit — whatever was embedded before the cancel stays.
+
+---
+
+## Wiring into the getbased PWA
+
+In the PWA: **Settings → AI → Knowledge Base → External server**
+
+| Field | Value |
+|---|---|
+| URL | `http://127.0.0.1:8322` |
+| API key | output of `lens key` |
+
+Click **Save**, then **Test connection**. `rag_ready: false` is expected before you ingest anything.
+
+### Agent access (Claude Code, Hermes, OpenClaw, etc.)
+
+Pair this server with [getbased-mcp](https://github.com/elkimek/getbased-agents/tree/main/packages/mcp) to expose `knowledge_search`, `knowledge_list_libraries`, `knowledge_activate_library`, and `knowledge_stats` as MCP tools. Typical setup: run both the lens server and getbased-mcp on the same VM, point MCP's `LENS_URL` at `http://localhost:8322`.
+
+### Browser UI
+
+Install [getbased-dashboard](https://github.com/elkimek/getbased-agents/tree/main/packages/dashboard) for a web UI on top of this server — library management, drag-drop ingest with live progress pill, search preview, MCP config generator.
+
+---
+
+## Configuration
+
+Every setting is an environment variable. Defaults in parentheses.
+
+| Variable | Purpose |
+|---|---|
+| `LENS_HOST` (`127.0.0.1`) | Bind interface. Change to `0.0.0.0` only if you really want LAN access |
+| `LENS_PORT` (`8322`) | TCP port |
+| `LENS_DATA_DIR` (platform default) | Where Qdrant DB, API key, and model cache live |
+| `LENS_EMBEDDING_MODEL` (`sentence-transformers/all-MiniLM-L6-v2`) | Default model for new libraries (overridable per library) |
+| `LENS_SIMILARITY_FLOOR` (`0.55`) | Minimum cosine score for a returned chunk |
+| `LENS_ONNX_PROVIDER` (auto) | `cuda` \| `rocm` \| `openvino` \| `coreml` \| `cpu` |
+| `LENS_RERANKER` (`false`) | Enable reranking of top candidates |
+| `LENS_MAX_INGEST_BYTES` (`268435456` — 256 MB) | Cap on a single ingest upload's total size |
+| `LENS_CHUNK_MAX_SIZE` (`800`) | Max chunk size in characters |
+| `LENS_CORS_ORIGINS` (empty) | Comma-separated extra CORS origins to allow, in addition to the PWA + loopback defaults |
+
+Default data dir:
+
+- Linux: `$XDG_DATA_HOME/getbased/lens` or `~/.local/share/getbased/lens`
+- macOS: `~/Library/Application Support/getbased/lens`
+- Windows: `%APPDATA%\getbased\lens`
+
+A legacy `~/.getbased/lens` is honored if it already exists, so pre-v1.21 installs don't lose their data.
+
+### GPU acceleration
+
+Install the matching `onnxruntime-*` wheel (e.g. `onnxruntime-gpu` for CUDA), then:
+
+```bash
+LENS_ONNX_PROVIDER=cuda lens serve
+```
+
+---
+
+## HTTP API
+
+All endpoints except `/`, `/health` require `Authorization: Bearer <key>`.
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/health` | Liveness + `rag_ready` + chunk count. Public |
+| `GET` | `/info` | Embedder engine/model/dim, active library, similarity floor. For UI engine badges |
+| `GET` | `/models` | Curated model picker list (id, label, dim, size_mb) plus the server's default |
+| `POST` | `/query` | `{ query, top_k }` → top-k chunks from the active library, encoded with that library's model |
+| `POST` | `/ingest` | Multipart upload; accepts streaming NDJSON progress when `Accept: application/x-ndjson` |
+| `GET` | `/stats` | Per-source chunk counts for the active library |
+| `DELETE` | `/sources/{source}` | Drop one source from the active library |
+| `DELETE` | `/sources` | Clear the active library's chunks (library stays) |
+| `GET` | `/libraries` | List libraries + active id. Each row includes `chunks`, `lastIngestAt`, `embedding_model` |
+| `POST` | `/libraries` | `{ name, embedding_model? }` → create. 409 on duplicate name (case-insensitive) |
+| `POST` | `/libraries/{id}/activate` | Set active |
+| `PATCH` | `/libraries/{id}` | Rename |
+| `DELETE` | `/libraries/{id}` | Delete (drops Qdrant collection) |
+
+---
+
+## Security notes
+
+- Default bind is `127.0.0.1` — queries never leak to the LAN unless you explicitly set `LENS_HOST=0.0.0.0`.
+- The API key file is mode `0600` and never exposed over HTTP. Use `lens key` locally to read it.
+- Bearer comparison uses `secrets.compare_digest` — constant-time, no timing-leak class of bug.
+- Upload paths are basename-sanitised server-side (so `../../etc/passwd` can't escape the ingest temp dir).
+- Zip uploads are zip-slip-guarded — each archive entry must resolve inside its own per-zip subdirectory AND inside the overall ingest root.
+- If you expose the server to a LAN or the internet, front it with a reverse proxy that terminates TLS and rate-limits.
+
+---
+
+## CLI
+
+```
+lens serve            Start the HTTP server (default)
+lens ingest <path>    Index files into the active library
+lens stats            List indexed sources + chunk counts
+lens delete <source>  Drop chunks belonging to one source
+lens clear            Wipe the active library
+lens info             Show config + API key
+lens key              Print the API key (creates one if missing)
+```
+
+---
+
+## License
+
+GPL-3.0-only.
+
+---
+
+## Lineage
+
+This repo is the Python portion lifted out of [getbased](https://github.com/elkimek/getbased) after the Electron desktop app was retired. The PWA's `external-server` lens backend speaks this same HTTP contract unchanged.

getbased_rag-0.6.1.dist-info/RECORD

@@ -0,0 +1,15 @@
+getbased_rag-0.6.1.dist-info/licenses/LICENSE,sha256=K-IjLWkez1gJQMrlqA5zgyw8vh19mDzk4hKM9Dslmts,1024
+lens/__init__.py,sha256=E4lyDbXhAwhHbsIONEtqzrBs02OXlA0Z8ds-VihY5pg,75
+lens/api_key.py,sha256=6qyZSGoWzCrVViBQm-X2mA_wz3BK2bUCgmBNxZpSSjA,1693
+lens/cli.py,sha256=-tA5a7VtSiWt--N6lcVSnehX5GY76cIIbqW_CyjTIuc,7072
+lens/config.py,sha256=jhlRf6gzMpWNjDsyonQaHV-rQN-lFQk24Sq6_O5XpKE,6009
+lens/embedder.py,sha256=VVKOE29209FvIa7gJ6J5pwkfACPXFoi8uQRVysYqDrU,19848
+lens/ingest.py,sha256=bRt2RYaM_v03lxnBNuMSqlDn-wGD24eftKjaWwGlKjE,14374
+lens/registry.py,sha256=jgcle03t7tMoODLBUzMuy78-i_hNINLtmlbwJ07i8Qw,10304
+lens/server.py,sha256=hv2wgAwrTJX3Gw8rVOawz61y9I9jV1v--dI_bK51zu0,33648
+lens/store.py,sha256=kiQo6AcQQ3ptHJ2oG-bYkJe2-AE_D6fwv6h97OBTm5M,10308
+getbased_rag-0.6.1.dist-info/METADATA,sha256=S9rCw9FsorD44LA7cWwUyA9zliTqIHMkFraMqHfd2Fk,9655
+getbased_rag-0.6.1.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+getbased_rag-0.6.1.dist-info/entry_points.txt,sha256=4ExZzbsCpB1cRlUc4ph9CGGtCsQMABcMs7gvr-yOIkk,38
+getbased_rag-0.6.1.dist-info/top_level.txt,sha256=weQcpg8Ic-kDl-iV7HcZ-TOJbXaihpy6vVm771yjAyw,5
+getbased_rag-0.6.1.dist-info/RECORD,,

getbased_rag-0.6.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

getbased_rag-0.6.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+lens = lens.cli:app

getbased_rag-0.6.1.dist-info/licenses/LICENSE

@@ -0,0 +1,22 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  For the full license text, see <https://www.gnu.org/licenses/gpl-3.0.txt>

getbased_rag-0.6.1.dist-info/top_level.txt

@@ -0,0 +1 @@
+lens

lens/__init__.py

@@ -0,0 +1,3 @@
+"""getbased-lens — local RAG knowledge server."""
+
+__version__ = "0.2.0"

lens/api_key.py

@@ -0,0 +1,55 @@
+"""API key generation + persistence for the Lens HTTP server.
+
+Auto-generates a key on first start if one doesn't exist. The desktop wrapper
+(Tauri) reads this file via `getbased_lens_config` MCP tool to display the
+key for the user to paste into the getbased web app's Custom Knowledge Source.
+"""
+
+from __future__ import annotations
+
+import os
+import secrets
+from pathlib import Path
+
+
+def get_or_create_api_key(key_file: Path) -> str:
+    """Read the API key from disk; generate + write one if missing.
+
+    Creates the file with O_EXCL + mode 0o600 in one syscall, so the key
+    is never briefly present with loose permissions (the race the old
+    write_text → chmod sequence had).
+    """
+    if key_file.exists():
+        try:
+            key = key_file.read_text().strip()
+            if key:
+                return key
+        except OSError:
+            pass
+
+    key_file.parent.mkdir(parents=True, exist_ok=True)
+    key = secrets.token_urlsafe(32)
+    flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
+    try:
+        fd = os.open(str(key_file), flags, 0o600)
+    except FileExistsError:
+        # Another process beat us to it — trust whatever they wrote rather
+        # than clobbering it with a fresh key.
+        existing = key_file.read_text().strip()
+        if existing:
+            return existing
+        raise
+    with os.fdopen(fd, "w") as f:
+        f.write(key + "\n")
+    return key
+
+
+def load_api_key(key_file: Path) -> str | None:
+    """Read existing API key without generating one."""
+    try:
+        if key_file.exists():
+            key = key_file.read_text().strip()
+            return key if key else None
+    except OSError:
+        pass
+    return None

lens/cli.py

@@ -0,0 +1,225 @@
+"""Lens CLI — typer-based commands.
+
+  lens serve            Start the HTTP server (default if no command)
+  lens ingest <path>    Index files into the local store
+  lens info             Show config + key + status
+  lens key              Print the API key (creates one if missing)
+
+Configuration comes from environment variables — see config.py for the full list.
+The Tauri desktop wrapper sets LENS_HOST, LENS_PORT, LENS_DATA_DIR,
+LENS_EMBEDDING_MODEL, and LENS_ONNX_PROVIDER for you.
+"""
+
+from __future__ import annotations
+
+import logging
+import sys
+from pathlib import Path
+
+import typer
+from rich.console import Console
+from rich.table import Table
+
+from .api_key import get_or_create_api_key
+from .config import LensConfig
+from .server import run_server
+
+console = Console()
+app = typer.Typer(
+    name="lens",
+    help="getbased-lens — local RAG knowledge server.",
+    no_args_is_help=False,
+    add_completion=False,
+)
+
+
+def _setup_logging(verbose: bool) -> None:
+    level = logging.DEBUG if verbose else logging.INFO
+    logging.basicConfig(
+        level=level,
+        format="%(asctime)s [%(name)s] %(levelname)s: %(message)s",
+        datefmt="%H:%M:%S",
+    )
+
+
+@app.callback(invoke_without_command=True)
+def _default(
+    ctx: typer.Context,
+    verbose: bool = typer.Option(False, "--verbose", "-v", help="Verbose logging"),
+):
+    """When invoked with no subcommand, run `serve`."""
+    _setup_logging(verbose)
+    if ctx.invoked_subcommand is None:
+        ctx.invoke(serve)
+
+
+@app.command()
+def serve():
+    """Start the HTTP server (uvicorn). Blocking."""
+    config = LensConfig.from_env()
+    console.print(f"[bold cyan]getbased-lens[/] starting on http://{config.host}:{config.port}")
+    console.print(f"  Data dir:    {config.data_dir}")
+    console.print(f"  Model:       {config.embedding_model}")
+    console.print(f"  Collection:  {config.collection}")
+    if config.onnx_provider:
+        console.print(f"  ONNX:        {config.onnx_provider}")
+    try:
+        run_server(config)
+    except KeyboardInterrupt:
+        console.print("\n[yellow]Stopped.[/]")
+
+
+@app.command()
+def ingest(
+    path: Path = typer.Argument(..., help="File or directory to ingest"),
+    json_out: bool = typer.Option(False, "--json", help="Emit machine-parseable JSON"),
+):
+    """Index documents from a path into the local store."""
+    from .ingest import ingest_path  # lazy import (heavy deps)
+    import json as _json
+
+    config = LensConfig.from_env()
+    if not json_out:
+        console.print(f"[bold cyan]Ingesting[/] {path}…")
+    try:
+        # JSONL progress is only useful for a parent process — emit it
+        # exactly when --json was requested. Human runs stay clean.
+        result = ingest_path(config, path, emit_progress=json_out)
+    except FileNotFoundError as e:
+        if json_out:
+            print(_json.dumps({"error": str(e)}))
+        else:
+            console.print(f"[red]Error:[/] {e}")
+        raise typer.Exit(1)
+
+    if json_out:
+        print(_json.dumps(result))
+        return
+
+    table = Table(title="Ingest result", show_header=False, box=None)
+    table.add_row("Files scanned", str(result["files_seen"]))
+    table.add_row("Chunks indexed", str(result["chunks_indexed"]))
+    if result["skipped"]:
+        table.add_row("Skipped", str(len(result["skipped"])))
+    console.print(table)
+
+
+def _active_store(config: LensConfig):
+    """CLI helper — resolve a Store bound to the active library.
+
+    Matches how the server's active_store() works: bootstrap a "Default"
+    library on first use so a fresh shell command doesn't 404 on the
+    registry being empty."""
+    from .registry import Registry
+    from .store import QdrantBackend, Store
+
+    registry = Registry(config)
+    registry.ensure_default()
+    return Store(
+        config,
+        collection=registry.active_collection(),
+        backend=QdrantBackend(config),
+    )
+
+
+@app.command()
+def stats(json_out: bool = typer.Option(False, "--json", help="Emit JSON")):
+    """List knowledge base contents: per-source chunk counts."""
+    import json as _json
+
+    config = LensConfig.from_env()
+    store = _active_store(config)
+    try:
+        docs = store.list_sources()
+    except Exception as e:
+        if json_out:
+            print(_json.dumps({"error": str(e), "total_chunks": 0, "documents": []}))
+        else:
+            console.print(f"[red]Error:[/] {e}")
+        raise typer.Exit(1)
+
+    total_chunks = sum(d["chunks"] for d in docs)
+    if json_out:
+        print(_json.dumps({"total_chunks": total_chunks, "documents": docs}))
+        return
+    if not docs:
+        console.print("No documents indexed yet. Use [bold]lens ingest <path>[/] to add some.")
+        return
+    table = Table(title=f"Indexed: {len(docs)} sources, {total_chunks} chunks")
+    table.add_column("Source")
+    table.add_column("Chunks", justify="right")
+    for d in docs:
+        table.add_row(d["source"], str(d["chunks"]))
+    console.print(table)
+
+
+@app.command()
+def delete(
+    source: str = typer.Argument(..., help="Source filename to delete (exact match)"),
+    json_out: bool = typer.Option(False, "--json", help="Emit JSON"),
+):
+    """Delete all chunks belonging to a source from the knowledge base."""
+    import json as _json
+
+    config = LensConfig.from_env()
+    store = _active_store(config)
+    deleted = store.delete_by_source(source)
+    if json_out:
+        print(_json.dumps({"source": source, "deleted_chunks": deleted}))
+        return
+    console.print(f"Deleted {deleted} chunks matching source '{source}'")
+
+
+@app.command()
+def clear(
+    json_out: bool = typer.Option(False, "--json", help="Emit JSON"),
+    yes: bool = typer.Option(False, "--yes", "-y", help="Skip interactive confirmation"),
+):
+    """Delete ALL chunks from the knowledge base (drops the collection)."""
+    import json as _json
+
+    config = LensConfig.from_env()
+    store = _active_store(config)
+
+    if not yes and not json_out:
+        console.print(f"[yellow]This will delete all chunks from[/] {config.qdrant_path}")
+        confirm = typer.confirm("Proceed?")
+        if not confirm:
+            console.print("Aborted.")
+            raise typer.Exit(0)
+
+    deleted = store.clear()
+    if json_out:
+        print(_json.dumps({"deleted_chunks": deleted}))
+        return
+    console.print(f"Cleared {deleted} chunks.")
+
+
+@app.command()
+def info():
+    """Show current configuration + status."""
+    config = LensConfig.from_env()
+    console.print(config.display())
+    console.print()
+    key = get_or_create_api_key(config.api_key_file)
+    console.print(f"  api_key:           {key[:8]}…{key[-4:]} (file: {config.api_key_file})")
+
+
+@app.command()
+def key():
+    """Print the API key (generates one on first invocation)."""
+    config = LensConfig.from_env()
+    print(get_or_create_api_key(config.api_key_file))
+
+
+def main():
+    """Entry point for `python -m lens`."""
+    try:
+        app()
+    except Exception as e:  # noqa: BLE001
+        console.print(f"[red]Error:[/] {e}")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()