askme-rag 0.1.1__tar.gz

askme_rag-0.1.1/PKG-INFO

@@ -0,0 +1,245 @@
+Metadata-Version: 2.4
+Name: askme-rag
+Version: 0.1.1
+Summary: CLI RAG tool for local codebases
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: qdrant-client
+Requires-Dist: openai
+Requires-Dist: rich
+Requires-Dist: python-dotenv
+Requires-Dist: prompt-toolkit
+Requires-Dist: python-docx
+Requires-Dist: pypdf
+
+# askme - Codebase RAG CLI
+
+A Command Line Interface (CLI) tool for local Retrieval-Augmented Generation (RAG) over your codebase. It scans, indexes
+and lets you chat with your project using local LLMs (via an OpenAI-compatible API) and a Qdrant vector database,
+returning context-aware answers with source citations.
+
+The Python package is named `askme` and exposes a single `askme` console script.
+
+---
+
+### Key Features
+
+- Local-first architecture
+  - Runs against any OpenAI-compatible LLM/embedding endpoint (e.g. Ollama) and a Qdrant instance you control.
+  - Code never has to leave your machine if you self-host the services.
+- Hybrid retrieval (dense + sparse)
+  - Dense vectors are produced by the configured embedding model.
+  - Sparse vectors are computed locally with a built-in BM25 encoder (`src/askme/sparse.py`) for keyword/identifier
+    recall.
+- Smart incremental indexing
+  - MD5 hashing of files detects changes, so only new or modified files are re-embedded.
+  - `/reindex` forces a full rebuild on demand.
+- Source citations
+  - Answers include inline `[File: ...]` citations and a citation list, backed by chunk metadata stored in Qdrant.
+- Conversation history
+  - Sessions are auto-saved as JSON in `.cfg/history/` and can be listed, loaded or deleted via `/history`.
+- Rich interactive prompt
+  - Powered by `prompt-toolkit` with multiline and paste modes.
+
+---
+
+### Technology Stack
+
+- Language: Python 3.11+
+- Vector Database: Qdrant (1.x; remote, via Docker, or local file-based)
+- LLM / Embeddings: any OpenAI-compatible API (tested with Ollama)
+- Key libraries:
+  - `qdrant-client` - Qdrant access (dense + sparse vectors).
+  - `openai` - client for the OpenAI-compatible LLM and embedding APIs.
+  - `rich` - terminal UI, panels, markdown rendering.
+  - `prompt-toolkit` - interactive prompt with multiline support.
+  - `python-dotenv` - environment variable loading.
+
+---
+
+### Project Layout
+
+```
+src/askme/
+  config.py      # ConfigManager - .cfg/settings.json and chat history
+  scanner.py     # FileScanner  - traversal, filtering, MD5 hashing
+  vector_db.py   # VectorDBConnector - chunking, embeddings, Qdrant I/O
+  sparse.py      # BM25SparseEncoder - local sparse vector generation
+  llm.py         # LLMInterface - prompt building and chat completion
+  list_files.py  # /index command helpers
+  ui.py          # rich-based UI helpers
+  utils.py       # shared utilities
+  models.py      # data models
+  main.py        # entry point and interactive loop
+tests/           # pytest test suite
+```
+
+#### Data Flow
+
+`Codebase` -> `FileScanner (hash + filter)` -> `VectorDBConnector (chunk + dense embed + BM25 sparse)` ->
+`Qdrant (hybrid index)` -> `LLMInterface (query + retrieved context)` -> `User`
+
+---
+
+### Setup
+
+#### Prerequisites
+
+- Python 3.11+
+- A Qdrant vector database, in one of two modes:
+  - `server` - a running Qdrant instance (local Docker or remote).
+  - `local` - a file-based store on disk, no server or Docker required (quick start).
+- An OpenAI-compatible LLM and embeddings endpoint (e.g. Ollama).
+
+#### 1. Start Qdrant (optional)
+
+In `local` mode you can skip this step entirely - the index is stored on disk under
+`qdrant_local_path` (default `./data/vector_store`) and persists between runs.
+
+For `server` mode, run Qdrant locally via Docker:
+
+```bash
+docker run -p 6333:6333 -p 6334:6334 \
+    -v $(pwd)/qdrant_storage:/qdrant/storage:z \
+    qdrant/qdrant
+```
+
+If you start in `server` mode but the server is unreachable, `askme` offers to fall back to `local` file-based mode
+(default answer: yes) and remembers the choice in
+`.cfg/settings.json`.
+
+#### 2. Prepare models (example with Ollama)
+
+```bash
+ollama pull llama3
+ollama pull embeddinggemma:300m
+```
+
+#### 3. Install
+
+Using `uv` (recommended for development):
+
+```bash
+uv sync
+uv run askme
+```
+
+Or with pip from the project root:
+
+```bash
+pip install .
+askme
+```
+
+Or directly from GitHub:
+
+```bash
+pip install git+https://github.com/varsey/codebase-rag.git
+```
+
+---
+
+### Configuration
+
+On the first run inside a project directory, `askme` prompts for configuration and stores it in `./.cfg/settings.json`.
+Conversation histories are stored next to it under `./.cfg/history/`.
+
+| Option              | Default                               | Description                                                          |
+|:--------------------|:--------------------------------------|:---------------------------------------------------------------------|
+| `qdrant_mode`       | `server`                              | Connection mode: `server` (remote/Docker) or `local` (file-based).   |
+| `qdrant_local_path` | `./data/vector_store`                 | On-disk path for the local file-based store (used in `local` mode).  |
+| `qdrant_host`       | `localhost`                           | Hostname of the Qdrant service (used in `server` mode).              |
+| `qdrant_port`       | `6333`                                | Port of the Qdrant service (used in `server` mode).                  |
+| `llm_api_base`      | `http://localhost:11434/v1`           | Base URL of the OpenAI-compatible LLM API.                           |
+| `llm_api_base_cert` | ``                                    | Optional path to custom cert/CA bundle for LLM API TLS verification. |
+| `vdb_api_base`      | `http://localhost:11434/v1`           | Base URL of the OpenAI-compatible embeddings API.                    |
+| `api_key`           | `sk-...`                              | API key passed to the OpenAI-compatible client.                      |
+| `llm_model`         | `llama3`                              | LLM model name.                                                      |
+| `embedding_model`   | `embeddinggemma:300m`                 | Embedding model name.                                                |
+| `chunk_size`        | `750`                                 | Chunk size in characters.                                            |
+| `chunk_overlap`     | `250`                                 | Overlap between chunks in characters.                                |
+| `buffer_size`       | `1048576`                             | Read buffer size for file scanning.                                  |
+| `top_n`             | `10`                                  | Number of chunks retrieved per query.                                |
+| `collection_name`   | auto-generated                        | Qdrant collection used for this project.                             |
+| `file_extensions`   | `.py, .md, .js, .ts, .go, .java, ...` | File types to index.                                                 |
+| `excluded_dirs`     | `.git, .venv, node_modules, ...`      | Directories skipped during scanning.                                 |
+
+A real example lives in `.cfg/settings.json`.
+
+---
+
+### Usage
+
+Inside the codebase you want to query:
+
+```bash
+askme
+```
+
+On first launch the tool guides you through configuration, scans the project and builds the Qdrant collection.
+Subsequent runs reuse the existing index and only re-embed changed files.
+
+#### Interactive commands
+
+- `/new` - start a fresh conversation (resets the context window).
+- `/history` - list, load or delete saved sessions in `.cfg/history/`.
+- `/reindex` - clear the collection and re-embed the codebase from scratch.
+- `/index` - show the files currently indexed.
+- `/multiline` or `/m` - toggle persistent multiline input (submit with Alt+Enter).
+- `/paste` - one-shot multiline input for a single query.
+- `/exit` or `/quit` - end the session.
+
+---
+
+### Storage Schema
+
+Each point in the Qdrant collection holds a dense vector, a BM25 sparse vector and the following payload:
+
+```json
+{
+    "path": "string (relative path to file)",
+    "content": "string (the actual code chunk)",
+    "hash": "string (MD5 hash of the original file)",
+    "chunk_index": "int",
+    "total_chunks": "int"
+}
+```
+
+---
+
+### Development
+
+- Install dev dependencies and run tests with `uv`:
+
+  ```bash
+  uv sync
+  uv run pytest
+  ```
+
+- The test suite covers config defaults, connection checks, scanner behaviour, sparse BM25 encoding, history/context
+  handling and the main module wiring.
+
+---
+
+### Known Limitations
+
+- Very large files can be memory-heavy during scanning and embedding.
+- Answer quality is bounded by the local LLM's context window.
+- Only text-based source files are supported; binaries are skipped.
+- Hybrid search quality depends on the corpus the BM25 encoder was fit on (the current project).
+
+---
+
+### Contributing
+
+1. Fork the repository.
+2. Create a feature branch (`git checkout -b feature/your-change`).
+3. Keep changes focused and follow the existing code style.
+4. Add or update tests under `tests/` and make sure `uv run pytest` passes.
+5. Open a Pull Request with a clear description.
+
+---
+
+### License
+
+MIT License - see the `LICENSE` file for details (or standard MIT terms if the file is missing).

askme_rag-0.1.1/README.md

@@ -0,0 +1,231 @@
+# askme - Codebase RAG CLI
+
+A Command Line Interface (CLI) tool for local Retrieval-Augmented Generation (RAG) over your codebase. It scans, indexes
+and lets you chat with your project using local LLMs (via an OpenAI-compatible API) and a Qdrant vector database,
+returning context-aware answers with source citations.
+
+The Python package is named `askme` and exposes a single `askme` console script.
+
+---
+
+### Key Features
+
+- Local-first architecture
+  - Runs against any OpenAI-compatible LLM/embedding endpoint (e.g. Ollama) and a Qdrant instance you control.
+  - Code never has to leave your machine if you self-host the services.
+- Hybrid retrieval (dense + sparse)
+  - Dense vectors are produced by the configured embedding model.
+  - Sparse vectors are computed locally with a built-in BM25 encoder (`src/askme/sparse.py`) for keyword/identifier
+    recall.
+- Smart incremental indexing
+  - MD5 hashing of files detects changes, so only new or modified files are re-embedded.
+  - `/reindex` forces a full rebuild on demand.
+- Source citations
+  - Answers include inline `[File: ...]` citations and a citation list, backed by chunk metadata stored in Qdrant.
+- Conversation history
+  - Sessions are auto-saved as JSON in `.cfg/history/` and can be listed, loaded or deleted via `/history`.
+- Rich interactive prompt
+  - Powered by `prompt-toolkit` with multiline and paste modes.
+
+---
+
+### Technology Stack
+
+- Language: Python 3.11+
+- Vector Database: Qdrant (1.x; remote, via Docker, or local file-based)
+- LLM / Embeddings: any OpenAI-compatible API (tested with Ollama)
+- Key libraries:
+  - `qdrant-client` - Qdrant access (dense + sparse vectors).
+  - `openai` - client for the OpenAI-compatible LLM and embedding APIs.
+  - `rich` - terminal UI, panels, markdown rendering.
+  - `prompt-toolkit` - interactive prompt with multiline support.
+  - `python-dotenv` - environment variable loading.
+
+---
+
+### Project Layout
+
+```
+src/askme/
+  config.py      # ConfigManager - .cfg/settings.json and chat history
+  scanner.py     # FileScanner  - traversal, filtering, MD5 hashing
+  vector_db.py   # VectorDBConnector - chunking, embeddings, Qdrant I/O
+  sparse.py      # BM25SparseEncoder - local sparse vector generation
+  llm.py         # LLMInterface - prompt building and chat completion
+  list_files.py  # /index command helpers
+  ui.py          # rich-based UI helpers
+  utils.py       # shared utilities
+  models.py      # data models
+  main.py        # entry point and interactive loop
+tests/           # pytest test suite
+```
+
+#### Data Flow
+
+`Codebase` -> `FileScanner (hash + filter)` -> `VectorDBConnector (chunk + dense embed + BM25 sparse)` ->
+`Qdrant (hybrid index)` -> `LLMInterface (query + retrieved context)` -> `User`
+
+---
+
+### Setup
+
+#### Prerequisites
+
+- Python 3.11+
+- A Qdrant vector database, in one of two modes:
+  - `server` - a running Qdrant instance (local Docker or remote).
+  - `local` - a file-based store on disk, no server or Docker required (quick start).
+- An OpenAI-compatible LLM and embeddings endpoint (e.g. Ollama).
+
+#### 1. Start Qdrant (optional)
+
+In `local` mode you can skip this step entirely - the index is stored on disk under
+`qdrant_local_path` (default `./data/vector_store`) and persists between runs.
+
+For `server` mode, run Qdrant locally via Docker:
+
+```bash
+docker run -p 6333:6333 -p 6334:6334 \
+    -v $(pwd)/qdrant_storage:/qdrant/storage:z \
+    qdrant/qdrant
+```
+
+If you start in `server` mode but the server is unreachable, `askme` offers to fall back to `local` file-based mode
+(default answer: yes) and remembers the choice in
+`.cfg/settings.json`.
+
+#### 2. Prepare models (example with Ollama)
+
+```bash
+ollama pull llama3
+ollama pull embeddinggemma:300m
+```
+
+#### 3. Install
+
+Using `uv` (recommended for development):
+
+```bash
+uv sync
+uv run askme
+```
+
+Or with pip from the project root:
+
+```bash
+pip install .
+askme
+```
+
+Or directly from GitHub:
+
+```bash
+pip install git+https://github.com/varsey/codebase-rag.git
+```
+
+---
+
+### Configuration
+
+On the first run inside a project directory, `askme` prompts for configuration and stores it in `./.cfg/settings.json`.
+Conversation histories are stored next to it under `./.cfg/history/`.
+
+| Option              | Default                               | Description                                                          |
+|:--------------------|:--------------------------------------|:---------------------------------------------------------------------|
+| `qdrant_mode`       | `server`                              | Connection mode: `server` (remote/Docker) or `local` (file-based).   |
+| `qdrant_local_path` | `./data/vector_store`                 | On-disk path for the local file-based store (used in `local` mode).  |
+| `qdrant_host`       | `localhost`                           | Hostname of the Qdrant service (used in `server` mode).              |
+| `qdrant_port`       | `6333`                                | Port of the Qdrant service (used in `server` mode).                  |
+| `llm_api_base`      | `http://localhost:11434/v1`           | Base URL of the OpenAI-compatible LLM API.                           |
+| `llm_api_base_cert` | ``                                    | Optional path to custom cert/CA bundle for LLM API TLS verification. |
+| `vdb_api_base`      | `http://localhost:11434/v1`           | Base URL of the OpenAI-compatible embeddings API.                    |
+| `api_key`           | `sk-...`                              | API key passed to the OpenAI-compatible client.                      |
+| `llm_model`         | `llama3`                              | LLM model name.                                                      |
+| `embedding_model`   | `embeddinggemma:300m`                 | Embedding model name.                                                |
+| `chunk_size`        | `750`                                 | Chunk size in characters.                                            |
+| `chunk_overlap`     | `250`                                 | Overlap between chunks in characters.                                |
+| `buffer_size`       | `1048576`                             | Read buffer size for file scanning.                                  |
+| `top_n`             | `10`                                  | Number of chunks retrieved per query.                                |
+| `collection_name`   | auto-generated                        | Qdrant collection used for this project.                             |
+| `file_extensions`   | `.py, .md, .js, .ts, .go, .java, ...` | File types to index.                                                 |
+| `excluded_dirs`     | `.git, .venv, node_modules, ...`      | Directories skipped during scanning.                                 |
+
+A real example lives in `.cfg/settings.json`.
+
+---
+
+### Usage
+
+Inside the codebase you want to query:
+
+```bash
+askme
+```
+
+On first launch the tool guides you through configuration, scans the project and builds the Qdrant collection.
+Subsequent runs reuse the existing index and only re-embed changed files.
+
+#### Interactive commands
+
+- `/new` - start a fresh conversation (resets the context window).
+- `/history` - list, load or delete saved sessions in `.cfg/history/`.
+- `/reindex` - clear the collection and re-embed the codebase from scratch.
+- `/index` - show the files currently indexed.
+- `/multiline` or `/m` - toggle persistent multiline input (submit with Alt+Enter).
+- `/paste` - one-shot multiline input for a single query.
+- `/exit` or `/quit` - end the session.
+
+---
+
+### Storage Schema
+
+Each point in the Qdrant collection holds a dense vector, a BM25 sparse vector and the following payload:
+
+```json
+{
+    "path": "string (relative path to file)",
+    "content": "string (the actual code chunk)",
+    "hash": "string (MD5 hash of the original file)",
+    "chunk_index": "int",
+    "total_chunks": "int"
+}
+```
+
+---
+
+### Development
+
+- Install dev dependencies and run tests with `uv`:
+
+  ```bash
+  uv sync
+  uv run pytest
+  ```
+
+- The test suite covers config defaults, connection checks, scanner behaviour, sparse BM25 encoding, history/context
+  handling and the main module wiring.
+
+---
+
+### Known Limitations
+
+- Very large files can be memory-heavy during scanning and embedding.
+- Answer quality is bounded by the local LLM's context window.
+- Only text-based source files are supported; binaries are skipped.
+- Hybrid search quality depends on the corpus the BM25 encoder was fit on (the current project).
+
+---
+
+### Contributing
+
+1. Fork the repository.
+2. Create a feature branch (`git checkout -b feature/your-change`).
+3. Keep changes focused and follow the existing code style.
+4. Add or update tests under `tests/` and make sure `uv run pytest` passes.
+5. Open a Pull Request with a clear description.
+
+---
+
+### License
+
+MIT License - see the `LICENSE` file for details (or standard MIT terms if the file is missing).

askme_rag-0.1.1/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "askme-rag"
+version = "0.1.1"
+description = "CLI RAG tool for local codebases"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "qdrant-client",
+    "openai",
+    "rich",
+    "python-dotenv",
+    "prompt-toolkit",
+    "python-docx",
+    "pypdf",
+]
+
+[project.scripts]
+askme = "askme.main:main"
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.2",
+]
+publish = [
+    "twine>=6.2.0",
+]

askme_rag-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

askme_rag-0.1.1/src/askme/config.py

@@ -0,0 +1,73 @@
+import json
+from pathlib import Path
+from typing import Dict, Any, List
+
+QDRANT_MODE_SERVER = "server"
+QDRANT_MODE_LOCAL = "local"
+DEFAULT_QDRANT_MODE = QDRANT_MODE_SERVER
+DEFAULT_QDRANT_LOCAL_PATH = "./data/vector_store"
+
+
+class ConfigManager:
+    def __init__(self, root_dir: Path = None):
+        self.root_dir = root_dir or Path.cwd()
+        self.cfg_path = self.root_dir / ".cfg"
+        self.config_file = self.cfg_path / "settings.json"
+        self.history_dir = self.cfg_path / "history"
+        self.ensure_dirs()
+
+    def ensure_dirs(self):
+        """Create the .cfg folder and history subfolder if they don't exist."""
+        self.cfg_path.mkdir(exist_ok=True, parents=True)
+        self.history_dir.mkdir(exist_ok=True, parents=True)
+
+    def is_configured(self) -> bool:
+        """Check if the tool is already configured."""
+        return self.config_file.exists()
+
+    @staticmethod
+    def get_qdrant_mode(settings: Dict[str, Any]) -> str:
+        """Return the validated Qdrant connection mode ('server' or 'local')."""
+        mode = str(settings.get("qdrant_mode", DEFAULT_QDRANT_MODE)).lower()
+        return mode if mode in (QDRANT_MODE_SERVER, QDRANT_MODE_LOCAL) else DEFAULT_QDRANT_MODE
+
+    @staticmethod
+    def get_qdrant_local_path(settings: Dict[str, Any]) -> str:
+        """Return the on-disk path used for the local file-based Qdrant store."""
+        return settings.get("qdrant_local_path", DEFAULT_QDRANT_LOCAL_PATH)
+
+    def save_config(self, settings: Dict[str, Any]):
+        """Save user configuration to settings.json."""
+        with open(self.config_file, "w") as f:
+            json.dump(settings, f, indent=4)
+
+    def load_config(self) -> Dict[str, Any]:
+        """Load configuration from settings.json."""
+        if not self.is_configured():
+            return {}
+        with open(self.config_file, "r") as f:
+            return json.load(f)
+
+    def save_chat_history(self, chat_id: str, history_data: List[Dict[str, str]]):
+        """Save a conversation history to a JSON file in .cfg/history."""
+        history_file = self.history_dir / f"{chat_id}.json"
+        with open(history_file, "w") as f:
+            json.dump(history_data, f, indent=4)
+
+    def load_chat_history(self, chat_id: str) -> List[Dict[str, str]]:
+        """Load a conversation history from a JSON file in .cfg/history."""
+        history_file = self.history_dir / f"{chat_id}.json"
+        if not history_file.exists():
+            return []
+        with open(history_file, "r") as f:
+            return json.load(f)
+
+    def list_histories(self):
+        """List all conversation histories."""
+        return list(self.history_dir.glob("*.json"))
+
+    def delete_history(self, chat_id: str):
+        """Delete a conversation history file."""
+        history_file = self.history_dir / f"{chat_id}.json"
+        if history_file.exists():
+            history_file.unlink()

askme_rag-0.1.1/src/askme/file_processors.py

@@ -0,0 +1,40 @@
+from abc import ABC, abstractmethod
+from pathlib import Path
+
+import docx
+from pypdf import PdfReader
+
+
+class FileProcessor(ABC):
+    @abstractmethod
+    def extract_text(self, file_path: Path) -> str:
+        """Extract text content from the file."""
+        pass
+
+    @abstractmethod
+    def get_supported_extensions(self) -> set[str]:
+        """Return a set of supported file extensions."""
+        pass
+
+
+class DocxProcessor(FileProcessor):
+    def extract_text(self, file_path: Path) -> str:
+        """Extract text from a .docx file."""
+        doc = docx.Document(file_path)
+        return "\n".join([paragraph.text for paragraph in doc.paragraphs])
+
+    def get_supported_extensions(self) -> set[str]:
+        return {".docx"}
+
+
+class PdfProcessor(FileProcessor):
+    def extract_text(self, file_path: Path) -> str:
+        """Extract text from a .pdf file."""
+        reader = PdfReader(file_path)
+        text_parts = []
+        for page in reader.pages:
+            text_parts.append(page.extract_text() or "")
+        return "\n".join(text_parts)
+
+    def get_supported_extensions(self) -> set[str]:
+        return {".pdf"}

askme_rag-0.1.1/src/askme/list_files.py

@@ -0,0 +1,49 @@
+import json
+import os
+
+from askme.config import ConfigManager
+from askme.vector_db import VectorDBConnector
+
+
+def list_indexed_files():
+    """
+    Retrieves and displays the list of currently indexed files in the vector database.
+    """
+    # Load the configuration from settings.json
+    config_path = 'settings.json'
+    if not os.path.exists(config_path):
+        print(f"Error: {config_path} not found.")
+        return
+
+    with open(config_path, 'r') as config_file:
+        settings = json.load(config_file)
+
+    # Initialize the VectorDBConnector with the settings
+    # We use .get() for optional parameters to handle cases where they might be missing
+    vdb = VectorDBConnector(
+        host=settings.get("qdrant_host", "localhost"),
+        port=settings.get("qdrant_port", 6333),
+        collection_name=settings.get("collection_name", "codebase"),
+        vdb_api_base=settings.get("vdb_api_base", "http://localhost:11434/v1"),
+        model_name=settings.get("embedding_model", "embeddinggemma:300m"),
+        chunk_size=settings.get("chunk_size", 1000),
+        chunk_overlap=settings.get("chunk_overlap", 100),
+        mode=ConfigManager.get_qdrant_mode(settings),
+        local_path=ConfigManager.get_qdrant_local_path(settings)
+    )
+
+    # Fetch the list of indexed files
+    # get_indexed_files returns a dict {path: hash}
+    indexed_files = vdb.get_indexed_files()
+
+    # Display the list of indexed files
+    if not indexed_files:
+        print("No files currently indexed.")
+    else:
+        print("Currently Indexed Files:")
+        # Sort paths for consistent output
+        for file_path in sorted(indexed_files.keys()):
+            print(f"- {file_path}")
+
+if __name__ == "__main__":
+    list_indexed_files()