ragpy-core 1.0.0__tar.gz

ragpy_core-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 William Klusman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the “Software”), to deal
+in the Software without restriction, including without limitation the rights  
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell  
+copies of the Software, and to permit persons to whom the Software is  
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in  
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR  
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,  
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE  
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER  
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING  
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER  
+DEALINGS IN THE SOFTWARE.

ragpy_core-1.0.0/PKG-INFO

@@ -0,0 +1,175 @@
+Metadata-Version: 2.4
+Name: ragpy-core
+Version: 1.0.0
+Summary: A modular Retrieval-Augmented Generation (RAG) pipeline for Python.
+Author: William Klusman
+Author-email: William Klusman <klusmannwilliam@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 William Klusman
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the “Software”), to deal
+        in the Software without restriction, including without limitation the rights  
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell  
+        copies of the Software, and to permit persons to whom the Software is  
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in  
+        all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR  
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,  
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE  
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER  
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING  
+        FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER  
+        DEALINGS IN THE SOFTWARE.
+Keywords: RAG,retrieval-augmented-generation,LLM,vector-database,azure-openai,machine-learning,nlp
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: chromadb>=0.4.0
+Requires-Dist: numpy>=1.20
+Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: requests>=2.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: pypdf>=4.0.0
+Dynamic: author
+Dynamic: license-file
+Dynamic: requires-python
+
+RAGpy
+	RAGpy is a lightweight, modular Retrieval-Augmented Generation (RAG) pipeline for Python. It provides a clear and testable architecture for document ingestion, chunking, embedding, retrieval, reranking, context compression, and grounded answer generation using Azure OpenAI and ChromaDB.
+
+	RAGpy is designed for developers who want a transparent, hackable RAG system without the complexity of large frameworks.
+
+Features
+	Modular ingestion pipeline for text and PDF documents
+
+	Chunking and batching utilities for efficient embedding
+
+	Azure OpenAI embeddings and chat completions
+
+ChromaDB vector database integration
+
+LLM-based reranking for improved retrieval quality
+
+Context compression to reduce token usage
+
+Fully monkeypatch-friendly design for offline testing
+
+Clean architecture suitable for extension and customization
+
+Installation
+Once published to PyPI:
+
+Code
+pip install ragpy
+For development:
+
+Code
+git clone https://github.com/yourusername/ragpy
+cd ragpy
+pip install -e .
+Quickstart Example
+python
+from ragpy.RAGOrchestrator import IngestFile, GenerateAnswer
+from ragpy.VectorDatabase import OpenDatabase
+
+OpenDatabase("AeroDB", "./vectorDB")
+IngestFile("engine_vibration.pdf", "AeroDB")
+
+answer = GenerateAnswer("What causes engine vibration?", "AeroDB")
+print(answer)
+How RAGpy Works
+1. Ingestion
+Load text or PDF using FileLoader
+
+Chunk text using TextChunker
+
+Batch chunks using ChunkBatcher
+
+Generate embeddings with Azure OpenAI
+
+Store vectors and metadata in ChromaDB
+
+2. Retrieval
+Embed the user query
+
+Retrieve top-K candidates from the vector database
+
+3. Reranking
+Use an LLM-based reranker to reorder retrieved chunks by relevance
+
+4. Compression
+Summarize top chunks into a compact context block
+
+5. Answer Generation
+Build a prompt using compressed context
+
+Generate a grounded answer using Azure OpenAI
+
+Project Structure
+Code
+ragpy/
+    AzureOpenAIRelay.py
+    RAGOrchestrator.py
+    VectorDatabase.py
+    Reranker.py
+    ChunkCompressor.py
+    loaders/
+        FileLoader.py
+        TextChunker.py
+    batching/
+        ChunkBatcher.py
+tests/
+docs/
+Requirements
+Python 3.9+
+
+ChromaDB
+
+numpy
+
+tiktoken
+
+pypdf
+
+openai (Azure OpenAI SDK)
+
+Testing
+RAGpy includes a full pytest suite. All Azure calls are monkeypatch-friendly, allowing offline testing with mock LLMs.
+
+Run tests:
+
+Code
+pytest -q
+Contributing
+Contributions are welcome.
+Please open an issue or submit a pull request on GitHub.
+
+Planned enhancements include:
+
+Local embedding support (sentence-transformers)
+
+Hybrid retrieval (vector + keyword)
+
+Multimodal RAG (image + text)
+
+Evaluation tools for relevance and faithfulness
+
+Agentic RAG extensions
+
+License
+RAGpy is released under the MIT License.

ragpy_core-1.0.0/README.md

@@ -0,0 +1,124 @@
+RAGpy
+	RAGpy is a lightweight, modular Retrieval-Augmented Generation (RAG) pipeline for Python. It provides a clear and testable architecture for document ingestion, chunking, embedding, retrieval, reranking, context compression, and grounded answer generation using Azure OpenAI and ChromaDB.
+
+	RAGpy is designed for developers who want a transparent, hackable RAG system without the complexity of large frameworks.
+
+Features
+	Modular ingestion pipeline for text and PDF documents
+
+	Chunking and batching utilities for efficient embedding
+
+	Azure OpenAI embeddings and chat completions
+
+ChromaDB vector database integration
+
+LLM-based reranking for improved retrieval quality
+
+Context compression to reduce token usage
+
+Fully monkeypatch-friendly design for offline testing
+
+Clean architecture suitable for extension and customization
+
+Installation
+Once published to PyPI:
+
+Code
+pip install ragpy
+For development:
+
+Code
+git clone https://github.com/yourusername/ragpy
+cd ragpy
+pip install -e .
+Quickstart Example
+python
+from ragpy.RAGOrchestrator import IngestFile, GenerateAnswer
+from ragpy.VectorDatabase import OpenDatabase
+
+OpenDatabase("AeroDB", "./vectorDB")
+IngestFile("engine_vibration.pdf", "AeroDB")
+
+answer = GenerateAnswer("What causes engine vibration?", "AeroDB")
+print(answer)
+How RAGpy Works
+1. Ingestion
+Load text or PDF using FileLoader
+
+Chunk text using TextChunker
+
+Batch chunks using ChunkBatcher
+
+Generate embeddings with Azure OpenAI
+
+Store vectors and metadata in ChromaDB
+
+2. Retrieval
+Embed the user query
+
+Retrieve top-K candidates from the vector database
+
+3. Reranking
+Use an LLM-based reranker to reorder retrieved chunks by relevance
+
+4. Compression
+Summarize top chunks into a compact context block
+
+5. Answer Generation
+Build a prompt using compressed context
+
+Generate a grounded answer using Azure OpenAI
+
+Project Structure
+Code
+ragpy/
+    AzureOpenAIRelay.py
+    RAGOrchestrator.py
+    VectorDatabase.py
+    Reranker.py
+    ChunkCompressor.py
+    loaders/
+        FileLoader.py
+        TextChunker.py
+    batching/
+        ChunkBatcher.py
+tests/
+docs/
+Requirements
+Python 3.9+
+
+ChromaDB
+
+numpy
+
+tiktoken
+
+pypdf
+
+openai (Azure OpenAI SDK)
+
+Testing
+RAGpy includes a full pytest suite. All Azure calls are monkeypatch-friendly, allowing offline testing with mock LLMs.
+
+Run tests:
+
+Code
+pytest -q
+Contributing
+Contributions are welcome.
+Please open an issue or submit a pull request on GitHub.
+
+Planned enhancements include:
+
+Local embedding support (sentence-transformers)
+
+Hybrid retrieval (vector + keyword)
+
+Multimodal RAG (image + text)
+
+Evaluation tools for relevance and faithfulness
+
+Agentic RAG extensions
+
+License
+RAGpy is released under the MIT License.

ragpy_core-1.0.0/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ragpy-core"
+version = "1.0.0"
+description = "A modular Retrieval-Augmented Generation (RAG) pipeline for Python."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { file = "LICENSE" }
+
+authors = [
+    { name = "William Klusman", email = "klusmannwilliam@gmail.com" }
+]
+
+keywords = [
+    "RAG",
+    "retrieval-augmented-generation",
+    "LLM",
+    "vector-database",
+    "azure-openai",
+    "machine-learning",
+    "nlp"
+]
+
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Software Development :: Libraries :: Python Modules"
+]
+
+dependencies = [
+    "chromadb>=0.4.0",
+    "numpy>=1.20",
+    "tiktoken>=0.5.0",
+    "requests>=2.0",
+    "openai>=1.0.0",
+    "pypdf>=4.0.0"
+]
+
+[project.urls]
+
+[tool.setuptools]
+packages = ["ragpy-core"]
+
+[tool.setuptools.package-data]
+ragpy = ["*.txt", "*.md"]

ragpy_core-1.0.0/ragpy-core/AzureOpenAIRelay.py

@@ -0,0 +1,206 @@
+# -*- coding: utf-8 -*-
+"""
+Azure OpenAI relay utilities for the RAGpy pipeline.
+
+This module provides a thin, deterministic wrapper around the Azure OpenAI
+Python SDK. It exposes simple embedding and chat‑completion functions used
+throughout the RAG workflow, while keeping configuration isolated and
+monkeypatch‑friendly for unit testing.
+
+Two independent clients are supported:
+    • Embedding client — for generating text embeddings.
+    • Chat completion client — for LLM‑based scoring, compression, and
+      answer generation.
+
+The relay is intentionally minimal to ensure predictable behavior and easy
+replacement during offline tests.
+
+"""
+from openai import AzureOpenAI
+
+#new Azure OpenAI SDK
+embedding_Client = None
+chat_Client = None
+embedding_Deployment = None
+completion_Deployment = None
+
+def SetCompletionEndpointInfo(completion_Endpoint, completion_Deployment_Name, completion_Api_Version, api_Key):
+    """
+    Configure the Azure OpenAI chat completion client.
+
+    This sets up the client used for all LLM‑based operations in the RAG
+    pipeline, including reranking, compression, and final answer generation.
+
+    Args:
+        completion_Endpoint (str):
+            The Azure endpoint URL for chat completions.
+        completion_Deployment_Name (str):
+            The name of the deployed chat model.
+        completion_Api_Version (str):
+            The API version to use.
+        api_Key (str):
+            The Azure OpenAI API key.
+
+    Returns:
+        None
+    """
+    global chat_Client, completion_Deployment
+    completion_Deployment = completion_Deployment_Name
+
+    chat_Client = AzureOpenAI(
+        azure_endpoint=completion_Endpoint,
+        api_version=completion_Api_Version,
+        api_key=api_Key
+            
+    )
+    
+    return
+
+def SetEmbeddingEndpointInfo(embedding_Endpoint, embedding_Deployment_Name, embedding_Api_Version, azure_Api_Key):
+    """
+    Configure the Azure OpenAI embedding client.
+
+    This sets up the client used for generating embeddings during ingestion
+    and query processing.
+
+    Args:
+        embedding_Endpoint (str):
+            The Azure endpoint URL for embeddings.
+        embedding_Deployment_Name (str):
+            The name of the deployed embedding model.
+        embedding_Api_Version (str):
+            The API version to use.
+        azure_Api_Key (str):
+            The Azure OpenAI API key.
+
+    Returns:
+        None
+    """
+    global embedding_Client, embedding_Deployment
+    embedding_Deployment = embedding_Deployment_Name
+    
+    embedding_Client = AzureOpenAI(
+        azure_endpoint=embedding_Endpoint,
+        api_version=embedding_Api_Version,
+        api_key = azure_Api_Key
+        )
+    
+    return
+    
+def EmbedText(text):
+    """
+    Generate an embedding vector for a single text string.
+
+    Args:
+        text (str):
+            The input text to embed.
+
+    Returns:
+        list[float]:
+            A 3072‑dimensional embedding vector produced by the configured
+            Azure OpenAI embedding model.
+
+    Raises:
+        ValueError:
+            If the returned embedding dimension is unexpected.
+
+    Notes:
+        - This function is intentionally simple to support monkeypatching
+          during unit tests.
+        - The embedding client must be configured before calling this
+          function.
+    """
+    vector = embedding_Client.embeddings.create(
+        model=embedding_Deployment,
+        input=text
+    ).data[0].embedding
+
+    if len(vector) != 3072:
+        raise ValueError(f"Unexpected embedding dimension: {len(vector)}")
+
+    return vector 
+
+def EmbedChunksInBatches(batched_chunks, batch_size=16):
+    """
+    Embed multiple batches of text chunks.
+
+    Each batch is sent to the embedding model as a single request, and all
+    resulting vectors are returned in a flat list (one embedding per chunk).
+
+    Args:
+        batched_chunks (list[list[str]]):
+            A list of batches, each containing text chunks.
+        batch_size (int):
+            Optional batch size hint (unused but kept for compatibility).
+
+    Returns:
+        list[list[float]]:
+            A flat list of embedding vectors, one per chunk.
+
+    Notes:
+        - Ordering is preserved: embeddings appear in the same order as
+          the input chunks.
+        - This function is intentionally simple to support monkeypatching.
+    """
+    all_vectors = []
+
+    for batch in batched_chunks:
+        response = embedding_Client.embeddings.create(
+            model=embedding_Deployment,
+            input=batch
+        )
+
+        for item in response.data:
+            all_vectors.append(item.embedding)
+
+    return all_vectors
+
+def ChatCompletion(prompt):
+    """
+    Generate a chat completion response using the configured Azure OpenAI client.
+
+    Args:
+        prompt (str):
+            The user prompt to send to the model.
+
+    Returns:
+        str:
+            The model's response text. Returns an empty string if no chat
+            client is configured.
+
+    Notes:
+        - This function is intentionally simple to support monkeypatching
+          during unit tests.
+        - No additional formatting or metadata is returned.
+    """
+    client = GetChatClient()
+    if client is None:
+        # In tests, FakeChat will override this entirely.
+        return ""
+
+    response = client.chat.completions.create(
+        model=GetChatModel(),
+        messages=[{"role": "user", "content": prompt}]
+    )
+
+    return response.choices[0].message.content
+
+def GetChatClient():
+    """
+    Retrieve the configured Azure OpenAI chat client.
+
+    Returns:
+        AzureOpenAI | None:
+            The chat client instance, or None if not configured.
+    """
+    return chat_Client
+
+def GetChatModel():
+    """
+    Retrieve the configured chat model deployment name.
+
+    Returns:
+        str | None:
+            The model deployment name, or None if not configured.
+    """
+    return completion_Deployment

ragpy_core-1.0.0/ragpy-core/ChunkCompressor.py

@@ -0,0 +1,64 @@
+# -*- coding: utf-8 -*-
+"""
+LLM-based context compressor for the RAGpy pipeline.
+
+This module provides a lightweight, deterministic interface for reducing
+multiple retrieved chunks into a single compact summary. The compressor
+uses AzureOpenAIRelay to generate a concise, query‑aware context block
+that preserves meaning while reducing token usage. The design is
+intentionally simple to support both production usage and easy
+monkeypatching during unit tests.
+
+"""
+from ragpy import AzureOpenAIRelay as AI
+
+def CompressChunks(query, chunks):
+    """
+    Compress a list of retrieved chunks into a shorter, unified context block.
+
+    This function sends the provided chunks and user query to an LLM,
+    requesting a concise summary that preserves the information most
+    relevant to the query. It is primarily used to reduce token usage
+    during prompt construction while maintaining grounding in the
+    retrieved content.
+
+    Args:
+        query (str):
+            The user query that determines what information is relevant.
+        chunks (list[dict]):
+            A list of chunk objects, each containing at least a "chunk"
+            field (or "text") holding the raw text to be compressed.
+
+    Returns:
+        str:
+            A compressed context string. Returns an empty string if the
+            compression model is not configured or returns no output.
+
+    Notes:
+        - The function is intentionally simple to allow monkeypatching
+          during unit tests.
+        - The compression prompt is defined inline for clarity and
+          isolation.
+        - If AzureOpenAIRelay is not configured, the function safely
+          returns an empty string.
+    """
+    # If monkeypatched, FakeCompress will run instead of this function.
+    if AI.GetChatClient() is None:
+        return ""
+
+    text = "\n\n".join(c.get("chunk") or c.get("text", "") for c in chunks)
+
+    prompt = f"""
+    You are a compression model. Your job is to compress the following text
+    into a concise summary that preserves meaning and focuses on information
+    relevant to the user query:
+
+    Query:
+    {query}
+
+    {text}
+
+    Return ONLY the compressed summary.
+    """
+
+    return AI.ChatCompletion(prompt)