rag-ingestion 0.1.0__tar.gz

rag_ingestion-0.1.0/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.3
+Name: rag-ingestion
+Version: 0.1.0
+Summary: A lightweight document ingestion component for RAG pipelines.
+Author: satyampandya
+Requires-Dist: docx2txt>=0.9
+Requires-Dist: langchain-community>=0.4.2
+Requires-Dist: langchain-text-splitters>=1.1.2
+Requires-Dist: openpyxl>=3.1.5
+Requires-Dist: pypdf>=6.18.0
+Requires-Dist: python-docx>=1.2.0
+Requires-Dist: tiktoken>=0.14.0
+Requires-Dist: unstructured[xlsx]>=0.27.5
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+
+# RAG Ingestion
+
+A lightweight document ingestion component for Retrieval-Augmented Generation (RAG) pipelines.
+
+RAG Ingestion handles the initial document-processing stage of a RAG workflow: loading supported files, extracting their content, tracking token usage, cleaning extracted text, and splitting documents into chunks.
+
+The package is designed to work with LangChain `Document` objects and can be used as a standalone component or as part of a larger RAG pipeline.
+
+## Features
+
+* Load multiple documents from file paths
+* Support for PDF, DOCX, and XLSX files
+* Extract content into LangChain `Document` objects
+* Token counting with `tiktoken`
+* Optional maximum token budget
+* Optional strict error handling
+* Basic text cleaning while preserving document metadata
+* Configurable LangChain text splitter
+* Configurable chunk size and chunk overlap
+* Compatible with custom tokenizers and text splitters
+
+## Supported File Types
+
+| File type | Loader                    |
+| --------- | ------------------------- |
+| `.pdf`    | `PyPDFLoader`             |
+| `.docx`   | `Docx2txtLoader`          |
+| `.xlsx`   | `UnstructuredExcelLoader` |
+
+## Installation
+
+```bash
+pip install rag-ingestion
+```
+
+## Basic Usage
+
+```python
+from rag_ingestion import RAGIngection
+
+ingestion = RAGIngection()
+
+result = ingestion.load([
+    "document.pdf",
+    "document.docx",
+    "spreadsheet.xlsx",
+])
+
+documents = result["documents"]
+
+chunks = ingestion.chunk(documents)
+```
+
+The `load()` method returns:
+
+```python
+{
+    "documents": [...],
+    "total_tokens": ...,
+    "files_processed": ...
+}
+```
+
+## Configuration
+
+The ingestion component can be configured with optional parameters:
+
+```python
+ingestion = RAGIngection(
+    tokenizer=custom_tokenizer,
+    max_tokens=4000,
+    strict=True,
+    chunk_size=1000,
+    chunk_overlap=200,
+    clean=True,
+)
+```
+
+### Tokenizer
+
+A custom tokenizer can be supplied when required.
+
+If no tokenizer is provided, the package uses the default `cl100k_base` encoding from `tiktoken`.
+
+### Token Budget
+
+`max_tokens` can be used to limit the total number of extracted tokens processed by the loader.
+
+```python
+ingestion = RAGIngection(max_tokens=4000)
+```
+
+If no limit is provided, token processing is unlimited.
+
+### Strict Mode
+
+By default, unsupported or failed files are skipped.
+
+To raise an exception instead:
+
+```python
+ingestion = RAGIngection(strict=True)
+```
+
+### Chunking
+
+Documents can be split using the default recursive character text splitter:
+
+```python
+chunks = ingestion.chunk(documents)
+```
+
+Chunking can be configured with:
+
+```python
+ingestion = RAGIngection(
+    chunk_size=1000,
+    chunk_overlap=200,
+)
+```
+
+A custom LangChain text splitter can also be supplied.
+
+## Pipeline
+
+The basic processing flow is:
+
+```text
+Files
+  ↓
+Load & Extract
+  ↓
+Token Counting
+  ↓
+Text Cleaning
+  ↓
+Chunking
+  ↓
+LangChain Documents
+```
+
+The resulting chunks can then be passed to a retrieval system, vector store, hybrid retriever, or another downstream RAG component.
+
+## Dependencies
+
+This package uses components from the LangChain ecosystem together with document loaders and tokenization utilities.
+
+The package dependencies are defined in `pyproject.toml`.
+
+## Status
+
+This is the initial version of the RAG ingestion component, developed as a modular building block for a larger RAG system.
+
+The package is intentionally focused on document ingestion and preprocessing rather than retrieval or generation.
+
+## License
+
+This project is licensed under the MIT License. See the `LICENSE` file for details.

rag_ingestion-0.1.0/README.md

@@ -0,0 +1,158 @@
+# RAG Ingestion
+
+A lightweight document ingestion component for Retrieval-Augmented Generation (RAG) pipelines.
+
+RAG Ingestion handles the initial document-processing stage of a RAG workflow: loading supported files, extracting their content, tracking token usage, cleaning extracted text, and splitting documents into chunks.
+
+The package is designed to work with LangChain `Document` objects and can be used as a standalone component or as part of a larger RAG pipeline.
+
+## Features
+
+* Load multiple documents from file paths
+* Support for PDF, DOCX, and XLSX files
+* Extract content into LangChain `Document` objects
+* Token counting with `tiktoken`
+* Optional maximum token budget
+* Optional strict error handling
+* Basic text cleaning while preserving document metadata
+* Configurable LangChain text splitter
+* Configurable chunk size and chunk overlap
+* Compatible with custom tokenizers and text splitters
+
+## Supported File Types
+
+| File type | Loader                    |
+| --------- | ------------------------- |
+| `.pdf`    | `PyPDFLoader`             |
+| `.docx`   | `Docx2txtLoader`          |
+| `.xlsx`   | `UnstructuredExcelLoader` |
+
+## Installation
+
+```bash
+pip install rag-ingestion
+```
+
+## Basic Usage
+
+```python
+from rag_ingestion import RAGIngection
+
+ingestion = RAGIngection()
+
+result = ingestion.load([
+    "document.pdf",
+    "document.docx",
+    "spreadsheet.xlsx",
+])
+
+documents = result["documents"]
+
+chunks = ingestion.chunk(documents)
+```
+
+The `load()` method returns:
+
+```python
+{
+    "documents": [...],
+    "total_tokens": ...,
+    "files_processed": ...
+}
+```
+
+## Configuration
+
+The ingestion component can be configured with optional parameters:
+
+```python
+ingestion = RAGIngection(
+    tokenizer=custom_tokenizer,
+    max_tokens=4000,
+    strict=True,
+    chunk_size=1000,
+    chunk_overlap=200,
+    clean=True,
+)
+```
+
+### Tokenizer
+
+A custom tokenizer can be supplied when required.
+
+If no tokenizer is provided, the package uses the default `cl100k_base` encoding from `tiktoken`.
+
+### Token Budget
+
+`max_tokens` can be used to limit the total number of extracted tokens processed by the loader.
+
+```python
+ingestion = RAGIngection(max_tokens=4000)
+```
+
+If no limit is provided, token processing is unlimited.
+
+### Strict Mode
+
+By default, unsupported or failed files are skipped.
+
+To raise an exception instead:
+
+```python
+ingestion = RAGIngection(strict=True)
+```
+
+### Chunking
+
+Documents can be split using the default recursive character text splitter:
+
+```python
+chunks = ingestion.chunk(documents)
+```
+
+Chunking can be configured with:
+
+```python
+ingestion = RAGIngection(
+    chunk_size=1000,
+    chunk_overlap=200,
+)
+```
+
+A custom LangChain text splitter can also be supplied.
+
+## Pipeline
+
+The basic processing flow is:
+
+```text
+Files
+  ↓
+Load & Extract
+  ↓
+Token Counting
+  ↓
+Text Cleaning
+  ↓
+Chunking
+  ↓
+LangChain Documents
+```
+
+The resulting chunks can then be passed to a retrieval system, vector store, hybrid retriever, or another downstream RAG component.
+
+## Dependencies
+
+This package uses components from the LangChain ecosystem together with document loaders and tokenization utilities.
+
+The package dependencies are defined in `pyproject.toml`.
+
+## Status
+
+This is the initial version of the RAG ingestion component, developed as a modular building block for a larger RAG system.
+
+The package is intentionally focused on document ingestion and preprocessing rather than retrieval or generation.
+
+## License
+
+This project is licensed under the MIT License. See the `LICENSE` file for details.

rag_ingestion-0.1.0/pyproject.toml

@@ -0,0 +1,23 @@
+[project]
+name = "rag-ingestion"
+version = "0.1.0"
+description = "A lightweight document ingestion component for RAG pipelines."
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "docx2txt>=0.9",
+    "langchain-community>=0.4.2",
+    "langchain-text-splitters>=1.1.2",
+    "openpyxl>=3.1.5",
+    "pypdf>=6.18.0",
+    "python-docx>=1.2.0",
+    "tiktoken>=0.14.0",
+    "unstructured[xlsx]>=0.27.5",
+]
+
+[[project.authors]]
+name = "satyampandya"
+
+[build-system]
+requires = ["uv_build>=0.12.10,<0.13.0"]
+build-backend = "uv_build"

rag_ingestion-0.1.0/pyproject.toml.orig

@@ -0,0 +1,23 @@
+[project]
+name = "rag-ingestion"
+version = "0.1.0"
+description = "A lightweight document ingestion component for RAG pipelines."
+readme = "README.md"
+authors = [
+    { name = "satyampandya" }
+]
+requires-python = ">=3.13"
+dependencies = [
+    "docx2txt>=0.9",
+    "langchain-community>=0.4.2",
+    "langchain-text-splitters>=1.1.2",
+    "openpyxl>=3.1.5",
+    "pypdf>=6.18.0",
+    "python-docx>=1.2.0",
+    "tiktoken>=0.14.0",
+    "unstructured[xlsx]>=0.27.5",
+]
+
+[build-system]
+requires = ["uv_build>=0.12.10,<0.13.0"]
+build-backend = "uv_build"

rag_ingestion-0.1.0/src/rag_ingestion/__init__.py

@@ -0,0 +1,7 @@
+"""Public API for the RAG ingestion package."""
+
+# The implementation module/class retain their original spelling for
+# compatibility; export the correctly spelled public API used by callers.
+from .ingestion import RAGIngestion
+
+__all__ = ["RAGIngestion"]

rag_ingestion-0.1.0/src/rag_ingestion/ingestion.py

@@ -0,0 +1,166 @@
+import tiktoken
+from pathlib import Path
+from typing import Any, Sequence
+from langchain_core.documents import Document
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+from langchain_community.document_loaders import (
+    Docx2txtLoader,
+    PyPDFLoader,
+    UnstructuredExcelLoader
+) 
+
+class RAGIngestion:
+    """Identify uploaded files and extract their content."""
+    LOADERS = {
+        ".pdf": PyPDFLoader,
+        ".docx": Docx2txtLoader,
+        ".xlsx": UnstructuredExcelLoader
+    }
+
+    def __init__(
+              self,
+              tokenizer: Any | None = None,
+              max_tokens: int | None = None,
+              strict: bool = False,
+              text_splitter: Any | None = None,
+              chunk_size: int = 1000,
+              chunk_overlap: int = 200,
+              clean: bool = True
+    ) -> None:
+         
+         """
+        Initialize the ingestion pipeline.
+
+        Args:
+            tokenizer:
+                Optional tokenizer. If omitted, the default tiktoken
+                encoding is used when token counting is required.
+
+            max_tokens:
+                Maximum number of tokens to process. None means unlimited.
+
+            strict:
+                If True, raise an exception when a file cannot be processed.
+                If False, skip unsupported/failed files.
+
+            text_splitter:
+                Optional custom LangChain text splitter.
+
+            chunk_size:
+                Default chunk size.
+
+            chunk_overlap:
+                Default chunk overlap.
+
+            clean:
+                If True, clean documents before chunking.      
+         """
+         
+         self.max_tokens = max_tokens
+         self.strict = strict
+         self.tokenizer = tokenizer or tiktoken.get_encoding("cl100k_base")
+
+         self.text_splitter = text_splitter or RecursiveCharacterTextSplitter(
+             chunk_size=chunk_size,
+             chunk_overlap=chunk_overlap
+         )
+         self.clean_enabled = clean
+
+    def load(
+              self, 
+              files: str | Path | Sequence[str | Path]
+              ) -> dict:
+         """
+        Load one or multiple files and extract their content.
+
+        Args:
+            files:
+                A single file path or a list of file paths.
+
+        Returns:
+            Dictionary containing:
+                documents:
+                    Extracted LangChain Document objects.
+
+                total_tokens:
+                    Total number of extracted tokens.
+
+                files_processed:
+                    Number of successfully processed files.
+        """
+         if isinstance(files, (str, Path)):
+            files = [files]
+
+         documents = []
+         total_tokens = 0
+         files_processed = 0   
+
+         for file in files:
+             file_path = Path(file)
+             extension = file_path.suffix.lower()
+
+             if extension not in self.LOADERS:
+                 if self.strict:
+                     raise ValueError(
+                         f"Unsupported file type: {extension}"
+                         )
+                 continue
+
+             try:
+                 loader = self.LOADERS[extension](str(file_path))
+                 file_documents = loader.load()
+                 file_tokens = sum(len(self.tokenizer.encode(document.page_content))for document in file_documents) 
+                 if (
+                     self.max_tokens is not None and
+                     total_tokens + file_tokens > self.max_tokens
+                 ):
+                     break
+
+                 documents.extend(file_documents)
+                 total_tokens += file_tokens
+                 files_processed += 1
+             except Exception:
+                 if self.strict:
+                     raise 
+                     
+         return {
+             "documents": documents,
+             "total_tokens": total_tokens,
+             "files_processed": files_processed
+         }   
+
+    def clean(self, documents: list[Document]) -> list[Document]:
+        """Perform basic cleaning while preserving document metadata."""
+        cleaned_documents = []
+
+        for document in documents:
+            text = document.page_content
+            text = text.replace("\r\n", "\n").replace("\r", "\n")
+            lines = [line.strip() for line in text.split("\n")]
+            text = "\n".join(line for line in lines if line).strip()
+
+            if not text:
+                continue
+
+            cleaned_documents.append(
+                Document(
+                    page_content=text,
+                    metadata=document.metadata.copy()
+                )
+            )
+
+        return cleaned_documents
+        
+    def chunk(
+            self, 
+            documents: list[Document],
+            text_splitter: Any | None = None,
+            ) -> list[Document]:
+        """Split documents using a LangChain text splitter."""
+
+        if self.clean_enabled:
+            documents = self.clean(documents)
+
+        splitter = text_splitter or self.text_splitter
+
+        return splitter.split_documents(documents)