vectoriz 0.0.3__tar.gz → 0.0.5__tar.gz

vectoriz-0.0.5/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: vectoriz
+Version: 0.0.5
+Summary: Python library for creating vectorized data from text or files.
+Home-page: https://github.com/PedroHenriqueDevBR/vectoriz
+Author: PedroHenriqueDevBR
+Author-email: pedro.henrique.particular@gmail.com
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: faiss-cpu==1.10.0
+Requires-Dist: numpy==2.2.4
+Requires-Dist: sentence-transformers==4.0.2
+Requires-Dist: python-docx==1.1.2
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Vectoriz
+
+[![PyPI version](https://badge.fury.io/py/vectoriz.svg)](https://pypi.org/project/vectoriz/)
+
+[![GitHub license](https://img.shields.io/github/license/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/blob/main/LICENSE)
+
+[![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)
+
+[![GitHub issues](https://img.shields.io/github/issues/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/issues)
+
+[![GitHub stars](https://img.shields.io/github/stars/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/stargazers)
+
+[![GitHub forks](https://img.shields.io/github/forks/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/network)
+
+Vectoriz is available on PyPI and can be installed via pip:
+
+```bash
+pip install vectoriz
+```
+
+A tool for generating vector embeddings for Retrieval-Augmented Generation (RAG) applications.
+
+## Overview
+
+This project provides utilities to create, manage, and optimize vector embeddings for use in RAG systems. It streamlines the process of converting documents and data sources into vector representations suitable for semantic search and retrieval.
+
+## Features
+
+- Document processing and chunking
+- Vector embedding generation using various models
+- Vector database integration
+- Optimization tools for RAG performance
+- Easy-to-use API for embedding creation
+
+## Installation
+
+```bash
+git clone https://github.com/PedroHenriqueDevBR/vectoriz.git
+cd vectoriz
+pip install -r requirements.txt
+```
+
+## Usage
+
+```python
+# initial informations
+index_db_path = "./data/faiss_db.index" # path to save/load index
+np_db_path = "./data/np_db.npz" # path to save/load numpy data
+directory_path = "/home/username/Documents/" # Path where the files (.txt, .docx) are saved
+
+# Class instance
+transformer = TokenTransformer()
+files_features = FilesFeature()
+
+# Load files and create a argument class (pack with embedings, chunk_names and text_list)
+argument = files_features.load_all_files_from_directory(directory_path)
+
+# Created FAISS index to be used in queries
+token_data = transformer.create_index(argument.text_list)
+index = token_data.index
+
+# To load files from VectorDB use
+vector_client = VectorDBClient()
+vector_client.load_data(self.index_db_path, self.np_db_path)
+index = vector_client.faiss_index
+argument = vector_client.file_argument
+
+# To save data on VectorDB use
+vector_client = VectorDBClient(index, argument)
+vector_client.save_data(index_db_path, np_db_path)
+
+# To search information on index
+query = input(">>> ")
+amoount_content = 1
+response = self.transformer.search(query, self.index, self.argument.text_list, amoount_content)
+print(response)
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

vectoriz-0.0.5/README.md

@@ -0,0 +1,85 @@
+# Vectoriz
+
+[![PyPI version](https://badge.fury.io/py/vectoriz.svg)](https://pypi.org/project/vectoriz/)
+
+[![GitHub license](https://img.shields.io/github/license/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/blob/main/LICENSE)
+
+[![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)
+
+[![GitHub issues](https://img.shields.io/github/issues/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/issues)
+
+[![GitHub stars](https://img.shields.io/github/stars/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/stargazers)
+
+[![GitHub forks](https://img.shields.io/github/forks/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/network)
+
+Vectoriz is available on PyPI and can be installed via pip:
+
+```bash
+pip install vectoriz
+```
+
+A tool for generating vector embeddings for Retrieval-Augmented Generation (RAG) applications.
+
+## Overview
+
+This project provides utilities to create, manage, and optimize vector embeddings for use in RAG systems. It streamlines the process of converting documents and data sources into vector representations suitable for semantic search and retrieval.
+
+## Features
+
+- Document processing and chunking
+- Vector embedding generation using various models
+- Vector database integration
+- Optimization tools for RAG performance
+- Easy-to-use API for embedding creation
+
+## Installation
+
+```bash
+git clone https://github.com/PedroHenriqueDevBR/vectoriz.git
+cd vectoriz
+pip install -r requirements.txt
+```
+
+## Usage
+
+```python
+# initial informations
+index_db_path = "./data/faiss_db.index" # path to save/load index
+np_db_path = "./data/np_db.npz" # path to save/load numpy data
+directory_path = "/home/username/Documents/" # Path where the files (.txt, .docx) are saved
+
+# Class instance
+transformer = TokenTransformer()
+files_features = FilesFeature()
+
+# Load files and create a argument class (pack with embedings, chunk_names and text_list)
+argument = files_features.load_all_files_from_directory(directory_path)
+
+# Created FAISS index to be used in queries
+token_data = transformer.create_index(argument.text_list)
+index = token_data.index
+
+# To load files from VectorDB use
+vector_client = VectorDBClient()
+vector_client.load_data(self.index_db_path, self.np_db_path)
+index = vector_client.faiss_index
+argument = vector_client.file_argument
+
+# To save data on VectorDB use
+vector_client = VectorDBClient(index, argument)
+vector_client.save_data(index_db_path, np_db_path)
+
+# To search information on index
+query = input(">>> ")
+amoount_content = 1
+response = self.transformer.search(query, self.index, self.argument.text_list, amoount_content)
+print(response)
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

{vectoriz-0.0.3 → vectoriz-0.0.5}/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup, find_packages
 
 setup(
     name="vectoriz",
-    version="0.0.3",
+    version="0.0.5",
     author="PedroHenriqueDevBR",
     author_email="pedro.henrique.particular@gmail.com",
     description="Python library for creating vectorized data from text or files.",

{vectoriz-0.0.3 → vectoriz-0.0.5}/vectoriz/files.py

@@ -2,7 +2,7 @@ import os
 import docx
 import numpy as np
 from typing import Optional
-from vectoriz.token_transformer import TokenTransformer
+from token_transformer import TokenTransformer
 
 class FileArgument:
     def __init__(
@@ -127,7 +127,7 @@ class FilesFeature:
             full_text.append(paragraph.text)
         return "\n".join(full_text)
 
-    def load_txt_files_from_directory(self, directory: str) -> FileArgument:
+    def load_txt_files_from_directory(self, directory: str, verbose: bool = False) -> FileArgument:
         """
         Load all text files from the specified directory and extract their content.
         This method scans the specified directory for files with the '.txt' extension
@@ -145,16 +145,22 @@ class FilesFeature:
         argument: FileArgument = FileArgument([], [], [])
         for file in os.listdir(directory):
             if not file.endswith(".txt"):
+                if verbose:
+                    print(f"Error file: {file}")
                 continue
             
             text = self._extract_txt_content(directory, file)
             if text is None:
+                if verbose:
+                    print(f"Error file: {file}")
                 continue
             
             argument.add_data(file, text)
+            if verbose:
+                    print(f"Loaded txt file: {file}")
         return argument
 
-    def load_docx_files_from_directory(self, directory: str) -> FileArgument:
+    def load_docx_files_from_directory(self, directory: str, verbose: bool = False) -> FileArgument:
         """
         Load all Word (.docx) files from the specified directory and extract their content.
 
@@ -174,16 +180,22 @@ class FilesFeature:
         argument: FileArgument = FileArgument([], [], [])
         for file in os.listdir(directory):
             if not file.endswith(".docx"):
+                if verbose:
+                    print(f"Error file: {file}")
                 continue
             
             text = self._extract_docx_content(directory, file)
             if text is None:
+                if verbose:
+                    print(f"Error file: {file}")
                 continue
             
             argument.add_data(file, text)
+            if verbose:
+                print(f"Loaded Word file: {file}")
         return argument
 
-    def load_all_files_from_directory(self, directory: str) -> FileArgument:
+    def load_all_files_from_directory(self, directory: str, verbose: bool =  False) -> FileArgument:
         """
         Load all supported files (.txt and .docx) from the specified directory and its subdirectories.
 
@@ -199,15 +211,23 @@ class FilesFeature:
         argument: FileArgument = FileArgument([], [], [])
         for root, _, files in os.walk(directory):
             for file in files:
+                readed = False
                 if file.endswith(".txt"):
                     text = self._extract_txt_content(root, file)
                     if text is not None:
                         argument.add_data(file, text)
+                        readed = True
                 elif file.endswith(".docx"):
                     try:
                         text = self._extract_docx_content(root, file)
                         if text is not None:
                             argument.add_data(file, text)
+                            readed = True
                     except Exception as e:
                         print(f"Error processing {file}: {str(e)}")
+
+                if verbose and readed:
+                    print(f"Loaded file: {file}")
+                elif verbose and not readed:
+                    print(f"Error file: {file}")
         return argument

{vectoriz-0.0.3 → vectoriz-0.0.5}/vectoriz/token_transformer.py

@@ -73,15 +73,16 @@ class TokenTransformer:
     def search(
         self,
         query: str,
-        data: TokenData,
+        index: faiss.IndexFlatL2,
+        texts: list[str],
         context_amount: int = 1,
     ) -> str:
         query_embedding = self._query_to_embeddings(query)
-        _, I = data.index.search(query_embedding, k=context_amount)
+        _, I = index.search(query_embedding, k=context_amount)
         context = ""
 
         for i in I[0]:
-            context += data.texts[i] + "\n"
+            context += texts[i] + "\n"
 
         return context.strip()
 

{vectoriz-0.0.3 → vectoriz-0.0.5}/vectoriz/vector_db.py

@@ -3,8 +3,8 @@ import faiss
 import numpy as np
 from typing import Optional
 
-from vectoriz.files import FileArgument
-from vectoriz.token_transformer import TokenTransformer
+from files import FileArgument
+from token_transformer import TokenTransformer
 
 
 class VectorDBClient:
@@ -54,15 +54,6 @@ class VectorDBClient:
 
 class VectorDB:
 
-    def __init__(self):
-        """
-        Constructor for the class.
-
-        Initializes the following attributes:
-        - transformer: A TokenTransformer instance for text transformation.
-        """
-        self.transformer = TokenTransformer()
-
     def load_saved_data(
         self, faiss_db_path: str, np_db_path: str
     ) -> Optional[VectorDBClient]:
@@ -158,13 +149,14 @@ class VectorDB:
             - 'chunk_names': The chunk names
             - 'texts': The text content
         """
+        transformer = TokenTransformer()
         np_db_path = np_db_path if np_db_path.endswith(".npz") else np_db_path + ".npz"
 
         embeddings_np: np.ndarray = None
         if argument.ndarray_data is not None:
             embeddings_np = argument.ndarray_data
         else:
-            embeddings_np = self.transformer.get_np_vectors(argument.embeddings)
+            embeddings_np = transformer.get_np_vectors(argument.embeddings)
 
         np.savez(
             np_db_path,

vectoriz-0.0.5/vectoriz.egg-info/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: vectoriz
+Version: 0.0.5
+Summary: Python library for creating vectorized data from text or files.
+Home-page: https://github.com/PedroHenriqueDevBR/vectoriz
+Author: PedroHenriqueDevBR
+Author-email: pedro.henrique.particular@gmail.com
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: faiss-cpu==1.10.0
+Requires-Dist: numpy==2.2.4
+Requires-Dist: sentence-transformers==4.0.2
+Requires-Dist: python-docx==1.1.2
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Vectoriz
+
+[![PyPI version](https://badge.fury.io/py/vectoriz.svg)](https://pypi.org/project/vectoriz/)
+
+[![GitHub license](https://img.shields.io/github/license/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/blob/main/LICENSE)
+
+[![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)
+
+[![GitHub issues](https://img.shields.io/github/issues/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/issues)
+
+[![GitHub stars](https://img.shields.io/github/stars/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/stargazers)
+
+[![GitHub forks](https://img.shields.io/github/forks/PedroHenriqueDevBR/vectoriz)](https://github.com/PedroHenriqueDevBR/vectoriz/network)
+
+Vectoriz is available on PyPI and can be installed via pip:
+
+```bash
+pip install vectoriz
+```
+
+A tool for generating vector embeddings for Retrieval-Augmented Generation (RAG) applications.
+
+## Overview
+
+This project provides utilities to create, manage, and optimize vector embeddings for use in RAG systems. It streamlines the process of converting documents and data sources into vector representations suitable for semantic search and retrieval.
+
+## Features
+
+- Document processing and chunking
+- Vector embedding generation using various models
+- Vector database integration
+- Optimization tools for RAG performance
+- Easy-to-use API for embedding creation
+
+## Installation
+
+```bash
+git clone https://github.com/PedroHenriqueDevBR/vectoriz.git
+cd vectoriz
+pip install -r requirements.txt
+```
+
+## Usage
+
+```python
+# initial informations
+index_db_path = "./data/faiss_db.index" # path to save/load index
+np_db_path = "./data/np_db.npz" # path to save/load numpy data
+directory_path = "/home/username/Documents/" # Path where the files (.txt, .docx) are saved
+
+# Class instance
+transformer = TokenTransformer()
+files_features = FilesFeature()
+
+# Load files and create a argument class (pack with embedings, chunk_names and text_list)
+argument = files_features.load_all_files_from_directory(directory_path)
+
+# Created FAISS index to be used in queries
+token_data = transformer.create_index(argument.text_list)
+index = token_data.index
+
+# To load files from VectorDB use
+vector_client = VectorDBClient()
+vector_client.load_data(self.index_db_path, self.np_db_path)
+index = vector_client.faiss_index
+argument = vector_client.file_argument
+
+# To save data on VectorDB use
+vector_client = VectorDBClient(index, argument)
+vector_client.save_data(index_db_path, np_db_path)
+
+# To search information on index
+query = input(">>> ")
+amoount_content = 1
+response = self.transformer.search(query, self.index, self.argument.text_list, amoount_content)
+print(response)
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

vectoriz-0.0.3/PKG-INFO

@@ -1,60 +0,0 @@
-Metadata-Version: 2.4
-Name: vectoriz
-Version: 0.0.3
-Summary: Python library for creating vectorized data from text or files.
-Home-page: https://github.com/PedroHenriqueDevBR/vectoriz
-Author: PedroHenriqueDevBR
-Author-email: pedro.henrique.particular@gmail.com
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.12
-Description-Content-Type: text/markdown
-Requires-Dist: faiss-cpu==1.10.0
-Requires-Dist: numpy==2.2.4
-Requires-Dist: sentence-transformers==4.0.2
-Requires-Dist: python-docx==1.1.2
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
-# RAG-vector-creator
-
-## Overview
-This project implements a RAG (Retrieval-Augmented Generation) system for creating and managing vector embeddings from documents using FAISS and NumPy libraries. It efficiently transforms text data into high-dimensional vector representations that enable semantic search capabilities, similarity matching, and context-aware document retrieval for enhanced question answering applications.
-
-## Features
-
-- Document ingestion and preprocessing
-- Vector embedding generation using state-of-the-art models
-- Efficient storage and retrieval of embeddings
-- Integration with LLM-based generation systems
-
-## Installation
-
-```bash
-pip install -r requirements.txt
-python app.py
-```
-
-## Build lib
-
-To build the lib run the commands:
-
-```
-python setup.py sdist bdist_wheel
-```
-
-To test the install run:
-```
-pip install .
-```
-
-## License
-
-MIT

vectoriz-0.0.3/README.md

@@ -1,35 +0,0 @@
-# RAG-vector-creator
-
-## Overview
-This project implements a RAG (Retrieval-Augmented Generation) system for creating and managing vector embeddings from documents using FAISS and NumPy libraries. It efficiently transforms text data into high-dimensional vector representations that enable semantic search capabilities, similarity matching, and context-aware document retrieval for enhanced question answering applications.
-
-## Features
-
-- Document ingestion and preprocessing
-- Vector embedding generation using state-of-the-art models
-- Efficient storage and retrieval of embeddings
-- Integration with LLM-based generation systems
-
-## Installation
-
-```bash
-pip install -r requirements.txt
-python app.py
-```
-
-## Build lib
-
-To build the lib run the commands:
-
-```
-python setup.py sdist bdist_wheel
-```
-
-To test the install run:
-```
-pip install .
-```
-
-## License
-
-MIT

vectoriz-0.0.3/vectoriz.egg-info/PKG-INFO

@@ -1,60 +0,0 @@
-Metadata-Version: 2.4
-Name: vectoriz
-Version: 0.0.3
-Summary: Python library for creating vectorized data from text or files.
-Home-page: https://github.com/PedroHenriqueDevBR/vectoriz
-Author: PedroHenriqueDevBR
-Author-email: pedro.henrique.particular@gmail.com
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.12
-Description-Content-Type: text/markdown
-Requires-Dist: faiss-cpu==1.10.0
-Requires-Dist: numpy==2.2.4
-Requires-Dist: sentence-transformers==4.0.2
-Requires-Dist: python-docx==1.1.2
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
-# RAG-vector-creator
-
-## Overview
-This project implements a RAG (Retrieval-Augmented Generation) system for creating and managing vector embeddings from documents using FAISS and NumPy libraries. It efficiently transforms text data into high-dimensional vector representations that enable semantic search capabilities, similarity matching, and context-aware document retrieval for enhanced question answering applications.
-
-## Features
-
-- Document ingestion and preprocessing
-- Vector embedding generation using state-of-the-art models
-- Efficient storage and retrieval of embeddings
-- Integration with LLM-based generation systems
-
-## Installation
-
-```bash
-pip install -r requirements.txt
-python app.py
-```
-
-## Build lib
-
-To build the lib run the commands:
-
-```
-python setup.py sdist bdist_wheel
-```
-
-To test the install run:
-```
-pip install .
-```
-
-## License
-
-MIT