collections-cache 0.1.0__tar.gz

collections_cache-0.1.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.1
+Name: collections-cache
+Version: 0.1.0
+Summary: Collection Cache is a Python package for managing data collections across multiple SQLite databases. It allows efficient storage, retrieval, and updating of key-value pairs, supporting various data types serialized with pickle. The package uses parallel processing for fast access and manipulation of large collections.
+License: MIT
+Author: Luiz-Trindade
+Author-email: luiz.gabriel.m.trindade@gmail.com
+Requires-Python: >=3.12,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Description-Content-Type: text/markdown
+
+# Collection Cache
+
+`Collection Cache` is a simple and efficient caching solution built using SQLite databases. It allows for storing, updating, and retrieving data using unique keys, supporting complex data types through the use of `pickle`. It is designed to scale across multiple CPU cores by distributing the data across multiple SQLite databases.
+
+## Features
+
+- **Multiple SQLite databases**: Distributes data across multiple databases for better scalability.
+- **Key-value store**: Store data as key-value pairs.
+- **Supports complex data types**: Data is serialized using `pickle`, so you can store lists, dictionaries, and other complex Python objects.
+- **Parallel processing**: Uses Python’s `multiprocessing` to handle large collections in parallel across multiple CPU cores.
+- **Efficient data retrieval**: Retrieves stored data based on the key using an efficient search across the collection.
+
+## Installation
+
+To install the `Collection Cache` package, use [Poetry](https://python-poetry.org/) for managing dependencies.
+
+1. Clone the repository:
+
+    ```bash
+    git clone https://github.com/Luiz-Trindade/collections_cache.git
+    cd collection-cache
+    ```
+
+2. Install the package with Poetry:
+
+    ```bash
+    poetry install
+    ```
+
+## Usage
+
+To use the `Collection Cache` package, you can import the main class `Collection_Cache` and interact with your collection.
+
+### Example:
+
+```python
+from collection_cache import Collection_Cache
+
+# Create a new collection
+cache = Collection_Cache("STORE")
+
+# Set a key-value pair
+cache.set_key("alunos", ["Luiz", "Marcos", "João"])
+
+# Get the value by key
+students = cache.get_key("alunos")
+print(students)  # Output: ['Luiz', 'Marcos', 'João']
+```
+
+### Methods:
+
+- **`set_key(key, value)`**: Set a key-value pair in the cache. If the key already exists, it will be updated.
+- **`get_key(key)`**: Retrieve the value associated with a key.
+- **`create_collection()`**: Initializes the collection and sets up databases for key-value storage.
+- **`get_all_keys()`**: Retrieves all keys from the collection.
+
+## Development
+
+To contribute or run tests:
+
+1. Install development dependencies:
+
+    ```bash
+    poetry install --dev
+    ```
+
+2. Run tests:
+
+    ```bash
+    poetry run pytest
+    ```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgements
+
+- This package was created to demonstrate how to work with SQLite, `pickle`, and Python's `multiprocessing` module.
+
+

collections_cache-0.1.0/README.md

@@ -0,0 +1,80 @@
+# Collection Cache
+
+`Collection Cache` is a simple and efficient caching solution built using SQLite databases. It allows for storing, updating, and retrieving data using unique keys, supporting complex data types through the use of `pickle`. It is designed to scale across multiple CPU cores by distributing the data across multiple SQLite databases.
+
+## Features
+
+- **Multiple SQLite databases**: Distributes data across multiple databases for better scalability.
+- **Key-value store**: Store data as key-value pairs.
+- **Supports complex data types**: Data is serialized using `pickle`, so you can store lists, dictionaries, and other complex Python objects.
+- **Parallel processing**: Uses Python’s `multiprocessing` to handle large collections in parallel across multiple CPU cores.
+- **Efficient data retrieval**: Retrieves stored data based on the key using an efficient search across the collection.
+
+## Installation
+
+To install the `Collection Cache` package, use [Poetry](https://python-poetry.org/) for managing dependencies.
+
+1. Clone the repository:
+
+    ```bash
+    git clone https://github.com/Luiz-Trindade/collections_cache.git
+    cd collection-cache
+    ```
+
+2. Install the package with Poetry:
+
+    ```bash
+    poetry install
+    ```
+
+## Usage
+
+To use the `Collection Cache` package, you can import the main class `Collection_Cache` and interact with your collection.
+
+### Example:
+
+```python
+from collection_cache import Collection_Cache
+
+# Create a new collection
+cache = Collection_Cache("STORE")
+
+# Set a key-value pair
+cache.set_key("alunos", ["Luiz", "Marcos", "João"])
+
+# Get the value by key
+students = cache.get_key("alunos")
+print(students)  # Output: ['Luiz', 'Marcos', 'João']
+```
+
+### Methods:
+
+- **`set_key(key, value)`**: Set a key-value pair in the cache. If the key already exists, it will be updated.
+- **`get_key(key)`**: Retrieve the value associated with a key.
+- **`create_collection()`**: Initializes the collection and sets up databases for key-value storage.
+- **`get_all_keys()`**: Retrieves all keys from the collection.
+
+## Development
+
+To contribute or run tests:
+
+1. Install development dependencies:
+
+    ```bash
+    poetry install --dev
+    ```
+
+2. Run tests:
+
+    ```bash
+    poetry run pytest
+    ```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgements
+
+- This package was created to demonstrate how to work with SQLite, `pickle`, and Python's `multiprocessing` module.
+

collections_cache-0.1.0/collections_cache/collections_cahce.py

@@ -0,0 +1,106 @@
+import sqlite3
+from multiprocessing import Pool
+from os import cpu_count, path, makedirs, scandir
+from itertools import chain
+from random import choice
+#from threading import Thread as task
+import pickle
+
+class Collection_Cache:
+    def __init__(self, collection_name):
+        self.collection_name        = collection_name
+        self.cpu_cores              = cpu_count()
+        self.collection_dir         = path.join("./Collections", self.collection_name)
+        self.databases_list         = []
+        self.keys_databases         = {}
+
+        #print(f"Collection '{self.collection_name}' created!")
+        #print(f"Number of cpu cores: {self.cpu_cores}")
+        self.create_collection()
+        self.get_all_databases()
+
+    def create_collection(self):
+        makedirs(self.collection_dir, exist_ok=True)
+        
+        for core in range(self.cpu_cores):
+            db_path = path.join(self.collection_dir, f"database_{core}.db")
+            self.initialize_databases(db_path)
+
+    def initialize_databases(self, db_path):
+        conn = sqlite3.connect(db_path)
+        conn.execute("PRAGMA journal_mode=WAL;")
+        conn.execute("""
+            CREATE TABLE IF NOT EXISTS data(
+                key     TEXT,
+                value   BLOB
+            );
+        """)
+        conn.close()
+
+    def get_all_databases(self):
+        #print("Obtaining all keys...")
+        with scandir(self.collection_dir) as contents:
+            self.databases_list = [path.join(self.collection_dir, content.name) for content in contents]
+            
+        with Pool(self.cpu_cores) as pool:
+            self.keys_databases = dict(chain.from_iterable(pool.map(self.get_all_keys, self.databases_list)))
+        #print(self.keys_databases)
+
+    def get_all_keys(self, database):
+        conn    = sqlite3.connect(database)
+        cursor  = conn.cursor()
+        cursor.execute("PRAGMA journal_mode=WAL;")
+        cursor.execute("SELECT key FROM data;")
+        result  = cursor.fetchall()
+        keys    = [(line[0], database) for line in result]
+        conn.close()
+        return keys
+
+    def set_key(self, key, value):
+        if key not in self.keys_databases:
+            database_to_insert = choice(self.databases_list)
+            #print(f"Inserting in {database_to_insert}")
+            conn = sqlite3.connect(database_to_insert)
+            cursor = conn.cursor()
+            cursor.execute("PRAGMA journal_mode=WAL;")
+            cursor.execute("INSERT INTO data(key, value) VALUES (?, ?);", (key, pickle.dumps(value)))
+            conn.commit()
+            conn.close()
+            self.add_to_keys_database(key, database_to_insert)
+
+        else:
+            #print(f"Updating key '{key}' in {self.keys_databases[key]}...")
+            database_to_update = self.keys_databases[key]
+            conn = sqlite3.connect(database_to_update)
+            cursor = conn.cursor()
+            cursor.execute("PRAGMA journal_mode=WAL;")
+            cursor.execute("UPDATE data SET value = ? WHERE key = ?;", (pickle.dumps(value), key))
+            conn.commit()
+            conn.close()
+            #print(f"Key '{key}' updated successfully in {database_to_update}")
+
+    def add_to_keys_database(self, key, database):
+        self.keys_databases[key] = database
+        #print(self.keys_databases)
+
+    def get_key(self, key):
+        try:
+            database_to_search = self.keys_databases[key]
+            #print(database_to_search)
+
+            conn = sqlite3.connect(database_to_search)
+            cursor = conn.cursor()
+            cursor.execute("PRAGMA journal_mode=WAL;")
+            cursor.execute("SELECT value FROM data WHERE key = ?", (key,))
+            result = cursor.fetchall()
+            conn.close()
+            return pickle.loads(result[0][0])
+            
+        except Exception as error:
+            return error
+    
+conn = Collection_Cache("STORE")
+conn.set_key("alunos", ["Luiz", "Marcos", "João"])
+
+alunos = conn.get_key("alunos")
+print(alunos)

collections_cache-0.1.0/pyproject.toml

@@ -0,0 +1,15 @@
+[tool.poetry]
+name = "collections-cache"
+version = "0.1.0"
+description = "Collection Cache is a Python package for managing data collections across multiple SQLite databases. It allows efficient storage, retrieval, and updating of key-value pairs, supporting various data types serialized with pickle. The package uses parallel processing for fast access and manipulation of large collections."
+authors = ["Luiz-Trindade <luiz.gabriel.m.trindade@gmail.com>"]
+license = "MIT"
+readme = "README.md"
+
+[tool.poetry.dependencies]
+python = "^3.12"
+
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"