sqlite-forge 1.1.0__tar.gz

sqlite_forge-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Tom Freeman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sqlite_forge-1.1.0/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.1
+Name: sqlite-forge
+Version: 1.1.0
+Summary: Common tool for forging and maintaining sqlite databases
+Home-page: https://github.com/Tom3man/sqlite-forge
+License: MIT
+Keywords: sqlite,database,etl,data-ingestion
+Author: Tom
+Author-email: tomrfreeman3@gmail.com
+Requires-Python: >=3.10,<4.0
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Requires-Dist: pandas (>=2.2.2,<3.0.0)
+Project-URL: Documentation, https://tom3man.github.io/sqlite-forge/
+Project-URL: Issues, https://github.com/Tom3man/sqlite-forge/issues
+Project-URL: Repository, https://github.com/Tom3man/sqlite-forge
+Description-Content-Type: text/markdown
+
+# SQLite Forge
+
+SQLite Forge is a lightweight toolkit that helps you declare and maintain SQLite tables from Python. Define your schema once, then manage tables, run queries, ingest pandas `DataFrame` objects, and export results.
+
+## Highlights
+
+- Declarative table definitions with schemas and optional multi-column primary keys
+- Safe helpers to create/drop tables and check existence
+- DataFrame ingestion with optional incremental overwrite support
+- Query execution that returns pandas `DataFrame` objects
+- Table export helpers for `csv`, `json`, and `parquet`
+
+## Installation
+
+```bash
+pip install sqlite-forge
+```
+
+For development:
+
+```bash
+git clone https://github.com/Tom3man/sqlite-forge.git
+cd sqlite-forge
+poetry install --with dev --with docs
+```
+
+## Quick Start
+
+```python
+from pathlib import Path
+
+import pandas as pd
+
+from sqlite_forge import SqliteDatabase
+
+
+class ExampleTable(SqliteDatabase):
+    DEFAULT_PATH = "example_table"
+    PRIMARY_KEY = ("id",)
+    DEFAULT_SCHEMA = {
+        "id": "INTEGER",
+        "name": "TEXT",
+        "score": "REAL",
+    }
+
+
+db = ExampleTable(database_path=Path("./data"))
+db.create_table(overwrite=True)
+
+db.ingest_dataframe(
+    pd.DataFrame(
+        [
+            {"id": 1, "name": "Alice", "score": 9.2},
+            {"id": 2, "name": "Bob", "score": 8.7},
+        ]
+    )
+)
+
+print(db.fetch_table())
+db.export_table("./data/example_table.csv", format="csv")
+```
+
+## Development
+
+```bash
+poetry run pytest
+poetry run ruff check .
+poetry run mypy
+poetry build
+```
+
+## Documentation
+
+- Docs site: https://tom3man.github.io/sqlite-forge/
+- Build locally:
+
+```bash
+poetry run mkdocs serve
+```
+
+## Release
+
+1. Bump version in `pyproject.toml`.
+2. Update `CHANGELOG.md`.
+3. Publish:
+
+```bash
+poetry publish --build
+```
+
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md).
+
+## Licence
+
+MIT. See [LICENSE](LICENSE).
+

sqlite_forge-1.1.0/README.md

@@ -0,0 +1,97 @@
+# SQLite Forge
+
+SQLite Forge is a lightweight toolkit that helps you declare and maintain SQLite tables from Python. Define your schema once, then manage tables, run queries, ingest pandas `DataFrame` objects, and export results.
+
+## Highlights
+
+- Declarative table definitions with schemas and optional multi-column primary keys
+- Safe helpers to create/drop tables and check existence
+- DataFrame ingestion with optional incremental overwrite support
+- Query execution that returns pandas `DataFrame` objects
+- Table export helpers for `csv`, `json`, and `parquet`
+
+## Installation
+
+```bash
+pip install sqlite-forge
+```
+
+For development:
+
+```bash
+git clone https://github.com/Tom3man/sqlite-forge.git
+cd sqlite-forge
+poetry install --with dev --with docs
+```
+
+## Quick Start
+
+```python
+from pathlib import Path
+
+import pandas as pd
+
+from sqlite_forge import SqliteDatabase
+
+
+class ExampleTable(SqliteDatabase):
+    DEFAULT_PATH = "example_table"
+    PRIMARY_KEY = ("id",)
+    DEFAULT_SCHEMA = {
+        "id": "INTEGER",
+        "name": "TEXT",
+        "score": "REAL",
+    }
+
+
+db = ExampleTable(database_path=Path("./data"))
+db.create_table(overwrite=True)
+
+db.ingest_dataframe(
+    pd.DataFrame(
+        [
+            {"id": 1, "name": "Alice", "score": 9.2},
+            {"id": 2, "name": "Bob", "score": 8.7},
+        ]
+    )
+)
+
+print(db.fetch_table())
+db.export_table("./data/example_table.csv", format="csv")
+```
+
+## Development
+
+```bash
+poetry run pytest
+poetry run ruff check .
+poetry run mypy
+poetry build
+```
+
+## Documentation
+
+- Docs site: https://tom3man.github.io/sqlite-forge/
+- Build locally:
+
+```bash
+poetry run mkdocs serve
+```
+
+## Release
+
+1. Bump version in `pyproject.toml`.
+2. Update `CHANGELOG.md`.
+3. Publish:
+
+```bash
+poetry publish --build
+```
+
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md).
+
+## Licence
+
+MIT. See [LICENSE](LICENSE).

sqlite_forge-1.1.0/pyproject.toml

@@ -0,0 +1,63 @@
+[tool.poetry]
+name = "sqlite-forge"
+version = "1.1.0"
+description = "Common tool for forging and maintaining sqlite databases"
+authors = ["Tom <tomrfreeman3@gmail.com>"]
+readme = "README.md"
+license = "MIT"
+homepage = "https://github.com/Tom3man/sqlite-forge"
+repository = "https://github.com/Tom3man/sqlite-forge"
+documentation = "https://tom3man.github.io/sqlite-forge/"
+packages = [{include = "sqlite_forge"}]
+keywords = ["sqlite", "database", "etl", "data-ingestion"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "License :: OSI Approved :: MIT License",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Database",
+]
+
+[tool.poetry.urls]
+Issues = "https://github.com/Tom3man/sqlite-forge/issues"
+
+[tool.poetry.dependencies]
+python = ">=3.10,<4.0"
+pandas = "^2.2.2"
+
+
+[tool.poetry.group.dev.dependencies]
+ipykernel = "^6.28.0"
+pytest = "^7.4.0"
+pytest-cov = "^6.0.0"
+ruff = "^0.9.7"
+mypy = "^1.15.0"
+
+[tool.poetry.group.docs.dependencies]
+mkdocs = "^1.6.1"
+mkdocs-material = "^9.6.20"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.pytest.ini_options]
+addopts = "-q --cov=sqlite_forge --cov-report=term-missing"
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I"]
+ignore = ["E402"]
+
+[tool.mypy]
+python_version = "3.10"
+warn_unused_configs = true
+ignore_missing_imports = true
+files = ["sqlite_forge"]

sqlite_forge-1.1.0/sqlite_forge/__init__.py

@@ -0,0 +1,27 @@
+import logging
+from importlib.metadata import PackageNotFoundError, version
+from pathlib import Path
+
+MODULE_PATH = Path(__file__).resolve().parent
+REPO_PATH = MODULE_PATH.parent
+DATABASE_PATH = str(REPO_PATH)
+
+log = logging.getLogger("sqlite_forge")
+log.addHandler(logging.NullHandler())
+
+try:
+    __version__ = version("sqlite-forge")
+except PackageNotFoundError:
+    __version__ = "0.0.0"
+
+from .database import SqliteDatabase
+from .forger import BuildDatabase, sqlite3_process
+
+__all__ = [
+    "BuildDatabase",
+    "SqliteDatabase",
+    "sqlite3_process",
+    "log",
+    "DATABASE_PATH",
+    "__version__",
+]

sqlite_forge-1.1.0/sqlite_forge/database.py

@@ -0,0 +1,236 @@
+import sqlite3
+from abc import ABC
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, List, Literal, Optional, Sequence, Union
+
+import pandas as pd
+
+from sqlite_forge import log
+from sqlite_forge.forger import BuildDatabase, sqlite3_process
+
+
+class SqliteDatabase(BuildDatabase, ABC):
+    """
+    Class for managing SQLite database operations.
+    """
+
+    # Table name for database
+    DEFAULT_PATH: Optional[str] = None
+
+    # Schema dictionary for database
+    DEFAULT_SCHEMA: Optional[Dict[str, str]] = None
+
+    # Optional primary key(s) for database
+    PRIMARY_KEY: Optional[Sequence[str]] = None
+
+    def __init__(self, *args, **kwargs):
+        """
+        Initialise the SqliteDatabase class.
+        """
+        super().__init__(*args, **kwargs)
+
+    @sqlite3_process
+    def drop_table(self, cursor: sqlite3.Cursor) -> None:
+        """
+        Drop the specified table from the database if it exists.
+        """
+        # Drop the table if it exists and overwrite is True
+        drop_query = f"DROP TABLE IF EXISTS {self.db_name};"
+        cursor.execute(drop_query)
+        log.info(f"Dropped existing table {self.db_name}.")
+
+    @sqlite3_process
+    def create_table(self, cursor: sqlite3.Cursor, overwrite: bool = False) -> None:
+        """
+        Create a table in the database with composite primary keys.
+        """
+        # Check if the table already exists
+        if self.exists():
+            if overwrite:
+                # Drop table
+                self.drop_table()
+            else:
+                # Log and return if the table exists and overwrite is False
+                log.info(f"Table {self.db_name} already exists and will not be overwritten.")
+                return
+
+        if self.DEFAULT_SCHEMA is None:
+            raise ValueError("DEFAULT_SCHEMA must be set before creating a table.")
+
+        # Define individual columns with their data types
+        columns_definitions = [
+            f'{column_name} {column_type}' for column_name, column_type in self.DEFAULT_SCHEMA.items()
+        ]
+
+        # Include primary key in the column definitions if specified
+        if self.PRIMARY_KEY:
+            primary_key_clause = f"PRIMARY KEY ({', '.join(self.PRIMARY_KEY)})"
+            columns_definitions.append(primary_key_clause)
+
+        columns_definitions_str = ', '.join(columns_definitions)
+
+        # Create the table with "IF NOT EXISTS" for safety
+        create_table_query = f'CREATE TABLE IF NOT EXISTS {self.db_name} ({columns_definitions_str})'
+
+        cursor.execute(create_table_query)
+        if self.PRIMARY_KEY:
+            log.info(
+                "Table %s created successfully with primary key(s): %s.",
+                self.db_name,
+                ", ".join(self.PRIMARY_KEY),
+            )
+        else:
+            log.info("Table %s created successfully.", self.db_name)
+
+    @sqlite3_process
+    def get_columns(self, cursor: sqlite3.Cursor) -> List[str]:
+        """
+        Retrieve column names from the database.
+        """
+        cursor.execute(f"PRAGMA table_info({self.db_name})")
+        columns_info = cursor.fetchall()
+        return [column_info[1] for column_info in columns_info]
+
+    @sqlite3_process
+    def execute_query(self, cursor: sqlite3.Cursor, query: str) -> pd.DataFrame:
+        """
+        Execute a query and return results as a DataFrame.
+        """
+        cursor.execute(query)
+        data = cursor.fetchall()
+        columns = [description[0] for description in cursor.description] if cursor.description else []
+        return pd.DataFrame(data, columns=columns)
+
+    @staticmethod
+    def _validate_headers(headers: List[str], schema: Dict[str, str]) -> None:
+        """
+        Validate column headers against the schema.
+        """
+        mismatched_headers = [
+            header for header in headers if header.lower() not in map(
+                str.lower, schema.keys())]
+        if mismatched_headers:
+            mismatched = ', '.join(mismatched_headers)
+            raise ValueError(
+                f"Following column(s) in imported file do not match the DEFAULT_SCHEMA: {mismatched}")
+
+    @sqlite3_process
+    def ingest_dataframe(
+        self,
+        cursor: sqlite3.Cursor,
+        df: pd.DataFrame,
+        load_date: bool = False,
+        overwrite: bool = False,
+    ) -> None:
+        """
+        Ingest a pandas dataframe into the database.
+
+        If load_date is True, adds a LOAD_DATE column with the current datetime.
+        If overwrite is True, updates existing records based on PRIMARY_KEY.
+        """
+
+        if load_date:
+            df = df.copy()
+            df['LOAD_DATE'] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+
+        if self.DEFAULT_SCHEMA is None:
+            raise ValueError("DEFAULT_SCHEMA must be set before ingesting a dataframe.")
+
+        headers = df.columns.tolist()
+        self._validate_headers(headers, self.DEFAULT_SCHEMA)
+
+        if self.PRIMARY_KEY:
+            # Pre-build the WHERE clause for existence check and update
+            where_clause = " AND ".join([f"{key} = ?" for key in self.PRIMARY_KEY])
+
+        insert_count = 0
+        for _, row in df.iterrows():
+
+            if self.PRIMARY_KEY:
+                where_values = tuple(row[key] for key in self.PRIMARY_KEY)
+
+                # Check if the record already exists
+                cursor.execute(f"SELECT COUNT(*) FROM {self.db_name} WHERE {where_clause}", where_values)
+                exists = cursor.fetchone()[0]
+
+                if exists and overwrite:
+                    # Record exists, update it
+                    update_clause = ", ".join([f"{header} = ?" for header in headers if header not in self.PRIMARY_KEY])
+                    update_values = tuple(row[header] for header in headers if header not in self.PRIMARY_KEY)
+                    update_query = f"""
+                        UPDATE {self.db_name}
+                        SET {update_clause}
+                        WHERE {where_clause}"""
+                    cursor.execute(update_query, update_values + where_values)
+                    insert_count += 1
+                elif not exists:
+                    # Record does not exist, insert it
+                    insert_query = f"""
+                        INSERT INTO {self.db_name} ({', '.join(headers)})
+                        VALUES ({', '.join(['?' for _ in range(len(headers))])})"""
+                    cursor.execute(insert_query, tuple(row[header] for header in headers))
+                    insert_count += 1
+            else:
+                # No primary key provided, insert directly
+                insert_query = f"""
+                    INSERT INTO {self.db_name} ({', '.join(headers)})
+                    VALUES ({', '.join(['?' for _ in range(len(headers))])})"""
+                cursor.execute(insert_query, tuple(row[header] for header in headers))
+                insert_count += 1
+
+        log.info("%s rows written to %s; table now has %s rows.", insert_count, self.db_name, self.table_length)
+
+    @property
+    @sqlite3_process
+    def table_length(self, cursor: sqlite3.Cursor) -> int:
+        """
+        Return the number of rows in the table.
+        """
+        cursor.execute(f"SELECT COUNT(*) FROM {self.db_name}")
+        length = cursor.fetchone()[0]
+        return length
+
+    def fetch_table(self, limit: Optional[int] = None) -> pd.DataFrame:
+        """
+        Return rows from the managed table as a DataFrame.
+        """
+        query = f"SELECT * FROM {self.db_name}"
+        if limit is not None:
+            query += f" LIMIT {int(limit)}"
+        return self.execute_query(query)
+
+    def export_table(
+        self,
+        output_path: Union[str, Path],
+        format: Literal["csv", "json", "parquet"] = "csv",
+        limit: Optional[int] = None,
+    ) -> Path:
+        """
+        Export table rows to a file.
+
+        Args:
+            output_path: Destination file path.
+            format: Export format ("csv", "json", or "parquet").
+            limit: Optional row limit before exporting.
+        """
+        df = self.fetch_table(limit=limit)
+        path = Path(output_path).expanduser()
+        path.parent.mkdir(parents=True, exist_ok=True)
+
+        if format == "csv":
+            df.to_csv(path, index=False)
+        elif format == "json":
+            df.to_json(path, orient="records", indent=2)
+        elif format == "parquet":
+            try:
+                df.to_parquet(path, index=False)
+            except ImportError as exc:
+                raise ImportError(
+                    "Parquet export requires an engine such as 'pyarrow' or 'fastparquet'."
+                ) from exc
+        else:
+            raise ValueError("format must be one of: csv, json, parquet")
+
+        log.info("Exported %s rows from %s to %s", len(df), self.db_name, path)
+        return path

sqlite_forge-1.1.0/sqlite_forge/forger.py

@@ -0,0 +1,83 @@
+import logging
+import sqlite3
+from abc import ABC
+from functools import wraps
+from pathlib import Path
+from typing import Callable, Dict, Optional, TypeVar, Union
+
+log = logging.getLogger(__name__)
+
+T = TypeVar("T")
+DatabasePath = Union[str, Path]
+
+
+def sqlite3_process(func: Callable[..., T]) -> Callable[..., T]:
+    """
+    Decorator to manage SQLite database connection.
+    """
+    @wraps(func)
+    def func_wrapper(self, *args, **kwargs) -> T:
+        database_dir = Path(self.database_path)
+        database_dir.mkdir(parents=True, exist_ok=True)
+        db_file = database_dir / f"{self.db_name}.db"
+
+        conn = sqlite3.connect(str(db_file))
+        try:
+            cursor = conn.cursor()
+            result = func(self, cursor, *args, **kwargs)
+        except Exception:
+            if conn.in_transaction:
+                conn.rollback()
+            raise
+        else:
+            conn.commit()
+            return result
+        finally:
+            conn.close()
+
+    return func_wrapper
+
+
+class BuildDatabase(ABC):
+    """
+    Abstract base class for building a SQLite database.
+    """
+
+    DEFAULT_PATH: Optional[str] = None
+    DEFAULT_SCHEMA: Optional[Dict[str, str]] = None
+
+    def __init__(self, database_path: DatabasePath, database_name: Optional[str] = None) -> None:
+        """
+        Initialize the BuildDatabase class.
+        """
+        if not self.DEFAULT_PATH or not self.DEFAULT_SCHEMA:
+            raise ValueError("Both DEFAULT_PATH and DEFAULT_SCHEMA must be implemented in the inheriting child class!")
+        self.db_name = database_name or self.DEFAULT_PATH
+        self.database_path = Path(database_path).expanduser()
+
+    @property
+    def database(self) -> str:
+        """
+        Get the full path of the database file.
+        """
+        db_path = self.database_path / f"{self.db_name}.db"
+        if not db_path.exists():
+            raise FileNotFoundError(
+                f"Database file '{db_path}' does not exist, please create first!")
+        return str(db_path)
+
+    @property
+    def conn(self) -> sqlite3.Connection:
+        """
+        Establish a connection to the SQLite database.
+        """
+        return sqlite3.connect(self.database)
+
+    @sqlite3_process
+    def exists(self, cursor: sqlite3.Cursor) -> bool:
+        """
+        Check if a specified table exists in the database.
+        """
+        query = f"SELECT name FROM sqlite_master WHERE type='table' AND name='{self.db_name}';"
+        cursor.execute(query)
+        return cursor.fetchone() is not None