hdx-python-database 1.2.9__tar.gz → 1.3.1__tar.gz

hdx_python_database-1.3.1/.config/pre-commit-config.yaml

@@ -0,0 +1,26 @@
+default_language_version:
+    python: python3.11
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-ast
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.4.2
+    hooks:
+      # Run the linter.
+      - id: ruff
+        args: [--config, .config/ruff.toml, --fix]
+      # Run the formatter.
+      - id: ruff-format
+        args: [--config, .config/ruff.toml]
+  - repo: https://github.com/astral-sh/uv-pre-commit
+    rev: 0.1.38
+    hooks:
+      # Run the pip compile
+      - id: pip-compile
+        name: pip-compile requirements.txt
+        files: pyproject.toml
+        args: [ pyproject.toml, --resolver=backtracking, --all-extras, --upgrade, -q, -o, requirements.txt ]

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/.config/ruff.toml

@@ -1,14 +1,16 @@
 line-length = 79
 exclude = ["_version.py"]
-ignore = [
-    "E501" # Line too long
-]
-# List of rules: https://beta.ruff.rs/docs/rules/
+
+[lint]
+# List of rules: https://docs.astral.sh/ruff/rules/
 select = [
   "E",   # pycodestyle - default
   "F",   # pyflakes - default
   "I"    # isort
 ]
+ignore = [
+    "E501" # Line too long
+]
 
-[isort]
+[lint.isort]
 known-local-folder = ["hdx"]

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/CONTRIBUTING.md

@@ -21,14 +21,13 @@ you make a git commit:
     pre-commit install
 
 The configuration file for this project is in a
-non-start location. Thus, you will need to edit your
+non-standard location. Thus, you will need to edit your
 `.git/hooks/pre-commit` file to reflect this. Change
 the line that begins with `ARGS` to:
 
     ARGS=(hook-impl --config=.config/pre-commit-config.yaml --hook-type=pre-commit)
 
 With pre-commit, all code is formatted according to
-[black]("https://github.com/psf/black") and
 [ruff]("https://github.com/charliermarsh/ruff") guidelines.
 
 To check if your changes pass pre-commit without committing, run:
@@ -41,7 +40,7 @@ To run the tests and view coverage, execute:
 
     pytest -c .config/pytest.ini --cov hdx --cov-config .config/coveragerc
 
-Follow the example set out already in ``api.rst`` as you write the documentation.
+Follow the example set out already in ``documentation/main.md`` as you write the documentation.
 
 ## Packages
 

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.3
 Name: hdx-python-database
-Version: 1.2.9
+Version: 1.3.1
 Summary: HDX Python database utilities
 Project-URL: Homepage, https://github.com/OCHA-DAP/hdx-python-database
 Author-email: Michael Rans <rans@email.com>
@@ -39,8 +39,7 @@ Description-Content-Type: text/markdown
 
 [![Build Status](https://github.com/OCHA-DAP/hdx-python-database/actions/workflows/run-python-tests.yaml/badge.svg)](https://github.com/OCHA-DAP/hdx-python-database/actions/workflows/run-python-tests.yaml)
 [![Coverage Status](https://coveralls.io/repos/github/OCHA-DAP/hdx-python-database/badge.svg?branch=main&ts=1)](https://coveralls.io/github/OCHA-DAP/hdx-python-database?branch=main)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![Downloads](https://img.shields.io/pypi/dm/hdx-python-database.svg)](https://pypistats.org/packages/hdx-python-database)
 
 The HDX Python Database Library provides utilities for connecting to databases in a standard way including

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/README.md

@@ -1,7 +1,6 @@
 [![Build Status](https://github.com/OCHA-DAP/hdx-python-database/actions/workflows/run-python-tests.yaml/badge.svg)](https://github.com/OCHA-DAP/hdx-python-database/actions/workflows/run-python-tests.yaml)
 [![Coverage Status](https://coveralls.io/repos/github/OCHA-DAP/hdx-python-database/badge.svg?branch=main&ts=1)](https://coveralls.io/github/OCHA-DAP/hdx-python-database?branch=main)
-[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![Downloads](https://img.shields.io/pypi/dm/hdx-python-database.svg)](https://pypistats.org/packages/hdx-python-database)
 
 The HDX Python Database Library provides utilities for connecting to databases in a standard way including

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/documentation/.readthedocs.yaml

@@ -11,5 +11,6 @@ build:
   jobs:
     pre_build:
       - pip install --upgrade mkdocs
+      - pip install mkdocs-material
       - pip install pydoc-markdown
       - pydoc-markdown documentation/pydoc-markdown.yaml

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/documentation/main.md

@@ -22,6 +22,13 @@ installed with extra `postgresql`:
     pip install hdx-python-database[postgresql]
 
 ## Breaking changes
+From 1.3.1, Database class refactored. With returns Database not Session
+object and can accept a prepare function called before
+Base.metadata.create_all. get_session can be used to obtain the session.
+get_engine, get_reflected_classes, get_prepare_results methods added.
+build_view, build_views move into Database class and renamed prepare_view and
+prepare_views.
+
 From 1.2.8, the sshtunnel dependency is optional.
 
 From 1.2.7, default table names are no longer plural. The camel case class name
@@ -51,8 +58,13 @@ Versions from 1.0.6 no longer support Python 2.7.
 
 ## Database
 
+A fresh schema can be created using the function `recreate_schema`. Any
+existing schema with that name will be dropped eg.
+
+    Database.recreate_schema(engine, db_uri)
+
 Your SQLAlchemy database tables must inherit from `Base` in
-`hdx.database.no_timezone` or `hdx.database.with_timezone` eg. :
+`hdx.database.no_timezone` or `hdx.database.with_timezone` eg.
 
     from hdx.database.no_timezone import Base
     class MyTable(Base):
@@ -70,9 +82,14 @@ tunnel (which requires installing `hdx-python-database[sshtunnel]`):
     from hdx.database import Database
     with Database(database="db", host="1.2.3.4", username="user",
                   password="pass", dialect="dialect", driver="driver",
+
                   ssh_host="5.6.7.8", ssh_port=2222, ssh_username="sshuser",
                   ssh_private_key="path_to_key", db_has_tz=True,
-                  reflect=False) as session:
+                  reflect=True, prepare_fn=prepare_views) as database:
+        engine = database.get_engine()
+        session = database.get_session()
+        reflected_classes = database.get_reflected_classes()
+        views = database.get_prepare_results()
         session.query(...)
 
 `db_has_tz` which defaults to `False` indicates whether database datetime
@@ -83,11 +100,20 @@ between Python datetimes with timezones to timezoneless database columns.
 
 If `reflect` (which defaults to `False`) is `True`, classes will be reflected
 from an existing database and the reflected classes are returned in a variable
-`reflected_classes` in the returned Session object. Note that type annotation
+`reflected_classes` in the Database object. Note that type annotation
 maps don't work with reflection and hence `db_has_tz` will be ignored ie.
 there will be no conversion between Python datetimes with timezones to
 timezoneless database columns.
 
+There is an option to wipe and create an empty schema in the database by
+setting `recreate_schema` to `True` and setting a `schema_name` ("public" is
+the default).
+
+If a prepare function is supplied in prepare_fn, it will be executed before
+Base.metadata.create_all and the results of it returned in instance variable
+prepare_results.
+
+
 ## Connection URI
 
 There are functions to handle converting from connection URIs to parameters and
@@ -117,10 +143,9 @@ URI that contains both dialect and driver.
 
 ## Views
 
-The method to make views described [here](https://github.com/sqlalchemy/sqlalchemy/wiki/Views#sqlalchemy-14-20-version)
-is available in this library.
+The method to prepare views described [here](https://github.com/sqlalchemy/sqlalchemy/wiki/Views#sqlalchemy-14-20-version) is available in this library.
 
-This allows creating views like this:
+This allows preparing views like this:
 ```
 class DBOrgType(Base):
     __tablename__ = "org_type"
@@ -144,11 +169,13 @@ date_view_params = {
     "selectable": select(*DBTestDate.__table__.columns),
 }
 
-date_view = build_view(date_view_params)
+date_view = Database.prepare_view(date_view_params)
 
-date_view = build_views([date_view_params])[0]
+date_view = Database.prepare_views([date_view_params])[0]
 ```
 
+View preparation must be done before calling Base.metadata.create_all.
+
 ## PostgreSQL specific
 
 There is a PostgreSQL specific call that only returns when the PostgreSQL database

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/documentation/pydoc-markdown.yaml

@@ -9,7 +9,7 @@ renderer:
   output_directory: docs
   mkdocs_config:
     site_name: HDX Python Database
-    theme: mkdocs
+    theme: material
     repo_url: "https://github.com/OCHA-DAP/hdx-python-database"
   markdown:
     source_linker:

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/pyproject.toml

@@ -86,22 +86,14 @@ test = """
        """
 
 [[tool.hatch.envs.test.matrix]]
-python = ["3.11"]
+python = ["3.12"]
 
 [tool.hatch.envs.lint]
 detached = true
-dependencies = [
-  "black",
-  "ruff",
-]
+dependencies = ["ruff"]
 
 [tool.hatch.envs.lint.scripts]
 style = [
-  "ruff --config .config/ruff.toml {args:.}",
-  "black --config .config/black.toml --check --diff {args:.}",
-]
-# Not used for anything at the moment
-fmt = [
-  "black --config .config/black.toml {args:.}",
-  "ruff --config .config/ruff.toml --fix {args:.}",
+  "ruff check --config .config/ruff.toml --diff {args:.}",
+  "ruff format --config .config/ruff.toml --diff {args:.}",
 ]

hdx_python_database-1.3.1/requirements.txt

@@ -0,0 +1,57 @@
+# This file was autogenerated by uv via the following command:
+#    uv pip compile pyproject.toml --resolver=backtracking --all-extras -o requirements.txt
+bcrypt==4.1.3
+    # via paramiko
+cffi==1.16.0
+    # via
+    #   cryptography
+    #   pynacl
+cfgv==3.4.0
+    # via pre-commit
+coverage==7.5.1
+    # via pytest-cov
+cryptography==42.0.7
+    # via paramiko
+distlib==0.3.8
+    # via virtualenv
+filelock==3.14.0
+    # via virtualenv
+greenlet==3.0.3
+    # via sqlalchemy
+identify==2.5.36
+    # via pre-commit
+iniconfig==2.0.0
+    # via pytest
+nodeenv==1.8.0
+    # via pre-commit
+packaging==24.0
+    # via pytest
+paramiko==3.4.0
+    # via sshtunnel
+platformdirs==4.2.1
+    # via virtualenv
+pluggy==1.5.0
+    # via pytest
+pre-commit==3.7.0
+psycopg==3.1.18
+psycopg-binary==3.1.18
+    # via psycopg
+pycparser==2.22
+    # via cffi
+pynacl==1.5.0
+    # via paramiko
+pytest==8.2.0
+    # via pytest-cov
+pytest-cov==5.0.0
+pyyaml==6.0.1
+    # via pre-commit
+setuptools==69.5.1
+    # via nodeenv
+sqlalchemy==2.0.30
+sshtunnel==0.4.0
+typing-extensions==4.11.0
+    # via
+    #   psycopg
+    #   sqlalchemy
+virtualenv==20.26.1
+    # via pre-commit

hdx_python_database-1.3.1/src/hdx/database/__init__.py

@@ -0,0 +1,294 @@
+"""Database utilities"""
+
+import logging
+from typing import Any, Dict, List, Optional, Tuple, Type, Union
+
+from sqlalchemy import Engine, TableClause, create_engine
+from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.ext.automap import automap_base
+from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
+from sqlalchemy.pool import NullPool
+from sqlalchemy.sql.ddl import CreateSchema, DropSchema
+from sqlalchemy.util import Properties
+
+from ._version import version as __version__  # noqa: F401
+from .dburi import get_connection_uri
+from .no_timezone import Base as NoTZBase
+from .postgresql import wait_for_postgresql
+from .views import view
+from .with_timezone import Base as TZBase
+
+logger = logging.getLogger(__name__)
+
+
+class DatabaseError(Exception):
+    pass
+
+
+def do_nothing_fn():
+    return None
+
+
+class Database:
+    """Database helper class to handle ssh tunnels, waiting for PostgreSQL to
+    be up etc. Can be used in a with statement returning a Database object that
+    if reflect is True will have a variable reflected_classes containing the
+    reflected classes. Either an SQLAlchemy engine, database uri or various
+    database parameters must be supplied.
+
+    db_has_tz which defaults to False indicates whether database datetime
+    columns have timezones. If not, conversion occurs between Python datetimes
+    with timezones to timezoneless database columns (but not when using
+    reflection). If table_base is supplied, db_has_tz is ignored.
+
+    There is an option to wipe and create an empty schema in the database by
+    setting recreate_schema to True and setting a schema_name ("public" is the
+    default).
+
+    If a prepare function is supplied in prepare_fn, it will be executed before
+    Base.metadata.create_all and the results of it returned in instance variable
+    prepare_results.
+
+    Args:
+        engine (Optional[Engine]): SQLAlchemy engine to use.
+        db_uri (Optional[str]): Connection URI.
+        database (Optional[str]): Database name
+        host (Optional[str]): Host where database is located
+        port (Union[int, str, None]): Database port
+        username (Optional[str]): Username to log into database
+        password (Optional[str]): Password to log into database
+        dialect (str): Database dialect. Defaults to "postgresql".
+        driver (Optional[str]): Database driver. Defaults to None (psycopg if postgresql or None)
+        db_has_tz (bool): True if db datetime columns have timezone. Defaults to False.
+        table_base (Optional[Type[DeclarativeBase]]): Override table base. Defaults to None.
+        reflect (bool): Whether to reflect existing tables. Defaults to False.
+        **kwargs: See below
+        recreate_schema (bool): Whether to recreate schema
+        schema_name (str): Database schema name. Defaults to "public".
+        prepare_fn (Callable[[], None]]): Function to call before Base.metadata.create_all.
+        ssh_host (str): SSH host (the server to connect to)
+        ssh_port (int): SSH port. Defaults to 22.
+        ssh_username (str): SSH username
+        ssh_password (str): SSH password
+        ssh_private_key: Location of SSH private key (instead of ssh_password)
+        For more advanced usage, see SSHTunnelForwarder documentation.
+
+    """
+
+    def __init__(
+        self,
+        engine: Optional[Engine] = None,
+        db_uri: Optional[str] = None,
+        database: Optional[str] = None,
+        host: Optional[str] = None,
+        port: Union[int, str, None] = None,
+        username: Optional[str] = None,
+        password: Optional[str] = None,
+        dialect: str = "postgresql",
+        driver: Optional[str] = None,
+        db_has_tz: bool = False,
+        table_base: Optional[Type[DeclarativeBase]] = None,
+        reflect: bool = False,
+        **kwargs: Any,
+    ) -> None:
+        if port is not None:
+            port = int(port)
+        schema_name = None
+        if len(kwargs) == 0:
+            recreate_schema = False
+            prepare_fn = do_nothing_fn
+        else:
+            recreate_schema = kwargs.pop("recreate_schema", False)
+            schema_name = kwargs.pop("schema", "public")
+            prepare_fn = kwargs.pop("prepare_fn", do_nothing_fn)
+        if len(kwargs) != 0:
+            try:
+                import sshtunnel
+            except ImportError:
+                # dependency missing, log an error
+                logger.error(
+                    "sshtunnel not found! Please install hdx-python-database[sshtunnel] to enable."
+                )
+                raise
+            ssh_host = kwargs["ssh_host"]
+            del kwargs["ssh_host"]
+            ssh_port = kwargs.get("ssh_port")
+            if ssh_port is not None:
+                ssh_port = int(ssh_port)
+                del kwargs["ssh_port"]
+            else:
+                ssh_port = 22
+            self.server = sshtunnel.SSHTunnelForwarder(
+                (ssh_host, ssh_port),
+                remote_bind_address=(host, port),
+                **kwargs,
+            )
+            self.server.start()
+            host = self.server.local_bind_host
+            port = self.server.local_bind_port
+        else:
+            self.server = None
+        if not engine and not db_uri:
+            db_uri = get_connection_uri(
+                database,
+                host,
+                port,
+                username,
+                password,
+                dialect=dialect,
+                driver=driver,
+            )
+            if not db_uri:
+                raise DatabaseError(
+                    "No engine, database uri or connection parameters provided!"
+                )
+        if not engine and dialect == "postgresql":
+            wait_for_postgresql(db_uri)
+        if not table_base:
+            if db_has_tz:
+                table_base = TZBase
+            else:
+                table_base = NoTZBase
+        if not engine:
+            engine = create_engine(db_uri, poolclass=NullPool, echo=False)
+        self.engine: Engine = engine
+        if recreate_schema:
+            self.recreate_schema(engine, schema_name)
+        self.prepare_results = prepare_fn()
+        self.session, self.reflected_classes = self.create_session(
+            engine,
+            table_base=table_base,
+            reflect=reflect,
+        )
+
+    def __enter__(self) -> "Database":
+        return self
+
+    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
+        self.session.close()
+        if self.server is not None:
+            self.server.stop()
+
+    def get_engine(self) -> Engine:
+        """Returns SQLAlchemy engine.
+
+        Returns:
+            sqlalchemy.Engine: SQLAlchemy engine
+        """
+        return self.engine
+
+    def get_session(self) -> Session:
+        """Returns SQLAlchemy session.
+
+        Returns:
+            sqlalchemy.orm.Session: SQLAlchemy session
+        """
+        return self.session
+
+    def get_reflected_classes(self) -> Any:
+        """Gets reflected classes.
+
+        Returns:
+            Any: Reflected classes
+        """
+        return self.reflected_classes
+
+    def get_prepare_results(self) -> Any:
+        """Returns results from prepare function.
+
+        Returns:
+            Any: Results from prepare function
+        """
+        return self.prepare_results
+
+    @staticmethod
+    def create_session(
+        engine: Optional[Engine] = None,
+        db_uri: Optional[str] = None,
+        table_base: Type[DeclarativeBase] = NoTZBase,
+        reflect: bool = False,
+    ) -> Tuple[Session, Optional[Properties]]:
+        """Creates SQLAlchemy session given SQLAlchemy engine or database uri
+        (one of which must be supplied). Tables must inherit from Base in
+        hdx.utilities.database unless base is defined. If reflect is True,
+        classes will be reflected from an existing database and the reflected
+        classes will be returned. Note that type annotation maps don't work
+        with reflection.
+
+        Args:
+            engine (Optional[Engine]): SQLAlchemy engine to use. Defaults to None (create from db_uri).
+            db_uri (Optional[str]): Connection URI. Defaults to None (use engine).
+            table_base (Type[DeclarativeBase]): Base database table class. Defaults to NoTZBase.
+            reflect (bool): Whether to reflect existing tables. Defaults to False.
+
+        Returns:
+            Tuple[Session, Optional[Properties]]: (SQLAlchemy session, reflected classes if available)
+        """
+        if not engine:
+            if db_uri is None:
+                raise DatabaseError("No engine or database uri provided!")
+            engine = create_engine(db_uri, poolclass=NullPool, echo=False)
+        Session = sessionmaker(bind=engine)
+        if reflect:
+            Base = automap_base(declarative_base=table_base)
+            Base.prepare(autoload_with=engine)
+            reflected_classes = Base.classes
+        else:
+            table_base.metadata.create_all(engine)
+            reflected_classes = None
+        session = Session()
+        return session, reflected_classes
+
+    @staticmethod
+    def recreate_schema(engine: Engine, schema_name: str = "public") -> bool:
+        """Wipe and create empty schema in database using SQLAlchemy.
+
+        Args:
+            engine (Engine): SQLAlchemy engine to use.
+            schema_name (str): Schema name. Defaults to "public".
+
+        Returns:
+            bool: True if all successful, False if not
+        """
+        # Wipe and create an empty schema
+        try:
+            with engine.connect() as connection:
+                connection.execute(
+                    DropSchema(schema_name, cascade=True, if_exists=True)
+                )
+                connection.commit()
+                connection.execute(
+                    CreateSchema(schema_name, if_not_exists=True)
+                )
+                connection.commit()
+                return True
+        except SQLAlchemyError:
+            return False
+
+    @staticmethod
+    def prepare_view(view_params: Dict) -> TableClause:
+        """Prepare SQLAlchemy view from dictionary with keys: name, metadata and
+        selectable. Must be run before Base.metadata.create_all.
+
+        Args:
+            view_params (Dict): Dictionary with keys name, metadata, selectable
+
+        Returns:
+            TableClause: SQLAlchemy View
+        """
+        return view(**view_params)
+
+    @classmethod
+    def prepare_views(cls, view_params_list: List[Dict]) -> List[TableClause]:
+        """Prepare SQLAlchemy views from a list of dictionaries with keys: name,
+        metadata and selectable. Must be run before Base.metadata.create_all.
+
+        Args:
+            view_params_list (List[Dict]): List of dictionaries with view parameters
+        Returns:
+            List[TableClause]: SQLAlchemy Views
+        """
+        results = []
+        for view_params in view_params_list:
+            results.append(cls.prepare_view(view_params))
+        return results

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/src/hdx/database/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '1.2.9'
-__version_tuple__ = version_tuple = (1, 2, 9)
+__version__ = version = '1.3.1'
+__version_tuple__ = version_tuple = (1, 3, 1)

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/src/hdx/database/dburi.py

@@ -62,6 +62,8 @@ def get_connection_uri(
     Returns:
         str: Connection URI
     """
+    if not database:
+        return ""
     strings = [dialect]
     if include_driver:
         if dialect == "postgresql" and driver is None:

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/src/hdx/database/no_timezone.py

@@ -1,4 +1,5 @@
 """Utilities for database datetime columns without timezone"""
+
 from datetime import datetime, timezone
 from typing import Optional
 

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/src/hdx/database/postgresql.py

@@ -1,4 +1,5 @@
 """PostgreSQL specific utilities"""
+
 import logging
 import time
 

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/src/hdx/database/views.py

@@ -1,10 +1,10 @@
 """
-Create a DB View.
+Prepare database views ins SQLAlchemy. Must be run before calling
+Base.metadata.create_all.
 
 Copied from:
 https://github.com/sqlalchemy/sqlalchemy/wiki/Views#sqlalchemy-14-20-version
 """
-from typing import Dict, List
 
 import sqlalchemy as sa
 from sqlalchemy import MetaData, Selectable, TableClause
@@ -46,7 +46,7 @@ def view_doesnt_exist(ddl, target, connection, **kw):
 
 
 def view(name: str, metadata: MetaData, selectable: Selectable) -> TableClause:
-    """Create SQLAlchemy view
+    """Prepare SQLAlchemy view. Must be run before Base.metadata.create_all.
 
     Args:
         name (str): View name
@@ -73,31 +73,3 @@ def view(name: str, metadata: MetaData, selectable: Selectable) -> TableClause:
         DropView(name).execute_if(callable_=view_exists),
     )
     return t
-
-
-def build_view(view_params: Dict) -> TableClause:
-    """Create SQLAlchemy view from dictionary with keys: name, metadata and
-    selectable
-
-    Args:
-        view_params (Dict): Dictionary with keys name, metadata, selectable
-
-    Returns:
-        TableClause: SQLAlchemy View
-    """
-    return view(**view_params)
-
-
-def build_views(view_params_list: List[Dict]) -> List[TableClause]:
-    """Create SQLAlchemy views from a list of dictionaries with keys: name,
-    metadata and selectable
-
-    Args:
-        view_params_list (List[Dict]): List of dictionaries with view parameters
-    Returns:
-        List[TableClause]: SQLAlchemy Views
-    """
-    results = []
-    for view_params in view_params_list:
-        results.append(build_view(view_params))
-    return results

hdx_python_database-1.3.1/tests/hdx/database/conftest.py

@@ -0,0 +1,86 @@
+from collections import namedtuple
+
+import psycopg
+import pytest
+from sqlalchemy.exc import SQLAlchemyError
+from sshtunnel import SSHTunnelForwarder
+
+from . import PsycopgConnection
+from .test_database import TestDatabase
+from hdx.database import Database
+
+
+@pytest.fixture(scope="function")
+def mock_psycopg(monkeypatch):
+    def connect(*args, **kwargs):
+        if PsycopgConnection.connected:
+
+            class PGConnection:
+                @staticmethod
+                def close():
+                    return
+
+            return PGConnection()
+        else:
+            PsycopgConnection.connected = True
+            raise psycopg.OperationalError
+
+    monkeypatch.setattr(psycopg, "connect", connect)
+
+
+@pytest.fixture(scope="function")
+def mock_SSHTunnelForwarder(monkeypatch):
+    def init(*args, **kwargs):
+        return None
+
+    def start(_):
+        TestDatabase.started = True
+
+    def stop(_):
+        TestDatabase.stopped = True
+
+    monkeypatch.setattr(SSHTunnelForwarder, "__init__", init)
+    monkeypatch.setattr(SSHTunnelForwarder, "start", start)
+    monkeypatch.setattr(SSHTunnelForwarder, "stop", stop)
+    monkeypatch.setattr(SSHTunnelForwarder, "local_bind_host", "0.0.0.0")
+    monkeypatch.setattr(SSHTunnelForwarder, "local_bind_port", 12345)
+
+    def create_session(_, engine, table_base, reflect):
+        class Session:
+            bind = namedtuple("Bind", "engine")
+
+            def close(self):
+                return None
+
+        Session.bind.engine = engine
+        TestDatabase.table_base = table_base
+        return Session(), None
+
+    monkeypatch.setattr(Database, "create_session", create_session)
+
+
+@pytest.fixture(scope="function")
+def mock_engine():
+    class SQAConnection:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, exc_type, exc_value, traceback) -> None:
+            pass
+
+        @staticmethod
+        def execute(statement):
+            if "mydatabase" in str(statement):
+                return
+            raise SQLAlchemyError("Fail!")
+
+        @staticmethod
+        def commit():
+            return
+
+    class MockEngine:
+        @staticmethod
+        def connect():
+            return SQAConnection()
+
+    return MockEngine()

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/tests/hdx/database/dbtestdate.py

@@ -1,12 +1,11 @@
-"""SQLAlchemy class representing DBTestDate row. Holds test data for dates.
-"""
+"""SQLAlchemy class representing DBTestDate row. Holds test data for dates."""
+
 from datetime import datetime
 
 from sqlalchemy import select
 from sqlalchemy.orm import Mapped, mapped_column
 
 from hdx.database.no_timezone import Base
-from hdx.database.views import build_views
 
 
 class DBTestDate(Base):
@@ -30,5 +29,3 @@ date_view_params = {
     "metadata": Base.metadata,
     "selectable": select(*DBTestDate.__table__.columns),
 }
-
-date_view = build_views([date_view_params])[0]

{hdx_python_database-1.2.9 → hdx_python_database-1.3.1}/tests/hdx/database/test_database.py

@@ -1,5 +1,5 @@
 """Database Utility Tests"""
-from collections import namedtuple
+
 from copy import deepcopy
 from datetime import datetime, timezone
 from os import remove
@@ -8,10 +8,9 @@ from shutil import copyfile
 
 import pytest
 from sqlalchemy import select
-from sshtunnel import SSHTunnelForwarder
 
-from .dbtestdate import DBTestDate, date_view
-from hdx.database import Database
+from .dbtestdate import DBTestDate, date_view_params
+from hdx.database import Database, DatabaseError
 from hdx.database.no_timezone import Base as NoTZBase
 from hdx.database.with_timezone import Base as TZBase
 
@@ -49,42 +48,19 @@ class TestDatabase:
             pass
         return f"sqlite:///{TestDatabase.dbpath}"
 
-    @pytest.fixture(scope="function")
-    def mock_SSHTunnelForwarder(self, monkeypatch):
-        def init(*args, **kwargs):
-            return None
-
-        def start(_):
-            TestDatabase.started = True
-
-        def stop(_):
-            TestDatabase.stopped = True
-
-        monkeypatch.setattr(SSHTunnelForwarder, "__init__", init)
-        monkeypatch.setattr(SSHTunnelForwarder, "start", start)
-        monkeypatch.setattr(SSHTunnelForwarder, "stop", stop)
-        monkeypatch.setattr(SSHTunnelForwarder, "local_bind_host", "0.0.0.0")
-        monkeypatch.setattr(SSHTunnelForwarder, "local_bind_port", 12345)
-
-        def get_session(_, db_url, table_base, reflect):
-            class Session:
-                bind = namedtuple("Bind", "engine")
-
-                def close(self):
-                    return None
-
-            Session.bind.engine = namedtuple("Engine", "url")
-            Session.bind.engine.url = db_url
-            TestDatabase.table_base = table_base
-            return Session()
-
-        monkeypatch.setattr(Database, "get_session", get_session)
-
     def test_get_session(self, nodatabase):
         assert DBTestDate.__tablename__ == "db_test_date"
+
+        def prepare_fn():
+            return Database.prepare_views([date_view_params])
+
         with Database(
-            database=TestDatabase.dbpath, port=None, dialect="sqlite"
-        ) as dbsession:
+            database=TestDatabase.dbpath,
+            port=None,
+            dialect="sqlite",
+            prepare_fn=prepare_fn,
+        ) as dbdatabase:
+            dbsession = dbdatabase.session
             assert str(dbsession.bind.engine.url) == nodatabase
             assert TestDatabase.table_base == NoTZBase
             now = datetime(2022, 10, 20, 22, 35, 55, tzinfo=timezone.utc)
@@ -94,8 +70,14 @@ class TestDatabase:
             dbsession.commit()
             dbtestdate = dbsession.execute(select(DBTestDate)).scalar_one()
             assert dbtestdate.test_date == now
+            date_view = dbdatabase.get_prepare_results()[0]
             dbtestdate = dbsession.execute(select(date_view)).scalar_one()
             assert dbtestdate == now
+        with pytest.raises(DatabaseError):
+            Database.create_session()
+        with pytest.raises(DatabaseError):
+            with Database():
+                pass
 
     def test_get_reflect_session(self, database_to_reflect):
         with Database(
@@ -103,10 +85,13 @@ class TestDatabase:
             port=None,
             dialect="sqlite",
             reflect=True,
-        ) as dbsession:
+        ) as dbdatabase:
+            dbengine = dbdatabase.get_engine()
+            assert str(dbengine.url) == "sqlite:///tests/test_database.db"
+            dbsession = dbdatabase.get_session()
             assert TestDatabase.table_base == NoTZBase
             assert str(dbsession.bind.engine.url) == database_to_reflect
-            Table1 = dbsession.reflected_classes.table1
+            Table1 = dbdatabase.get_reflected_classes().table1
             row = dbsession.execute(select(Table1)).scalar_one()
             assert row.id == "1"
             assert row.col1 == "wfrefds"
@@ -114,19 +99,25 @@ class TestDatabase:
             # we don't have a timezone here
             assert row.date1 == datetime(1993, 9, 23, 14, 12, 56, 111000)
 
-    def test_get_session_ssh(self, mock_psycopg, mock_SSHTunnelForwarder):
+    def test_get_session_ssh(
+        self,
+        mock_psycopg,
+        mock_SSHTunnelForwarder,
+    ):
         with Database(
             ssh_host="mysshhost", **TestDatabase.params_pg
-        ) as dbsession:
+        ) as dbdatabase:
+            dbsession = dbdatabase.get_session()
             assert (
                 str(dbsession.bind.engine.url)
-                == "postgresql+psycopg://myuser:mypass@0.0.0.0:12345/mydatabase"
+                == "postgresql+psycopg://myuser:***@0.0.0.0:12345/mydatabase"
             )
         params = deepcopy(TestDatabase.params_pg)
         del params["password"]
         with Database(
             ssh_host="mysshhost", ssh_port=25, **params
-        ) as dbsession:
+        ) as dbdatabase:
+            dbsession = dbdatabase.get_session()
             assert (
                 str(dbsession.bind.engine.url)
                 == "postgresql+psycopg://myuser@0.0.0.0:12345/mydatabase"
@@ -134,9 +125,25 @@ class TestDatabase:
         assert TestDatabase.table_base == NoTZBase
         with Database(
             ssh_host="mysshhost", ssh_port=25, db_has_tz=True, **params
-        ) as dbsession:
+        ) as dbdatabase:
+            dbsession = dbdatabase.get_session()
             assert (
                 str(dbsession.bind.engine.url)
                 == "postgresql+psycopg://myuser@0.0.0.0:12345/mydatabase"
             )
             assert TestDatabase.table_base == TZBase
+        with Database(
+            ssh_host="mysshhost", ssh_port=25, table_base=TZBase, **params
+        ) as dbdatabase:
+            dbsession = dbdatabase.get_session()
+            assert (
+                str(dbsession.bind.engine.url)
+                == "postgresql+psycopg://myuser@0.0.0.0:12345/mydatabase"
+            )
+            assert TestDatabase.table_base == TZBase
+
+    def test_recreate_schema(self, mock_engine):
+        db_uri = "postgresql+psycopg://myuser:mypass@0.0.0.0:12345/mydatabase"
+        assert Database.recreate_schema(mock_engine, db_uri) is True
+        db_uri = "Error"
+        assert Database.recreate_schema(mock_engine, db_uri) is False

hdx_python_database-1.2.9/.config/black.toml

@@ -1,2 +0,0 @@
-[tool.black]
-line-length = 79

hdx_python_database-1.2.9/.config/pre-commit-config.yaml

@@ -1,26 +0,0 @@
-default_language_version:
-    python: python3.11
-repos:
-  - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
-    hooks:
-      - id: trailing-whitespace
-      - id: end-of-file-fixer
-      - id: check-ast
-  - repo: https://github.com/psf/black
-    rev: 23.3.0
-    hooks:
-      - id: black
-        args: [--config, .config/black.toml]
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: v0.0.267
-    hooks:
-      - id: ruff
-        args: [--config, .config/ruff.toml, --fix, --exit-non-zero-on-fix]
-  - repo: https://github.com/jazzband/pip-tools
-    rev: 6.13.0
-    hooks:
-      - id: pip-compile
-        name: pip-compile requirements.txt
-        files: pyproject.toml
-        args: [pyproject.toml, --resolver=backtracking, --all-extras, --upgrade, -q, -o, requirements.txt]

hdx_python_database-1.2.9/requirements.txt

@@ -1,69 +0,0 @@
-#
-# This file is autogenerated by pip-compile with Python 3.11
-# by the following command:
-#
-#    pip-compile --all-extras --output-file=requirements.txt --resolver=backtracking pyproject.toml
-#
-bcrypt==4.0.1
-    # via paramiko
-cffi==1.16.0
-    # via
-    #   cryptography
-    #   pynacl
-cfgv==3.4.0
-    # via pre-commit
-coverage[toml]==7.3.2
-    # via pytest-cov
-cryptography==41.0.4
-    # via paramiko
-distlib==0.3.7
-    # via virtualenv
-filelock==3.12.4
-    # via virtualenv
-greenlet==3.0.0
-    # via sqlalchemy
-identify==2.5.30
-    # via pre-commit
-iniconfig==2.0.0
-    # via pytest
-nodeenv==1.8.0
-    # via pre-commit
-packaging==23.2
-    # via pytest
-paramiko==3.3.1
-    # via sshtunnel
-platformdirs==3.11.0
-    # via virtualenv
-pluggy==1.3.0
-    # via pytest
-pre-commit==3.4.0
-    # via hdx-python-database (pyproject.toml)
-psycopg[binary]==3.1.12
-    # via hdx-python-database (pyproject.toml)
-psycopg-binary==3.1.12
-    # via psycopg
-pycparser==2.21
-    # via cffi
-pynacl==1.5.0
-    # via paramiko
-pytest==7.4.2
-    # via
-    #   hdx-python-database (pyproject.toml)
-    #   pytest-cov
-pytest-cov==4.1.0
-    # via hdx-python-database (pyproject.toml)
-pyyaml==6.0.1
-    # via pre-commit
-sqlalchemy==2.0.21
-    # via hdx-python-database (pyproject.toml)
-sshtunnel==0.4.0
-    # via hdx-python-database (pyproject.toml)
-typing-extensions==4.8.0
-    # via
-    #   psycopg
-    #   sqlalchemy
-virtualenv==20.24.5
-    # via pre-commit
-
-# The following packages are considered to be unsafe in a requirements file:
-# setuptools

hdx_python_database-1.2.9/src/hdx/database/__init__.py

@@ -1,149 +0,0 @@
-"""Database utilities"""
-import logging
-from typing import Any, Optional, Type, Union
-
-from sqlalchemy import create_engine
-from sqlalchemy.ext.automap import automap_base
-from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
-from sqlalchemy.pool import NullPool
-
-from ._version import version as __version__  # noqa: F401
-from .dburi import get_connection_uri
-from .no_timezone import Base as NoTZBase
-from .postgresql import wait_for_postgresql
-from .with_timezone import Base as TZBase
-
-logger = logging.getLogger(__name__)
-
-
-class Database:
-    """Database helper class to handle ssh tunnels, waiting for PostgreSQL to
-    be up etc. Can be used in a with statement returning a Session object that
-    if reflect is True will have a variable reflected_classes containing the
-    reflected classes. db_has_tz which defaults to False indicates whether
-    database datetime columns have timezones. If not, conversion occurs between
-    Python datetimes with timezones to timezoneless database columns (but not
-    when using reflection).
-
-    Args:
-        database (Optional[str]): Database name
-        host (Optional[str]): Host where database is located
-        port (Union[int, str, None]): Database port
-        username (Optional[str]): Username to log into database
-        password (Optional[str]): Password to log into database
-        dialect (str): Database dialect. Defaults to "postgresql".
-        driver (Optional[str]): Database driver. Defaults to None (psycopg if postgresql or None)
-        db_has_tz (bool): True if db datetime columns have timezone. Defaults to False.
-        reflect (bool): Whether to reflect existing tables. Defaults to False.
-        **kwargs: See below
-        ssh_host (str): SSH host (the server to connect to)
-        ssh_port (int): SSH port. Defaults to 22.
-        ssh_username (str): SSH username
-        ssh_password (str): SSH password
-        ssh_private_key: Location of SSH private key (instead of ssh_password)
-        For more advanced usage, see SSHTunnelForwarder documentation.
-
-    """
-
-    def __init__(
-        self,
-        database: Optional[str] = None,
-        host: Optional[str] = None,
-        port: Union[int, str, None] = None,
-        username: Optional[str] = None,
-        password: Optional[str] = None,
-        dialect: str = "postgresql",
-        driver: Optional[str] = None,
-        db_has_tz: bool = False,
-        reflect: bool = False,
-        **kwargs: Any,
-    ) -> None:
-        if port is not None:
-            port = int(port)
-        if len(kwargs) != 0:
-            try:
-                import sshtunnel
-            except ImportError:
-                # dependency missing, log an error
-                logger.error(
-                    "sshtunnel not found! Please install hdx-python-database[sshtunnel] to enable."
-                )
-                raise
-            ssh_host = kwargs["ssh_host"]
-            del kwargs["ssh_host"]
-            ssh_port = kwargs.get("ssh_port")
-            if ssh_port is not None:
-                ssh_port = int(ssh_port)
-                del kwargs["ssh_port"]
-            else:
-                ssh_port = 22
-            self.server = sshtunnel.SSHTunnelForwarder(
-                (ssh_host, ssh_port),
-                remote_bind_address=(host, port),
-                **kwargs,
-            )
-            self.server.start()
-            host = self.server.local_bind_host
-            port = self.server.local_bind_port
-        else:
-            self.server = None
-        db_uri = get_connection_uri(
-            database,
-            host,
-            port,
-            username,
-            password,
-            dialect=dialect,
-            driver=driver,
-        )
-        if dialect == "postgresql":
-            wait_for_postgresql(db_uri)
-        if db_has_tz:
-            table_base = TZBase
-        else:
-            table_base = NoTZBase
-        self.session = self.get_session(
-            db_uri, table_base=table_base, reflect=reflect
-        )
-
-    def __enter__(self) -> Session:
-        return self.session
-
-    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
-        self.session.close()
-        if self.server is not None:
-            self.server.stop()
-
-    @staticmethod
-    def get_session(
-        db_uri: str,
-        table_base: Type[DeclarativeBase] = NoTZBase,
-        reflect: bool = False,
-    ) -> Session:
-        """Gets SQLAlchemy session given url. Tables must inherit from Base in
-        hdx.utilities.database unless base is defined. If reflect is True,
-        classes will be reflected from an existing database and the reflected
-        classes are returned in a variable reflected_classes in the returned
-        Session object. Note that type annotation maps don't work with
-        reflection.
-
-        Args:
-            db_uri (str): Connection URI
-            table_base (Type[DeclarativeBase]): Base database table class. Defaults to NoTZBase.
-            reflect (bool): Whether to reflect existing tables. Defaults to False.
-
-        Returns:
-            sqlalchemy.orm.Session: SQLAlchemy session
-        """
-        engine = create_engine(db_uri, poolclass=NullPool, echo=False)
-        Session = sessionmaker(bind=engine)
-        if reflect:
-            Base = automap_base(declarative_base=table_base)
-            Base.prepare(autoload_with=engine)
-            classes = Base.classes
-        else:
-            table_base.metadata.create_all(engine)
-            classes = None
-        session = Session()
-        session.reflected_classes = classes
-        return session

hdx_python_database-1.2.9/tests/hdx/database/conftest.py

@@ -1,22 +0,0 @@
-import psycopg
-import pytest
-
-from . import PsycopgConnection
-
-
-@pytest.fixture(scope="function")
-def mock_psycopg(monkeypatch):
-    def connect(*args, **kwargs):
-        if PsycopgConnection.connected:
-
-            class Connection:
-                @staticmethod
-                def close():
-                    return
-
-            return Connection()
-        else:
-            PsycopgConnection.connected = True
-            raise psycopg.OperationalError
-
-    monkeypatch.setattr(psycopg, "connect", connect)