alembic-utils-extended 0.0.0__tar.gz

alembic_utils_extended-0.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Candid Health
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

alembic_utils_extended-0.0.0/PKG-INFO

@@ -0,0 +1,165 @@
+Metadata-Version: 2.3
+Name: alembic-utils-extended
+Version: 0.0.0
+Summary: A sqlalchemy/alembic extension for migrating procedures and views
+License: MIT
+Author: Justin Malin
+Author-email: justin@joincandidhealth.com
+Requires-Python: >=3.9
+Classifier: Development Status :: 4 - Beta
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: SQL
+Requires-Dist: alembic (>=1.9)
+Requires-Dist: flupy
+Requires-Dist: parse (>=1.8.4)
+Requires-Dist: sqlalchemy (>=1.4)
+Requires-Dist: typing_extensions (>=0.1.0)
+Project-URL: GitHub, https://github.com/candidhealth/alembic-utils-extended
+Project-URL: PyPI, https://pypi.org/project/alembic-utils/
+Description-Content-Type: text/markdown
+
+# Alembic Utils
+
+<p>
+    <a href="https://github.com/candidhealth/alembic-utils-extended/actions">
+        <img src="https://github.com/candidhealth/alembic-utils-extended/workflows/Tests/badge.svg" alt="Test Status" height="18">
+    </a>
+    <a href="https://github.com/candidhealth/alembic-utils-extended/actions">
+        <img src="https://github.com/candidhealth/alembic-utils-extended/workflows/pre-commit%20hooks/badge.svg" alt="Pre-commit Status" height="18">
+    </a>
+    <a href="https://codecov.io/gh/candidhealth/alembic-utils-extended"><img src="https://codecov.io/gh/candidhealth/alembic-utils-extended/branch/master/graph/badge.svg" height="18"></a>
+</p>
+<p>
+    <a href="https://github.com/candidhealth/alembic-utils-extended/blob/master/LICENSE"><img src="https://img.shields.io/pypi/l/markdown-subtemplate.svg" alt="License" height="18"></a>
+    <a href="https://badge.fury.io/py/alembic_utils"><img src="https://badge.fury.io/py/alembic_utils.svg" alt="PyPI version" height="18"></a>
+    <a href="https://github.com/psf/black">
+        <img src="https://img.shields.io/badge/code%20style-black-000000.svg" alt="Codestyle Black" height="18">
+    </a>
+    <a href="https://pypi.org/project/alembic_utils/"><img src="https://img.shields.io/pypi/dm/alembic_utils.svg" alt="Download count" height="18"></a>
+</p>
+<p>
+    <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.6+-blue.svg" alt="Python version" height="18"></a>
+    <a href=""><img src="https://img.shields.io/badge/postgresql-11+-blue.svg" alt="PostgreSQL version" height="18"></a>
+</p>
+
+---
+
+**Documentation**: <a href="https://olirice.github.io/alembic_utils" target="_blank">https://olirice.github.io/alembic_utils</a>
+
+**Source Code**: <a href="https://github.com/candidhealth/alembic-utils-extended" target="_blank">https://github.com/candidhealth/alembic-utils-extended</a>
+
+---
+
+**Autogenerate Support for PostgreSQL Functions, Views, Materialized View, Triggers, and Policies**
+
+[Alembic](https://alembic.sqlalchemy.org/en/latest/) is the defacto migration tool for use with [SQLAlchemy](https://www.sqlalchemy.org/). Without extensions, alembic can detect local changes to SQLAlchemy models and autogenerate a database migration or "revision" script. That revision can be applied to update the database's schema to match the SQLAlchemy model definitions.
+
+Alembic Utils is an extension to alembic that adds support for autogenerating a larger number of [PostgreSQL](https://www.postgresql.org/) entity types, including [functions](https://www.postgresql.org/docs/current/sql-createfunction.html), [views](https://www.postgresql.org/docs/current/sql-createview.html), [materialized views](https://www.postgresql.org/docs/current/sql-creatematerializedview.html), [triggers](https://www.postgresql.org/docs/current/sql-createtrigger.html), and [policies](https://www.postgresql.org/docs/current/sql-createpolicy.html).
+
+### TL;DR
+
+Update alembic's `env.py` to register a function or view:
+
+```python
+# migrations/env.py
+from alembic_utils_extended.pg_function import PGFunction
+from alembic_utils_extended.replaceable_entity import register_entities
+
+to_upper = PGFunction(
+    schema='public',
+    signature='to_upper(some_text text)',
+    definition="""
+  RETURNS text as
+  $$
+    SELECT upper(some_text)
+  $$ language SQL;
+  """
+)
+
+register_entities([to_upper])
+```
+
+You're done!
+
+The next time you autogenerate a revision with
+```shell
+alembic revision --autogenerate -m 'create to_upper'
+```
+Alembic will detect if your entities are new, updated, or removed & populate the revison's `upgrade` and `downgrade` sections automatically.
+
+For example:
+
+```python
+"""create to_upper
+
+Revision ID: 8efi0da3a4
+Revises:
+Create Date: 2020-04-22 09:24:25.556995
+"""
+from alembic import op
+import sqlalchemy as sa
+from alembic_utils_extended.pg_function import PGFunction
+
+# revision identifiers, used by Alembic.
+revision = '8efi0da3a4'
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    public_to_upper_6fa0de = PGFunction(
+        schema="public",
+        signature="to_upper(some_text text)",
+        definition="""
+        returns text
+        as
+        $$ select upper(some_text) $$ language SQL;
+        """
+    )
+
+    op.create_entity(public_to_upper_6fa0de)
+
+
+def downgrade():
+    public_to_upper_6fa0de = PGFunction(
+        schema="public",
+        signature="to_upper(some_text text)",
+        definition="# Not Used"
+    )
+
+    op.drop_entity(public_to_upper_6fa0de)
+```
+
+
+Visit the [quickstart guide](https://candidhealth.github.io/alembic-utils-extended/quickstart/) for usage instructions.
+
+<p align="center">&mdash;&mdash;  &mdash;&mdash;</p>
+
+### Contributing
+
+To run the tests
+```
+# install pip dependencies
+pip install wheel && pip install -e ".[dev]"
+
+# run the tests
+pytest src/test
+```
+
+To invoke the linter automated formatting and generally make use of precommit checks:
+```
+pip install pre-commit
+pre-commit install
+
+# manually run
+pre-commit run --all
+```
+

alembic_utils_extended-0.0.0/README.md

@@ -0,0 +1,137 @@
+# Alembic Utils
+
+<p>
+    <a href="https://github.com/candidhealth/alembic-utils-extended/actions">
+        <img src="https://github.com/candidhealth/alembic-utils-extended/workflows/Tests/badge.svg" alt="Test Status" height="18">
+    </a>
+    <a href="https://github.com/candidhealth/alembic-utils-extended/actions">
+        <img src="https://github.com/candidhealth/alembic-utils-extended/workflows/pre-commit%20hooks/badge.svg" alt="Pre-commit Status" height="18">
+    </a>
+    <a href="https://codecov.io/gh/candidhealth/alembic-utils-extended"><img src="https://codecov.io/gh/candidhealth/alembic-utils-extended/branch/master/graph/badge.svg" height="18"></a>
+</p>
+<p>
+    <a href="https://github.com/candidhealth/alembic-utils-extended/blob/master/LICENSE"><img src="https://img.shields.io/pypi/l/markdown-subtemplate.svg" alt="License" height="18"></a>
+    <a href="https://badge.fury.io/py/alembic_utils"><img src="https://badge.fury.io/py/alembic_utils.svg" alt="PyPI version" height="18"></a>
+    <a href="https://github.com/psf/black">
+        <img src="https://img.shields.io/badge/code%20style-black-000000.svg" alt="Codestyle Black" height="18">
+    </a>
+    <a href="https://pypi.org/project/alembic_utils/"><img src="https://img.shields.io/pypi/dm/alembic_utils.svg" alt="Download count" height="18"></a>
+</p>
+<p>
+    <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.6+-blue.svg" alt="Python version" height="18"></a>
+    <a href=""><img src="https://img.shields.io/badge/postgresql-11+-blue.svg" alt="PostgreSQL version" height="18"></a>
+</p>
+
+---
+
+**Documentation**: <a href="https://olirice.github.io/alembic_utils" target="_blank">https://olirice.github.io/alembic_utils</a>
+
+**Source Code**: <a href="https://github.com/candidhealth/alembic-utils-extended" target="_blank">https://github.com/candidhealth/alembic-utils-extended</a>
+
+---
+
+**Autogenerate Support for PostgreSQL Functions, Views, Materialized View, Triggers, and Policies**
+
+[Alembic](https://alembic.sqlalchemy.org/en/latest/) is the defacto migration tool for use with [SQLAlchemy](https://www.sqlalchemy.org/). Without extensions, alembic can detect local changes to SQLAlchemy models and autogenerate a database migration or "revision" script. That revision can be applied to update the database's schema to match the SQLAlchemy model definitions.
+
+Alembic Utils is an extension to alembic that adds support for autogenerating a larger number of [PostgreSQL](https://www.postgresql.org/) entity types, including [functions](https://www.postgresql.org/docs/current/sql-createfunction.html), [views](https://www.postgresql.org/docs/current/sql-createview.html), [materialized views](https://www.postgresql.org/docs/current/sql-creatematerializedview.html), [triggers](https://www.postgresql.org/docs/current/sql-createtrigger.html), and [policies](https://www.postgresql.org/docs/current/sql-createpolicy.html).
+
+### TL;DR
+
+Update alembic's `env.py` to register a function or view:
+
+```python
+# migrations/env.py
+from alembic_utils_extended.pg_function import PGFunction
+from alembic_utils_extended.replaceable_entity import register_entities
+
+to_upper = PGFunction(
+    schema='public',
+    signature='to_upper(some_text text)',
+    definition="""
+  RETURNS text as
+  $$
+    SELECT upper(some_text)
+  $$ language SQL;
+  """
+)
+
+register_entities([to_upper])
+```
+
+You're done!
+
+The next time you autogenerate a revision with
+```shell
+alembic revision --autogenerate -m 'create to_upper'
+```
+Alembic will detect if your entities are new, updated, or removed & populate the revison's `upgrade` and `downgrade` sections automatically.
+
+For example:
+
+```python
+"""create to_upper
+
+Revision ID: 8efi0da3a4
+Revises:
+Create Date: 2020-04-22 09:24:25.556995
+"""
+from alembic import op
+import sqlalchemy as sa
+from alembic_utils_extended.pg_function import PGFunction
+
+# revision identifiers, used by Alembic.
+revision = '8efi0da3a4'
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    public_to_upper_6fa0de = PGFunction(
+        schema="public",
+        signature="to_upper(some_text text)",
+        definition="""
+        returns text
+        as
+        $$ select upper(some_text) $$ language SQL;
+        """
+    )
+
+    op.create_entity(public_to_upper_6fa0de)
+
+
+def downgrade():
+    public_to_upper_6fa0de = PGFunction(
+        schema="public",
+        signature="to_upper(some_text text)",
+        definition="# Not Used"
+    )
+
+    op.drop_entity(public_to_upper_6fa0de)
+```
+
+
+Visit the [quickstart guide](https://candidhealth.github.io/alembic-utils-extended/quickstart/) for usage instructions.
+
+<p align="center">&mdash;&mdash;  &mdash;&mdash;</p>
+
+### Contributing
+
+To run the tests
+```
+# install pip dependencies
+pip install wheel && pip install -e ".[dev]"
+
+# run the tests
+pytest src/test
+```
+
+To invoke the linter automated formatting and generally make use of precommit checks:
+```
+pip install pre-commit
+pre-commit install
+
+# manually run
+pre-commit run --all
+```

alembic_utils_extended-0.0.0/pyproject.toml

@@ -0,0 +1,92 @@
+[project]
+name = "alembic-utils-extended"
+version = "0.0.0"
+description = "A sqlalchemy/alembic extension for migrating procedures and views"
+requires-python = ">=3.9"
+authors = [{ "name" = "Justin Malin", "email" = "justin@joincandidhealth.com" }]
+license = "MIT"
+readme = "README.md"
+packages = [{ include = "alembic-utils-extended", from = "src" }]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "License :: OSI Approved :: MIT License",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: SQL",
+]
+dependencies = [
+    "alembic>=1.9",
+    "flupy",
+    "parse>=1.8.4",
+    "sqlalchemy>=1.4",
+    "typing_extensions>=0.1.0",
+]
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry.urls]
+"PyPI" = "https://pypi.org/project/alembic-utils/"
+"GitHub" = "https://github.com/candidhealth/alembic-utils-extended"
+
+[tool.poetry.group.dev.dependencies]
+black = "*"
+pylint = "*"
+pre-commit = "*"
+mypy = "*"
+psycopg2-binary = "*"
+pytest = "*"
+pytest-cov = "*"
+mkdocs = "*"
+
+[tool.poetry.group.nvim.dependencies]
+neovim = "*"
+python-language-server = "*"
+
+[tool.poetry.group.docs.dependencies]
+mkdocs = "*"
+pygments = "*"
+pymdown-extensions = "*"
+mkautodoc = "*"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+"*" = ["py.typed"]
+
+[tool.black]
+line-length = 100
+exclude = '''
+/(
+    \.git
+  | \.hg
+  | \.mypy_cache
+  | \.tox
+  | \.venv
+  | _build
+  | buck-out
+  | build
+  | dist
+)/
+'''
+
+[tool.isort]
+known_third_party = ["alembic", "flupy", "parse", "pytest", "setuptools", "sqlalchemy"]
+
+[tool.mypy]
+follow_imports = "skip"
+strict_optional = true
+warn_redundant_casts = true
+warn_unused_ignores = false
+disallow_any_generics = true
+check_untyped_defs = true
+no_implicit_reexport = true
+ignore_missing_imports = true
+# disallow_untyped_defs = true

alembic_utils_extended-0.0.0/src/alembic_utils_extended/depends.py

@@ -0,0 +1,145 @@
+import logging
+from contextlib import contextmanager
+from typing import Generator, List
+
+from sqlalchemy import exc as sqla_exc
+from sqlalchemy.orm import Session
+
+from alembic_utils_extended.simulate import simulate_entity
+
+logger = logging.getLogger(__name__)
+
+
+def solve_resolution_order(sess: Session, entities):
+    """Solve for an entity resolution order that increases the probability that
+    a migration will suceed if, for example, two new views are created and one
+    refers to the other
+
+    This strategy will only solve for simple cases
+    """
+
+    resolved = []
+
+    # Resolve the entities with 0 dependencies first (faster)
+    logger.info("Resolving entities with no dependencies")
+    for entity in entities:
+        try:
+            with simulate_entity(sess, entity):
+                resolved.append(entity)
+        except (sqla_exc.ProgrammingError, sqla_exc.InternalError) as exc:
+            continue
+
+    # Resolve entities with possible dependencies
+    logger.info("Resolving entities with dependencies. This may take a minute")
+    for _ in range(len(entities)):
+        n_resolved = len(resolved)
+
+        for entity in entities:
+            if entity in resolved:
+                continue
+
+            try:
+                with simulate_entity(sess, entity, dependencies=resolved):
+                    resolved.append(entity)
+            except (sqla_exc.ProgrammingError, sqla_exc.InternalError):
+                continue
+
+        if len(resolved) == n_resolved:
+            # No new entities resolved in the last iteration. Exit
+            break
+
+    for entity in entities:
+        if entity not in resolved:
+            resolved.append(entity)
+
+    return resolved
+
+
+@contextmanager
+def recreate_dropped(connection) -> Generator[Session, None, None]:
+    """Recreate any dropped all ReplaceableEntities that were dropped within block
+
+    This is useful for making cascading updates. For example, updating a table's column type when it has dependent views.
+
+    def upgrade() -> None:
+
+        my_view = PGView(...)
+
+        with recreate_dropped(op.get_bind()) as conn:
+
+            op.drop_entity(my_view)
+
+            # change an integer column to a bigint
+            op.alter_column(
+                table_name="account",
+                column_name="id",
+                schema="public"
+                type_=sa.BIGINT()
+                existing_type=sa.Integer(),
+            )
+    """
+    from alembic_utils_extended.pg_function import PGFunction
+    from alembic_utils_extended.pg_materialized_view import PGMaterializedView
+    from alembic_utils_extended.pg_trigger import PGTrigger
+    from alembic_utils_extended.pg_view import PGView
+    from alembic_utils_extended.replaceable_entity import ReplaceableEntity
+
+    # Do not include permissions here e.g. PGGrantTable. If columns granted to users are dropped, it will cause an error
+
+    def collect_all_db_entities(sess: Session) -> List[ReplaceableEntity]:
+        """Collect all entities from the database"""
+
+        return [
+            *PGFunction.from_database(sess, "%"),
+            *PGTrigger.from_database(sess, "%"),
+            *PGView.from_database(sess, "%"),
+            *PGMaterializedView.from_database(sess, "%"),
+        ]
+
+    sess = Session(bind=connection)
+
+    # All existing entities, before the upgrade
+    before = collect_all_db_entities(sess)
+
+    # In the yield, do a
+    #     op.drop_entity(my_mat_view, cascade=True)
+    #     op.create_entity(my_mat_view)
+    try:
+        yield sess
+    except:
+        sess.rollback()
+        raise
+
+    # All existing entities, after the upgrade
+    after = collect_all_db_entities(sess)
+    after_identities = {x.identity for x in after}
+
+    # Entities that were not impacted, or that we have "recovered"
+    resolved = []
+    unresolved = []
+
+    # First, ignore the ones that were not impacted by the upgrade
+    for ent in before:
+        if ent.identity in after_identities:
+            resolved.append(ent)
+        else:
+            unresolved.append(ent)
+
+    # Attempt to find an acceptable order of creation for the unresolved entities
+    ordered_unresolved = solve_resolution_order(sess, unresolved)
+
+    # Attempt to recreate the missing entities in the specified order
+    for ent in ordered_unresolved:
+        sess.execute(ent.to_sql_statement_create())
+
+    # Sanity check that everything is now fine
+    sanity_check = collect_all_db_entities(sess)
+    # Fail and rollback if the sanity check is wrong
+    try:
+        assert len(before) == len(sanity_check)
+    except:
+        sess.rollback()
+        raise
+
+    # Close out the session
+    sess.commit()

alembic_utils_extended-0.0.0/src/alembic_utils_extended/exceptions.py

@@ -0,0 +1,18 @@
+class AlembicUtilsException(Exception):
+    """Base exception for AlembicUtils package"""
+
+
+class SQLParseFailure(AlembicUtilsException):
+    """An entity could not be parsed"""
+
+
+class FailedToGenerateComparable(AlembicUtilsException):
+    """Failed to generate a comparable entity"""
+
+
+class UnreachableException(AlembicUtilsException):
+    """An exception no one should ever see"""
+
+
+class BadInputException(AlembicUtilsException):
+    """Invalid user input"""

alembic_utils_extended-0.0.0/src/alembic_utils_extended/experimental/__init__.py

@@ -0,0 +1,6 @@
+from alembic_utils_extended.experimental._collect_instances import (
+    collect_instances,
+    collect_subclasses,
+)
+
+__all__ = ["collect_instances", "collect_subclasses"]

alembic_utils_extended-0.0.0/src/alembic_utils_extended/experimental/_collect_instances.py

@@ -0,0 +1,90 @@
+import importlib
+import os
+from pathlib import Path
+from types import ModuleType
+from typing import Generator, List, Type, TypeVar
+
+from flupy import walk_files
+
+T = TypeVar("T")
+
+
+def walk_modules(module: ModuleType) -> Generator[ModuleType, None, None]:
+    """Recursively yield python import paths to submodules in *module*
+
+    Example:
+        module_iter = walk_modules(alembic_utils_extended)
+
+        for module_path in module_iter:
+            print(module_path)
+
+        # alembic_utils_extended.exceptions
+        # alembic_utils_extended.on_entity_mixin
+        # ...
+    """
+    top_module = module
+    top_path = Path(top_module.__path__[0])
+    top_path_absolute = top_path.resolve()
+
+    directories = (
+        walk_files(str(top_path_absolute))
+        .filter(lambda x: x.endswith(".py"))
+        .map(Path)
+        .group_by(lambda x: x.parent)
+        .map(lambda x: (x[0], x[1].collect()))
+    )
+
+    for base_path, files in directories:
+        if str(base_path / "__init__.py") in [str(x) for x in files]:
+            for module_path in files:
+                if "__init__.py" not in str(module_path):
+
+                    # Example: elt.settings
+                    module_import_path = str(module_path)[
+                        len(str(top_path_absolute)) - len(top_module.__name__) :
+                    ].replace(os.path.sep, ".")[:-3]
+
+                    module = importlib.import_module(module_import_path)
+                    yield module
+
+
+def collect_instances(module: ModuleType, class_: Type[T]) -> List[T]:
+    """Collect all instances of *class_* defined in *module*
+
+    Note: Will import all submodules in *module*. Beware of import side effects
+    """
+
+    found: List[T] = []
+
+    for module_ in walk_modules(module):
+
+        for _, variable in module_.__dict__.items():
+
+            if isinstance(variable, class_):
+                # Ensure variable is not a subclass
+                if variable.__class__ == class_:
+                    found.append(variable)
+    return found
+
+
+def collect_subclasses(module: ModuleType, class_: Type[T]) -> List[Type[T]]:
+    """Collect all subclasses of *class_* currently imported or defined in *module*
+
+    Note: Will import all submodules in *module*. Beware of import side effects
+    """
+
+    found: List[Type[T]] = []
+
+    for module_ in walk_modules(module):
+
+        for _, variable in module_.__dict__.items():
+            try:
+                if issubclass(variable, class_) and not class_ == variable:
+                    found.append(variable)
+            except TypeError:
+                # argument 2 to issubclass must be a class ....
+                pass
+
+    imported: List[Type[T]] = list(class_.__subclasses__())  # type: ignore
+
+    return list(set(found + imported))