django-postpone-index 0.0.1__py3-none-any.whl

django_postpone_index-0.0.1.dist-info/METADATA

@@ -0,0 +1,276 @@
+Metadata-Version: 2.4
+Name: django-postpone-index
+Version: 0.0.1
+Summary: Postpone index creation to provide Zero Downtime Migration feature
+Home-page: https://github.com/nnseva/django-postpone-index
+Author: Vsevolod Novikov
+Author-email: nnseva@gmail.com
+License: LGPLv3
+Project-URL: Source, https://github.com/nnseva/django-postpone-index
+Project-URL: Issues, https://github.com/nnseva/django-postpone-index/issues
+Keywords: zero-downtime-migration,django,migration,zero-downtime,downtime,postgres,postgresql
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 2.2
+Classifier: Framework :: Django :: 3.0
+Classifier: Framework :: Django :: 3.1
+Classifier: Framework :: Django :: 3.2
+Classifier: Framework :: Django :: 4.0
+Classifier: Framework :: Django :: 4.1
+Classifier: Framework :: Django :: 4.2
+Classifier: Framework :: Django :: 5.0
+Classifier: Framework :: Django :: 5.1
+Classifier: Framework :: Django :: 5.2
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: django2-postgres-backport
+Requires-Dist: django-unlimited-char
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+[![Tests](https://github.com/nnseva/django-postpone-index/actions/workflows/ci.yml/badge.svg)](https://github.com/nnseva/django-postpone-index/actions/workflows/ci.yml)
+
+# Django Postpone Index
+
+This package provides modules and tools to postpone any index creation instead doing it inside the migration,
+to provide *Zero Downtime Migration* feature.
+
+The package is now using the PostgresSQL-specific `CREATE INDEX CONCURRENTLY` SQL command, so is applicable
+only to the PostgreSQL backend.
+
+## Installation
+
+*Stable version* from the PyPi package repository
+```bash
+pip install django-postpone-index
+```
+
+*Last development version* from the GitHub source version control system
+```
+pip install git+git://github.com/nnseva/django-postpone-index.git
+```
+
+## Problem Description
+
+Large data leads to long index creation time.
+
+When the migration is automatically created, it executes all SQL commands creating index inside a transaction.
+
+Large data and index creation inside a transaction lead to long-term table lock which blocks any data writting to the table.
+
+On the other side, `CREATE INDEX CONCURRENTLY` SQL command may solve the problem, but this SQL command can not be executed inside a transaction block.
+
+The `AddIndexConcurrently` might be created in a separate migration, moving out the automatically generated `AddIndex` from the migration,
+but not all indexes are created using `AddIndex`.
+
+## Solution
+
+All index creation SQL commands (as well as unique constraints creation) are catched
+and postponed using a special `PostponedSQL` model (the `DROP INDEX` and `DROP CONSTRAINT` SQL commands
+are still executed immediately).
+
+When the migration is finished, the postponed indexes may be created in a separate process
+using `CREATE INDEX CONCURRENTLY` SQL command by the `apply_postponed` management command.
+Apart from the standard migration, this process doesn't lock the whole table for a long time.
+
+Failed index creation statements don't lead to the command failure
+(until a special command line parameter passed). Every failed statement is stored
+as erroneous instead. When the data is fixed, you can execute the `apply_postponed` management
+command again to restore the failed indexes.
+
+## Complex Use Cases
+
+The following complex use cases are processed by the package.
+
+- Several create/drop pairs. There can be several create/drop index pairs if several migrations applied at once.
+- Back Migration. The both, forward and backward migrations are processed.
+- Implicit index drop while removing the table. The Django doesn't issue a separate SQL to drop indexes of the dropped table.
+- Implicit index drop while removing the field. The Django doesn't issue a separate SQL to drop indexes related to the dropped column.
+
+## Using
+
+Include the `postpone_index` application in `setting.py`:
+
+```python
+INSTALLED_APPS = [
+    ...
+    'postpone_index',
+    ...
+]
+```
+
+Use `pospone_index.contrib.postgres` or `postpone_index.contrib.postgis` engines instead of the Django-provided in `settings.py`:
+
+```python
+DATABASES = {
+    'default': {
+        'ENGINE': 'postpone_index.contrib.postgres',
+        ...
+    }
+}
+```
+
+If you provide your own database engine instead of the Django-provided, you can also
+combine `pospone_index.contrib.postgres.schema.DatabaseSchemaEditorMixin` with your own Database Schema Editor, f.e.:
+
+`mybackend/schema.py`
+```python
+from django.db.backends.postgresql.schema import DatabaseSchemaEditor as _DatabaseSchemaEditor
+from pospone_index.contrib.postgres.schema import DatabaseSchemaEditorMixin
+
+class PostponeIndexDatabaseSchemaEditor(DatabaseSchemaEditorMixin, _DatabaseSchemaEditor):
+    # Your own code
+    ...
+```
+
+`mybackend/base.py`
+```python
+from django.db.backends.postgresql.base import (
+    DatabaseWrapper as _DatabaseWrapper,
+)
+
+from mybackend.schema import PostponeIndexDatabaseSchemaEditor
+
+
+class DatabaseWrapper(_DatabaseWrapper):
+    """Database wrapper"""
+
+    SchemaEditorClass = PostponeIndexDatabaseSchemaEditor
+    # Your own code
+    ...
+```
+
+Execute `apply_postponed` management command every time after the `migrate` management command to create new postponed indexes.
+
+Monitor `PostponedSQL` model instances to see errors on the SQL execution.
+
+After the data is fixed, you can try to recreate the postponed invalid indexes just
+calling the `apply_postponed` migration command again. All not-applied indexes will be tried to create again.
+
+**NOTICE** the `apply_postponed` management command doesn't have any explicit locking mechanics. Avoid starting this
+command concurrently with itself or another `migrate` command on the same database.
+
+## Django testing
+
+Django migrates testing database before tests. Always use `POSTPONE_INDEX_IGNORE = True` settings to avoid postpone index
+for the testing database.
+
+If you want to check your own migration with the postpone index switched on,
+use the `postpone_index.testing_utils.TestCase` and `override_settings` Django feature with the following trick:
+
+```python
+from django.core.management import call_command
+from django.test import override_settings
+from postpone_index.models import PostponedSQL
+from postpone_index import testing_utils
+
+class ModuleTest(testing_utils.TestCase):
+    # Notice that the base TestCase is TransactionalTestCase
+
+    @classmethod
+    def setUpClass(cls):
+        # If you want to have customized setUpClass, call the method of the base class
+        super().setUpClass()
+
+    @classmethod
+    def tearDownClass(cls):
+        # If you want to have customized tearDownClass, call the method of the base class
+        super().tearDownClass()
+
+    def test_my_special_migration_case(self):
+        """Explicitly check my migration with postpone_index"""
+
+        module_to_check = "my_module"           # Your Django App
+        migration_before_the_check = "0005"     # Just before your migration
+        migration_to_check = "0006"             # The migration you check
+
+        # Notice that POSTPONED_INDEX_IGNORE is True by default while testing
+        call_command('migrate', module_to_check, migration_before_the_check)
+
+        with override_settings(
+            POSTPONE_INDEX_IGNORE=False
+        ):
+            # Here we can check how it's going with `postpone_index` activated
+
+            # Check whether your migration works as expected with postponed indexes
+            call_command('migrate', module_to_check, migration_to_check)
+
+            # Here you can check how the module works before apply_postponed
+            ...
+
+            # Check whether the indexes applied properly. The `-x` parameter
+            # causes exception on errors
+            call_command('apply_postponed', 'run', '-x')
+```
+
+## Django settings
+
+### `POSTPONE_INDEX_IGNORE`
+
+The setting totally switches off the functionality of the package.
+
+Always use this setting in the test environment to avoid using postponed index creation for the test database.
+
+May be used in a heterogeneous database environment to switch off the package functionality on unsupported databases.
+
+### `POSTPONE_INDEX_ADMIN_IGNORE`
+
+The `PostponedSQL` model admin view is switched on by default. You can totally switch it off,
+or create your own admin class instead. Use `postpone_index.admin.PostponedSQLAdminMixin` as a base class if necessary.
+
+## Django database
+
+The Django supports heterogeneous database environment in a single project. Every single database has it's own
+state of migrations executed by the `manage.py migrate --database <alias>`.
+
+The `apply_postponed` command also supports selection of the database alias using similar syntax:
+
+```bash
+
+# The 'default' database alias is used as a default
+python manage.py migrate
+python manage.py apply_postponed
+
+# A non-default database alias parameter has similar syntax
+python manage.py migrate --database another-postgres-database
+python manage.py apply_postponed --database another-postgres-database
+```
+
+Use `POSTPONE_INDEX_IGNORE=1` environment to switch off the package functionality on migrations running on unsupported database engines like:
+
+```bash
+POSTPONE_INDEX_IGNORE=1 python manage.py migrate --database non-postgres-database
+```
+
+## Special migrations to avoid postpone index
+
+Sometimes you may need to avoid the `postpone_index` applied to a single migration.
+
+Just include the `PostponeIndexIgnoreMigrationMixin` into a base class list for your special migration:
+
+```python
+from django.db import migrations, models
+from postpone_index.migration_utils import PostponeIndexIgnoreMigrationMixin
+
+class Migration(PostponeIndexIgnoreMigrationMixin, migrations.Migration):
+    ...
+```

django_postpone_index-0.0.1.dist-info/RECORD

@@ -0,0 +1,12 @@
+postpone_index/__init__.py,sha256=s1NgdtOn7SO7scs-uAaq1134F7Luh-JjB5Ghra3D2fQ,84
+postpone_index/_version.py,sha256=qf6R-J7-UyuABBo8c0HgaquJ8bejVbf07HodXgwAwgQ,704
+postpone_index/admin.py,sha256=TAZxaciQkEBNhReYulsU5VopPCyABtMLZvCSxk0CfYE,820
+postpone_index/apps.py,sha256=1ap2HY49uXVXjY9UWK3___fjPe22wqFOSOnuH2kIAxs,200
+postpone_index/migration_utils.py,sha256=02rm77mz_MAufl5vdqKPV1CAd1-3bRk6kSy4MIxKsEo,752
+postpone_index/models.py,sha256=jMsA5hdiL8Xgl-hq7bgVl5NUX6LXlSYpMuX9pD68BbU,1771
+postpone_index/testing_utils.py,sha256=ycxp7ovY6JceOvNpotK7OGAebhdHicXJ-0fWJmaq9fo,1224
+postpone_index/utils.py,sha256=wYVPsQRs3LN-ZtvfSWrDDqFFp0OCkVxW9j7w9yqd47o,3823
+django_postpone_index-0.0.1.dist-info/METADATA,sha256=f6r6FaLnag7wQ4NjfFRPhMDZAffRmXtXK7Bg7o4viLU,10415
+django_postpone_index-0.0.1.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+django_postpone_index-0.0.1.dist-info/top_level.txt,sha256=SzF44QR8Fr9ArKXZPY7FSJUlTBKYo_dPNHn2hbqKOrc,15
+django_postpone_index-0.0.1.dist-info/RECORD,,

django_postpone_index-0.0.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.10.2)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

django_postpone_index-0.0.1.dist-info/top_level.txt

@@ -0,0 +1 @@
+postpone_index

postpone_index/__init__.py

@@ -0,0 +1,4 @@
+from ._version import *  # noqa
+
+
+default_app_config = 'postpone_index.apps.Config'

postpone_index/_version.py

@@ -0,0 +1,34 @@
+# file generated by setuptools-scm
+# don't change, don't track in version control
+
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
+
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Tuple
+    from typing import Union
+
+    VERSION_TUPLE = Tuple[Union[int, str], ...]
+    COMMIT_ID = Union[str, None]
+else:
+    VERSION_TUPLE = object
+    COMMIT_ID = object
+
+version: str
+__version__: str
+__version_tuple__: VERSION_TUPLE
+version_tuple: VERSION_TUPLE
+commit_id: COMMIT_ID
+__commit_id__: COMMIT_ID
+
+__version__ = version = '0.0.1'
+__version_tuple__ = version_tuple = (0, 0, 1)
+
+__commit_id__ = commit_id = None

postpone_index/admin.py

@@ -0,0 +1,29 @@
+"""
+Admin interface
+"""
+
+from django.conf import settings
+from django.contrib import admin
+from django.utils.translation import gettext_lazy as _
+
+from postpone_index.models import PostponedSQL
+
+
+class PostponedSQLAdminMixin:
+    list_display = ('d', 'description', 'table', 'db_index')
+    list_display_links = ('d', 'description')
+    search_fields = ('description', 'table', 'db_index')
+    list_filter = (
+        'table',
+        'done'
+    )
+
+    def d(self, obj):
+        return '✅' if obj.done else '🚫' if obj.error else '➡'
+    d.short_description = _('Status')
+
+
+if not getattr(settings, 'POSTPONE_INDEX_IGNORE', False) and not getattr(settings, 'POSTPONE_INDEX_ADMIN_IGNORE', False):
+    @admin.register(PostponedSQL)
+    class PostponedSQLAdmin(PostponedSQLAdminMixin, admin.ModelAdmin):
+        pass

postpone_index/apps.py

@@ -0,0 +1,8 @@
+"""App config"""
+from django.apps import AppConfig
+from django.utils.translation import gettext_lazy as _
+
+
+class Config(AppConfig):
+    verbose_name = _('Postpone Index')
+    name = 'postpone_index'

postpone_index/migration_utils.py

@@ -0,0 +1,20 @@
+"""Utility for special migration cases"""
+
+
+class PostponeIndexIgnoreMigrationMixin:
+    """Migration class mixin to avoid postpone index"""
+
+    def apply(self, project_state, schema_editor, *av, **kw):
+        """Override to avoid postpone index"""
+
+        schema_editor._postpone_index_ignore = True
+        ret = super().apply(project_state, schema_editor, *av, **kw)
+        schema_editor._postpone_index_ignore = False
+        return ret
+
+    def unapply(self, project_state, schema_editor, *av, **kw):
+        """Override to avoid postpone index"""
+        schema_editor._postpone_index_ignore = True
+        ret = super().unapply(project_state, schema_editor, *av, **kw)
+        schema_editor._postpone_index_ignore = False
+        return ret

postpone_index/models.py

@@ -0,0 +1,61 @@
+import time
+
+from unlimited_char.fields import CharField
+
+from django.db import models
+from django.utils.translation import gettext_lazy as _
+
+
+class PostponedSQL(models.Model):
+    ts = models.BigIntegerField(
+        primary_key=True, editable=False,
+        default=time.time_ns,
+        verbose_name=_('Timestamp ns'),
+        help_text=_('Timestamp in nanoseconds')
+    )
+    description = CharField(
+        verbose_name=_('Description'),
+        help_text=_('Free-form short description')
+    )
+    sql = models.TextField(
+        verbose_name=_('SQL'),
+        help_text=_('Original SQL generated on the migration stage'),
+    )
+    table = CharField(
+        blank=True, null=True, db_index=True,
+        verbose_name=_('Table'),
+        help_text=_('Table name if applied to the table')
+    )
+    db_index = CharField(
+        blank=True, null=True, db_index=True,
+        verbose_name=_('Index'),
+        help_text=_('Index name if applied to the index')
+    )
+    fields = CharField(
+        blank=True, null=True,
+        verbose_name=_('Fields'),
+        help_text=_('Field names separated by comma if applied to the fields')
+    )
+    done = models.BooleanField(
+        default=False,
+        verbose_name=_('Done'),
+        help_text=_('Has the command been applied')
+    )
+    error = models.TextField(
+        blank=True, null=True,
+        verbose_name=_('Error'),
+        help_text=_('Last error reported when tried to apply')
+    )
+
+    @property
+    def d(self):
+        """Status mark"""
+        return '[X]' if self.done else '[E]' if self.error else '[ ]'
+
+    class Meta:
+        managed = False
+        verbose_name = _('Postponed SQL')
+        verbose_name_plural = _('Postponed SQLs')
+
+    def __str__(self):
+        return self.description

postpone_index/testing_utils.py

@@ -0,0 +1,37 @@
+"""Testing Utilities"""
+
+from django.conf import settings
+from django.test import TransactionTestCase
+
+from postpone_index.models import PostponedSQL
+
+
+class TestCase(TransactionTestCase):
+    """Base module testcase class"""
+
+    @staticmethod
+    def _check_postponed_sql_empty(alias='default'):
+        try:
+            first = PostponedSQL.objects.using(alias).all().first()
+            if not first:
+                # No record
+                return True
+        except Exception:
+            # No table
+            return True
+        return False
+
+    @classmethod
+    def _assert_postponed_sql_empty(cls, message=None, alias='default'):
+        cls.assertTrue(cls._check_postponed_sql_empty(alias=alias), message or 'Postponed SQL list not empty')
+
+    @classmethod
+    def setUpClass(cls):
+        cls.assertTrue(settings.POSTPONE_INDEX_IGNORE, 'Tests should be started with POSTPONE_INDEX_IGNORE=True')
+
+    @classmethod
+    def tearDownClass(cls):
+        if not cls._check_postponed_sql_empty(alias='default'):
+            PostponedSQL.objects.using('default').all().delete()
+        if not cls._check_postponed_sql_empty(alias='additional'):
+            PostponedSQL.objects.using('additional').all().delete()

postpone_index/utils.py

@@ -0,0 +1,111 @@
+"""Some utilities"""
+
+import re
+from collections.abc import MappingView
+
+
+class ObjMap(MappingView):
+    """Gives access to any attribute using map protocol, combining with another map for defaults"""
+
+    def __init__(self, obj, defaults=None):
+        """Initialize by the original object"""
+        self._obj = obj
+        self._defaults = defaults or {}
+
+    def __getitem__(self, k):
+        """Override to get access"""
+        try:
+            return getattr(self._obj, k)
+        except AttributeError:
+            return self._defaults[k]
+
+
+class Utils:
+    """Utilities class"""
+    _create_index_re = re.compile(
+        r'^\s*CREATE\s+(?P<unique>UNIQUE\s+)?INDEX\s+(IF\s+NOT\s+EXISTS\s+)?'
+        r'(((?P<iq>")?(?P<index_nameq>[^"]+)(?P=iq))|(?P<index_name>[^\s]+))'
+        r'\s+ON\s+(?P<only>ONLY\s+)?'
+        r'(((?P<tq>")?(?P<table_nameq>[^"]+)(?P=tq))|(?P<table_name>[^\s]+))'
+        r'(?P<rest>.*)$',
+        re.IGNORECASE | re.MULTILINE
+    )
+    _drop_index_re = re.compile(
+        r'^\s*DROP\s+INDEX\s+(IF\s+EXISTS\s+)?'
+        r'(((?P<iq>")?(?P<index_nameq>[^"]+)(?P=iq))|(?P<index_name>[^\s]+))'
+        r'(?P<rest>.*)$',
+        re.IGNORECASE | re.MULTILINE
+    )
+    _drop_table_re = re.compile(
+        r'^\s*DROP\s+TABLE\s+(IF\s+EXISTS\s+)?'
+        r'(((?P<tq>")?(?P<table_nameq>[^"]+)(?P=tq))|(?P<table_name>[^\s]+))'
+        r'(?P<rest>.*)$',
+        re.IGNORECASE | re.MULTILINE
+    )
+    _add_constraint_re = re.compile(
+        r'^\s*ALTER\s+TABLE\s+'
+        r'(((?P<tq>")?(?P<table_nameq>[^"]+)(?P=tq))|(?P<table_name>[^\s]+))'
+        r'\s+ADD\s+CONSTRAINT\s+'
+        r'(((?P<iq>")?(?P<index_nameq>[^"]+)(?P=iq))|(?P<index_name>[^\s]+))'
+        r'\s+UNIQUE\s+'
+        r'(?P<rest>.*)$',
+        re.IGNORECASE | re.MULTILINE
+    )
+    _drop_constraint_re = re.compile(
+        r'^\s*ALTER\s+TABLE\s+'
+        r'(((?P<tq>")?(?P<table_nameq>[^"]+)(?P=tq))|(?P<table_name>[^\s]+))'
+        r'\s+DROP\s+CONSTRAINT\s+'
+        r'(((?P<iq>")?(?P<index_nameq>[^"]+)(?P=iq))|(?P<index_name>[^\s]+))'
+        r'(?P<rest>.*)$',
+        re.IGNORECASE | re.MULTILINE
+    )
+    _drop_column_re = re.compile(
+        r'^\s*ALTER\s+TABLE\s+'
+        r'(((?P<tq>")?(?P<table_nameq>[^"]+)(?P=tq))|(?P<table_name>[^\s]+))'
+        r'\s+DROP\s+COLUMN\s+'
+        r'(((?P<cq>")?(?P<column_nameq>[^"]+)(?P=cq))|(?P<column_name>[^\s]+))'
+        r'(?P<rest>.*)$',
+        re.IGNORECASE | re.MULTILINE
+    )
+    _column_name_re = re.compile(
+        r'(((?P<cq>")(?P<column_nameq>[^"]+)(?P=cq))|(?P<column_name>[^\s]+))',
+        re.IGNORECASE | re.MULTILINE
+    )
+    _enclosed_re = re.compile(
+        r'\((?P<enclosed>[^\(\)]*)\)',
+        re.IGNORECASE | re.MULTILINE
+    )
+
+    @classmethod
+    def _extract_column_names(cls, rest):
+        """Extracts column names from complex index suffix"""
+        rest = cls._extract_enclosed(rest)
+        while match := cls._enclosed_re.search(rest):
+            rest = rest[:match.start()] + rest[match.end():]
+        ret = []
+        for column in [c for c in rest.split(',') if c.strip()]:
+            match = cls._column_name_re.search(column)
+            if not match:
+                continue
+            ret.append('"%s"' % (match.group('column_nameq') or match.group('column_name')))
+        return ret
+
+    @classmethod
+    def _extract_enclosed(cls, rest):
+        """Extract first enclosed statement with recursive parentheses"""
+        deep = 0
+        start = 0
+        end = 0
+        for i, c in enumerate(rest):
+            if c == '(':
+                if deep == 0:
+                    start = i + 1
+                deep += 1
+            elif c == ')':
+                deep -= 1
+                if deep == 0:
+                    end = i
+                    break
+        if end == 0:
+            return rest
+        return rest[start:end]