morango 0.7.0__tar.gz → 0.7.1__tar.gz

{morango-0.7.0 → morango-0.7.1}/CHANGELOG.md

@@ -2,6 +2,25 @@
 
 List of the most important changes for each release.
 
+## 0.7.1
+ - Supersedes 0.7.0
+ - Drops support for Python 3.4 and 3.5
+ - Upgrades cryptography library to 3.3.2, the latest version that supports Python 2.7
+
+## 0.6.19
+- The `cleanupsyncs` management command now only cleans up sync sessions if also inactive for `expiration` amount of time
+- Fixes issue accessing index on queryset in `cleanupsyncs` management command
+
+## 0.6.18
+- Prevent creation of Deleted and HardDeleted models during deserialization to allow propagation of syncable objects that are recreated after a previous deletion without causing a merge conflict.
+
+## 0.6.17
+- Added `client-instance-id`, `server-instance-id`, `sync-filter`, `push` and `pull` arguments to `cleanupsyncs` management command
+- Added option for resuming a sync to ignore the `SyncSession`'s existing `process_id`
+- Added custom user agent to sync HTTP requests
+- Fixed documentation build issues
+- Makefile no longer defines shell with explicit path
+
 ## 0.6.16
 - Added dedicated `client_instance_id` and `server_instance_id` fields to `SyncSession`
 - Renamed `client_instance` and `server_instance` fields on `SyncSession` to `client_instance_json` and `server_instance_json` respectively

morango-0.7.1/PKG-INFO

@@ -0,0 +1,58 @@
+Metadata-Version: 2.1
+Name: morango
+Version: 0.7.1
+Summary: Pure Python sqlite-based Django DB replication engine.
+Home-page: https://github.com/learningequality/morango
+Author: Learning Equality
+Author-email: dev@learningequality.org
+License: MIT
+Keywords: database,syncing,morango
+Platform: UNKNOWN
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 2.7
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: AUTHORS.md
+
+# Morango
+
+[![build](https://github.com/learningequality/morango/actions/workflows/tox.yml/badge.svg?branch=master)](https://github.com/learningequality/morango/actions)
+[![image](http://codecov.io/github/learningequality/morango/coverage.svg?branch=master)](http://codecov.io/github/learningequality/morango?branch=master)
+[![image](https://readthedocs.org/projects/morango/badge/?version=latest)](http://morango.readthedocs.org/en/latest/)
+
+Morango is a pure-Python database replication engine for Django that supports peer-to-peer syncing of data. It is structured as a Django app that can be included in projects to make specific application models syncable.
+
+Developed in support of the [Kolibri](https://github.com/learningequality/kolibri) product ecosystem, Morango includes some important features including:
+
+-   A certificate-based authentication system to protect privacy and integrity of data
+-   A change-tracking system to support calculation of differences between databases across low-bandwidth connections
+-   A set of constructs to support data partitioning
+
+## Developer documentation
+
+See [morango.readthedocs.io](https://morango.readthedocs.io)
+
+To build and edit the docs, run:
+
+```bash
+# install requirements
+pip install -r requirements/docs.txt
+pip install -e .
+
+# build docs
+make docs
+
+# auto-build and refresh docs on edit
+make docs-autobuild
+```
+
+

{morango-0.7.0 → morango-0.7.1}/morango/__init__.py

@@ -3,4 +3,4 @@ from __future__ import print_function
 from __future__ import unicode_literals
 
 default_app_config = "morango.apps.MorangoConfig"
-__version__ = "0.7.0"
+__version__ = "0.7.1"

morango-0.7.1/morango/management/commands/cleanupsyncs.py

@@ -0,0 +1,134 @@
+import datetime
+import logging
+import uuid
+
+from django.core.management.base import BaseCommand
+from django.db import transaction
+from django.utils import timezone
+
+from morango.models import SyncSession
+from morango.models import TransferSession
+
+
+logger = logging.getLogger(__name__)
+
+
+class Command(BaseCommand):
+    help = "Closes and cleans up the data for any incomplete sync sessions older than a certain number of hours."
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "--ids",
+            type=lambda ids: ids.split(","),
+            default=None,
+            help="Comma separated list of SyncSession IDs to filter against"
+        )
+        parser.add_argument(
+            "--expiration",
+            action="store",
+            type=int,
+            default=6,
+            help="Number of hours of inactivity after which a session should be considered stale",
+        )
+        parser.add_argument(
+            "--client-instance-id",
+            type=uuid.UUID,
+            default=None,
+            help="Filters the SyncSession models to those with matching 'client_instance_id'",
+        )
+        parser.add_argument(
+            "--server-instance-id",
+            type=uuid.UUID,
+            default=None,
+            help="Filters the SyncSession models to those with matching 'server_instance_id'",
+        )
+        parser.add_argument(
+            "--sync-filter",
+            type=str,
+            default=None,
+            help="Filters the TransferSession models to those with 'filters' starting with 'sync_filter'",
+        )
+        parser.add_argument(
+            "--push",
+            type=bool,
+            default=None,
+            help="Filters the TransferSession models to those with 'push' set to True",
+        )
+        parser.add_argument(
+            "--pull",
+            type=bool,
+            default=None,
+            help="Filters the TransferSession models to those with 'push' set to False",
+        )
+
+    def handle(self, *args, **options):
+
+        # establish the cutoff time and date for stale sessions
+        cutoff = timezone.now() - datetime.timedelta(hours=options["expiration"])
+
+        sync_sessions = SyncSession.objects.filter(active=True)
+
+        # if ids arg was passed, filter down sessions to only those IDs
+        # if included by expiration filter
+        if options["ids"]:
+            sync_sessions = sync_sessions.filter(id__in=options["ids"])
+
+        if options["client_instance_id"]:
+            sync_sessions = sync_sessions.filter(client_instance_id=options["client_instance_id"])
+
+        if options["server_instance_id"]:
+            sync_sessions = sync_sessions.filter(server_instance_id=options["server_instance_id"])
+
+        # retrieve all sessions still marked as active but with no activity since the cutoff
+        transfer_sessions = TransferSession.objects.filter(
+            sync_session_id__in=sync_sessions.values("id"),
+            active=True,
+            last_activity_timestamp__lt=cutoff,
+        )
+
+        if options["sync_filter"]:
+            transfer_sessions = transfer_sessions.filter(filter__startswith=options["sync_filter"])
+
+        if options["push"] and not options["pull"]:
+            transfer_sessions = transfer_sessions.filter(push=True)
+
+        if options["pull"] and not options["push"]:
+            transfer_sessions = transfer_sessions.filter(push=False)
+
+        transfer_count = transfer_sessions.count()
+
+        # loop over the stale sessions one by one to close them out
+        for i, transfer_session in enumerate(transfer_sessions):
+            logger.info(
+                "TransferSession {} of {}: deleting {} Buffers and {} RMC Buffers...".format(
+                    i + 1,
+                    transfer_count,
+                    transfer_session.buffer_set.all().count(),
+                    transfer_session.recordmaxcounterbuffer_set.all().count(),
+                )
+            )
+
+            # delete buffer data and mark as inactive
+            with transaction.atomic():
+                transfer_session.delete_buffers()
+                transfer_session.active = False
+                transfer_session.save()
+
+        # in order to close a sync session, it must have no active transfer sessions
+        # and must have no activity since the cutoff
+        sync_sessions = sync_sessions.filter(
+            last_activity_timestamp__lt=cutoff,
+        ).exclude(
+            transfersession__active=True,
+        )
+        sync_count = sync_sessions.count()
+
+        for i, sync_session in enumerate(sync_sessions):
+            logger.info(
+                "Closing SyncSession {} of {}".format(
+                    i + 1,
+                    sync_count,
+                )
+            )
+            sync_session.active = False
+            sync_session.save()

{morango-0.7.0 → morango-0.7.1}/morango/models/core.py

@@ -17,6 +17,7 @@ from django.db.models import F
 from django.db.models import Func
 from django.db.models import Max
 from django.db.models import Q
+from django.db.models import signals
 from django.db.models import TextField
 from django.db.models import Value
 from django.db.models.deletion import Collector
@@ -468,13 +469,13 @@ class Store(AbstractStore):
         klass_model = syncable_models.get_model(self.profile, self.model_name)
         # if store model marked as deleted, attempt to delete in app layer
         if self.deleted:
-            # if hard deleted, propagate to related models
-            if self.hard_deleted:
-                try:
-                    klass_model.objects.get(id=self.id).delete(hard_delete=True)
-                except klass_model.DoesNotExist:
-                    pass
-            else:
+            # Don't differentiate between deletion and hard deletion here,
+            # as we don't want to add additional tracking for models in either case,
+            # just to actually delete them.
+            # Import here to avoid circular import, as the utils module
+            # imports core models.
+            from morango.sync.utils import mute_signals
+            with mute_signals(signals.post_delete):
                 klass_model.objects.filter(id=self.id).delete()
             return None, deferred_fks
         else:

{morango-0.7.0 → morango-0.7.1}/morango/sync/session.py

@@ -1,11 +1,13 @@
 import logging
 
 from requests import exceptions
+from morango import __version__
 from requests.sessions import Session
 from requests.utils import super_len
 from requests.packages.urllib3.util.url import parse_url
 
 from morango.utils import serialize_capabilities_to_client_request
+from morango.utils import SETTINGS
 
 
 logger = logging.getLogger(__name__)
@@ -35,6 +37,15 @@ class SessionWrapper(Session):
     bytes_sent = 0
     bytes_received = 0
 
+    def __init__(self):
+        super(SessionWrapper, self).__init__()
+        user_agent_header = "morango/{}".format(__version__)
+        if SETTINGS.CUSTOM_INSTANCE_INFO is not None:
+            instances = list(SETTINGS.CUSTOM_INSTANCE_INFO)
+            if instances:
+                user_agent_header += " " + "{}/{}".format(instances[0], SETTINGS.CUSTOM_INSTANCE_INFO.get(instances[0]))
+        self.headers["User-Agent"] = "{} {}".format(user_agent_header, self.headers["User-Agent"])
+
     def request(self, method, url, **kwargs):
         response = None
         try:

{morango-0.7.0 → morango-0.7.1}/morango/sync/syncsession.py

@@ -260,12 +260,16 @@ class NetworkSyncConnection(Connection):
         sync_session = SyncSession.objects.create(**data)
         return SyncSessionClient(self, sync_session)
 
-    def resume_sync_session(self, sync_session_id, chunk_size=None):
+    def resume_sync_session(self, sync_session_id, chunk_size=None, ignore_existing_process=False):
         """
         Resumes an existing sync session given an ID
 
         :param sync_session_id: The UUID of the `SyncSession` to resume
         :param chunk_size: An optional parameter specifying the size for each transferred chunk
+        :type chunk_size: int
+        :param ignore_existing_process:An optional parameter specifying whether to ignore an
+            existing active process ID
+        :type ignore_existing_process: bool
         :return: A SyncSessionClient instance
         :rtype: SyncSessionClient
         """
@@ -281,7 +285,8 @@ class NetworkSyncConnection(Connection):
 
         # check that process of existing session isn't still running
         if (
-            sync_session.process_id
+            not ignore_existing_process
+            and sync_session.process_id
             and sync_session.process_id != os.getpid()
             and pid_exists(sync_session.process_id)
         ):

morango-0.7.1/morango.egg-info/PKG-INFO

@@ -0,0 +1,58 @@
+Metadata-Version: 2.1
+Name: morango
+Version: 0.7.1
+Summary: Pure Python sqlite-based Django DB replication engine.
+Home-page: https://github.com/learningequality/morango
+Author: Learning Equality
+Author-email: dev@learningequality.org
+License: MIT
+Keywords: database,syncing,morango
+Platform: UNKNOWN
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 2.7
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: AUTHORS.md
+
+# Morango
+
+[![build](https://github.com/learningequality/morango/actions/workflows/tox.yml/badge.svg?branch=master)](https://github.com/learningequality/morango/actions)
+[![image](http://codecov.io/github/learningequality/morango/coverage.svg?branch=master)](http://codecov.io/github/learningequality/morango?branch=master)
+[![image](https://readthedocs.org/projects/morango/badge/?version=latest)](http://morango.readthedocs.org/en/latest/)
+
+Morango is a pure-Python database replication engine for Django that supports peer-to-peer syncing of data. It is structured as a Django app that can be included in projects to make specific application models syncable.
+
+Developed in support of the [Kolibri](https://github.com/learningequality/kolibri) product ecosystem, Morango includes some important features including:
+
+-   A certificate-based authentication system to protect privacy and integrity of data
+-   A change-tracking system to support calculation of differences between databases across low-bandwidth connections
+-   A set of constructs to support data partitioning
+
+## Developer documentation
+
+See [morango.readthedocs.io](https://morango.readthedocs.io)
+
+To build and edit the docs, run:
+
+```bash
+# install requirements
+pip install -r requirements/docs.txt
+pip install -e .
+
+# build docs
+make docs
+
+# auto-build and refresh docs on edit
+make docs-autobuild
+```
+
+

{morango-0.7.0 → morango-0.7.1}/requirements/docs.txt

@@ -15,5 +15,3 @@ djangorestframework==3.9.1
 django-mptt==0.9.1
 rsa==3.4.2
 ifcfg==0.21
-
--r ./accelerated.txt

{morango-0.7.0 → morango-0.7.1}/setup.cfg

@@ -36,6 +36,7 @@ exclude_lines =
 	raise NotImplementedError
 	raise AssertionError
 	raise NotImplementedError
+	
 	if __name__ == .__main__.:
 
 [egg_info]

morango-0.7.0/PKG-INFO

@@ -1,54 +0,0 @@
-Metadata-Version: 2.1
-Name: morango
-Version: 0.7.0
-Summary: Pure Python sqlite-based Django DB replication engine.
-Home-page: https://github.com/learningequality/morango
-Author: Learning Equality
-Author-email: dev@learningequality.org
-License: MIT
-Description: # Morango
-        
-        [![build](https://github.com/learningequality/morango/actions/workflows/tox.yml/badge.svg?branch=master)](https://github.com/learningequality/morango/actions)
-        [![image](http://codecov.io/github/learningequality/morango/coverage.svg?branch=master)](http://codecov.io/github/learningequality/morango?branch=master)
-        [![image](https://readthedocs.org/projects/morango/badge/?version=latest)](http://morango.readthedocs.org/en/latest/)
-        
-        Morango is a pure-Python database replication engine for Django that supports peer-to-peer syncing of data. It is structured as a Django app that can be included in projects to make specific application models syncable.
-        
-        Developed in support of the [Kolibri](https://github.com/learningequality/kolibri) product ecosystem, Morango includes some important features including:
-        
-        -   A certificate-based authentication system to protect privacy and integrity of data
-        -   A change-tracking system to support calculation of differences between databases across low-bandwidth connections
-        -   A set of constructs to support data partitioning
-        
-        ## Developer documentation
-        
-        See [morango.readthedocs.io](https://morango.readthedocs.io)
-        
-        To build and edit the docs, run:
-        
-        ```bash
-        # install requirements
-        pip install -r requirements/docs.txt
-        pip install -e .
-        
-        # build docs
-        make docs
-        
-        # auto-build and refresh docs on edit
-        make docs-autobuild
-        ```
-        
-Keywords: database,syncing,morango
-Platform: UNKNOWN
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Natural Language :: English
-Classifier: Development Status :: 4 - Beta
-Classifier: Programming Language :: Python :: 2.7
-Classifier: Programming Language :: Python :: 3.6
-Classifier: Programming Language :: Python :: 3.7
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Description-Content-Type: text/markdown

morango-0.7.0/morango/management/commands/cleanupsyncs.py

@@ -1,84 +0,0 @@
-import datetime
-import logging
-
-from django.core.management.base import BaseCommand
-from django.db import transaction
-from django.utils import timezone
-
-from morango.models import SyncSession
-from morango.models import TransferSession
-
-
-logger = logging.getLogger(__name__)
-
-
-class Command(BaseCommand):
-    help = "Closes and cleans up the data for any incomplete sync sessions older than a certain number of hours."
-
-    def add_arguments(self, parser):
-        parser.add_argument(
-            "--ids",
-            type=lambda ids: ids.split(","),
-            default=None,
-            help="Comma separated list of SyncSession IDs to filter against"
-        )
-        parser.add_argument(
-            "--expiration",
-            action="store",
-            type=int,
-            default=6,
-            help="Number of hours of inactivity after which a session should be considered stale",
-        )
-
-    def handle(self, *args, **options):
-
-        # establish the cutoff time and date for stale sessions
-        cutoff = timezone.now() - datetime.timedelta(hours=options["expiration"])
-
-        sync_sessions = SyncSession.objects.filter(active=True)
-
-        # if ids arg was passed, filter down sessions to only those IDs if included by expiration filter
-        if options["ids"]:
-            sync_sessions = sync_sessions.filter(id__in=options["ids"])
-
-        # retrieve all sessions still marked as active but with no activity since the cutoff
-        transfer_sessions = TransferSession.objects.filter(
-            sync_session_id__in=sync_sessions.values("id"),
-            active=True,
-            last_activity_timestamp__lt=cutoff,
-        )
-
-        transfer_count = transfer_sessions.count()
-
-        # loop over the stale sessions one by one to close them out
-        for i in range(transfer_count):
-            transfer_session = transfer_sessions[0]
-            logger.info(
-                "TransferSession {} of {}: deleting {} Buffers and {} RMC Buffers...".format(
-                    i + 1,
-                    transfer_count,
-                    transfer_session.buffer_set.all().count(),
-                    transfer_session.recordmaxcounterbuffer_set.all().count(),
-                )
-            )
-
-            # delete buffer data and mark as inactive
-            with transaction.atomic():
-                transfer_session.delete_buffers()
-                transfer_session.active = False
-                transfer_session.save()
-
-        sync_count = sync_sessions.count()
-
-        # finally loop over sync sessions and close out if there are no other active transfer sessions
-        for i in range(sync_count):
-            sync_session = sync_sessions[0]
-            if not sync_session.transfersession_set.filter(active=True).exists():
-                logger.info(
-                    "Closing SyncSession {} of {}".format(
-                        i + 1,
-                        sync_count,
-                    )
-                )
-                sync_session.active = False
-                sync_session.save()

morango-0.7.0/morango.egg-info/PKG-INFO

@@ -1,54 +0,0 @@
-Metadata-Version: 2.1
-Name: morango
-Version: 0.7.0
-Summary: Pure Python sqlite-based Django DB replication engine.
-Home-page: https://github.com/learningequality/morango
-Author: Learning Equality
-Author-email: dev@learningequality.org
-License: MIT
-Description: # Morango
-        
-        [![build](https://github.com/learningequality/morango/actions/workflows/tox.yml/badge.svg?branch=master)](https://github.com/learningequality/morango/actions)
-        [![image](http://codecov.io/github/learningequality/morango/coverage.svg?branch=master)](http://codecov.io/github/learningequality/morango?branch=master)
-        [![image](https://readthedocs.org/projects/morango/badge/?version=latest)](http://morango.readthedocs.org/en/latest/)
-        
-        Morango is a pure-Python database replication engine for Django that supports peer-to-peer syncing of data. It is structured as a Django app that can be included in projects to make specific application models syncable.
-        
-        Developed in support of the [Kolibri](https://github.com/learningequality/kolibri) product ecosystem, Morango includes some important features including:
-        
-        -   A certificate-based authentication system to protect privacy and integrity of data
-        -   A change-tracking system to support calculation of differences between databases across low-bandwidth connections
-        -   A set of constructs to support data partitioning
-        
-        ## Developer documentation
-        
-        See [morango.readthedocs.io](https://morango.readthedocs.io)
-        
-        To build and edit the docs, run:
-        
-        ```bash
-        # install requirements
-        pip install -r requirements/docs.txt
-        pip install -e .
-        
-        # build docs
-        make docs
-        
-        # auto-build and refresh docs on edit
-        make docs-autobuild
-        ```
-        
-Keywords: database,syncing,morango
-Platform: UNKNOWN
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Natural Language :: English
-Classifier: Development Status :: 4 - Beta
-Classifier: Programming Language :: Python :: 2.7
-Classifier: Programming Language :: Python :: 3.6
-Classifier: Programming Language :: Python :: 3.7
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Description-Content-Type: text/markdown