aws-python-helper 1.0.2__tar.gz → 1.0.3__tar.gz

{aws_python_helper-1.0.2 → aws_python_helper-1.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aws-python-helper
-Version: 1.0.2
+Version: 1.0.3
 Summary: AWS Python Helper Framework
 Author-email: Fabian Claros <neufabiae@gmail.com>
 License-Expression: MIT
@@ -39,7 +39,7 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
 - **LambdaInvoker**: Invoke other Lambda functions (sync or async) with built-in error handling
 - **ApiClient**: HTTP client for inter-service communication — resolves service URLs and auth token from environment variables automatically
 - **ModelQueryLambda**: Base class for MongoDB query-proxy Lambdas — declare an `allowed_collections` whitelist and the framework handles `find` and `aggregate` operations generically
-- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates every declared index across the `core` database and all active state databases, idempotently, in a single run (run once per deploy)
+- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates the declared indexes **and prunes orphans** across the `core` database and all active state databases, idempotently and in parallel, in a single run (run once per deploy)
 - **JSON utilities**: Automatic serialization of MongoDB types
 - **Type hints**: Modern Python with type annotations
 - **Async/await**: Full support for asynchronous operations
@@ -68,7 +68,7 @@ All available classes and functions:
 | `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
 | `Repository` | `aws_python_helper.repository.base` | Base class for MongoDB repositories |
 | `ModelQueryLambda` | `aws_python_helper` | Generic MongoDB query-proxy Lambda base class |
-| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates indexes across `core` + all state DBs) |
+| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates declared indexes + prunes orphans across `core` + all state DBs, in parallel) |
 | `LambdaInvoker` | `aws_python_helper` | Invoke Lambda functions (sync/async) with error handling |
 | `ApiClient` | `aws_python_helper` | HTTP client for inter-service communication |
 | `LambdaInvocationError` | `aws_python_helper` | Raised when boto3 fails to invoke a Lambda |
@@ -1360,10 +1360,16 @@ Always pass `self.session.to_dict()` when calling from an API or Lambda to ensur
 
 ## 🧱 ModelIndexSyncLambda
 
-`ModelIndexSyncLambda` is a ready-to-use base class for **creating all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
+`ModelIndexSyncLambda` is a ready-to-use base class for **synchronizing all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
 
 **Why:** in short-lived Lambda/API runtimes, creating indexes lazily on first access often never finishes (the function responds and freezes). This Lambda creates every declared index up front, `await`-ing each one, idempotently — and across **every** active state database, not just the one of the current request.
 
+It does a **two-way sync** per collection: it creates the declared indexes **and prunes orphans** — any index present in MongoDB that is no longer declared in the repository's `indexes` is dropped (the default `_id_` index and special text/geo/hashed indexes are always preserved).
+
+> ⚠️ **Orphan pruning drops indexes not declared in code.** Indexes created out-of-band (by a DBA, Atlas, or another tool) that aren't in any repository's `indexes` will be removed on every run. Declare every index you want to keep.
+
+**Concurrency & fault tolerance:** each `(collection, database)` unit is synced **in parallel** (bounded by `max_concurrency`, default 10); index work *within* a unit stays sequential. A failure in one unit is **captured and isolated** — it never aborts the others, so a single broken collection doesn't stop the rest of the sync. The top-level response is always `success: true`; per-unit outcomes are reported in `results[].success`.
+
 ### Creating an index-sync Lambda
 
 Create `src/lambda/model-index-sync/main.py` and declare which repositories to sync:
@@ -1409,14 +1415,20 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 
 ### Response format
 
+The top-level `success` is **always `true`** (a failing collection no longer aborts the run). Inspect `data.summary` for the OK/failed counts and `data.results[].success` for each unit; failed units include an `error` field.
+
 ```json
 {
   "success": true,
-  "data": [
-    {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"]}
-  ]
+  "data": {
+    "summary": { "total": 4, "succeeded": 3, "failed": 1 },
+    "results": [
+      {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": ["old_field_1"], "success": true},
+      {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "TokensRepository",  "database": "core",        "collection": "tokens",  "indexes": [], "orphan_indexes_dropped": [], "success": false, "error": "..."}
+    ]
+  }
 }
 ```
 
@@ -1427,11 +1439,14 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 | `repositories` | `[]` | **Required.** List of `Repository` subclasses (the classes, not instances) to sync. Empty raises a validation error |
 | `states_database` | `"core"` | Database holding the states collection |
 | `states_collection` | `"states"` | Collection listing the states (filtered by `is_active = true`, field `name`) |
+| `max_concurrency` | `10` | Max `(collection, database)` units synced in parallel. `0` (or negative) = unlimited |
 | `requires_state` | `False` | Inherited override — the Lambda runs across all states, no session required |
 
 ### Running it once per deploy (Terraform)
 
-Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state). The `postcondition` fails the `apply` if the sync reports `success = false` (the framework handler returns HTTP 200 with `success = false` on internal errors, so this check is required to surface failures):
+Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state).
+
+> **Detecting failures in the deploy.** The top-level `success` is now **always `true`** — a single failed collection no longer aborts the run, so checking `success` alone won't catch partial failures. If you want the `apply` to fail when any collection failed, point the `postcondition` at `data.summary.failed` instead (shown below). If you'd rather the deploy never fail on index errors and only review them afterward (in the response/logs), simply omit the `lifecycle` block.
 
 ```hcl
 module "lambda" {
@@ -1449,10 +1464,13 @@ resource "aws_lambda_invocation" "sync_on_deploy" {
 
   depends_on = [module.lambda]                                # run after the code is updated
 
+  # Optional: fail the apply if any collection failed to sync.
+  # Omit this whole lifecycle block if you prefer the deploy to never fail on
+  # index errors (review data.results[].success / logs instead).
   lifecycle {
     postcondition {
-      condition     = try(jsondecode(self.result).success, false) == true
-      error_message = "Index sync failed during apply: ${self.result}"
+      condition     = try(jsondecode(self.result).data.summary.failed, 1) == 0
+      error_message = "Index sync had failures during apply: ${self.result}"
     }
   }
 }

{aws_python_helper-1.0.2 → aws_python_helper-1.0.3}/README.md

@@ -19,7 +19,7 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
 - **LambdaInvoker**: Invoke other Lambda functions (sync or async) with built-in error handling
 - **ApiClient**: HTTP client for inter-service communication — resolves service URLs and auth token from environment variables automatically
 - **ModelQueryLambda**: Base class for MongoDB query-proxy Lambdas — declare an `allowed_collections` whitelist and the framework handles `find` and `aggregate` operations generically
-- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates every declared index across the `core` database and all active state databases, idempotently, in a single run (run once per deploy)
+- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates the declared indexes **and prunes orphans** across the `core` database and all active state databases, idempotently and in parallel, in a single run (run once per deploy)
 - **JSON utilities**: Automatic serialization of MongoDB types
 - **Type hints**: Modern Python with type annotations
 - **Async/await**: Full support for asynchronous operations
@@ -48,7 +48,7 @@ All available classes and functions:
 | `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
 | `Repository` | `aws_python_helper.repository.base` | Base class for MongoDB repositories |
 | `ModelQueryLambda` | `aws_python_helper` | Generic MongoDB query-proxy Lambda base class |
-| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates indexes across `core` + all state DBs) |
+| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates declared indexes + prunes orphans across `core` + all state DBs, in parallel) |
 | `LambdaInvoker` | `aws_python_helper` | Invoke Lambda functions (sync/async) with error handling |
 | `ApiClient` | `aws_python_helper` | HTTP client for inter-service communication |
 | `LambdaInvocationError` | `aws_python_helper` | Raised when boto3 fails to invoke a Lambda |
@@ -1340,10 +1340,16 @@ Always pass `self.session.to_dict()` when calling from an API or Lambda to ensur
 
 ## 🧱 ModelIndexSyncLambda
 
-`ModelIndexSyncLambda` is a ready-to-use base class for **creating all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
+`ModelIndexSyncLambda` is a ready-to-use base class for **synchronizing all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
 
 **Why:** in short-lived Lambda/API runtimes, creating indexes lazily on first access often never finishes (the function responds and freezes). This Lambda creates every declared index up front, `await`-ing each one, idempotently — and across **every** active state database, not just the one of the current request.
 
+It does a **two-way sync** per collection: it creates the declared indexes **and prunes orphans** — any index present in MongoDB that is no longer declared in the repository's `indexes` is dropped (the default `_id_` index and special text/geo/hashed indexes are always preserved).
+
+> ⚠️ **Orphan pruning drops indexes not declared in code.** Indexes created out-of-band (by a DBA, Atlas, or another tool) that aren't in any repository's `indexes` will be removed on every run. Declare every index you want to keep.
+
+**Concurrency & fault tolerance:** each `(collection, database)` unit is synced **in parallel** (bounded by `max_concurrency`, default 10); index work *within* a unit stays sequential. A failure in one unit is **captured and isolated** — it never aborts the others, so a single broken collection doesn't stop the rest of the sync. The top-level response is always `success: true`; per-unit outcomes are reported in `results[].success`.
+
 ### Creating an index-sync Lambda
 
 Create `src/lambda/model-index-sync/main.py` and declare which repositories to sync:
@@ -1389,14 +1395,20 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 
 ### Response format
 
+The top-level `success` is **always `true`** (a failing collection no longer aborts the run). Inspect `data.summary` for the OK/failed counts and `data.results[].success` for each unit; failed units include an `error` field.
+
 ```json
 {
   "success": true,
-  "data": [
-    {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"]}
-  ]
+  "data": {
+    "summary": { "total": 4, "succeeded": 3, "failed": 1 },
+    "results": [
+      {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": ["old_field_1"], "success": true},
+      {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "TokensRepository",  "database": "core",        "collection": "tokens",  "indexes": [], "orphan_indexes_dropped": [], "success": false, "error": "..."}
+    ]
+  }
 }
 ```
 
@@ -1407,11 +1419,14 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 | `repositories` | `[]` | **Required.** List of `Repository` subclasses (the classes, not instances) to sync. Empty raises a validation error |
 | `states_database` | `"core"` | Database holding the states collection |
 | `states_collection` | `"states"` | Collection listing the states (filtered by `is_active = true`, field `name`) |
+| `max_concurrency` | `10` | Max `(collection, database)` units synced in parallel. `0` (or negative) = unlimited |
 | `requires_state` | `False` | Inherited override — the Lambda runs across all states, no session required |
 
 ### Running it once per deploy (Terraform)
 
-Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state). The `postcondition` fails the `apply` if the sync reports `success = false` (the framework handler returns HTTP 200 with `success = false` on internal errors, so this check is required to surface failures):
+Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state).
+
+> **Detecting failures in the deploy.** The top-level `success` is now **always `true`** — a single failed collection no longer aborts the run, so checking `success` alone won't catch partial failures. If you want the `apply` to fail when any collection failed, point the `postcondition` at `data.summary.failed` instead (shown below). If you'd rather the deploy never fail on index errors and only review them afterward (in the response/logs), simply omit the `lifecycle` block.
 
 ```hcl
 module "lambda" {
@@ -1429,10 +1444,13 @@ resource "aws_lambda_invocation" "sync_on_deploy" {
 
   depends_on = [module.lambda]                                # run after the code is updated
 
+  # Optional: fail the apply if any collection failed to sync.
+  # Omit this whole lifecycle block if you prefer the deploy to never fail on
+  # index errors (review data.results[].success / logs instead).
   lifecycle {
     postcondition {
-      condition     = try(jsondecode(self.result).success, false) == true
-      error_message = "Index sync failed during apply: ${self.result}"
+      condition     = try(jsondecode(self.result).data.summary.failed, 1) == 0
+      error_message = "Index sync had failures during apply: ${self.result}"
     }
   }
 }

aws_python_helper-1.0.3/aws_python_helper/model_index_sync/base.py

@@ -0,0 +1,191 @@
+"""
+ModelIndexSyncLambda - Synchronizes MongoDB indexes for a service's repositories.
+
+Creates (idempotently) every index declared in the service's repositories, in
+the correct databases:
+  - Repositories with a fixed database_key (e.g. 'core') → that database.
+  - State-scoped repositories (database_key is None) → every active state
+    database, read from core.states (is_active=True). A state-scoped collection
+    (e.g. 'parcels') therefore gets its indexes created in 'connecticut',
+    'new_jersey', etc.
+
+It also prunes orphan indexes: any index present in MongoDB but no longer
+declared in a repository's `indexes` is dropped (see Repository.prune_orphan_indexes).
+
+Designed to run once per deploy (and on demand), guaranteeing index creation
+without the per-request overhead of creating indexes at runtime.
+
+Each (collection, database) unit is synced concurrently (bounded by
+`max_concurrency`); index creation WITHIN a unit stays sequential. A failure in
+one unit is captured and never aborts the others, so a single broken collection
+does not stop the rest of the sync — the failure is reported instead.
+
+Usage — subclass and declare the repositories to sync:
+
+    import aws_python_helper
+    from repositories.parcels import ParcelsRepository
+    from repositories.users import UsersRepository
+
+    class PropertiesModelIndexSyncLambda(aws_python_helper.ModelIndexSyncLambda):
+        @property
+        def repositories(self) -> list:
+            return [ParcelsRepository, UsersRepository]
+
+This Lambda does not require a session state (requires_state = False); it
+operates across all active states.
+
+Response (via Lambda base run()) — always success=True at the top level; per-unit
+outcome is reported in results[].success:
+    {"success": True, "data": {
+        "summary": {"total": 3, "succeeded": 2, "failed": 1},
+        "results": [
+            {"repository": "ParcelsRepository", "database": "connecticut",
+             "collection": "parcels", "indexes": ["parcel_id_1", ...],
+             "orphan_indexes_dropped": ["old_field_1"], "success": True},
+            {"repository": "UsersRepository", "database": "core",
+             "collection": "users", "indexes": ["email_1"],
+             "orphan_indexes_dropped": [], "success": True},
+            {"repository": "TokensRepository", "database": "core",
+             "collection": "tokens", "indexes": [], "orphan_indexes_dropped": [],
+             "success": False, "error": "..."},
+        ]
+    }}
+"""
+
+import asyncio
+from typing import Any, List, Type
+
+from ..lambda_standalone.base import Lambda
+from ..repository.base import Repository
+from ..database.mongo_manager import MongoManager
+
+
+class ModelIndexSyncLambda(Lambda):
+    """
+    Base class for index-synchronization Lambdas.
+
+    Subclass and override `repositories` to declare which repositories to sync.
+    Everything else (resolving core vs state-scoped, reading active states and
+    creating the indexes) is handled automatically.
+    """
+
+    @property
+    def requires_state(self) -> bool:
+        # Operates across all states; not tied to a single one.
+        return False
+
+    @property
+    def repositories(self) -> List[Type[Repository]]:
+        """
+        Repositories whose indexes should be synchronized.
+
+        Override returning a list of Repository subclasses (the classes, not
+        instances). Declare them explicitly rather than auto-discovering, so the
+        synced set is intentional and nothing is silently missed or added.
+        """
+        return []
+
+    @property
+    def states_database(self) -> str:
+        """Database holding the states collection. Default: 'core'."""
+        return "core"
+
+    @property
+    def states_collection(self) -> str:
+        """Collection listing the states. Default: 'states'."""
+        return "states"
+
+    @property
+    def max_concurrency(self) -> int:
+        """
+        Max number of (collection, database) index syncs running in parallel.
+        0 (or negative) means unlimited. Default: 10.
+        """
+        return 10
+
+    async def validate(self):
+        if not self.repositories:
+            raise ValueError(
+                "repositories is not configured on this ModelIndexSyncLambda"
+            )
+
+    async def _active_states(self) -> List[str]:
+        """Names of active states — i.e. the state-scoped database names."""
+        db = MongoManager.get_database(self.states_database)
+        cursor = db[self.states_collection].find(
+            {"is_active": True}, {"name": 1, "_id": 0}
+        )
+        return [doc["name"] async for doc in cursor if doc.get("name")]
+
+    async def _sync_target(self, repo_cls: Type[Repository], db_name: str) -> dict:
+        """
+        Sync one collection in one database: create declared indexes and drop
+        orphan ones.
+
+        Captures its own errors so a single failure never aborts the rest of the
+        sync — a failed unit returns success=False with the error message, and
+        any indexes created before the failure are still reported.
+        """
+        result = {
+            "repository": repo_cls.__name__,
+            "database": db_name,
+            "collection": None,
+            "indexes": [],
+            "orphan_indexes_dropped": [],
+            "success": True,
+        }
+        try:
+            repo = repo_cls()
+            result["collection"] = repo.collection_name
+            result["indexes"] = await repo.ensure_indexes(database_name=db_name)
+            result["orphan_indexes_dropped"] = await repo.prune_orphan_indexes(
+                database_name=db_name, dry_run=False
+            )
+        except Exception as e:
+            self.logger.error(
+                f"Index sync failed for {repo_cls.__name__} on '{db_name}': {e}"
+            )
+            result["success"] = False
+            result["error"] = str(e)
+        return result
+
+    async def process(self) -> Any:
+        active_states = await self._active_states()
+
+        # Plan all independent (repo_cls, db_name) work units.
+        targets: List[tuple] = []
+        for repo_cls in self.repositories:
+            repo = repo_cls()
+            # database_key is None -> state-scoped (collection repeats per state db).
+            is_state_scoped = repo.database_key is None and not repo.is_external
+            target_databases = active_states if is_state_scoped else [repo.database_name]
+            for db_name in target_databases:
+                targets.append((repo_cls, db_name))
+
+        # Each (collection, database) runs concurrently; index creation WITHIN
+        # each one stays sequential (via await). _sync_target captures its own
+        # errors, so one failed unit never aborts the others.
+        limit = self.max_concurrency
+        if limit and limit > 0:
+            semaphore = asyncio.Semaphore(limit)
+
+            async def run(repo_cls, db_name):
+                async with semaphore:
+                    return await self._sync_target(repo_cls, db_name)
+        else:
+            async def run(repo_cls, db_name):
+                return await self._sync_target(repo_cls, db_name)
+
+        results = list(await asyncio.gather(
+            *(run(repo_cls, db_name) for repo_cls, db_name in targets)
+        ))
+
+        failed = [r for r in results if not r["success"]]
+        return {
+            "summary": {
+                "total": len(results),
+                "succeeded": len(results) - len(failed),
+                "failed": len(failed),
+            },
+            "results": results,
+        }

{aws_python_helper-1.0.2 → aws_python_helper-1.0.3}/aws_python_helper/repository/base.py

@@ -255,3 +255,71 @@ class Repository(ABC):
 
         self._indexes_created[key] = True
         return created
+
+    async def prune_orphan_indexes(
+        self, database_name: Optional[str] = None, dry_run: bool = True
+    ) -> List[str]:
+        """
+        Detects and (optionally) drops indexes that exist in MongoDB but are NOT
+        declared in the `indexes` property — i.e. orphan indexes left behind
+        after an index was removed from the code.
+
+        Matching is done by index key spec (fields + direction), never by name,
+        since index names are auto-generated by MongoDB. The default '_id_' index
+        is always preserved, and special indexes (text/geo/hashed, whose key spec
+        MongoDB rewrites internally) are skipped to avoid false positives.
+
+        Args:
+            database_name: Target database. Defaults to the resolved
+                database_name (database_key or current session state).
+            dry_run: If True (default), only reports orphans without dropping.
+                If False, drops each orphan index.
+
+        Returns:
+            List of orphan index names (dropped when dry_run=False).
+        """
+        db_name = database_name or self.database_name
+
+        if self.is_external:
+            if not self.cluster_name:
+                raise ValueError(
+                    f"{self.__class__.__name__}: 'cluster_name' is required when is_external=True"
+                )
+            db = ExternalMongoManager.get_database(self.cluster_name, db_name)
+        else:
+            db = MongoManager.get_database(db_name)
+        collection = db[self.collection_name]
+
+        declared = {
+            tuple(tuple(field) for field in index_def["key"])
+            for index_def in self.indexes
+            if index_def.get("key")
+        }
+
+        orphans: List[str] = []
+        async for existing in collection.list_indexes():
+            name = existing.get("name")
+            if name == "_id_":  # the default _id index can never be dropped
+                continue
+
+            key_spec = list(existing.get("key", {}).items())
+            # Skip special indexes (text/geo/hashed): MongoDB rewrites their key
+            # spec ("text"/2dsphere/"hashed", _fts/_ftsx), so a key-based
+            # comparison would wrongly flag them as orphans.
+            if any(isinstance(direction, str) for _, direction in key_spec) \
+                    or any(field in ("_fts", "_ftsx") for field, _ in key_spec):
+                continue
+
+            if tuple(key_spec) not in declared:
+                orphans.append(name)
+                if not dry_run:
+                    await collection.drop_index(name)
+                    self.logger.info(
+                        f"Dropped orphan index '{name}' on '{db_name}.{self.collection_name}'"
+                    )
+                else:
+                    self.logger.info(
+                        f"[dry-run] orphan index '{name}' on '{db_name}.{self.collection_name}'"
+                    )
+
+        return orphans

{aws_python_helper-1.0.2 → aws_python_helper-1.0.3}/aws_python_helper.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aws-python-helper
-Version: 1.0.2
+Version: 1.0.3
 Summary: AWS Python Helper Framework
 Author-email: Fabian Claros <neufabiae@gmail.com>
 License-Expression: MIT
@@ -39,7 +39,7 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
 - **LambdaInvoker**: Invoke other Lambda functions (sync or async) with built-in error handling
 - **ApiClient**: HTTP client for inter-service communication — resolves service URLs and auth token from environment variables automatically
 - **ModelQueryLambda**: Base class for MongoDB query-proxy Lambdas — declare an `allowed_collections` whitelist and the framework handles `find` and `aggregate` operations generically
-- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates every declared index across the `core` database and all active state databases, idempotently, in a single run (run once per deploy)
+- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates the declared indexes **and prunes orphans** across the `core` database and all active state databases, idempotently and in parallel, in a single run (run once per deploy)
 - **JSON utilities**: Automatic serialization of MongoDB types
 - **Type hints**: Modern Python with type annotations
 - **Async/await**: Full support for asynchronous operations
@@ -68,7 +68,7 @@ All available classes and functions:
 | `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
 | `Repository` | `aws_python_helper.repository.base` | Base class for MongoDB repositories |
 | `ModelQueryLambda` | `aws_python_helper` | Generic MongoDB query-proxy Lambda base class |
-| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates indexes across `core` + all state DBs) |
+| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates declared indexes + prunes orphans across `core` + all state DBs, in parallel) |
 | `LambdaInvoker` | `aws_python_helper` | Invoke Lambda functions (sync/async) with error handling |
 | `ApiClient` | `aws_python_helper` | HTTP client for inter-service communication |
 | `LambdaInvocationError` | `aws_python_helper` | Raised when boto3 fails to invoke a Lambda |
@@ -1360,10 +1360,16 @@ Always pass `self.session.to_dict()` when calling from an API or Lambda to ensur
 
 ## 🧱 ModelIndexSyncLambda
 
-`ModelIndexSyncLambda` is a ready-to-use base class for **creating all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
+`ModelIndexSyncLambda` is a ready-to-use base class for **synchronizing all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
 
 **Why:** in short-lived Lambda/API runtimes, creating indexes lazily on first access often never finishes (the function responds and freezes). This Lambda creates every declared index up front, `await`-ing each one, idempotently — and across **every** active state database, not just the one of the current request.
 
+It does a **two-way sync** per collection: it creates the declared indexes **and prunes orphans** — any index present in MongoDB that is no longer declared in the repository's `indexes` is dropped (the default `_id_` index and special text/geo/hashed indexes are always preserved).
+
+> ⚠️ **Orphan pruning drops indexes not declared in code.** Indexes created out-of-band (by a DBA, Atlas, or another tool) that aren't in any repository's `indexes` will be removed on every run. Declare every index you want to keep.
+
+**Concurrency & fault tolerance:** each `(collection, database)` unit is synced **in parallel** (bounded by `max_concurrency`, default 10); index work *within* a unit stays sequential. A failure in one unit is **captured and isolated** — it never aborts the others, so a single broken collection doesn't stop the rest of the sync. The top-level response is always `success: true`; per-unit outcomes are reported in `results[].success`.
+
 ### Creating an index-sync Lambda
 
 Create `src/lambda/model-index-sync/main.py` and declare which repositories to sync:
@@ -1409,14 +1415,20 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 
 ### Response format
 
+The top-level `success` is **always `true`** (a failing collection no longer aborts the run). Inspect `data.summary` for the OK/failed counts and `data.results[].success` for each unit; failed units include an `error` field.
+
 ```json
 {
   "success": true,
-  "data": [
-    {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"]}
-  ]
+  "data": {
+    "summary": { "total": 4, "succeeded": 3, "failed": 1 },
+    "results": [
+      {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": ["old_field_1"], "success": true},
+      {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "TokensRepository",  "database": "core",        "collection": "tokens",  "indexes": [], "orphan_indexes_dropped": [], "success": false, "error": "..."}
+    ]
+  }
 }
 ```
 
@@ -1427,11 +1439,14 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 | `repositories` | `[]` | **Required.** List of `Repository` subclasses (the classes, not instances) to sync. Empty raises a validation error |
 | `states_database` | `"core"` | Database holding the states collection |
 | `states_collection` | `"states"` | Collection listing the states (filtered by `is_active = true`, field `name`) |
+| `max_concurrency` | `10` | Max `(collection, database)` units synced in parallel. `0` (or negative) = unlimited |
 | `requires_state` | `False` | Inherited override — the Lambda runs across all states, no session required |
 
 ### Running it once per deploy (Terraform)
 
-Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state). The `postcondition` fails the `apply` if the sync reports `success = false` (the framework handler returns HTTP 200 with `success = false` on internal errors, so this check is required to surface failures):
+Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state).
+
+> **Detecting failures in the deploy.** The top-level `success` is now **always `true`** — a single failed collection no longer aborts the run, so checking `success` alone won't catch partial failures. If you want the `apply` to fail when any collection failed, point the `postcondition` at `data.summary.failed` instead (shown below). If you'd rather the deploy never fail on index errors and only review them afterward (in the response/logs), simply omit the `lifecycle` block.
 
 ```hcl
 module "lambda" {
@@ -1449,10 +1464,13 @@ resource "aws_lambda_invocation" "sync_on_deploy" {
 
   depends_on = [module.lambda]                                # run after the code is updated
 
+  # Optional: fail the apply if any collection failed to sync.
+  # Omit this whole lifecycle block if you prefer the deploy to never fail on
+  # index errors (review data.results[].success / logs instead).
   lifecycle {
     postcondition {
-      condition     = try(jsondecode(self.result).success, false) == true
-      error_message = "Index sync failed during apply: ${self.result}"
+      condition     = try(jsondecode(self.result).data.summary.failed, 1) == 0
+      error_message = "Index sync had failures during apply: ${self.result}"
     }
   }
 }

{aws_python_helper-1.0.2 → aws_python_helper-1.0.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aws-python-helper"
-version = "1.0.2"
+version = "1.0.3"
 description = "AWS Python Helper Framework"
 readme = "README.md"
 requires-python = ">=3.9"

aws_python_helper-1.0.2/aws_python_helper/model_index_sync/base.py

@@ -1,114 +0,0 @@
-"""
-ModelIndexSyncLambda - Synchronizes MongoDB indexes for a service's repositories.
-
-Creates (idempotently) every index declared in the service's repositories, in
-the correct databases:
-  - Repositories with a fixed database_key (e.g. 'core') → that database.
-  - State-scoped repositories (database_key is None) → every active state
-    database, read from core.states (is_active=True). A state-scoped collection
-    (e.g. 'parcels') therefore gets its indexes created in 'connecticut',
-    'new_jersey', etc.
-
-Designed to run once per deploy (and on demand), guaranteeing index creation
-without the per-request overhead of creating indexes at runtime.
-
-Usage — subclass and declare the repositories to sync:
-
-    import aws_python_helper
-    from repositories.parcels import ParcelsRepository
-    from repositories.users import UsersRepository
-
-    class PropertiesModelIndexSyncLambda(aws_python_helper.ModelIndexSyncLambda):
-        @property
-        def repositories(self) -> list:
-            return [ParcelsRepository, UsersRepository]
-
-This Lambda does not require a session state (requires_state = False); it
-operates across all active states.
-
-Response (via Lambda base run()):
-    {"success": True, "data": [
-        {"repository": "ParcelsRepository", "database": "connecticut",
-         "collection": "parcels", "indexes": ["parcel_id_1", ...]},
-        {"repository": "UsersRepository", "database": "core",
-         "collection": "users", "indexes": ["email_1"]},
-        ...
-    ]}
-"""
-
-from typing import Any, List, Type
-
-from ..lambda_standalone.base import Lambda
-from ..repository.base import Repository
-from ..database.mongo_manager import MongoManager
-
-
-class ModelIndexSyncLambda(Lambda):
-    """
-    Base class for index-synchronization Lambdas.
-
-    Subclass and override `repositories` to declare which repositories to sync.
-    Everything else (resolving core vs state-scoped, reading active states and
-    creating the indexes) is handled automatically.
-    """
-
-    @property
-    def requires_state(self) -> bool:
-        # Operates across all states; not tied to a single one.
-        return False
-
-    @property
-    def repositories(self) -> List[Type[Repository]]:
-        """
-        Repositories whose indexes should be synchronized.
-
-        Override returning a list of Repository subclasses (the classes, not
-        instances). Declare them explicitly rather than auto-discovering, so the
-        synced set is intentional and nothing is silently missed or added.
-        """
-        return []
-
-    @property
-    def states_database(self) -> str:
-        """Database holding the states collection. Default: 'core'."""
-        return "core"
-
-    @property
-    def states_collection(self) -> str:
-        """Collection listing the states. Default: 'states'."""
-        return "states"
-
-    async def validate(self):
-        if not self.repositories:
-            raise ValueError(
-                "repositories is not configured on this ModelIndexSyncLambda"
-            )
-
-    async def _active_states(self) -> List[str]:
-        """Names of active states — i.e. the state-scoped database names."""
-        db = MongoManager.get_database(self.states_database)
-        cursor = db[self.states_collection].find(
-            {"is_active": True}, {"name": 1, "_id": 0}
-        )
-        return [doc["name"] async for doc in cursor if doc.get("name")]
-
-    async def process(self) -> Any:
-        active_states = await self._active_states()
-        report: List[dict] = []
-
-        for repo_cls in self.repositories:
-            repo = repo_cls()
-            # database_key is None -> state-scoped (collection repeats per state db).
-            is_state_scoped = repo.database_key is None and not repo.is_external
-            target_databases = active_states if is_state_scoped else [repo.database_name]
-
-            for db_name in target_databases:
-                created = await repo.ensure_indexes(database_name=db_name)
-                report.append({
-                    "repository": repo_cls.__name__,
-                    "database": db_name,
-                    "collection": repo.collection_name,
-                    "indexes": created,
-                })
-
-        return report