aws-python-helper 1.0.2__tar.gz → 1.1.0__tar.gz

{aws_python_helper-1.0.2/aws_python_helper.egg-info → aws_python_helper-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aws-python-helper
-Version: 1.0.2
+Version: 1.1.0
 Summary: AWS Python Helper Framework
 Author-email: Fabian Claros <neufabiae@gmail.com>
 License-Expression: MIT
@@ -17,6 +17,8 @@ Requires-Dist: motor==3.3.2
 Requires-Dist: pymongo==4.6.1
 Requires-Dist: pydantic>=2.0
 Requires-Dist: httpx>=0.27.0
+Provides-Extra: jwt
+Requires-Dist: PyJWT[crypto]>=2.9.0; extra == "jwt"
 
 # AWS Python Framework
 
@@ -39,7 +41,7 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
 - **LambdaInvoker**: Invoke other Lambda functions (sync or async) with built-in error handling
 - **ApiClient**: HTTP client for inter-service communication — resolves service URLs and auth token from environment variables automatically
 - **ModelQueryLambda**: Base class for MongoDB query-proxy Lambdas — declare an `allowed_collections` whitelist and the framework handles `find` and `aggregate` operations generically
-- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates every declared index across the `core` database and all active state databases, idempotently, in a single run (run once per deploy)
+- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates the declared indexes **and prunes orphans** across the `core` database and all active state databases, idempotently and in parallel, in a single run (run once per deploy)
 - **JSON utilities**: Automatic serialization of MongoDB types
 - **Type hints**: Modern Python with type annotations
 - **Async/await**: Full support for asynchronous operations
@@ -68,7 +70,7 @@ All available classes and functions:
 | `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
 | `Repository` | `aws_python_helper.repository.base` | Base class for MongoDB repositories |
 | `ModelQueryLambda` | `aws_python_helper` | Generic MongoDB query-proxy Lambda base class |
-| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates indexes across `core` + all state DBs) |
+| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates declared indexes + prunes orphans across `core` + all state DBs, in parallel) |
 | `LambdaInvoker` | `aws_python_helper` | Invoke Lambda functions (sync/async) with error handling |
 | `ApiClient` | `aws_python_helper` | HTTP client for inter-service communication |
 | `LambdaInvocationError` | `aws_python_helper` | Raised when boto3 fails to invoke a Lambda |
@@ -1360,10 +1362,16 @@ Always pass `self.session.to_dict()` when calling from an API or Lambda to ensur
 
 ## 🧱 ModelIndexSyncLambda
 
-`ModelIndexSyncLambda` is a ready-to-use base class for **creating all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
+`ModelIndexSyncLambda` is a ready-to-use base class for **synchronizing all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
 
 **Why:** in short-lived Lambda/API runtimes, creating indexes lazily on first access often never finishes (the function responds and freezes). This Lambda creates every declared index up front, `await`-ing each one, idempotently — and across **every** active state database, not just the one of the current request.
 
+It does a **two-way sync** per collection: it creates the declared indexes **and prunes orphans** — any index present in MongoDB that is no longer declared in the repository's `indexes` is dropped (the default `_id_` index and special text/geo/hashed indexes are always preserved).
+
+> ⚠️ **Orphan pruning drops indexes not declared in code.** Indexes created out-of-band (by a DBA, Atlas, or another tool) that aren't in any repository's `indexes` will be removed on every run. Declare every index you want to keep.
+
+**Concurrency & fault tolerance:** each `(collection, database)` unit is synced **in parallel** (bounded by `max_concurrency`, default 10); index work *within* a unit stays sequential. A failure in one unit is **captured and isolated** — it never aborts the others, so a single broken collection doesn't stop the rest of the sync. The top-level response is always `success: true`; per-unit outcomes are reported in `results[].success`.
+
 ### Creating an index-sync Lambda
 
 Create `src/lambda/model-index-sync/main.py` and declare which repositories to sync:
@@ -1409,14 +1417,20 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 
 ### Response format
 
+The top-level `success` is **always `true`** (a failing collection no longer aborts the run). Inspect `data.summary` for the OK/failed counts and `data.results[].success` for each unit; failed units include an `error` field.
+
 ```json
 {
   "success": true,
-  "data": [
-    {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"]}
-  ]
+  "data": {
+    "summary": { "total": 4, "succeeded": 3, "failed": 1 },
+    "results": [
+      {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": ["old_field_1"], "success": true},
+      {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "TokensRepository",  "database": "core",        "collection": "tokens",  "indexes": [], "orphan_indexes_dropped": [], "success": false, "error": "..."}
+    ]
+  }
 }
 ```
 
@@ -1427,11 +1441,14 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 | `repositories` | `[]` | **Required.** List of `Repository` subclasses (the classes, not instances) to sync. Empty raises a validation error |
 | `states_database` | `"core"` | Database holding the states collection |
 | `states_collection` | `"states"` | Collection listing the states (filtered by `is_active = true`, field `name`) |
+| `max_concurrency` | `10` | Max `(collection, database)` units synced in parallel. `0` (or negative) = unlimited |
 | `requires_state` | `False` | Inherited override — the Lambda runs across all states, no session required |
 
 ### Running it once per deploy (Terraform)
 
-Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state). The `postcondition` fails the `apply` if the sync reports `success = false` (the framework handler returns HTTP 200 with `success = false` on internal errors, so this check is required to surface failures):
+Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state).
+
+> **Detecting failures in the deploy.** The top-level `success` is now **always `true`** — a single failed collection no longer aborts the run, so checking `success` alone won't catch partial failures. If you want the `apply` to fail when any collection failed, point the `postcondition` at `data.summary.failed` instead (shown below). If you'd rather the deploy never fail on index errors and only review them afterward (in the response/logs), simply omit the `lifecycle` block.
 
 ```hcl
 module "lambda" {
@@ -1449,10 +1466,13 @@ resource "aws_lambda_invocation" "sync_on_deploy" {
 
   depends_on = [module.lambda]                                # run after the code is updated
 
+  # Optional: fail the apply if any collection failed to sync.
+  # Omit this whole lifecycle block if you prefer the deploy to never fail on
+  # index errors (review data.results[].success / logs instead).
   lifecycle {
     postcondition {
-      condition     = try(jsondecode(self.result).success, false) == true
-      error_message = "Index sync failed during apply: ${self.result}"
+      condition     = try(jsondecode(self.result).data.summary.failed, 1) == 0
+      error_message = "Index sync had failures during apply: ${self.result}"
     }
   }
 }

aws_python_helper-1.0.2/PKG-INFO → aws_python_helper-1.1.0/README.md

@@ -1,23 +1,3 @@
-Metadata-Version: 2.4
-Name: aws-python-helper
-Version: 1.0.2
-Summary: AWS Python Helper Framework
-Author-email: Fabian Claros <neufabiae@gmail.com>
-License-Expression: MIT
-Project-URL: Homepage, https://github.com/fabiae/aws-python-framework
-Project-URL: Source Code, https://github.com/fabiae/aws-python-framework
-Project-URL: Bug Tracker, https://github.com/fabiae/aws-python-framework/issues
-Project-URL: Documentation, https://github.com/fabiae/aws-python-framework/blob/main/README.md
-Keywords: aws,python,framework,helper,mongodb,sqs,sns,fargate,lambda
-Classifier: Programming Language :: Python :: 3
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-Requires-Dist: motor==3.3.2
-Requires-Dist: pymongo==4.6.1
-Requires-Dist: pydantic>=2.0
-Requires-Dist: httpx>=0.27.0
-
 # AWS Python Framework
 
 Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks, and Standalone Lambdas with Python in AWS Lambda.
@@ -39,7 +19,7 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
 - **LambdaInvoker**: Invoke other Lambda functions (sync or async) with built-in error handling
 - **ApiClient**: HTTP client for inter-service communication — resolves service URLs and auth token from environment variables automatically
 - **ModelQueryLambda**: Base class for MongoDB query-proxy Lambdas — declare an `allowed_collections` whitelist and the framework handles `find` and `aggregate` operations generically
-- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates every declared index across the `core` database and all active state databases, idempotently, in a single run (run once per deploy)
+- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates the declared indexes **and prunes orphans** across the `core` database and all active state databases, idempotently and in parallel, in a single run (run once per deploy)
 - **JSON utilities**: Automatic serialization of MongoDB types
 - **Type hints**: Modern Python with type annotations
 - **Async/await**: Full support for asynchronous operations
@@ -68,7 +48,7 @@ All available classes and functions:
 | `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
 | `Repository` | `aws_python_helper.repository.base` | Base class for MongoDB repositories |
 | `ModelQueryLambda` | `aws_python_helper` | Generic MongoDB query-proxy Lambda base class |
-| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates indexes across `core` + all state DBs) |
+| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates declared indexes + prunes orphans across `core` + all state DBs, in parallel) |
 | `LambdaInvoker` | `aws_python_helper` | Invoke Lambda functions (sync/async) with error handling |
 | `ApiClient` | `aws_python_helper` | HTTP client for inter-service communication |
 | `LambdaInvocationError` | `aws_python_helper` | Raised when boto3 fails to invoke a Lambda |
@@ -1360,10 +1340,16 @@ Always pass `self.session.to_dict()` when calling from an API or Lambda to ensur
 
 ## 🧱 ModelIndexSyncLambda
 
-`ModelIndexSyncLambda` is a ready-to-use base class for **creating all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
+`ModelIndexSyncLambda` is a ready-to-use base class for **synchronizing all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
 
 **Why:** in short-lived Lambda/API runtimes, creating indexes lazily on first access often never finishes (the function responds and freezes). This Lambda creates every declared index up front, `await`-ing each one, idempotently — and across **every** active state database, not just the one of the current request.
 
+It does a **two-way sync** per collection: it creates the declared indexes **and prunes orphans** — any index present in MongoDB that is no longer declared in the repository's `indexes` is dropped (the default `_id_` index and special text/geo/hashed indexes are always preserved).
+
+> ⚠️ **Orphan pruning drops indexes not declared in code.** Indexes created out-of-band (by a DBA, Atlas, or another tool) that aren't in any repository's `indexes` will be removed on every run. Declare every index you want to keep.
+
+**Concurrency & fault tolerance:** each `(collection, database)` unit is synced **in parallel** (bounded by `max_concurrency`, default 10); index work *within* a unit stays sequential. A failure in one unit is **captured and isolated** — it never aborts the others, so a single broken collection doesn't stop the rest of the sync. The top-level response is always `success: true`; per-unit outcomes are reported in `results[].success`.
+
 ### Creating an index-sync Lambda
 
 Create `src/lambda/model-index-sync/main.py` and declare which repositories to sync:
@@ -1409,14 +1395,20 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 
 ### Response format
 
+The top-level `success` is **always `true`** (a failing collection no longer aborts the run). Inspect `data.summary` for the OK/failed counts and `data.results[].success` for each unit; failed units include an `error` field.
+
 ```json
 {
   "success": true,
-  "data": [
-    {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"]}
-  ]
+  "data": {
+    "summary": { "total": 4, "succeeded": 3, "failed": 1 },
+    "results": [
+      {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": ["old_field_1"], "success": true},
+      {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "TokensRepository",  "database": "core",        "collection": "tokens",  "indexes": [], "orphan_indexes_dropped": [], "success": false, "error": "..."}
+    ]
+  }
 }
 ```
 
@@ -1427,11 +1419,14 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 | `repositories` | `[]` | **Required.** List of `Repository` subclasses (the classes, not instances) to sync. Empty raises a validation error |
 | `states_database` | `"core"` | Database holding the states collection |
 | `states_collection` | `"states"` | Collection listing the states (filtered by `is_active = true`, field `name`) |
+| `max_concurrency` | `10` | Max `(collection, database)` units synced in parallel. `0` (or negative) = unlimited |
 | `requires_state` | `False` | Inherited override — the Lambda runs across all states, no session required |
 
 ### Running it once per deploy (Terraform)
 
-Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state). The `postcondition` fails the `apply` if the sync reports `success = false` (the framework handler returns HTTP 200 with `success = false` on internal errors, so this check is required to surface failures):
+Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state).
+
+> **Detecting failures in the deploy.** The top-level `success` is now **always `true`** — a single failed collection no longer aborts the run, so checking `success` alone won't catch partial failures. If you want the `apply` to fail when any collection failed, point the `postcondition` at `data.summary.failed` instead (shown below). If you'd rather the deploy never fail on index errors and only review them afterward (in the response/logs), simply omit the `lifecycle` block.
 
 ```hcl
 module "lambda" {
@@ -1449,10 +1444,13 @@ resource "aws_lambda_invocation" "sync_on_deploy" {
 
   depends_on = [module.lambda]                                # run after the code is updated
 
+  # Optional: fail the apply if any collection failed to sync.
+  # Omit this whole lifecycle block if you prefer the deploy to never fail on
+  # index errors (review data.results[].success / logs instead).
   lifecycle {
     postcondition {
-      condition     = try(jsondecode(self.result).success, false) == true
-      error_message = "Index sync failed during apply: ${self.result}"
+      condition     = try(jsondecode(self.result).data.summary.failed, 1) == 0
+      error_message = "Index sync had failures during apply: ${self.result}"
     }
   }
 }

{aws_python_helper-1.0.2 → aws_python_helper-1.1.0}/aws_python_helper/api/auth_validators.py

@@ -141,3 +141,112 @@ class TokenValidator(AuthValidator):
             'token_data': token_doc,
             'is_bypass': False
         }
+
+
+class JWTValidator(AuthValidator):
+    """
+    Validates RS256 JWTs issued by constitution-core.
+
+    Stateless: the signature and the claims are enough, so no database is
+    touched. That is what lets any microservice authenticate a request without
+    calling the issuer.
+
+    Environment:
+        JWT_PUBLIC_KEY: RSA public key in PEM, raw or base64-encoded.
+        JWT_ISSUER: expected `iss`. Defaults to 'constitution-core'.
+        JWT_AUDIENCE: expected `aud`. Only verified when set.
+        AUTH_BYPASS_TOKEN: still honoured, same as TokenValidator.
+    """
+
+    _public_key_cache: Optional[str] = None
+
+    @classmethod
+    def _public_key(cls) -> str:
+        """The configured public key, decoded once per container."""
+        if cls._public_key_cache:
+            return cls._public_key_cache
+
+        raw = os.getenv('JWT_PUBLIC_KEY')
+        if not raw:
+            raise ValueError(
+                "JWT_PUBLIC_KEY environment variable not set. "
+                "Required when AUTH_STRATEGY=jwt."
+            )
+
+        key = raw.strip()
+        if not key.startswith('-----BEGIN'):
+            # PEMs are multi-line, so they travel base64-encoded in env vars.
+            import base64
+            key = base64.b64decode(key).decode('utf-8')
+
+        cls._public_key_cache = key
+        return key
+
+    async def validate_token(self, token: str) -> Dict[str, Any]:
+        bypass_token = os.getenv('AUTH_BYPASS_TOKEN')
+        if bypass_token and token == bypass_token:
+            logger.info("Bypass token used - skipping JWT validation")
+            return {
+                'user_id': 'bypass',
+                'user': {
+                    'email': 'bypass@system',
+                    'role': 'admin',
+                    'name': 'Bypass User',
+                    '_id': 'bypass'
+                },
+                'is_bypass': True,
+                'token_data': None
+            }
+
+        try:
+            import jwt
+        except ImportError as exc:
+            raise RuntimeError(
+                "PyJWT is required for AUTH_STRATEGY=jwt. Install aws-python-helper[jwt]."
+            ) from exc
+
+        audience = os.getenv('JWT_AUDIENCE')
+        try:
+            claims = jwt.decode(
+                token,
+                self._public_key(),
+                algorithms=['RS256'],
+                issuer=os.getenv('JWT_ISSUER', 'constitution-core'),
+                audience=audience,
+                options={'verify_aud': bool(audience)},
+            )
+        except jwt.ExpiredSignatureError:
+            logger.warning("JWT expired")
+            raise UnauthorizedError("Token has expired")
+        except jwt.InvalidTokenError as exc:
+            logger.warning("JWT rejected: %s", exc)
+            raise UnauthorizedError("Invalid token")
+
+        if not claims.get('sub'):
+            raise UnauthorizedError("Invalid token")
+
+        # Same shape TokenValidator returns, so nothing downstream changes.
+        return {
+            'user_id': str(claims['sub']),
+            'user': {
+                '_id': claims['sub'],
+                'email': claims.get('email'),
+                'name': claims.get('name', ''),
+                'role': claims.get('role', 'user'),
+                **(claims.get('extra') or {}),
+            },
+            'token_data': claims,
+            'is_bypass': False,
+        }
+
+
+def get_auth_validator() -> AuthValidator:
+    """The validator this service is configured to use.
+
+    AUTH_STRATEGY=jwt switches to stateless validation. Anything else keeps the
+    database lookup, so a service only migrates when its environment says so.
+    """
+    strategy = (os.getenv('AUTH_STRATEGY') or 'db').strip().lower()
+    if strategy == 'jwt':
+        return JWTValidator()
+    return TokenValidator()

{aws_python_helper-1.0.2 → aws_python_helper-1.1.0}/aws_python_helper/api/dispatcher.py

@@ -11,7 +11,7 @@ from .fetcher import Fetcher
 from .base import API
 from .exceptions import UnauthorizedError, ForbiddenError, AuthenticationError
 from .auth_middleware import AuthMiddleware
-from .auth_validators import TokenValidator
+from .auth_validators import get_auth_validator
 from ..context.session import get_session
 from ..context.state_validator import StateValidator, InvalidStateError
 
@@ -240,8 +240,8 @@ class Dispatcher:
         """
         Execute authentication middleware
         
-        Uses TokenValidator to validate tokens against AUTH_BYPASS_TOKEN
-        or MongoDB database.
+        The validator is chosen by AUTH_STRATEGY: stateless JWT, or the
+        database lookup used so far.
         
         Args:
             api: API instance to inject authentication data into
@@ -249,8 +249,8 @@ class Dispatcher:
         Raises:
             UnauthorizedError: If authentication fails
         """
-        # Use unified token validator
-        validator = TokenValidator()
+        # Strategy comes from configuration, per service
+        validator = get_auth_validator()
         
         # Create middleware and authenticate
         middleware = AuthMiddleware(validator)

{aws_python_helper-1.0.2 → aws_python_helper-1.1.0}/aws_python_helper/api/fetcher.py

@@ -91,7 +91,7 @@ class Fetcher:
     
     def _looks_like_id(self, part: str) -> bool:
         """
-        Check if a URL part looks like an ID (numeric or UUID format)
+        Check if a URL part looks like an ID (numeric, ObjectId or UUID format)
         
         Args:
             part: URL part to check
@@ -105,6 +105,15 @@ class Fetcher:
         # Check if it's numeric
         if part.isdigit():
             return True
+
+        # ObjectId de MongoDB: 24 caracteres hexadecimales. Sin esto, un
+        # PATCH /users/<oid> busca un directorio con el id como nombre.
+        if len(part) == 24:
+            try:
+                int(part, 16)
+                return True
+            except ValueError:
+                pass
         
         # Check if it's a UUID format (8-4-4-4-12 hex digits)
         if len(part) == 36 and part.count('-') == 4:

aws_python_helper-1.1.0/aws_python_helper/model_index_sync/base.py

@@ -0,0 +1,191 @@
+"""
+ModelIndexSyncLambda - Synchronizes MongoDB indexes for a service's repositories.
+
+Creates (idempotently) every index declared in the service's repositories, in
+the correct databases:
+  - Repositories with a fixed database_key (e.g. 'core') → that database.
+  - State-scoped repositories (database_key is None) → every active state
+    database, read from core.states (is_active=True). A state-scoped collection
+    (e.g. 'parcels') therefore gets its indexes created in 'connecticut',
+    'new_jersey', etc.
+
+It also prunes orphan indexes: any index present in MongoDB but no longer
+declared in a repository's `indexes` is dropped (see Repository.prune_orphan_indexes).
+
+Designed to run once per deploy (and on demand), guaranteeing index creation
+without the per-request overhead of creating indexes at runtime.
+
+Each (collection, database) unit is synced concurrently (bounded by
+`max_concurrency`); index creation WITHIN a unit stays sequential. A failure in
+one unit is captured and never aborts the others, so a single broken collection
+does not stop the rest of the sync — the failure is reported instead.
+
+Usage — subclass and declare the repositories to sync:
+
+    import aws_python_helper
+    from repositories.parcels import ParcelsRepository
+    from repositories.users import UsersRepository
+
+    class PropertiesModelIndexSyncLambda(aws_python_helper.ModelIndexSyncLambda):
+        @property
+        def repositories(self) -> list:
+            return [ParcelsRepository, UsersRepository]
+
+This Lambda does not require a session state (requires_state = False); it
+operates across all active states.
+
+Response (via Lambda base run()) — always success=True at the top level; per-unit
+outcome is reported in results[].success:
+    {"success": True, "data": {
+        "summary": {"total": 3, "succeeded": 2, "failed": 1},
+        "results": [
+            {"repository": "ParcelsRepository", "database": "connecticut",
+             "collection": "parcels", "indexes": ["parcel_id_1", ...],
+             "orphan_indexes_dropped": ["old_field_1"], "success": True},
+            {"repository": "UsersRepository", "database": "core",
+             "collection": "users", "indexes": ["email_1"],
+             "orphan_indexes_dropped": [], "success": True},
+            {"repository": "TokensRepository", "database": "core",
+             "collection": "tokens", "indexes": [], "orphan_indexes_dropped": [],
+             "success": False, "error": "..."},
+        ]
+    }}
+"""
+
+import asyncio
+from typing import Any, List, Type
+
+from ..lambda_standalone.base import Lambda
+from ..repository.base import Repository
+from ..database.mongo_manager import MongoManager
+
+
+class ModelIndexSyncLambda(Lambda):
+    """
+    Base class for index-synchronization Lambdas.
+
+    Subclass and override `repositories` to declare which repositories to sync.
+    Everything else (resolving core vs state-scoped, reading active states and
+    creating the indexes) is handled automatically.
+    """
+
+    @property
+    def requires_state(self) -> bool:
+        # Operates across all states; not tied to a single one.
+        return False
+
+    @property
+    def repositories(self) -> List[Type[Repository]]:
+        """
+        Repositories whose indexes should be synchronized.
+
+        Override returning a list of Repository subclasses (the classes, not
+        instances). Declare them explicitly rather than auto-discovering, so the
+        synced set is intentional and nothing is silently missed or added.
+        """
+        return []
+
+    @property
+    def states_database(self) -> str:
+        """Database holding the states collection. Default: 'core'."""
+        return "core"
+
+    @property
+    def states_collection(self) -> str:
+        """Collection listing the states. Default: 'states'."""
+        return "states"
+
+    @property
+    def max_concurrency(self) -> int:
+        """
+        Max number of (collection, database) index syncs running in parallel.
+        0 (or negative) means unlimited. Default: 10.
+        """
+        return 10
+
+    async def validate(self):
+        if not self.repositories:
+            raise ValueError(
+                "repositories is not configured on this ModelIndexSyncLambda"
+            )
+
+    async def _active_states(self) -> List[str]:
+        """Names of active states — i.e. the state-scoped database names."""
+        db = MongoManager.get_database(self.states_database)
+        cursor = db[self.states_collection].find(
+            {"is_active": True}, {"name": 1, "_id": 0}
+        )
+        return [doc["name"] async for doc in cursor if doc.get("name")]
+
+    async def _sync_target(self, repo_cls: Type[Repository], db_name: str) -> dict:
+        """
+        Sync one collection in one database: create declared indexes and drop
+        orphan ones.
+
+        Captures its own errors so a single failure never aborts the rest of the
+        sync — a failed unit returns success=False with the error message, and
+        any indexes created before the failure are still reported.
+        """
+        result = {
+            "repository": repo_cls.__name__,
+            "database": db_name,
+            "collection": None,
+            "indexes": [],
+            "orphan_indexes_dropped": [],
+            "success": True,
+        }
+        try:
+            repo = repo_cls()
+            result["collection"] = repo.collection_name
+            result["indexes"] = await repo.ensure_indexes(database_name=db_name)
+            result["orphan_indexes_dropped"] = await repo.prune_orphan_indexes(
+                database_name=db_name, dry_run=False
+            )
+        except Exception as e:
+            self.logger.error(
+                f"Index sync failed for {repo_cls.__name__} on '{db_name}': {e}"
+            )
+            result["success"] = False
+            result["error"] = str(e)
+        return result
+
+    async def process(self) -> Any:
+        active_states = await self._active_states()
+
+        # Plan all independent (repo_cls, db_name) work units.
+        targets: List[tuple] = []
+        for repo_cls in self.repositories:
+            repo = repo_cls()
+            # database_key is None -> state-scoped (collection repeats per state db).
+            is_state_scoped = repo.database_key is None and not repo.is_external
+            target_databases = active_states if is_state_scoped else [repo.database_name]
+            for db_name in target_databases:
+                targets.append((repo_cls, db_name))
+
+        # Each (collection, database) runs concurrently; index creation WITHIN
+        # each one stays sequential (via await). _sync_target captures its own
+        # errors, so one failed unit never aborts the others.
+        limit = self.max_concurrency
+        if limit and limit > 0:
+            semaphore = asyncio.Semaphore(limit)
+
+            async def run(repo_cls, db_name):
+                async with semaphore:
+                    return await self._sync_target(repo_cls, db_name)
+        else:
+            async def run(repo_cls, db_name):
+                return await self._sync_target(repo_cls, db_name)
+
+        results = list(await asyncio.gather(
+            *(run(repo_cls, db_name) for repo_cls, db_name in targets)
+        ))
+
+        failed = [r for r in results if not r["success"]]
+        return {
+            "summary": {
+                "total": len(results),
+                "succeeded": len(results) - len(failed),
+                "failed": len(failed),
+            },
+            "results": results,
+        }

{aws_python_helper-1.0.2 → aws_python_helper-1.1.0}/aws_python_helper/repository/base.py

@@ -255,3 +255,71 @@ class Repository(ABC):
 
         self._indexes_created[key] = True
         return created
+
+    async def prune_orphan_indexes(
+        self, database_name: Optional[str] = None, dry_run: bool = True
+    ) -> List[str]:
+        """
+        Detects and (optionally) drops indexes that exist in MongoDB but are NOT
+        declared in the `indexes` property — i.e. orphan indexes left behind
+        after an index was removed from the code.
+
+        Matching is done by index key spec (fields + direction), never by name,
+        since index names are auto-generated by MongoDB. The default '_id_' index
+        is always preserved, and special indexes (text/geo/hashed, whose key spec
+        MongoDB rewrites internally) are skipped to avoid false positives.
+
+        Args:
+            database_name: Target database. Defaults to the resolved
+                database_name (database_key or current session state).
+            dry_run: If True (default), only reports orphans without dropping.
+                If False, drops each orphan index.
+
+        Returns:
+            List of orphan index names (dropped when dry_run=False).
+        """
+        db_name = database_name or self.database_name
+
+        if self.is_external:
+            if not self.cluster_name:
+                raise ValueError(
+                    f"{self.__class__.__name__}: 'cluster_name' is required when is_external=True"
+                )
+            db = ExternalMongoManager.get_database(self.cluster_name, db_name)
+        else:
+            db = MongoManager.get_database(db_name)
+        collection = db[self.collection_name]
+
+        declared = {
+            tuple(tuple(field) for field in index_def["key"])
+            for index_def in self.indexes
+            if index_def.get("key")
+        }
+
+        orphans: List[str] = []
+        async for existing in collection.list_indexes():
+            name = existing.get("name")
+            if name == "_id_":  # the default _id index can never be dropped
+                continue
+
+            key_spec = list(existing.get("key", {}).items())
+            # Skip special indexes (text/geo/hashed): MongoDB rewrites their key
+            # spec ("text"/2dsphere/"hashed", _fts/_ftsx), so a key-based
+            # comparison would wrongly flag them as orphans.
+            if any(isinstance(direction, str) for _, direction in key_spec) \
+                    or any(field in ("_fts", "_ftsx") for field, _ in key_spec):
+                continue
+
+            if tuple(key_spec) not in declared:
+                orphans.append(name)
+                if not dry_run:
+                    await collection.drop_index(name)
+                    self.logger.info(
+                        f"Dropped orphan index '{name}' on '{db_name}.{self.collection_name}'"
+                    )
+                else:
+                    self.logger.info(
+                        f"[dry-run] orphan index '{name}' on '{db_name}.{self.collection_name}'"
+                    )
+
+        return orphans

aws_python_helper-1.0.2/README.md → aws_python_helper-1.1.0/aws_python_helper.egg-info/PKG-INFO

@@ -1,3 +1,25 @@
+Metadata-Version: 2.4
+Name: aws-python-helper
+Version: 1.1.0
+Summary: AWS Python Helper Framework
+Author-email: Fabian Claros <neufabiae@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/fabiae/aws-python-framework
+Project-URL: Source Code, https://github.com/fabiae/aws-python-framework
+Project-URL: Bug Tracker, https://github.com/fabiae/aws-python-framework/issues
+Project-URL: Documentation, https://github.com/fabiae/aws-python-framework/blob/main/README.md
+Keywords: aws,python,framework,helper,mongodb,sqs,sns,fargate,lambda
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: motor==3.3.2
+Requires-Dist: pymongo==4.6.1
+Requires-Dist: pydantic>=2.0
+Requires-Dist: httpx>=0.27.0
+Provides-Extra: jwt
+Requires-Dist: PyJWT[crypto]>=2.9.0; extra == "jwt"
+
 # AWS Python Framework
 
 Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks, and Standalone Lambdas with Python in AWS Lambda.
@@ -19,7 +41,7 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
 - **LambdaInvoker**: Invoke other Lambda functions (sync or async) with built-in error handling
 - **ApiClient**: HTTP client for inter-service communication — resolves service URLs and auth token from environment variables automatically
 - **ModelQueryLambda**: Base class for MongoDB query-proxy Lambdas — declare an `allowed_collections` whitelist and the framework handles `find` and `aggregate` operations generically
-- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates every declared index across the `core` database and all active state databases, idempotently, in a single run (run once per deploy)
+- **ModelIndexSyncLambda**: Base class for index-synchronization Lambdas — declare your `repositories` and the framework creates the declared indexes **and prunes orphans** across the `core` database and all active state databases, idempotently and in parallel, in a single run (run once per deploy)
 - **JSON utilities**: Automatic serialization of MongoDB types
 - **Type hints**: Modern Python with type annotations
 - **Async/await**: Full support for asynchronous operations
@@ -48,7 +70,7 @@ All available classes and functions:
 | `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
 | `Repository` | `aws_python_helper.repository.base` | Base class for MongoDB repositories |
 | `ModelQueryLambda` | `aws_python_helper` | Generic MongoDB query-proxy Lambda base class |
-| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates indexes across `core` + all state DBs) |
+| `ModelIndexSyncLambda` | `aws_python_helper` | Index-sync Lambda base class (creates declared indexes + prunes orphans across `core` + all state DBs, in parallel) |
 | `LambdaInvoker` | `aws_python_helper` | Invoke Lambda functions (sync/async) with error handling |
 | `ApiClient` | `aws_python_helper` | HTTP client for inter-service communication |
 | `LambdaInvocationError` | `aws_python_helper` | Raised when boto3 fails to invoke a Lambda |
@@ -1340,10 +1362,16 @@ Always pass `self.session.to_dict()` when calling from an API or Lambda to ensur
 
 ## 🧱 ModelIndexSyncLambda
 
-`ModelIndexSyncLambda` is a ready-to-use base class for **creating all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
+`ModelIndexSyncLambda` is a ready-to-use base class for **synchronizing all of a service's MongoDB indexes deterministically**, in one run. Because index creation is no longer triggered at request time (see the [Repository Pattern](#️-repository-pattern) note), this Lambda is the single source of truth for materializing indexes — typically invoked **once per deploy**.
 
 **Why:** in short-lived Lambda/API runtimes, creating indexes lazily on first access often never finishes (the function responds and freezes). This Lambda creates every declared index up front, `await`-ing each one, idempotently — and across **every** active state database, not just the one of the current request.
 
+It does a **two-way sync** per collection: it creates the declared indexes **and prunes orphans** — any index present in MongoDB that is no longer declared in the repository's `indexes` is dropped (the default `_id_` index and special text/geo/hashed indexes are always preserved).
+
+> ⚠️ **Orphan pruning drops indexes not declared in code.** Indexes created out-of-band (by a DBA, Atlas, or another tool) that aren't in any repository's `indexes` will be removed on every run. Declare every index you want to keep.
+
+**Concurrency & fault tolerance:** each `(collection, database)` unit is synced **in parallel** (bounded by `max_concurrency`, default 10); index work *within* a unit stays sequential. A failure in one unit is **captured and isolated** — it never aborts the others, so a single broken collection doesn't stop the rest of the sync. The top-level response is always `success: true`; per-unit outcomes are reported in `results[].success`.
+
 ### Creating an index-sync Lambda
 
 Create `src/lambda/model-index-sync/main.py` and declare which repositories to sync:
@@ -1389,14 +1417,20 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 
 ### Response format
 
+The top-level `success` is **always `true`** (a failing collection no longer aborts the run). Inspect `data.summary` for the OK/failed counts and `data.results[].success` for each unit; failed units include an `error` field.
+
 ```json
 {
   "success": true,
-  "data": [
-    {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."]},
-    {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"]}
-  ]
+  "data": {
+    "summary": { "total": 4, "succeeded": 3, "failed": 1 },
+    "results": [
+      {"repository": "ParcelsRepository", "database": "connecticut", "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": ["old_field_1"], "success": true},
+      {"repository": "ParcelsRepository", "database": "new_jersey",  "collection": "parcels", "indexes": ["parcel_id_1", "..."], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "UsersRepository",   "database": "core",        "collection": "users",   "indexes": ["email_1"], "orphan_indexes_dropped": [], "success": true},
+      {"repository": "TokensRepository",  "database": "core",        "collection": "tokens",  "indexes": [], "orphan_indexes_dropped": [], "success": false, "error": "..."}
+    ]
+  }
 }
 ```
 
@@ -1407,11 +1441,14 @@ This Lambda sets `requires_state = False`: it is not tied to a single state and
 | `repositories` | `[]` | **Required.** List of `Repository` subclasses (the classes, not instances) to sync. Empty raises a validation error |
 | `states_database` | `"core"` | Database holding the states collection |
 | `states_collection` | `"states"` | Collection listing the states (filtered by `is_active = true`, field `name`) |
+| `max_concurrency` | `10` | Max `(collection, database)` units synced in parallel. `0` (or negative) = unlimited |
 | `requires_state` | `False` | Inherited override — the Lambda runs across all states, no session required |
 
 ### Running it once per deploy (Terraform)
 
-Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state). The `postcondition` fails the `apply` if the sync reports `success = false` (the framework handler returns HTTP 200 with `success = false` on internal errors, so this check is required to surface failures):
+Since deploys are applied manually with Terraform, attach an `aws_lambda_invocation` that runs **on every `apply`**, right after the function code is updated. `timestamp()` changes on every plan, so it always re-runs (creation is idempotent and also picks up any newly-activated state).
+
+> **Detecting failures in the deploy.** The top-level `success` is now **always `true`** — a single failed collection no longer aborts the run, so checking `success` alone won't catch partial failures. If you want the `apply` to fail when any collection failed, point the `postcondition` at `data.summary.failed` instead (shown below). If you'd rather the deploy never fail on index errors and only review them afterward (in the response/logs), simply omit the `lifecycle` block.
 
 ```hcl
 module "lambda" {
@@ -1429,10 +1466,13 @@ resource "aws_lambda_invocation" "sync_on_deploy" {
 
   depends_on = [module.lambda]                                # run after the code is updated
 
+  # Optional: fail the apply if any collection failed to sync.
+  # Omit this whole lifecycle block if you prefer the deploy to never fail on
+  # index errors (review data.results[].success / logs instead).
   lifecycle {
     postcondition {
-      condition     = try(jsondecode(self.result).success, false) == true
-      error_message = "Index sync failed during apply: ${self.result}"
+      condition     = try(jsondecode(self.result).data.summary.failed, 1) == 0
+      error_message = "Index sync had failures during apply: ${self.result}"
     }
   }
 }

{aws_python_helper-1.0.2 → aws_python_helper-1.1.0}/aws_python_helper.egg-info/requires.txt

@@ -2,3 +2,6 @@ motor==3.3.2
 pymongo==4.6.1
 pydantic>=2.0
 httpx>=0.27.0
+
+[jwt]
+PyJWT[crypto]>=2.9.0

{aws_python_helper-1.0.2 → aws_python_helper-1.1.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aws-python-helper"
-version = "1.0.2"
+version = "1.1.0"
 description = "AWS Python Helper Framework"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -31,6 +31,10 @@ keywords = [
     "aws", "python", "framework", "helper", "mongodb", "sqs", "sns", "fargate", "lambda"
 ]
 
+# Only needed by services running AUTH_STRATEGY=jwt.
+[project.optional-dependencies]
+jwt = ["PyJWT[crypto]>=2.9.0"]
+
 [project.urls]
 "Homepage" = "https://github.com/fabiae/aws-python-framework"
 "Source Code" = "https://github.com/fabiae/aws-python-framework"

aws_python_helper-1.0.2/aws_python_helper/model_index_sync/base.py

@@ -1,114 +0,0 @@
-"""
-ModelIndexSyncLambda - Synchronizes MongoDB indexes for a service's repositories.
-
-Creates (idempotently) every index declared in the service's repositories, in
-the correct databases:
-  - Repositories with a fixed database_key (e.g. 'core') → that database.
-  - State-scoped repositories (database_key is None) → every active state
-    database, read from core.states (is_active=True). A state-scoped collection
-    (e.g. 'parcels') therefore gets its indexes created in 'connecticut',
-    'new_jersey', etc.
-
-Designed to run once per deploy (and on demand), guaranteeing index creation
-without the per-request overhead of creating indexes at runtime.
-
-Usage — subclass and declare the repositories to sync:
-
-    import aws_python_helper
-    from repositories.parcels import ParcelsRepository
-    from repositories.users import UsersRepository
-
-    class PropertiesModelIndexSyncLambda(aws_python_helper.ModelIndexSyncLambda):
-        @property
-        def repositories(self) -> list:
-            return [ParcelsRepository, UsersRepository]
-
-This Lambda does not require a session state (requires_state = False); it
-operates across all active states.
-
-Response (via Lambda base run()):
-    {"success": True, "data": [
-        {"repository": "ParcelsRepository", "database": "connecticut",
-         "collection": "parcels", "indexes": ["parcel_id_1", ...]},
-        {"repository": "UsersRepository", "database": "core",
-         "collection": "users", "indexes": ["email_1"]},
-        ...
-    ]}
-"""
-
-from typing import Any, List, Type
-
-from ..lambda_standalone.base import Lambda
-from ..repository.base import Repository
-from ..database.mongo_manager import MongoManager
-
-
-class ModelIndexSyncLambda(Lambda):
-    """
-    Base class for index-synchronization Lambdas.
-
-    Subclass and override `repositories` to declare which repositories to sync.
-    Everything else (resolving core vs state-scoped, reading active states and
-    creating the indexes) is handled automatically.
-    """
-
-    @property
-    def requires_state(self) -> bool:
-        # Operates across all states; not tied to a single one.
-        return False
-
-    @property
-    def repositories(self) -> List[Type[Repository]]:
-        """
-        Repositories whose indexes should be synchronized.
-
-        Override returning a list of Repository subclasses (the classes, not
-        instances). Declare them explicitly rather than auto-discovering, so the
-        synced set is intentional and nothing is silently missed or added.
-        """
-        return []
-
-    @property
-    def states_database(self) -> str:
-        """Database holding the states collection. Default: 'core'."""
-        return "core"
-
-    @property
-    def states_collection(self) -> str:
-        """Collection listing the states. Default: 'states'."""
-        return "states"
-
-    async def validate(self):
-        if not self.repositories:
-            raise ValueError(
-                "repositories is not configured on this ModelIndexSyncLambda"
-            )
-
-    async def _active_states(self) -> List[str]:
-        """Names of active states — i.e. the state-scoped database names."""
-        db = MongoManager.get_database(self.states_database)
-        cursor = db[self.states_collection].find(
-            {"is_active": True}, {"name": 1, "_id": 0}
-        )
-        return [doc["name"] async for doc in cursor if doc.get("name")]
-
-    async def process(self) -> Any:
-        active_states = await self._active_states()
-        report: List[dict] = []
-
-        for repo_cls in self.repositories:
-            repo = repo_cls()
-            # database_key is None -> state-scoped (collection repeats per state db).
-            is_state_scoped = repo.database_key is None and not repo.is_external
-            target_databases = active_states if is_state_scoped else [repo.database_name]
-
-            for db_name in target_databases:
-                created = await repo.ensure_indexes(database_name=db_name)
-                report.append({
-                    "repository": repo_cls.__name__,
-                    "database": db_name,
-                    "collection": repo.collection_name,
-                    "indexes": created,
-                })
-
-        return report