aws-python-helper 0.30.0__tar.gz → 0.30.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/PKG-INFO +441 -94
  2. aws_python_helper-0.30.1/README.md +1039 -0
  3. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/PKG-INFO +441 -94
  4. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/pyproject.toml +1 -1
  5. aws_python_helper-0.30.0/README.md +0 -692
  6. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/__init__.py +0 -0
  7. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/__init__.py +0 -0
  8. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/auth_middleware.py +0 -0
  9. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/auth_validators.py +0 -0
  10. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/base.py +0 -0
  11. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/dispatcher.py +0 -0
  12. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/exceptions.py +0 -0
  13. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/fetcher.py +0 -0
  14. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/handler.py +0 -0
  15. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/__init__.py +0 -0
  16. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/database_proxy.py +0 -0
  17. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/external_database_proxy.py +0 -0
  18. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/external_mongo_manager.py +0 -0
  19. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/mongo_manager.py +0 -0
  20. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/__init__.py +0 -0
  21. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/executor.py +0 -0
  22. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/fetcher.py +0 -0
  23. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/handler.py +0 -0
  24. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/task_base.py +0 -0
  25. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/__init__.py +0 -0
  26. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/base.py +0 -0
  27. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/fetcher.py +0 -0
  28. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/handler.py +0 -0
  29. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sns/__init__.py +0 -0
  30. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sns/publisher.py +0 -0
  31. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/__init__.py +0 -0
  32. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/consumer_base.py +0 -0
  33. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/fetcher.py +0 -0
  34. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/handler.py +0 -0
  35. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/__init__.py +0 -0
  36. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/json_encoder.py +0 -0
  37. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/response.py +0 -0
  38. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/serializer.py +0 -0
  39. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/SOURCES.txt +0 -0
  40. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/dependency_links.txt +0 -0
  41. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/requires.txt +0 -0
  42. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/top_level.txt +0 -0
  43. {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/setup.cfg +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: aws-python-helper
3
- Version: 0.30.0
3
+ Version: 0.30.1
4
4
  Summary: AWS Python Helper Framework
5
5
  Author-email: Fabian Claros <neufabiae@gmail.com>
6
6
  License: MIT
@@ -28,23 +28,44 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
28
28
  - **Dynamic controller loading**: Routing based on convention
29
29
  - **OOP structure**: Object-oriented programming for your code
30
30
  - **Flexible MongoDB**: Direct access to multiple databases without models
31
- - **SQS Consumers**: Same pattern to process SQS messages
31
+ - **External MongoDB**: Connect to multiple MongoDB clusters simultaneously
32
+ - **SQS Consumers**: Same pattern to process SQS messages (single or batch mode)
32
33
  - **SNS Publishers**: Same pattern to publish messages to SNS topics
33
34
  - **Fargate Tasks**: Same pattern to run tasks in Fargate containers
34
35
  - **Standalone Lambdas**: Create lambdas invocable directly with AWS SDK
36
+ - **Authentication middleware**: Built-in token-based authentication
37
+ - **JSON utilities**: Automatic serialization of MongoDB types
35
38
  - **Type hints**: Modern Python with type annotations
36
39
  - **Async/await**: Full support for asynchronous operations
37
40
 
38
41
  ## 🔧 Installation
39
42
 
40
43
  ```bash
41
- # Install dependencies
42
- pip install -r requirements.txt
43
-
44
- # Configure MongoDB URI
45
- export MONGODB_URI="mongodb://localhost:27017"
44
+ pip install aws-python-helper
46
45
  ```
47
46
 
47
+ ## 📦 Quick Reference
48
+
49
+ All available classes and functions:
50
+
51
+ | Class / Function | Import | Purpose |
52
+ |------------------|--------|---------|
53
+ | `API` | `aws_python_helper.api.base` | Base class for REST endpoints |
54
+ | `api_handler` | `aws_python_helper.api.handler` | Generic handler for API Gateway |
55
+ | `SQSConsumer` | `aws_python_helper.sqs.consumer_base` | Base class for SQS consumers |
56
+ | `sqs_handler` | `aws_python_helper.sqs.handler` | Factory handler for SQS |
57
+ | `SNSPublisher` | `aws_python_helper.sns.publisher` | Base class for SNS publishers |
58
+ | `Lambda` | `aws_python_helper.lambda_standalone.base` | Base class for Standalone Lambdas |
59
+ | `lambda_handler` | `aws_python_helper.lambda_standalone.handler` | Factory handler for Lambda |
60
+ | `FargateTask` | `aws_python_helper.fargate.task_base` | Base class for Fargate tasks |
61
+ | `FargateExecutor` | `aws_python_helper.fargate.executor` | Launches Fargate tasks from Lambda |
62
+ | `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
63
+ | `MongoJSONEncoder` | `aws_python_helper.utils.json_encoder` | JSON encoder for MongoDB types |
64
+ | `mongo_json_dumps` | `aws_python_helper.utils.json_encoder` | Helper to serialize MongoDB types |
65
+ | `serialize_mongo_types` | `aws_python_helper.utils.serializer` | Recursively serialize MongoDB types |
66
+ | `UnauthorizedError` | `aws_python_helper.api.exceptions` | 401 authentication exception |
67
+ | `ForbiddenError` | `aws_python_helper.api.exceptions` | 403 authorization exception |
68
+
48
69
  ## 📂 Project Structure
49
70
 
50
71
  This framework follows a convention-based folder structure. Here's the recommended organization:
@@ -73,13 +94,16 @@ your-project/
73
94
  │ └── process-payment/ # process-payment -> ProcessPaymentLambda
74
95
  │ └── main.py
75
96
 
76
- └── task/ # Fargate Tasks (folders)
77
- ├── search-tax-by-town/ # search-tax-by-town -> SearchTaxByTownTask
78
- │ ├── main.py # Entry point
79
- │ └── task.py # Task class
80
- └── process-data/ # process-data -> ProcessDataTask
81
- ├── main.py
82
- └── task.py
97
+ ├── task/ # Fargate Tasks (folders)
98
+ ├── search-tax-by-town/ # search-tax-by-town -> SearchTaxByTownTask
99
+ ├── main.py # Entry point
100
+ └── task.py # Task class
101
+ └── process-data/ # process-data -> ProcessDataTask
102
+ ├── main.py
103
+ └── task.py
104
+
105
+ └── topic/ # SNS Publishers
106
+ └── order_created.py # OrderCreatedTopic
83
107
  ```
84
108
 
85
109
  ### Naming Conventions
@@ -137,7 +161,7 @@ from aws_python_helper.sqs.consumer_base import SQSConsumer
137
161
 
138
162
  class TitleIndexedConsumer(SQSConsumer):
139
163
  async def process_record(self, record):
140
- body = self.parse_body(record)
164
+ body = self.extract_content_message(record)
141
165
  # Your logic here
142
166
  await self.db.constitution_db.titles.insert_one(body)
143
167
  ```
@@ -174,22 +198,22 @@ class GenerateRouteLambda(Lambda):
174
198
  # Validate input data
175
199
  if 'shipping_id' not in self.data:
176
200
  raise ValueError("shipping_id is required")
177
-
201
+
178
202
  if not isinstance(self.data['shipping_id'], str):
179
203
  raise TypeError("shipping_id must be a string")
180
-
204
+
181
205
  async def process(self):
182
206
  # Your business logic here
183
207
  shipping_id = self.data['shipping_id']
184
-
208
+
185
209
  # Access to MongoDB
186
210
  shipping = await self.db.deliveries.shippings.find_one(
187
211
  {'_id': shipping_id}
188
212
  )
189
-
213
+
190
214
  if not shipping:
191
215
  raise ValueError(f"Shipping {shipping_id} not found")
192
-
216
+
193
217
  # Create route
194
218
  route = {
195
219
  'shipping_id': shipping_id,
@@ -197,11 +221,11 @@ class GenerateRouteLambda(Lambda):
197
221
  'status': 'pending',
198
222
  'created_at': datetime.utcnow()
199
223
  }
200
-
224
+
201
225
  result = await self.db.deliveries.routes.insert_one(route)
202
-
226
+
203
227
  self.logger.info(f"Route created: {result.inserted_id}")
204
-
228
+
205
229
  # Return result
206
230
  return {
207
231
  'route_id': str(result.inserted_id),
@@ -305,23 +329,47 @@ class TitleIndexedTopic(SNSPublisher):
305
329
  super().__init__(
306
330
  topic_arn=os.getenv('TITLE_INDEXED_SNS_TOPIC_ARN')
307
331
  )
308
-
309
- async def publish_message(self, constitution_id, title):
310
- await self.publish({
311
- 'constitution_id': constitution_id,
312
- 'title': title,
313
- 'event_type': 'title_indexed'
314
- })
332
+
333
+ def build_message(self, constitution_id, title, event_type='title_indexed'):
334
+ return {
335
+ 'content': {
336
+ 'constitution_id': constitution_id,
337
+ 'title': title,
338
+ 'event_type': event_type
339
+ },
340
+ 'attributes': {
341
+ 'event_type': event_type # Used for SNS subscription filtering
342
+ }
343
+ }
315
344
  ```
316
345
 
317
346
  **2. Use the topic** from anywhere:
318
347
 
319
348
  ```python
320
- from src.topics.title_indexed import TitleIndexedTopic
349
+ from src.topic.title_indexed import TitleIndexedTopic
321
350
 
322
351
  # In a consumer, API or task
323
352
  topic = TitleIndexedTopic()
324
- await topic.publish_indexed('123', 'My Constitution')
353
+
354
+ # Publish a single message
355
+ await topic.publish(topic.build_message('123', 'My Constitution'))
356
+
357
+ # Publish multiple messages in batch
358
+ messages = [
359
+ topic.build_message('id1', 'Constitution A'),
360
+ topic.build_message('id2', 'Constitution B'),
361
+ ]
362
+ await topic.publish(messages)
363
+ ```
364
+
365
+ **Message format** — every message must have a `content` key:
366
+
367
+ ```python
368
+ {
369
+ 'content': {...}, # Required: message body (any dict)
370
+ 'attributes': {...}, # Optional: SNS message attributes for filtering
371
+ 'subject': 'Optional subject' # Optional: message subject
372
+ }
325
373
  ```
326
374
 
327
375
  ### Run a Fargate Task
@@ -336,10 +384,10 @@ class SearchTaxByTownTask(FargateTask):
336
384
  async def execute(self):
337
385
  town = self.require_env('TOWN')
338
386
  self.logger.info(f"Processing town: {town}")
339
-
387
+
340
388
  # Access to DB
341
389
  docs = await self.db.smart_data.address.find({'town': town}).to_list()
342
-
390
+
343
391
  # Your logic here
344
392
  for doc in docs:
345
393
  # Process document
@@ -388,7 +436,7 @@ def handler(event, context):
388
436
  executor = FargateExecutor()
389
437
  task_arn = executor.run_task(
390
438
  'search-tax-by-town',
391
- envs={'town': 'Norwalk', 'only_tax': 'true'}
439
+ envs={'TOWN': 'Norwalk', 'ONLY_TAX': 'true'}
392
440
  )
393
441
  return {'taskArn': task_arn}
394
442
  ```
@@ -400,17 +448,47 @@ The framework provides flexible access to multiple databases:
400
448
  ```python
401
449
  class MyAPI(API):
402
450
  async def process(self):
403
- # Access to different databases
451
+ # Access to different databases on the same cluster
404
452
  user = await self.db.users_db.users.find_one({'_id': user_id})
405
-
453
+
406
454
  # Another database
407
455
  await self.db.analytics_db.logs.insert_one({'action': 'view'})
408
-
456
+
409
457
  # Multiple collections
410
458
  titles = await self.db.constitution_db.titles.find().to_list(100)
411
459
  articles = await self.db.constitution_db.articles.find().to_list(100)
412
460
  ```
413
461
 
462
+ The pattern is always: `self.db.<database_name>.<collection_name>.<motor_operation>()`
463
+
464
+ ### External MongoDB Clusters
465
+
466
+ Connect to additional MongoDB clusters using `EXTERNAL_MONGODB_CONNECTIONS`:
467
+
468
+ ```bash
469
+ EXTERNAL_MONGODB_CONNECTIONS='[
470
+ {"name": "ClusterDockets", "connection_string": "mongodb+srv://cluster.mongodb.net"},
471
+ {"name": "ClusterAnalytics", "connection_string": "mongodb+srv://analytics.mongodb.net"}
472
+ ]'
473
+ ```
474
+
475
+ The credentials from `MONGO_DB_USER` / `MONGO_DB_PASSWORD` are automatically injected into the connection strings.
476
+
477
+ Access external clusters via `self.external_db`:
478
+
479
+ ```python
480
+ class AddressAPI(API):
481
+ async def process(self):
482
+ # Access external cluster: self.external_db.<ClusterName>.<database>.<collection>
483
+ addresses = await self.external_db.ClusterDockets.smart_data.addresses.find(
484
+ {'town': self.data['town']}
485
+ ).to_list(100)
486
+
487
+ self.set_body({'addresses': addresses})
488
+ ```
489
+
490
+ `self.external_db` is available in `API`, `SQSConsumer`, `Lambda`, and `FargateTask`.
491
+
414
492
  ## 🔄 Routing Convention
415
493
 
416
494
  The framework uses convention over configuration for the routing:
@@ -432,6 +510,105 @@ The framework uses convention over configuration for the routing:
432
510
  - `GET` with **even number of parts** → **get** method
433
511
  - Other methods use their name directly
434
512
 
513
+ ## 🧩 API Class Reference
514
+
515
+ All properties and methods available inside an `API` subclass:
516
+
517
+ ### Request Properties
518
+
519
+ | Property | Type | Description |
520
+ |----------|------|-------------|
521
+ | `self.data` | `dict` | Request body (POST/PUT) or query params (GET) |
522
+ | `self.headers` | `dict` | HTTP request headers |
523
+ | `self.path_parameters` | `dict` | URL path parameters (e.g. `/users/123` → `{'id': '123'}`) |
524
+ | `self.query_parameters` | `dict` | Query string parameters |
525
+ | `self.db` | `DatabaseProxy` | Access to main MongoDB cluster |
526
+ | `self.external_db` | `ExternalDatabaseProxy` | Access to external MongoDB clusters |
527
+ | `self.current_user` | `dict \| None` | Authenticated user document (requires `REQUIRE_AUTH=true`) |
528
+ | `self.is_authenticated` | `bool` | Whether the request is authenticated |
529
+ | `self.auth_data` | `dict \| None` | Full authentication data |
530
+
531
+ ### Response Methods
532
+
533
+ | Method | Description |
534
+ |--------|-------------|
535
+ | `self.set_code(code: int)` | Set HTTP response status code |
536
+ | `self.set_body(body: Any)` | Set response body (auto-serialized to JSON) |
537
+ | `self.set_header(key: str, value: str)` | Add a single response header |
538
+ | `self.set_headers(headers: dict)` | Set multiple response headers at once |
539
+
540
+ ### Methods to Override
541
+
542
+ | Method | Required | Description |
543
+ |--------|----------|-------------|
544
+ | `async validate()` | Optional | Validate request data, raise exceptions to reject |
545
+ | `async process()` | **Required** | Main business logic |
546
+
547
+ ```python
548
+ class UserGetAPI(API):
549
+ async def validate(self):
550
+ # Access path params: /users/123 → self.path_parameters = {'id': '123'}
551
+ if not self.path_parameters.get('id'):
552
+ raise ValueError("User ID is required")
553
+
554
+ async def process(self):
555
+ user_id = self.path_parameters['id']
556
+ user = await self.db.users_db.users.find_one({'_id': user_id})
557
+
558
+ if not user:
559
+ self.set_code(404)
560
+ self.set_body({'error': 'User not found'})
561
+ return
562
+
563
+ self.set_code(200)
564
+ self.set_body({'data': user})
565
+ self.set_header('X-Resource-Id', user_id)
566
+ ```
567
+
568
+ ## 🔐 Authentication
569
+
570
+ The framework includes a built-in token-based authentication middleware.
571
+
572
+ ### Configuration
573
+
574
+ ```bash
575
+ REQUIRE_AUTH=true # Enable authentication (default: false)
576
+ AUTH_DB_NAME=my_database # MongoDB database where tokens are stored
577
+ AUTH_BYPASS_TOKEN=secret123 # Master token to bypass auth (for internal use)
578
+ ```
579
+
580
+ ### Using the authenticated user
581
+
582
+ When `REQUIRE_AUTH=true`, every request must include a valid `Authorization: Bearer <token>` header. The authenticated user is available via `self.current_user`:
583
+
584
+ ```python
585
+ class OrderListAPI(API):
586
+ async def process(self):
587
+ # self.current_user contains the user document from MongoDB
588
+ user_id = self.current_user['_id']
589
+
590
+ orders = await self.db.orders_db.orders.find(
591
+ {'user_id': user_id}
592
+ ).to_list(100)
593
+
594
+ self.set_body({'data': orders})
595
+ ```
596
+
597
+ ### Auth exceptions
598
+
599
+ Use these exceptions in your `validate()` or `process()` methods:
600
+
601
+ ```python
602
+ from aws_python_helper.api.exceptions import UnauthorizedError, ForbiddenError
603
+
604
+ class AdminOnlyAPI(API):
605
+ async def validate(self):
606
+ if not self.is_authenticated:
607
+ raise UnauthorizedError("Authentication required") # Returns 401
608
+
609
+ if self.current_user.get('role') != 'admin':
610
+ raise ForbiddenError("Admin access required") # Returns 403
611
+ ```
435
612
 
436
613
  ## 🎯 Complete Example
437
614
 
@@ -445,28 +622,28 @@ class ConstitutionListAPI(API):
445
622
  limit = int(self.data['limit'])
446
623
  if limit > 1000:
447
624
  raise ValueError("Limit cannot exceed 1000")
448
-
625
+
449
626
  async def process(self):
450
627
  # Build filters
451
628
  filters = {}
452
629
  if 'country' in self.data:
453
630
  filters['country'] = self.data['country']
454
-
631
+
455
632
  # Query MongoDB
456
633
  limit = int(self.data.get('limit', 100))
457
634
  results = await self.db.constitution_db.constitutions.find(
458
635
  filters
459
636
  ).limit(limit).to_list(limit)
460
-
637
+
461
638
  # Count total
462
639
  total = await self.db.constitution_db.constitutions.count_documents(filters)
463
-
640
+
464
641
  # Register in analytics
465
642
  await self.db.analytics_db.searches.insert_one({
466
643
  'filters': filters,
467
644
  'result_count': len(results)
468
645
  })
469
-
646
+
470
647
  # Response
471
648
  self.set_body({
472
649
  'data': results,
@@ -494,7 +671,7 @@ class ShippingPostAPI(API):
494
671
  for field in required_fields:
495
672
  if field not in self.data:
496
673
  raise ValueError(f"{field} is required")
497
-
674
+
498
675
  async def process(self):
499
676
  # Create shipping in database
500
677
  shipping = {
@@ -504,10 +681,10 @@ class ShippingPostAPI(API):
504
681
  'status': 'pending',
505
682
  'route_pending': True
506
683
  }
507
-
684
+
508
685
  result = await self.db.deliveries.shippings.insert_one(shipping)
509
686
  shipping_id = str(result.inserted_id)
510
-
687
+
511
688
  # Invoke standalone lambda asynchronously to generate route
512
689
  lambda_client = boto3.client('lambda')
513
690
  lambda_client.invoke(
@@ -517,7 +694,7 @@ class ShippingPostAPI(API):
517
694
  'data': {'shipping_id': shipping_id}
518
695
  })
519
696
  )
520
-
697
+
521
698
  self.set_code(201)
522
699
  self.set_body({
523
700
  'shipping_id': shipping_id,
@@ -535,24 +712,24 @@ class GenerateRouteLambda(Lambda):
535
712
  async def validate(self):
536
713
  if 'shipping_id' not in self.data:
537
714
  raise ValueError("shipping_id is required")
538
-
715
+
539
716
  async def process(self):
540
717
  shipping_id = self.data['shipping_id']
541
-
718
+
542
719
  # Get shipping details
543
720
  shipping = await self.db.deliveries.shippings.find_one(
544
721
  {'_id': shipping_id}
545
722
  )
546
-
723
+
547
724
  if not shipping:
548
725
  raise ValueError(f"Shipping {shipping_id} not found")
549
-
726
+
550
727
  # Generate optimal route
551
728
  route = await self.calculate_optimal_route(shipping)
552
-
729
+
553
730
  # Save route
554
731
  route_result = await self.db.deliveries.routes.insert_one(route)
555
-
732
+
556
733
  # Update shipping
557
734
  await self.db.deliveries.shippings.update_one(
558
735
  {'_id': shipping_id},
@@ -562,12 +739,12 @@ class GenerateRouteLambda(Lambda):
562
739
  'status': 'scheduled'
563
740
  }}
564
741
  )
565
-
742
+
566
743
  return {
567
744
  'route_id': str(route_result.inserted_id),
568
745
  'shipping_id': shipping_id
569
746
  }
570
-
747
+
571
748
  async def calculate_optimal_route(self, shipping):
572
749
  # Your route calculation logic here
573
750
  return {
@@ -595,48 +772,89 @@ __all__ = ['generate_route_handler']
595
772
  - Can retry lambda independently if it fails
596
773
  - Scalable architecture
597
774
 
775
+ ## 🏗️ Architecture Overview
776
+
777
+ Typical flow for event-driven architectures using this framework:
778
+
779
+ ```
780
+ ┌──────────┐ ┌─────────────┐ ┌──────────────────────────────────────┐
781
+ │ Client │────▶│ API Gateway │────▶│ Lambda: api_handler │
782
+ └──────────┘ └─────────────┘ │ (src/api/resource/post.py) │
783
+ │ → validates, queries MongoDB, │
784
+ │ publishes to SNS │
785
+ └────────────────┬─────────────────────┘
786
+
787
+
788
+ ┌─────────────────┐
789
+ │ SNS Topic │
790
+ │ (fanout/filter) │
791
+ └────────┬────────┘
792
+ ┌───────────────┼───────────────┐
793
+ ▼ ▼ ▼
794
+ ┌────────────┐ ┌────────────┐ ┌────────────┐
795
+ │ SQS Queue │ │ SQS Queue │ │ SQS Queue │
796
+ │ Platform A│ │ Platform B│ │ Platform C│
797
+ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
798
+ │ │ │
799
+ ▼ ▼ ▼
800
+ ┌──────────────────────────────────────────────┐
801
+ │ Lambda: sqs_handler │
802
+ │ (src/consumer/platform_consumer.py) │
803
+ │ → groups messages, acquires sessions, │
804
+ │ launches Fargate tasks │
805
+ └───────────────────┬──────────────────────────┘
806
+ │ FargateExecutor.run_task()
807
+
808
+ ┌──────────────────────────────────────────────┐
809
+ │ Fargate Task: fargate_handler │
810
+ │ (src/task/my-task/task.py) │
811
+ │ → scrapes/processes data, │
812
+ │ writes results to MongoDB │
813
+ └──────────────────────────────────────────────┘
814
+ ```
815
+
598
816
  ## 🔐 Environment Variables
599
817
 
600
818
  ### MongoDB Configuration
601
819
 
602
- El framework soporta dos formas de configurar MongoDB:
820
+ The framework supports two ways to configure MongoDB:
603
821
 
604
- #### Opción 1: Connection String Completa
822
+ #### Option 1: Full Connection String
605
823
 
606
824
  ```bash
607
- # URI completa con credenciales incluidas
825
+ # Full URI with embedded credentials
608
826
  MONGODB_URI=mongodb+srv://user:password@cluster.mongodb.net/dbname?retryWrites=true&w=majority
609
- # o
827
+ # or
610
828
  MONGO_DB_URI=mongodb+srv://user:password@cluster.mongodb.net/dbname
611
829
  ```
612
830
 
613
- #### Opción 2: Componentes Separados (Recomendado para Terraform)
831
+ #### Option 2: Separate Components (Recommended for Terraform)
614
832
 
615
833
  ```bash
616
- # Host sin credenciales
834
+ # Host without credentials
617
835
  MONGO_DB_HOST=mongodb+srv://cluster.mongodb.net
618
836
 
619
- # Credenciales separadas (más seguro)
837
+ # Credentials (more secure)
620
838
  MONGO_DB_USER=admin
621
839
  MONGO_DB_PASSWORD=my-secure-password
622
840
 
623
- # Opcionales
841
+ # Optional
624
842
  MONGO_DB_NAME=my_database
625
843
  MONGO_DB_OPTIONS=retryWrites=true&w=majority
626
844
  ```
627
845
 
628
- **Ventajas de usar componentes separados:**
629
- - ✅ Mejor seguridad: credenciales separadas del host
630
- - ✅ Fácil integración con Terraform/AWS Secrets Manager
631
- - ✅ Contraseñas con caracteres especiales se manejan automáticamente
632
- - ✅ Más flexible para diferentes entornos
846
+ **Benefits of separate components:**
847
+ - ✅ Better security: credentials separate from host
848
+ - ✅ Easy integration with Terraform/AWS Secrets Manager
849
+ - ✅ Passwords with special characters are handled automatically
850
+ - ✅ More flexible for different environments
633
851
 
634
- El framework automáticamente:
635
- 1. URL-encodea la contraseña (maneja `@`, `:`, `/`, etc.)
636
- 2. Construye la URI completa
637
- 3. Inicializa la conexión
852
+ The framework automatically:
853
+ 1. URL-encodes the password (handles `@`, `:`, `/`, etc.)
854
+ 2. Builds the full URI
855
+ 3. Initializes the connection
638
856
 
639
- ### Ejemplo en Terraform
857
+ #### Terraform Example
640
858
 
641
859
  ```hcl
642
860
  environment_variables = {
@@ -646,22 +864,119 @@ environment_variables = {
646
864
  }
647
865
  ```
648
866
 
649
- ## Rest Environment Variables
867
+ ### All Environment Variables
868
+
869
+ | Variable | Required | Description |
870
+ |----------|----------|-------------|
871
+ | `MONGODB_URI` or `MONGO_DB_URI` | One of these or components below | Full MongoDB connection string |
872
+ | `MONGO_DB_HOST` | Alt. to URI | MongoDB host (e.g. `mongodb+srv://cluster.net`) |
873
+ | `MONGO_DB_USER` | Alt. to URI | MongoDB username |
874
+ | `MONGO_DB_PASSWORD` | Alt. to URI | MongoDB password |
875
+ | `MONGO_DB_NAME` | Optional | Default database name |
876
+ | `MONGO_DB_OPTIONS` | Optional | Connection options (e.g. `retryWrites=true&w=majority`) |
877
+ | `EXTERNAL_MONGODB_CONNECTIONS` | Optional | JSON array of external cluster configurations |
878
+ | `REQUIRE_AUTH` | Optional | Enable authentication middleware (`true`/`false`) |
879
+ | `AUTH_DB_NAME` | If `REQUIRE_AUTH=true` | MongoDB database for token validation |
880
+ | `AUTH_BYPASS_TOKEN` | Optional | Master token to bypass authentication |
881
+ | `ECS_CLUSTER` | Fargate only | ECS cluster name for `FargateExecutor` |
882
+ | `ECS_SUBNETS` | Fargate only | Comma-separated subnet IDs for Fargate tasks |
883
+ | `AWS_REGION` | Fargate/SNS/SQS | AWS region |
884
+ | `AWS_ACCOUNT_ID` | SQS `get_queue_url` | AWS account ID |
885
+ | `SERVICE_NAME` | SQS `get_queue_url` | Service name prefix for queue name |
886
+ | `QUEUE_NAME` | SQS `get_queue_url` | Queue name segment |
887
+ | `ENV` | SQS `get_queue_url` | Environment suffix (e.g. `prod`, `dev`) |
650
888
 
651
889
  ## 📊 Advanced Features
652
890
 
891
+ ### SQS Consumer - Batch Mode
892
+
893
+ By default, consumers process messages one by one (`"single"` mode). Use `"batch"` mode when you need to group or bulk-process messages:
894
+
895
+ ```python
896
+ from aws_python_helper.sqs.consumer_base import SQSConsumer
897
+
898
+ class OrderConsumer(SQSConsumer):
899
+
900
+ @property
901
+ def processing_mode(self) -> str:
902
+ return "batch"
903
+
904
+ async def process_batch(self, records):
905
+ # Group records by some key before processing
906
+ grouped = {}
907
+ for record in records:
908
+ message_id = record.get('messageId')
909
+ body = self.extract_content_message(record)
910
+ key = body.get('region', 'default')
911
+ grouped.setdefault(key, []).append((message_id, body))
912
+
913
+ for region, messages in grouped.items():
914
+ try:
915
+ # Bulk operation for the whole group
916
+ docs = [msg[1] for msg in messages]
917
+ await self.db.orders_db.orders.insert_many(docs)
918
+ except Exception as e:
919
+ # Mark individual messages as failed
920
+ for message_id, _ in messages:
921
+ self.add_message_failed(message_id, str(e))
922
+ ```
923
+
924
+ **Key methods in SQSConsumer:**
925
+
926
+ | Method / Property | Description |
927
+ |-------------------|-------------|
928
+ | `self.extract_content_message(record)` | Parse message body (handles SNS → SQS wrapping automatically) |
929
+ | `self.parse_body(record)` | Alias for `extract_content_message` |
930
+ | `self.add_message_failed(message_id, error)` | Mark a message for retry (batch mode) |
931
+ | `self.get_queue_url()` | Get the SQS queue URL (uses `AWS_REGION`, `AWS_ACCOUNT_ID`, `SERVICE_NAME`, `QUEUE_NAME`, `ENV`) |
932
+ | `self.db` | Access to main MongoDB cluster |
933
+ | `self.external_db` | Access to external MongoDB clusters |
934
+
935
+ **Retry behavior:**
936
+ - Messages marked with `add_message_failed()` are reported via `reportBatchItemFailures`
937
+ - AWS SQS retries **only** the failed messages, not the whole batch
938
+ - Successful messages in the same batch are not retried
939
+
653
940
  ### SNS Publisher - Batch Publishing
654
941
 
655
942
  ```python
656
- # Publish multiple messages
657
943
  topic = TitleIndexedTopic()
658
- await topic.publish_batch_indexed([
659
- {'constitution_id': 'id1', 'title': 'Title 1'},
660
- {'constitution_id': 'id2', 'title': 'Title 2'},
661
- {'constitution_id': 'id3', 'title': 'Title 3'}
944
+
945
+ # Publish multiple messages in a single call
946
+ await topic.publish([
947
+ {'content': {'id': 'id1', 'title': 'Title 1'}, 'attributes': {'type': 'created'}},
948
+ {'content': {'id': 'id2', 'title': 'Title 2'}, 'attributes': {'type': 'updated'}},
949
+ {'content': {'id': 'id3', 'title': 'Title 3'}}, # attributes are optional
662
950
  ])
663
951
  ```
664
952
 
953
+ ### SNS - Message Attributes for Filtering
954
+
955
+ Use `attributes` to filter which SQS subscriptions receive each message:
956
+
957
+ ```python
958
+ class EventTopic(SNSPublisher):
959
+ def __init__(self):
960
+ super().__init__(topic_arn=os.getenv('EVENTS_SNS_TOPIC_ARN'))
961
+
962
+ def build_message(self, payload, event_type, priority='normal'):
963
+ return {
964
+ 'content': payload,
965
+ 'attributes': {
966
+ 'event_type': event_type, # SQS subscriptions can filter on this
967
+ 'priority': priority
968
+ }
969
+ }
970
+
971
+ # Usage
972
+ topic = EventTopic()
973
+ await topic.publish(topic.build_message(
974
+ payload={'order_id': '123', 'amount': 99.99},
975
+ event_type='order_created',
976
+ priority='high'
977
+ ))
978
+ ```
979
+
665
980
  ### Fargate - Run multiple tasks
666
981
 
667
982
  ```python
@@ -669,9 +984,9 @@ executor = FargateExecutor()
669
984
  task_arns = executor.run_task_batch(
670
985
  'search-tax-by-town',
671
986
  [
672
- {'town': 'Norwalk'},
673
- {'town': 'Stamford'},
674
- {'town': 'Bridgeport'}
987
+ {'TOWN': 'Norwalk'},
988
+ {'TOWN': 'Stamford'},
989
+ {'TOWN': 'Bridgeport'}
675
990
  ]
676
991
  )
677
992
  ```
@@ -680,7 +995,7 @@ task_arns = executor.run_task_batch(
680
995
 
681
996
  ```python
682
997
  executor = FargateExecutor()
683
- task_arn = executor.run_task('my-task', {'param': 'value'})
998
+ task_arn = executor.run_task('my-task', {'PARAM': 'value'})
684
999
 
685
1000
  # Check task status
686
1001
  status = executor.get_task_status(task_arn)
@@ -688,19 +1003,51 @@ print(f"Status: {status['status']}")
688
1003
  print(f"Started at: {status['started_at']}")
689
1004
  ```
690
1005
 
691
- ### SNS - Message Attributes
1006
+ ### JSON Utilities for MongoDB Types
1007
+
1008
+ When returning MongoDB documents in API responses or exporting data, use the built-in serializers to handle `ObjectId`, `datetime`, `Decimal128`, and other BSON types:
692
1009
 
693
1010
  ```python
694
- # Publish with attributes for SNS filtering
695
- topic = ConstitutionCreatedTopic()
696
- await topic.publish_created(
697
- constitution_id='123',
698
- title='New Constitution',
699
- country='Ecuador',
700
- year=2023,
701
- created_by='user_456',
702
- attributes={'priority': 'high', 'region': 'latam'}
703
- )
1011
+ import json
1012
+ from aws_python_helper.utils.json_encoder import MongoJSONEncoder, mongo_json_dumps
1013
+ from aws_python_helper.utils.serializer import serialize_mongo_types
1014
+
1015
+ # Use as json.dumps cls parameter
1016
+ json_str = json.dumps(my_mongo_doc, cls=MongoJSONEncoder)
1017
+
1018
+ # Helper function
1019
+ json_str = mongo_json_dumps(my_mongo_doc)
1020
+
1021
+ # Convert a document in-place (dict → JSON-serializable dict)
1022
+ clean_doc = serialize_mongo_types(my_mongo_doc)
1023
+ ```
1024
+
1025
+ Types automatically converted:
1026
+
1027
+ | MongoDB Type | Converts to |
1028
+ |-------------|-------------|
1029
+ | `ObjectId` | `str` |
1030
+ | `datetime` | ISO 8601 string |
1031
+ | `date` | ISO 8601 string |
1032
+ | `Decimal128` | `float` |
1033
+ | `Decimal` | `float` |
1034
+ | `Binary` | base64 `str` |
1035
+ | `UUID` | `str` |
1036
+ | `bytes` | base64 `str` |
1037
+ | `set` | `list` |
1038
+
1039
+ **Common use case** — exporting query results to JSON files:
1040
+
1041
+ ```python
1042
+ from aws_python_helper.utils.json_encoder import MongoJSONEncoder
1043
+
1044
+ class ExportResultsAPI(API):
1045
+ async def process(self):
1046
+ records = await self.db.orders_db.orders.find({}).to_list(1000)
1047
+
1048
+ # Write to file with MongoJSONEncoder
1049
+ with open('/tmp/export.json', 'w') as f:
1050
+ json.dump(records, f, cls=MongoJSONEncoder, ensure_ascii=False, indent=2)
704
1051
  ```
705
1052
 
706
1053
  ## 🤝 Contributing