aws-python-helper 0.30.0__tar.gz → 0.30.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/PKG-INFO +441 -94
- aws_python_helper-0.30.1/README.md +1039 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/PKG-INFO +441 -94
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/pyproject.toml +1 -1
- aws_python_helper-0.30.0/README.md +0 -692
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/auth_middleware.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/auth_validators.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/base.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/dispatcher.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/exceptions.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/fetcher.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/api/handler.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/database_proxy.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/external_database_proxy.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/external_mongo_manager.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/database/mongo_manager.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/executor.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/fetcher.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/handler.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/fargate/task_base.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/base.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/fetcher.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/lambda_standalone/handler.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sns/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sns/publisher.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/consumer_base.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/fetcher.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/sqs/handler.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/__init__.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/json_encoder.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/response.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper/utils/serializer.py +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/SOURCES.txt +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/dependency_links.txt +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/requires.txt +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/aws_python_helper.egg-info/top_level.txt +0 -0
- {aws_python_helper-0.30.0 → aws_python_helper-0.30.1}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: aws-python-helper
|
|
3
|
-
Version: 0.30.
|
|
3
|
+
Version: 0.30.1
|
|
4
4
|
Summary: AWS Python Helper Framework
|
|
5
5
|
Author-email: Fabian Claros <neufabiae@gmail.com>
|
|
6
6
|
License: MIT
|
|
@@ -28,23 +28,44 @@ Mini-framework to create REST APIs, SQS Consumers, SNS Publishers, Fargate Tasks
|
|
|
28
28
|
- **Dynamic controller loading**: Routing based on convention
|
|
29
29
|
- **OOP structure**: Object-oriented programming for your code
|
|
30
30
|
- **Flexible MongoDB**: Direct access to multiple databases without models
|
|
31
|
-
- **
|
|
31
|
+
- **External MongoDB**: Connect to multiple MongoDB clusters simultaneously
|
|
32
|
+
- **SQS Consumers**: Same pattern to process SQS messages (single or batch mode)
|
|
32
33
|
- **SNS Publishers**: Same pattern to publish messages to SNS topics
|
|
33
34
|
- **Fargate Tasks**: Same pattern to run tasks in Fargate containers
|
|
34
35
|
- **Standalone Lambdas**: Create lambdas invocable directly with AWS SDK
|
|
36
|
+
- **Authentication middleware**: Built-in token-based authentication
|
|
37
|
+
- **JSON utilities**: Automatic serialization of MongoDB types
|
|
35
38
|
- **Type hints**: Modern Python with type annotations
|
|
36
39
|
- **Async/await**: Full support for asynchronous operations
|
|
37
40
|
|
|
38
41
|
## 🔧 Installation
|
|
39
42
|
|
|
40
43
|
```bash
|
|
41
|
-
|
|
42
|
-
pip install -r requirements.txt
|
|
43
|
-
|
|
44
|
-
# Configure MongoDB URI
|
|
45
|
-
export MONGODB_URI="mongodb://localhost:27017"
|
|
44
|
+
pip install aws-python-helper
|
|
46
45
|
```
|
|
47
46
|
|
|
47
|
+
## 📦 Quick Reference
|
|
48
|
+
|
|
49
|
+
All available classes and functions:
|
|
50
|
+
|
|
51
|
+
| Class / Function | Import | Purpose |
|
|
52
|
+
|------------------|--------|---------|
|
|
53
|
+
| `API` | `aws_python_helper.api.base` | Base class for REST endpoints |
|
|
54
|
+
| `api_handler` | `aws_python_helper.api.handler` | Generic handler for API Gateway |
|
|
55
|
+
| `SQSConsumer` | `aws_python_helper.sqs.consumer_base` | Base class for SQS consumers |
|
|
56
|
+
| `sqs_handler` | `aws_python_helper.sqs.handler` | Factory handler for SQS |
|
|
57
|
+
| `SNSPublisher` | `aws_python_helper.sns.publisher` | Base class for SNS publishers |
|
|
58
|
+
| `Lambda` | `aws_python_helper.lambda_standalone.base` | Base class for Standalone Lambdas |
|
|
59
|
+
| `lambda_handler` | `aws_python_helper.lambda_standalone.handler` | Factory handler for Lambda |
|
|
60
|
+
| `FargateTask` | `aws_python_helper.fargate.task_base` | Base class for Fargate tasks |
|
|
61
|
+
| `FargateExecutor` | `aws_python_helper.fargate.executor` | Launches Fargate tasks from Lambda |
|
|
62
|
+
| `fargate_handler` | `aws_python_helper.fargate.handler` | Entry point handler for Fargate |
|
|
63
|
+
| `MongoJSONEncoder` | `aws_python_helper.utils.json_encoder` | JSON encoder for MongoDB types |
|
|
64
|
+
| `mongo_json_dumps` | `aws_python_helper.utils.json_encoder` | Helper to serialize MongoDB types |
|
|
65
|
+
| `serialize_mongo_types` | `aws_python_helper.utils.serializer` | Recursively serialize MongoDB types |
|
|
66
|
+
| `UnauthorizedError` | `aws_python_helper.api.exceptions` | 401 authentication exception |
|
|
67
|
+
| `ForbiddenError` | `aws_python_helper.api.exceptions` | 403 authorization exception |
|
|
68
|
+
|
|
48
69
|
## 📂 Project Structure
|
|
49
70
|
|
|
50
71
|
This framework follows a convention-based folder structure. Here's the recommended organization:
|
|
@@ -73,13 +94,16 @@ your-project/
|
|
|
73
94
|
│ └── process-payment/ # process-payment -> ProcessPaymentLambda
|
|
74
95
|
│ └── main.py
|
|
75
96
|
│
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
97
|
+
├── task/ # Fargate Tasks (folders)
|
|
98
|
+
│ ├── search-tax-by-town/ # search-tax-by-town -> SearchTaxByTownTask
|
|
99
|
+
│ │ ├── main.py # Entry point
|
|
100
|
+
│ │ └── task.py # Task class
|
|
101
|
+
│ └── process-data/ # process-data -> ProcessDataTask
|
|
102
|
+
│ ├── main.py
|
|
103
|
+
│ └── task.py
|
|
104
|
+
│
|
|
105
|
+
└── topic/ # SNS Publishers
|
|
106
|
+
└── order_created.py # OrderCreatedTopic
|
|
83
107
|
```
|
|
84
108
|
|
|
85
109
|
### Naming Conventions
|
|
@@ -137,7 +161,7 @@ from aws_python_helper.sqs.consumer_base import SQSConsumer
|
|
|
137
161
|
|
|
138
162
|
class TitleIndexedConsumer(SQSConsumer):
|
|
139
163
|
async def process_record(self, record):
|
|
140
|
-
body = self.
|
|
164
|
+
body = self.extract_content_message(record)
|
|
141
165
|
# Your logic here
|
|
142
166
|
await self.db.constitution_db.titles.insert_one(body)
|
|
143
167
|
```
|
|
@@ -174,22 +198,22 @@ class GenerateRouteLambda(Lambda):
|
|
|
174
198
|
# Validate input data
|
|
175
199
|
if 'shipping_id' not in self.data:
|
|
176
200
|
raise ValueError("shipping_id is required")
|
|
177
|
-
|
|
201
|
+
|
|
178
202
|
if not isinstance(self.data['shipping_id'], str):
|
|
179
203
|
raise TypeError("shipping_id must be a string")
|
|
180
|
-
|
|
204
|
+
|
|
181
205
|
async def process(self):
|
|
182
206
|
# Your business logic here
|
|
183
207
|
shipping_id = self.data['shipping_id']
|
|
184
|
-
|
|
208
|
+
|
|
185
209
|
# Access to MongoDB
|
|
186
210
|
shipping = await self.db.deliveries.shippings.find_one(
|
|
187
211
|
{'_id': shipping_id}
|
|
188
212
|
)
|
|
189
|
-
|
|
213
|
+
|
|
190
214
|
if not shipping:
|
|
191
215
|
raise ValueError(f"Shipping {shipping_id} not found")
|
|
192
|
-
|
|
216
|
+
|
|
193
217
|
# Create route
|
|
194
218
|
route = {
|
|
195
219
|
'shipping_id': shipping_id,
|
|
@@ -197,11 +221,11 @@ class GenerateRouteLambda(Lambda):
|
|
|
197
221
|
'status': 'pending',
|
|
198
222
|
'created_at': datetime.utcnow()
|
|
199
223
|
}
|
|
200
|
-
|
|
224
|
+
|
|
201
225
|
result = await self.db.deliveries.routes.insert_one(route)
|
|
202
|
-
|
|
226
|
+
|
|
203
227
|
self.logger.info(f"Route created: {result.inserted_id}")
|
|
204
|
-
|
|
228
|
+
|
|
205
229
|
# Return result
|
|
206
230
|
return {
|
|
207
231
|
'route_id': str(result.inserted_id),
|
|
@@ -305,23 +329,47 @@ class TitleIndexedTopic(SNSPublisher):
|
|
|
305
329
|
super().__init__(
|
|
306
330
|
topic_arn=os.getenv('TITLE_INDEXED_SNS_TOPIC_ARN')
|
|
307
331
|
)
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
'
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
332
|
+
|
|
333
|
+
def build_message(self, constitution_id, title, event_type='title_indexed'):
|
|
334
|
+
return {
|
|
335
|
+
'content': {
|
|
336
|
+
'constitution_id': constitution_id,
|
|
337
|
+
'title': title,
|
|
338
|
+
'event_type': event_type
|
|
339
|
+
},
|
|
340
|
+
'attributes': {
|
|
341
|
+
'event_type': event_type # Used for SNS subscription filtering
|
|
342
|
+
}
|
|
343
|
+
}
|
|
315
344
|
```
|
|
316
345
|
|
|
317
346
|
**2. Use the topic** from anywhere:
|
|
318
347
|
|
|
319
348
|
```python
|
|
320
|
-
from src.
|
|
349
|
+
from src.topic.title_indexed import TitleIndexedTopic
|
|
321
350
|
|
|
322
351
|
# In a consumer, API or task
|
|
323
352
|
topic = TitleIndexedTopic()
|
|
324
|
-
|
|
353
|
+
|
|
354
|
+
# Publish a single message
|
|
355
|
+
await topic.publish(topic.build_message('123', 'My Constitution'))
|
|
356
|
+
|
|
357
|
+
# Publish multiple messages in batch
|
|
358
|
+
messages = [
|
|
359
|
+
topic.build_message('id1', 'Constitution A'),
|
|
360
|
+
topic.build_message('id2', 'Constitution B'),
|
|
361
|
+
]
|
|
362
|
+
await topic.publish(messages)
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
**Message format** — every message must have a `content` key:
|
|
366
|
+
|
|
367
|
+
```python
|
|
368
|
+
{
|
|
369
|
+
'content': {...}, # Required: message body (any dict)
|
|
370
|
+
'attributes': {...}, # Optional: SNS message attributes for filtering
|
|
371
|
+
'subject': 'Optional subject' # Optional: message subject
|
|
372
|
+
}
|
|
325
373
|
```
|
|
326
374
|
|
|
327
375
|
### Run a Fargate Task
|
|
@@ -336,10 +384,10 @@ class SearchTaxByTownTask(FargateTask):
|
|
|
336
384
|
async def execute(self):
|
|
337
385
|
town = self.require_env('TOWN')
|
|
338
386
|
self.logger.info(f"Processing town: {town}")
|
|
339
|
-
|
|
387
|
+
|
|
340
388
|
# Access to DB
|
|
341
389
|
docs = await self.db.smart_data.address.find({'town': town}).to_list()
|
|
342
|
-
|
|
390
|
+
|
|
343
391
|
# Your logic here
|
|
344
392
|
for doc in docs:
|
|
345
393
|
# Process document
|
|
@@ -388,7 +436,7 @@ def handler(event, context):
|
|
|
388
436
|
executor = FargateExecutor()
|
|
389
437
|
task_arn = executor.run_task(
|
|
390
438
|
'search-tax-by-town',
|
|
391
|
-
envs={'
|
|
439
|
+
envs={'TOWN': 'Norwalk', 'ONLY_TAX': 'true'}
|
|
392
440
|
)
|
|
393
441
|
return {'taskArn': task_arn}
|
|
394
442
|
```
|
|
@@ -400,17 +448,47 @@ The framework provides flexible access to multiple databases:
|
|
|
400
448
|
```python
|
|
401
449
|
class MyAPI(API):
|
|
402
450
|
async def process(self):
|
|
403
|
-
# Access to different databases
|
|
451
|
+
# Access to different databases on the same cluster
|
|
404
452
|
user = await self.db.users_db.users.find_one({'_id': user_id})
|
|
405
|
-
|
|
453
|
+
|
|
406
454
|
# Another database
|
|
407
455
|
await self.db.analytics_db.logs.insert_one({'action': 'view'})
|
|
408
|
-
|
|
456
|
+
|
|
409
457
|
# Multiple collections
|
|
410
458
|
titles = await self.db.constitution_db.titles.find().to_list(100)
|
|
411
459
|
articles = await self.db.constitution_db.articles.find().to_list(100)
|
|
412
460
|
```
|
|
413
461
|
|
|
462
|
+
The pattern is always: `self.db.<database_name>.<collection_name>.<motor_operation>()`
|
|
463
|
+
|
|
464
|
+
### External MongoDB Clusters
|
|
465
|
+
|
|
466
|
+
Connect to additional MongoDB clusters using `EXTERNAL_MONGODB_CONNECTIONS`:
|
|
467
|
+
|
|
468
|
+
```bash
|
|
469
|
+
EXTERNAL_MONGODB_CONNECTIONS='[
|
|
470
|
+
{"name": "ClusterDockets", "connection_string": "mongodb+srv://cluster.mongodb.net"},
|
|
471
|
+
{"name": "ClusterAnalytics", "connection_string": "mongodb+srv://analytics.mongodb.net"}
|
|
472
|
+
]'
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
The credentials from `MONGO_DB_USER` / `MONGO_DB_PASSWORD` are automatically injected into the connection strings.
|
|
476
|
+
|
|
477
|
+
Access external clusters via `self.external_db`:
|
|
478
|
+
|
|
479
|
+
```python
|
|
480
|
+
class AddressAPI(API):
|
|
481
|
+
async def process(self):
|
|
482
|
+
# Access external cluster: self.external_db.<ClusterName>.<database>.<collection>
|
|
483
|
+
addresses = await self.external_db.ClusterDockets.smart_data.addresses.find(
|
|
484
|
+
{'town': self.data['town']}
|
|
485
|
+
).to_list(100)
|
|
486
|
+
|
|
487
|
+
self.set_body({'addresses': addresses})
|
|
488
|
+
```
|
|
489
|
+
|
|
490
|
+
`self.external_db` is available in `API`, `SQSConsumer`, `Lambda`, and `FargateTask`.
|
|
491
|
+
|
|
414
492
|
## 🔄 Routing Convention
|
|
415
493
|
|
|
416
494
|
The framework uses convention over configuration for the routing:
|
|
@@ -432,6 +510,105 @@ The framework uses convention over configuration for the routing:
|
|
|
432
510
|
- `GET` with **even number of parts** → **get** method
|
|
433
511
|
- Other methods use their name directly
|
|
434
512
|
|
|
513
|
+
## 🧩 API Class Reference
|
|
514
|
+
|
|
515
|
+
All properties and methods available inside an `API` subclass:
|
|
516
|
+
|
|
517
|
+
### Request Properties
|
|
518
|
+
|
|
519
|
+
| Property | Type | Description |
|
|
520
|
+
|----------|------|-------------|
|
|
521
|
+
| `self.data` | `dict` | Request body (POST/PUT) or query params (GET) |
|
|
522
|
+
| `self.headers` | `dict` | HTTP request headers |
|
|
523
|
+
| `self.path_parameters` | `dict` | URL path parameters (e.g. `/users/123` → `{'id': '123'}`) |
|
|
524
|
+
| `self.query_parameters` | `dict` | Query string parameters |
|
|
525
|
+
| `self.db` | `DatabaseProxy` | Access to main MongoDB cluster |
|
|
526
|
+
| `self.external_db` | `ExternalDatabaseProxy` | Access to external MongoDB clusters |
|
|
527
|
+
| `self.current_user` | `dict \| None` | Authenticated user document (requires `REQUIRE_AUTH=true`) |
|
|
528
|
+
| `self.is_authenticated` | `bool` | Whether the request is authenticated |
|
|
529
|
+
| `self.auth_data` | `dict \| None` | Full authentication data |
|
|
530
|
+
|
|
531
|
+
### Response Methods
|
|
532
|
+
|
|
533
|
+
| Method | Description |
|
|
534
|
+
|--------|-------------|
|
|
535
|
+
| `self.set_code(code: int)` | Set HTTP response status code |
|
|
536
|
+
| `self.set_body(body: Any)` | Set response body (auto-serialized to JSON) |
|
|
537
|
+
| `self.set_header(key: str, value: str)` | Add a single response header |
|
|
538
|
+
| `self.set_headers(headers: dict)` | Set multiple response headers at once |
|
|
539
|
+
|
|
540
|
+
### Methods to Override
|
|
541
|
+
|
|
542
|
+
| Method | Required | Description |
|
|
543
|
+
|--------|----------|-------------|
|
|
544
|
+
| `async validate()` | Optional | Validate request data, raise exceptions to reject |
|
|
545
|
+
| `async process()` | **Required** | Main business logic |
|
|
546
|
+
|
|
547
|
+
```python
|
|
548
|
+
class UserGetAPI(API):
|
|
549
|
+
async def validate(self):
|
|
550
|
+
# Access path params: /users/123 → self.path_parameters = {'id': '123'}
|
|
551
|
+
if not self.path_parameters.get('id'):
|
|
552
|
+
raise ValueError("User ID is required")
|
|
553
|
+
|
|
554
|
+
async def process(self):
|
|
555
|
+
user_id = self.path_parameters['id']
|
|
556
|
+
user = await self.db.users_db.users.find_one({'_id': user_id})
|
|
557
|
+
|
|
558
|
+
if not user:
|
|
559
|
+
self.set_code(404)
|
|
560
|
+
self.set_body({'error': 'User not found'})
|
|
561
|
+
return
|
|
562
|
+
|
|
563
|
+
self.set_code(200)
|
|
564
|
+
self.set_body({'data': user})
|
|
565
|
+
self.set_header('X-Resource-Id', user_id)
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
## 🔐 Authentication
|
|
569
|
+
|
|
570
|
+
The framework includes a built-in token-based authentication middleware.
|
|
571
|
+
|
|
572
|
+
### Configuration
|
|
573
|
+
|
|
574
|
+
```bash
|
|
575
|
+
REQUIRE_AUTH=true # Enable authentication (default: false)
|
|
576
|
+
AUTH_DB_NAME=my_database # MongoDB database where tokens are stored
|
|
577
|
+
AUTH_BYPASS_TOKEN=secret123 # Master token to bypass auth (for internal use)
|
|
578
|
+
```
|
|
579
|
+
|
|
580
|
+
### Using the authenticated user
|
|
581
|
+
|
|
582
|
+
When `REQUIRE_AUTH=true`, every request must include a valid `Authorization: Bearer <token>` header. The authenticated user is available via `self.current_user`:
|
|
583
|
+
|
|
584
|
+
```python
|
|
585
|
+
class OrderListAPI(API):
|
|
586
|
+
async def process(self):
|
|
587
|
+
# self.current_user contains the user document from MongoDB
|
|
588
|
+
user_id = self.current_user['_id']
|
|
589
|
+
|
|
590
|
+
orders = await self.db.orders_db.orders.find(
|
|
591
|
+
{'user_id': user_id}
|
|
592
|
+
).to_list(100)
|
|
593
|
+
|
|
594
|
+
self.set_body({'data': orders})
|
|
595
|
+
```
|
|
596
|
+
|
|
597
|
+
### Auth exceptions
|
|
598
|
+
|
|
599
|
+
Use these exceptions in your `validate()` or `process()` methods:
|
|
600
|
+
|
|
601
|
+
```python
|
|
602
|
+
from aws_python_helper.api.exceptions import UnauthorizedError, ForbiddenError
|
|
603
|
+
|
|
604
|
+
class AdminOnlyAPI(API):
|
|
605
|
+
async def validate(self):
|
|
606
|
+
if not self.is_authenticated:
|
|
607
|
+
raise UnauthorizedError("Authentication required") # Returns 401
|
|
608
|
+
|
|
609
|
+
if self.current_user.get('role') != 'admin':
|
|
610
|
+
raise ForbiddenError("Admin access required") # Returns 403
|
|
611
|
+
```
|
|
435
612
|
|
|
436
613
|
## 🎯 Complete Example
|
|
437
614
|
|
|
@@ -445,28 +622,28 @@ class ConstitutionListAPI(API):
|
|
|
445
622
|
limit = int(self.data['limit'])
|
|
446
623
|
if limit > 1000:
|
|
447
624
|
raise ValueError("Limit cannot exceed 1000")
|
|
448
|
-
|
|
625
|
+
|
|
449
626
|
async def process(self):
|
|
450
627
|
# Build filters
|
|
451
628
|
filters = {}
|
|
452
629
|
if 'country' in self.data:
|
|
453
630
|
filters['country'] = self.data['country']
|
|
454
|
-
|
|
631
|
+
|
|
455
632
|
# Query MongoDB
|
|
456
633
|
limit = int(self.data.get('limit', 100))
|
|
457
634
|
results = await self.db.constitution_db.constitutions.find(
|
|
458
635
|
filters
|
|
459
636
|
).limit(limit).to_list(limit)
|
|
460
|
-
|
|
637
|
+
|
|
461
638
|
# Count total
|
|
462
639
|
total = await self.db.constitution_db.constitutions.count_documents(filters)
|
|
463
|
-
|
|
640
|
+
|
|
464
641
|
# Register in analytics
|
|
465
642
|
await self.db.analytics_db.searches.insert_one({
|
|
466
643
|
'filters': filters,
|
|
467
644
|
'result_count': len(results)
|
|
468
645
|
})
|
|
469
|
-
|
|
646
|
+
|
|
470
647
|
# Response
|
|
471
648
|
self.set_body({
|
|
472
649
|
'data': results,
|
|
@@ -494,7 +671,7 @@ class ShippingPostAPI(API):
|
|
|
494
671
|
for field in required_fields:
|
|
495
672
|
if field not in self.data:
|
|
496
673
|
raise ValueError(f"{field} is required")
|
|
497
|
-
|
|
674
|
+
|
|
498
675
|
async def process(self):
|
|
499
676
|
# Create shipping in database
|
|
500
677
|
shipping = {
|
|
@@ -504,10 +681,10 @@ class ShippingPostAPI(API):
|
|
|
504
681
|
'status': 'pending',
|
|
505
682
|
'route_pending': True
|
|
506
683
|
}
|
|
507
|
-
|
|
684
|
+
|
|
508
685
|
result = await self.db.deliveries.shippings.insert_one(shipping)
|
|
509
686
|
shipping_id = str(result.inserted_id)
|
|
510
|
-
|
|
687
|
+
|
|
511
688
|
# Invoke standalone lambda asynchronously to generate route
|
|
512
689
|
lambda_client = boto3.client('lambda')
|
|
513
690
|
lambda_client.invoke(
|
|
@@ -517,7 +694,7 @@ class ShippingPostAPI(API):
|
|
|
517
694
|
'data': {'shipping_id': shipping_id}
|
|
518
695
|
})
|
|
519
696
|
)
|
|
520
|
-
|
|
697
|
+
|
|
521
698
|
self.set_code(201)
|
|
522
699
|
self.set_body({
|
|
523
700
|
'shipping_id': shipping_id,
|
|
@@ -535,24 +712,24 @@ class GenerateRouteLambda(Lambda):
|
|
|
535
712
|
async def validate(self):
|
|
536
713
|
if 'shipping_id' not in self.data:
|
|
537
714
|
raise ValueError("shipping_id is required")
|
|
538
|
-
|
|
715
|
+
|
|
539
716
|
async def process(self):
|
|
540
717
|
shipping_id = self.data['shipping_id']
|
|
541
|
-
|
|
718
|
+
|
|
542
719
|
# Get shipping details
|
|
543
720
|
shipping = await self.db.deliveries.shippings.find_one(
|
|
544
721
|
{'_id': shipping_id}
|
|
545
722
|
)
|
|
546
|
-
|
|
723
|
+
|
|
547
724
|
if not shipping:
|
|
548
725
|
raise ValueError(f"Shipping {shipping_id} not found")
|
|
549
|
-
|
|
726
|
+
|
|
550
727
|
# Generate optimal route
|
|
551
728
|
route = await self.calculate_optimal_route(shipping)
|
|
552
|
-
|
|
729
|
+
|
|
553
730
|
# Save route
|
|
554
731
|
route_result = await self.db.deliveries.routes.insert_one(route)
|
|
555
|
-
|
|
732
|
+
|
|
556
733
|
# Update shipping
|
|
557
734
|
await self.db.deliveries.shippings.update_one(
|
|
558
735
|
{'_id': shipping_id},
|
|
@@ -562,12 +739,12 @@ class GenerateRouteLambda(Lambda):
|
|
|
562
739
|
'status': 'scheduled'
|
|
563
740
|
}}
|
|
564
741
|
)
|
|
565
|
-
|
|
742
|
+
|
|
566
743
|
return {
|
|
567
744
|
'route_id': str(route_result.inserted_id),
|
|
568
745
|
'shipping_id': shipping_id
|
|
569
746
|
}
|
|
570
|
-
|
|
747
|
+
|
|
571
748
|
async def calculate_optimal_route(self, shipping):
|
|
572
749
|
# Your route calculation logic here
|
|
573
750
|
return {
|
|
@@ -595,48 +772,89 @@ __all__ = ['generate_route_handler']
|
|
|
595
772
|
- Can retry lambda independently if it fails
|
|
596
773
|
- Scalable architecture
|
|
597
774
|
|
|
775
|
+
## 🏗️ Architecture Overview
|
|
776
|
+
|
|
777
|
+
Typical flow for event-driven architectures using this framework:
|
|
778
|
+
|
|
779
|
+
```
|
|
780
|
+
┌──────────┐ ┌─────────────┐ ┌──────────────────────────────────────┐
|
|
781
|
+
│ Client │────▶│ API Gateway │────▶│ Lambda: api_handler │
|
|
782
|
+
└──────────┘ └─────────────┘ │ (src/api/resource/post.py) │
|
|
783
|
+
│ → validates, queries MongoDB, │
|
|
784
|
+
│ publishes to SNS │
|
|
785
|
+
└────────────────┬─────────────────────┘
|
|
786
|
+
│
|
|
787
|
+
▼
|
|
788
|
+
┌─────────────────┐
|
|
789
|
+
│ SNS Topic │
|
|
790
|
+
│ (fanout/filter) │
|
|
791
|
+
└────────┬────────┘
|
|
792
|
+
┌───────────────┼───────────────┐
|
|
793
|
+
▼ ▼ ▼
|
|
794
|
+
┌────────────┐ ┌────────────┐ ┌────────────┐
|
|
795
|
+
│ SQS Queue │ │ SQS Queue │ │ SQS Queue │
|
|
796
|
+
│ Platform A│ │ Platform B│ │ Platform C│
|
|
797
|
+
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘
|
|
798
|
+
│ │ │
|
|
799
|
+
▼ ▼ ▼
|
|
800
|
+
┌──────────────────────────────────────────────┐
|
|
801
|
+
│ Lambda: sqs_handler │
|
|
802
|
+
│ (src/consumer/platform_consumer.py) │
|
|
803
|
+
│ → groups messages, acquires sessions, │
|
|
804
|
+
│ launches Fargate tasks │
|
|
805
|
+
└───────────────────┬──────────────────────────┘
|
|
806
|
+
│ FargateExecutor.run_task()
|
|
807
|
+
▼
|
|
808
|
+
┌──────────────────────────────────────────────┐
|
|
809
|
+
│ Fargate Task: fargate_handler │
|
|
810
|
+
│ (src/task/my-task/task.py) │
|
|
811
|
+
│ → scrapes/processes data, │
|
|
812
|
+
│ writes results to MongoDB │
|
|
813
|
+
└──────────────────────────────────────────────┘
|
|
814
|
+
```
|
|
815
|
+
|
|
598
816
|
## 🔐 Environment Variables
|
|
599
817
|
|
|
600
818
|
### MongoDB Configuration
|
|
601
819
|
|
|
602
|
-
|
|
820
|
+
The framework supports two ways to configure MongoDB:
|
|
603
821
|
|
|
604
|
-
####
|
|
822
|
+
#### Option 1: Full Connection String
|
|
605
823
|
|
|
606
824
|
```bash
|
|
607
|
-
# URI
|
|
825
|
+
# Full URI with embedded credentials
|
|
608
826
|
MONGODB_URI=mongodb+srv://user:password@cluster.mongodb.net/dbname?retryWrites=true&w=majority
|
|
609
|
-
#
|
|
827
|
+
# or
|
|
610
828
|
MONGO_DB_URI=mongodb+srv://user:password@cluster.mongodb.net/dbname
|
|
611
829
|
```
|
|
612
830
|
|
|
613
|
-
####
|
|
831
|
+
#### Option 2: Separate Components (Recommended for Terraform)
|
|
614
832
|
|
|
615
833
|
```bash
|
|
616
|
-
# Host
|
|
834
|
+
# Host without credentials
|
|
617
835
|
MONGO_DB_HOST=mongodb+srv://cluster.mongodb.net
|
|
618
836
|
|
|
619
|
-
#
|
|
837
|
+
# Credentials (more secure)
|
|
620
838
|
MONGO_DB_USER=admin
|
|
621
839
|
MONGO_DB_PASSWORD=my-secure-password
|
|
622
840
|
|
|
623
|
-
#
|
|
841
|
+
# Optional
|
|
624
842
|
MONGO_DB_NAME=my_database
|
|
625
843
|
MONGO_DB_OPTIONS=retryWrites=true&w=majority
|
|
626
844
|
```
|
|
627
845
|
|
|
628
|
-
**
|
|
629
|
-
- ✅
|
|
630
|
-
- ✅
|
|
631
|
-
- ✅
|
|
632
|
-
- ✅
|
|
846
|
+
**Benefits of separate components:**
|
|
847
|
+
- ✅ Better security: credentials separate from host
|
|
848
|
+
- ✅ Easy integration with Terraform/AWS Secrets Manager
|
|
849
|
+
- ✅ Passwords with special characters are handled automatically
|
|
850
|
+
- ✅ More flexible for different environments
|
|
633
851
|
|
|
634
|
-
|
|
635
|
-
1. URL-
|
|
636
|
-
2.
|
|
637
|
-
3.
|
|
852
|
+
The framework automatically:
|
|
853
|
+
1. URL-encodes the password (handles `@`, `:`, `/`, etc.)
|
|
854
|
+
2. Builds the full URI
|
|
855
|
+
3. Initializes the connection
|
|
638
856
|
|
|
639
|
-
|
|
857
|
+
#### Terraform Example
|
|
640
858
|
|
|
641
859
|
```hcl
|
|
642
860
|
environment_variables = {
|
|
@@ -646,22 +864,119 @@ environment_variables = {
|
|
|
646
864
|
}
|
|
647
865
|
```
|
|
648
866
|
|
|
649
|
-
|
|
867
|
+
### All Environment Variables
|
|
868
|
+
|
|
869
|
+
| Variable | Required | Description |
|
|
870
|
+
|----------|----------|-------------|
|
|
871
|
+
| `MONGODB_URI` or `MONGO_DB_URI` | One of these or components below | Full MongoDB connection string |
|
|
872
|
+
| `MONGO_DB_HOST` | Alt. to URI | MongoDB host (e.g. `mongodb+srv://cluster.net`) |
|
|
873
|
+
| `MONGO_DB_USER` | Alt. to URI | MongoDB username |
|
|
874
|
+
| `MONGO_DB_PASSWORD` | Alt. to URI | MongoDB password |
|
|
875
|
+
| `MONGO_DB_NAME` | Optional | Default database name |
|
|
876
|
+
| `MONGO_DB_OPTIONS` | Optional | Connection options (e.g. `retryWrites=true&w=majority`) |
|
|
877
|
+
| `EXTERNAL_MONGODB_CONNECTIONS` | Optional | JSON array of external cluster configurations |
|
|
878
|
+
| `REQUIRE_AUTH` | Optional | Enable authentication middleware (`true`/`false`) |
|
|
879
|
+
| `AUTH_DB_NAME` | If `REQUIRE_AUTH=true` | MongoDB database for token validation |
|
|
880
|
+
| `AUTH_BYPASS_TOKEN` | Optional | Master token to bypass authentication |
|
|
881
|
+
| `ECS_CLUSTER` | Fargate only | ECS cluster name for `FargateExecutor` |
|
|
882
|
+
| `ECS_SUBNETS` | Fargate only | Comma-separated subnet IDs for Fargate tasks |
|
|
883
|
+
| `AWS_REGION` | Fargate/SNS/SQS | AWS region |
|
|
884
|
+
| `AWS_ACCOUNT_ID` | SQS `get_queue_url` | AWS account ID |
|
|
885
|
+
| `SERVICE_NAME` | SQS `get_queue_url` | Service name prefix for queue name |
|
|
886
|
+
| `QUEUE_NAME` | SQS `get_queue_url` | Queue name segment |
|
|
887
|
+
| `ENV` | SQS `get_queue_url` | Environment suffix (e.g. `prod`, `dev`) |
|
|
650
888
|
|
|
651
889
|
## 📊 Advanced Features
|
|
652
890
|
|
|
891
|
+
### SQS Consumer - Batch Mode
|
|
892
|
+
|
|
893
|
+
By default, consumers process messages one by one (`"single"` mode). Use `"batch"` mode when you need to group or bulk-process messages:
|
|
894
|
+
|
|
895
|
+
```python
|
|
896
|
+
from aws_python_helper.sqs.consumer_base import SQSConsumer
|
|
897
|
+
|
|
898
|
+
class OrderConsumer(SQSConsumer):
|
|
899
|
+
|
|
900
|
+
@property
|
|
901
|
+
def processing_mode(self) -> str:
|
|
902
|
+
return "batch"
|
|
903
|
+
|
|
904
|
+
async def process_batch(self, records):
|
|
905
|
+
# Group records by some key before processing
|
|
906
|
+
grouped = {}
|
|
907
|
+
for record in records:
|
|
908
|
+
message_id = record.get('messageId')
|
|
909
|
+
body = self.extract_content_message(record)
|
|
910
|
+
key = body.get('region', 'default')
|
|
911
|
+
grouped.setdefault(key, []).append((message_id, body))
|
|
912
|
+
|
|
913
|
+
for region, messages in grouped.items():
|
|
914
|
+
try:
|
|
915
|
+
# Bulk operation for the whole group
|
|
916
|
+
docs = [msg[1] for msg in messages]
|
|
917
|
+
await self.db.orders_db.orders.insert_many(docs)
|
|
918
|
+
except Exception as e:
|
|
919
|
+
# Mark individual messages as failed
|
|
920
|
+
for message_id, _ in messages:
|
|
921
|
+
self.add_message_failed(message_id, str(e))
|
|
922
|
+
```
|
|
923
|
+
|
|
924
|
+
**Key methods in SQSConsumer:**
|
|
925
|
+
|
|
926
|
+
| Method / Property | Description |
|
|
927
|
+
|-------------------|-------------|
|
|
928
|
+
| `self.extract_content_message(record)` | Parse message body (handles SNS → SQS wrapping automatically) |
|
|
929
|
+
| `self.parse_body(record)` | Alias for `extract_content_message` |
|
|
930
|
+
| `self.add_message_failed(message_id, error)` | Mark a message for retry (batch mode) |
|
|
931
|
+
| `self.get_queue_url()` | Get the SQS queue URL (uses `AWS_REGION`, `AWS_ACCOUNT_ID`, `SERVICE_NAME`, `QUEUE_NAME`, `ENV`) |
|
|
932
|
+
| `self.db` | Access to main MongoDB cluster |
|
|
933
|
+
| `self.external_db` | Access to external MongoDB clusters |
|
|
934
|
+
|
|
935
|
+
**Retry behavior:**
|
|
936
|
+
- Messages marked with `add_message_failed()` are reported via `reportBatchItemFailures`
|
|
937
|
+
- AWS SQS retries **only** the failed messages, not the whole batch
|
|
938
|
+
- Successful messages in the same batch are not retried
|
|
939
|
+
|
|
653
940
|
### SNS Publisher - Batch Publishing
|
|
654
941
|
|
|
655
942
|
```python
|
|
656
|
-
# Publish multiple messages
|
|
657
943
|
topic = TitleIndexedTopic()
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
661
|
-
{'
|
|
944
|
+
|
|
945
|
+
# Publish multiple messages in a single call
|
|
946
|
+
await topic.publish([
|
|
947
|
+
{'content': {'id': 'id1', 'title': 'Title 1'}, 'attributes': {'type': 'created'}},
|
|
948
|
+
{'content': {'id': 'id2', 'title': 'Title 2'}, 'attributes': {'type': 'updated'}},
|
|
949
|
+
{'content': {'id': 'id3', 'title': 'Title 3'}}, # attributes are optional
|
|
662
950
|
])
|
|
663
951
|
```
|
|
664
952
|
|
|
953
|
+
### SNS - Message Attributes for Filtering
|
|
954
|
+
|
|
955
|
+
Use `attributes` to filter which SQS subscriptions receive each message:
|
|
956
|
+
|
|
957
|
+
```python
|
|
958
|
+
class EventTopic(SNSPublisher):
|
|
959
|
+
def __init__(self):
|
|
960
|
+
super().__init__(topic_arn=os.getenv('EVENTS_SNS_TOPIC_ARN'))
|
|
961
|
+
|
|
962
|
+
def build_message(self, payload, event_type, priority='normal'):
|
|
963
|
+
return {
|
|
964
|
+
'content': payload,
|
|
965
|
+
'attributes': {
|
|
966
|
+
'event_type': event_type, # SQS subscriptions can filter on this
|
|
967
|
+
'priority': priority
|
|
968
|
+
}
|
|
969
|
+
}
|
|
970
|
+
|
|
971
|
+
# Usage
|
|
972
|
+
topic = EventTopic()
|
|
973
|
+
await topic.publish(topic.build_message(
|
|
974
|
+
payload={'order_id': '123', 'amount': 99.99},
|
|
975
|
+
event_type='order_created',
|
|
976
|
+
priority='high'
|
|
977
|
+
))
|
|
978
|
+
```
|
|
979
|
+
|
|
665
980
|
### Fargate - Run multiple tasks
|
|
666
981
|
|
|
667
982
|
```python
|
|
@@ -669,9 +984,9 @@ executor = FargateExecutor()
|
|
|
669
984
|
task_arns = executor.run_task_batch(
|
|
670
985
|
'search-tax-by-town',
|
|
671
986
|
[
|
|
672
|
-
{'
|
|
673
|
-
{'
|
|
674
|
-
{'
|
|
987
|
+
{'TOWN': 'Norwalk'},
|
|
988
|
+
{'TOWN': 'Stamford'},
|
|
989
|
+
{'TOWN': 'Bridgeport'}
|
|
675
990
|
]
|
|
676
991
|
)
|
|
677
992
|
```
|
|
@@ -680,7 +995,7 @@ task_arns = executor.run_task_batch(
|
|
|
680
995
|
|
|
681
996
|
```python
|
|
682
997
|
executor = FargateExecutor()
|
|
683
|
-
task_arn = executor.run_task('my-task', {'
|
|
998
|
+
task_arn = executor.run_task('my-task', {'PARAM': 'value'})
|
|
684
999
|
|
|
685
1000
|
# Check task status
|
|
686
1001
|
status = executor.get_task_status(task_arn)
|
|
@@ -688,19 +1003,51 @@ print(f"Status: {status['status']}")
|
|
|
688
1003
|
print(f"Started at: {status['started_at']}")
|
|
689
1004
|
```
|
|
690
1005
|
|
|
691
|
-
###
|
|
1006
|
+
### JSON Utilities for MongoDB Types
|
|
1007
|
+
|
|
1008
|
+
When returning MongoDB documents in API responses or exporting data, use the built-in serializers to handle `ObjectId`, `datetime`, `Decimal128`, and other BSON types:
|
|
692
1009
|
|
|
693
1010
|
```python
|
|
694
|
-
|
|
695
|
-
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
1011
|
+
import json
|
|
1012
|
+
from aws_python_helper.utils.json_encoder import MongoJSONEncoder, mongo_json_dumps
|
|
1013
|
+
from aws_python_helper.utils.serializer import serialize_mongo_types
|
|
1014
|
+
|
|
1015
|
+
# Use as json.dumps cls parameter
|
|
1016
|
+
json_str = json.dumps(my_mongo_doc, cls=MongoJSONEncoder)
|
|
1017
|
+
|
|
1018
|
+
# Helper function
|
|
1019
|
+
json_str = mongo_json_dumps(my_mongo_doc)
|
|
1020
|
+
|
|
1021
|
+
# Convert a document in-place (dict → JSON-serializable dict)
|
|
1022
|
+
clean_doc = serialize_mongo_types(my_mongo_doc)
|
|
1023
|
+
```
|
|
1024
|
+
|
|
1025
|
+
Types automatically converted:
|
|
1026
|
+
|
|
1027
|
+
| MongoDB Type | Converts to |
|
|
1028
|
+
|-------------|-------------|
|
|
1029
|
+
| `ObjectId` | `str` |
|
|
1030
|
+
| `datetime` | ISO 8601 string |
|
|
1031
|
+
| `date` | ISO 8601 string |
|
|
1032
|
+
| `Decimal128` | `float` |
|
|
1033
|
+
| `Decimal` | `float` |
|
|
1034
|
+
| `Binary` | base64 `str` |
|
|
1035
|
+
| `UUID` | `str` |
|
|
1036
|
+
| `bytes` | base64 `str` |
|
|
1037
|
+
| `set` | `list` |
|
|
1038
|
+
|
|
1039
|
+
**Common use case** — exporting query results to JSON files:
|
|
1040
|
+
|
|
1041
|
+
```python
|
|
1042
|
+
from aws_python_helper.utils.json_encoder import MongoJSONEncoder
|
|
1043
|
+
|
|
1044
|
+
class ExportResultsAPI(API):
|
|
1045
|
+
async def process(self):
|
|
1046
|
+
records = await self.db.orders_db.orders.find({}).to_list(1000)
|
|
1047
|
+
|
|
1048
|
+
# Write to file with MongoJSONEncoder
|
|
1049
|
+
with open('/tmp/export.json', 'w') as f:
|
|
1050
|
+
json.dump(records, f, cls=MongoJSONEncoder, ensure_ascii=False, indent=2)
|
|
704
1051
|
```
|
|
705
1052
|
|
|
706
1053
|
## 🤝 Contributing
|