lambdakiq 1.0.1 → 1.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -2
- data/Gemfile.lock +1 -1
- data/README.md +59 -42
- data/lib/lambdakiq/metrics.rb +5 -0
- data/lib/lambdakiq/version.rb +1 -1
- data/lib/lambdakiq/worker.rb +4 -0
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 66ee8bddcec048e20486b4c34b79650805407612f6b83edbb5dcfb4e436584b3
|
4
|
+
data.tar.gz: 91535d3c53e692e1f8c4d1426dd6543e1adf7dcba712028fa9b57d5e61e8cf4d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b2785043b2b96a82d9a0727d07edc7601673cf216d2e928418c203d1dc051a8e71bf95bef9dd23942a7477d1a585a8be135444b80fe6c4953ad2eaed3dec5f61
|
7
|
+
data.tar.gz: 9b4285d5881a4c335065edecc44e8b145b8ba15dc8881220784ad3c0ff9ab9ee7a65cd078efca27a02ce820812770ff947e1b521e2d887dfe476d4207196092a
|
data/CHANGELOG.md
CHANGED
@@ -1,10 +1,15 @@
|
|
1
|
-
|
2
1
|
# Keep A Changelog!
|
3
2
|
|
4
3
|
See this http://keepachangelog.com link for information on how we want this documented formatted.
|
5
4
|
|
5
|
+
## v1.0.2
|
6
|
+
|
7
|
+
#### Fixed
|
8
|
+
|
9
|
+
- Ensure `active_job` events are safe for non-Lambdakiq jobs.
|
10
|
+
|
6
11
|
## v1.0.0
|
7
12
|
|
8
13
|
#### Added
|
9
14
|
|
10
|
-
|
15
|
+
- Initial release.
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -1,7 +1,7 @@
|
|
1
|
-
![Test](https://github.com/customink/lambdakiq/workflows/Test/badge.svg)
|
2
|
-
|
3
1
|
![Lambdakiq: ActiveJob on SQS & Lambda](images/Lambdakiq.png)
|
4
2
|
|
3
|
+
![Test](https://github.com/customink/lambdakiq/workflows/Test/badge.svg)
|
4
|
+
|
5
5
|
# Lambdakiq
|
6
6
|
|
7
7
|
<a href="https://lamby.custominktech.com"><img src="https://user-images.githubusercontent.com/2381/59363668-89edeb80-8d03-11e9-9985-2ce14361b7e3.png" alt="Lamby: Simple Rails & AWS Lambda Integration using Rack." align="right" width="300" /></a>A drop-in replacement for [Sidekiq](https://github.com/mperham/sidekiq) when running Rails in AWS Lambda using the [Lamby](https://lamby.custominktech.com) gem.
|
@@ -10,19 +10,18 @@ Lambdakiq allows you to leverage AWS' managed infrastructure to the fullest exte
|
|
10
10
|
|
11
11
|
## Key Features
|
12
12
|
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
13
|
+
- Distinct web & jobs Lambda functions.
|
14
|
+
- AWS fully managed polling. Event-driven.
|
15
|
+
- Maximum 12 retries. Per job configurable.
|
16
|
+
- Mirror Sidekiq's retry [backoff](https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry) timing.
|
17
|
+
- Last retry is at 11 hours 30 minutes.
|
18
|
+
- Supports ActiveJob's wait/delay. Up to 15 minutes.
|
19
|
+
- Dead messages are stored for up to 14 days.
|
20
20
|
|
21
21
|
## Project Setup
|
22
22
|
|
23
23
|
This gem assumes your Rails application is on AWS Lambda, ideally with our [Lamby](https://lamby.custominktech.com) gem. It could be using Lambda's traditional zip package type or the newer [container](https://dev.to/aws-heroes/lambda-containers-with-rails-a-perfect-match-4lgb) format. If Rails on Lambda is new to you, consider following our [quick start](https://lamby.custominktech.com/docs/quick_start) guide to get your first application up and running. From there, to use Lambdakiq, here are steps to setup your project
|
24
24
|
|
25
|
-
|
26
25
|
### Bundle & Config
|
27
26
|
|
28
27
|
Add the Lambdakiq gem to your `Gemfile`.
|
@@ -46,6 +45,22 @@ class ApplicationJob < ActiveJob::Base
|
|
46
45
|
end
|
47
46
|
```
|
48
47
|
|
48
|
+
The same Docker image will be used for both your `web` and `jobs` functions (example setup in following sections) which means the same `app.rb` handler would be used. The [Lamby](https://lamby.custominktech.com) gem automatically detects if Lambdakiq is being used so the following handler works as is.
|
49
|
+
|
50
|
+
```ruby
|
51
|
+
def handler(event:, context:)
|
52
|
+
Lamby.handler $app, event, context
|
53
|
+
end
|
54
|
+
```
|
55
|
+
|
56
|
+
You can use the Lambdakiq handler directly in cases where your handler is a different method. Likewise there is a `Lambdakiq.jobs?(event)` helper function which returns true if the `messageAttributes` has a `lambdakiq` attribute.
|
57
|
+
|
58
|
+
```ruby
|
59
|
+
def jobs_handler(event:, context:)
|
60
|
+
Lambdakiq.handler(event)
|
61
|
+
end
|
62
|
+
```
|
63
|
+
|
49
64
|
### SQS Resources
|
50
65
|
|
51
66
|
Open up your project's SAM [`template.yaml`](https://lamby.custominktech.com/docs/anatomy#file-template-yaml) file and make the following additions and changes. First, we need to create your [SQS queues](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html) under the `Resources` section.
|
@@ -91,7 +106,7 @@ Both functions will need capabilities to access the SQS jobs queue. We can add o
|
|
91
106
|
|
92
107
|
```yaml
|
93
108
|
Policies:
|
94
|
-
- Version:
|
109
|
+
- Version: "2012-10-17"
|
95
110
|
Statement:
|
96
111
|
- Effect: Allow
|
97
112
|
Action:
|
@@ -100,6 +115,8 @@ Policies:
|
|
100
115
|
- !Sub arn:aws:sqs:${AWS::Region}:${AWS::AccountId}:${JobsQueue.QueueName}
|
101
116
|
```
|
102
117
|
|
118
|
+
### Overview
|
119
|
+
|
103
120
|
Now we can duplicate our `RailsLambda` resource YAML (except for the `Events` property) to a new `JobsLambda` one. This gives us a distinct Lambda function to process jobs whose events, memory, timeout, and more can be independently tuned. However, both the `web` and `jobs` functions will use the same ECR container image!
|
104
121
|
|
105
122
|
```yaml
|
@@ -119,7 +136,7 @@ JobsLambda:
|
|
119
136
|
MemorySize: 1792
|
120
137
|
PackageType: Image
|
121
138
|
Policies:
|
122
|
-
- Version:
|
139
|
+
- Version: "2012-10-17"
|
123
140
|
Statement:
|
124
141
|
- Effect: Allow
|
125
142
|
Action:
|
@@ -131,10 +148,10 @@ JobsLambda:
|
|
131
148
|
|
132
149
|
Here are some key aspects of our `JobsLambda` resource above:
|
133
150
|
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
151
|
+
- The `Events` property uses the [SQS Type](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html).
|
152
|
+
- Our [BatchSize](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html#sam-function-sqs-batchsize) is set to one so we can handle retrys more easily without worrying about idempotency in larger batches.
|
153
|
+
- The `Metadata`'s Docker properties must be the same as our web function except for the `DockerTag`. This is needed for the image to be shared. This works around a known [SAM issue](https://github.com/aws/aws-sam-cli/issues/2466) vs using the `ImageConfig` property.
|
154
|
+
- The jobs function `Timeout` must be lower than the `JobsQueue`'s `VisibilityTimeout` property. When the batch size is one, the queue's visibility is generally one second more.
|
138
155
|
|
139
156
|
🎉 Deploy your application and have fun with ActiveJob on SQS & Lambda.
|
140
157
|
|
@@ -148,9 +165,9 @@ Most general Lambdakiq configuration options are exposed via the Rails standard
|
|
148
165
|
config.lambdakiq
|
149
166
|
```
|
150
167
|
|
151
|
-
|
152
|
-
|
153
|
-
|
168
|
+
- `max_retries=` - Retries for all jobs. Default is the Lambdakiq maximum of `12`.
|
169
|
+
- `metrics_namespace=` - The CloudWatch Embedded Metrics namespace. Default is `Lambdakiq`.
|
170
|
+
- `metrics_logger=` - Set to the Rails logger which is STDOUT via Lamby/Lambda.
|
154
171
|
|
155
172
|
### ActiveJob Configs
|
156
173
|
|
@@ -162,7 +179,7 @@ class OrderProcessorJob < ApplicationJob
|
|
162
179
|
end
|
163
180
|
```
|
164
181
|
|
165
|
-
|
182
|
+
- `retry` - Overrides the default Lambdakiq `max_retries` for this one job.
|
166
183
|
|
167
184
|
## Concurrency & Limits
|
168
185
|
|
@@ -180,42 +197,43 @@ Metrics are published under the `Lambdakiq` namespace. This is configurable usin
|
|
180
197
|
|
181
198
|
### Metric Dimensions
|
182
199
|
|
183
|
-
|
184
|
-
|
185
|
-
|
200
|
+
- `AppName` - This is the name of your Rails application. Ex: `MyApp`
|
201
|
+
- `JobEvent` - Name of the ActiveSupport Notification. Ex: `*.active_job`.
|
202
|
+
- `JobName` - The class name of the ActiveSupport job. Ex: `NotificationJob`
|
186
203
|
|
187
204
|
### ActiveJob Event Names
|
205
|
+
|
188
206
|
For reference, here are the `JobEvent` names published by ActiveSupport. A few of these are instrumented by Lambdakiq since we use custom retry logic like Sidekiq. These event/metrics are found in the Rails application CloudWatch logs because they publish/enqueue jobs.
|
189
207
|
|
190
|
-
|
191
|
-
|
208
|
+
- `enqueue.active_job`
|
209
|
+
- `enqueue_at.active_job`
|
192
210
|
|
193
211
|
While these event/metrics can be found in the jobs function's log.
|
194
212
|
|
195
|
-
|
196
|
-
|
197
|
-
|
198
|
-
|
213
|
+
- `perform_start.active_job`
|
214
|
+
- `perform.active_job`
|
215
|
+
- `enqueue_retry.active_job`
|
216
|
+
- `retry_stopped.active_job`
|
199
217
|
|
200
218
|
### Metric Properties
|
201
219
|
|
202
220
|
These are the properties published with each metric. Remember, properties can not be used as metric data in charts but can be searched using [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html).
|
203
221
|
|
204
|
-
|
205
|
-
|
206
|
-
|
207
|
-
|
208
|
-
|
209
|
-
|
210
|
-
|
222
|
+
- `JobId` - ActiveJob Unique ID. Ex: `9f3b6977-6afc-4769-aed6-bab1ad9a0df5`
|
223
|
+
- `QueueName` - SQS Queue Name. Ex: `myapp-JobsQueue-14F18LG6XFUW5.fifo`
|
224
|
+
- `MessageId` - SQS Message ID. Ex: `5653246d-dc5e-4c95-9583-b6b83ec78602`
|
225
|
+
- `ExceptionName` - Class name of error raised. Present in perform and retry events.
|
226
|
+
- `EnqueuedAt` - When ActiveJob enqueued the message. Ex: `2021-01-14T01:43:38Z`
|
227
|
+
- `Executions` - The number of current executions. Counts from `1` and up.
|
228
|
+
- `JobArg#{n}` - Enumerated serialized arguments.
|
211
229
|
|
212
230
|
### Metric Data
|
213
231
|
|
214
232
|
And finally, here are the metrics which each dimension can chart using [CloudWatch Metrics & Dashboards](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html).
|
215
233
|
|
216
|
-
|
217
|
-
|
218
|
-
|
234
|
+
- `Duration` - Of the job event in milliseconds.
|
235
|
+
- `Count` - Of the event.
|
236
|
+
- `ExceptionCount` - Of the event. Useful with `ExceptionName`.
|
219
237
|
|
220
238
|
### CloudWatch Dashboard Examples
|
221
239
|
|
@@ -223,15 +241,14 @@ Please share how you are using CloudWatch to monitor and/or alert on your Active
|
|
223
241
|
|
224
242
|
💬 https://github.com/customink/lambdakiq/discussions/3
|
225
243
|
|
226
|
-
|
227
244
|
## Common Questions
|
228
245
|
|
229
246
|
**Are Scheduled Jobs Supported?** - No. If you need a scheduled job please use the [SAM Schedule](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-schedule.html) event source which invokes your function with an [Eventbridege AWS::Events::Rule](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html).
|
230
247
|
|
231
248
|
**Are FIFO Queues Supported?** - Yes. When you create your [AWS::SQS::Queue](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html) resources you can set the [FifoQueue](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-sqs-queues.html#aws-sqs-queue-fifoqueue) property to `true`. Remember that both your jobs queue and the redrive queue must be the same. When using FIFO we:
|
232
249
|
|
233
|
-
|
234
|
-
|
250
|
+
- Simulate `delay_seconds` for ActiveJob's wait by using visibility timeouts under the hood. We still cap it to non-FIFO's 15 minutes.
|
251
|
+
- Set both the messages `message_group_id` and `message_deduplication_id` to the unique job id provided by ActiveJob.
|
235
252
|
|
236
253
|
**Can I Use Multiple Queues?** - Yes. Nothing is stopping you from creating any number of queues and/or functions to process them. Your subclasses can use ActiveJob's `queue_as` method as needed. This is an easy way to handle job priorities too.
|
237
254
|
|
data/lib/lambdakiq/metrics.rb
CHANGED
@@ -16,6 +16,7 @@ module Lambdakiq
|
|
16
16
|
end
|
17
17
|
|
18
18
|
def log
|
19
|
+
return unless lambdakiq?
|
19
20
|
logger.info JSON.dump(message)
|
20
21
|
end
|
21
22
|
|
@@ -29,6 +30,10 @@ module Lambdakiq
|
|
29
30
|
job.class.name
|
30
31
|
end
|
31
32
|
|
33
|
+
def lambdakiq?
|
34
|
+
job.respond_to?(:lambdakiq?) && job.lambdakiq?
|
35
|
+
end
|
36
|
+
|
32
37
|
def logger
|
33
38
|
Lambdakiq.config.metrics_logger
|
34
39
|
end
|
data/lib/lambdakiq/version.rb
CHANGED
data/lib/lambdakiq/worker.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: lambdakiq
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.
|
4
|
+
version: 1.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ken Collins
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-10-04 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|