lambdakiq 1.0.4 → 2.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/Gemfile.lock +1 -1
- data/README.md +16 -9
- data/lib/lambdakiq/error.rb +0 -10
- data/lib/lambdakiq/job.rb +9 -4
- data/lib/lambdakiq/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 94804599e9c2e4f8f701c34cee61a7d18db8f9a20fee85a3e24681e2bd555076
|
4
|
+
data.tar.gz: 7550236dacfc649a36468bbc4b3a7116a03b90d5943df4cdb0c08f7521ad505f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e7592aaa326a0ed6fa6202c0877b61ed67f00d156078d37840dda13ee4a455ff4de1ac35a377dd95230288fdc6ae054b974ef990190c0a46b59a45eb81c10ee5
|
7
|
+
data.tar.gz: 2f8efaa743a20710fd57ad2aa1b0cf8983828100b6ec5880f0815320f186216a5420e362d24b1b21b4e0ae78187c75e517afc4e27ddf5f9d3c3b295803101c16
|
data/CHANGELOG.md
CHANGED
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
@@ -30,7 +30,7 @@ Add the Lambdakiq gem to your `Gemfile`.
|
|
30
30
|
gem 'lambdakiq'
|
31
31
|
```
|
32
32
|
|
33
|
-
Open `config/
|
33
|
+
Open `config/environments/production.rb` and set Lambdakiq as your ActiveJob queue adapter.
|
34
34
|
|
35
35
|
```ruby
|
36
36
|
config.active_job.queue_adapter = :lambdakiq
|
@@ -45,6 +45,17 @@ class ApplicationJob < ActiveJob::Base
|
|
45
45
|
end
|
46
46
|
```
|
47
47
|
|
48
|
+
Using ActionMailer's built-in deliver job with ActiveJob? Make sure to include the Lambdakiq worker and set the queue name depending on your Rails version. You can do this in a newly created `config/initializers//action_mailer.rb` or another initializer of your choice.
|
49
|
+
|
50
|
+
```ruby
|
51
|
+
# Rails 5.x
|
52
|
+
ActionMailer::DeliveryJob.include Lambdakiq::Worker
|
53
|
+
ActionMailer::DeliveryJob.queue_as ENV['JOBS_QUEUE_NAME']
|
54
|
+
# Rails 6.x
|
55
|
+
ActionMailer::MailDeliveryJob.include Lambdakiq::Worker
|
56
|
+
ActionMailer::MailDeliveryJob.queue_as ENV['JOBS_QUEUE_NAME']
|
57
|
+
```
|
58
|
+
|
48
59
|
The same Docker image will be used for both your `web` and `jobs` functions (example setup in following sections) which means the same `app.rb` handler would be used. The [Lamby](https://lamby.custominktech.com) gem automatically detects if Lambdakiq is being used so the following handler works as is.
|
49
60
|
|
50
61
|
```ruby
|
@@ -133,6 +144,8 @@ JobsLambda:
|
|
133
144
|
Properties:
|
134
145
|
Queue: !GetAtt JobsQueue.Arn
|
135
146
|
BatchSize: 1
|
147
|
+
FunctionResponseTypes:
|
148
|
+
- ReportBatchItemFailures
|
136
149
|
MemorySize: 1792
|
137
150
|
PackageType: Image
|
138
151
|
Policies:
|
@@ -149,9 +162,9 @@ JobsLambda:
|
|
149
162
|
Here are some key aspects of our `JobsLambda` resource above:
|
150
163
|
|
151
164
|
- The `Events` property uses the [SQS Type](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html).
|
152
|
-
-
|
165
|
+
- The [BatchSize](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html#sam-function-sqs-batchsize) can be any number you like. Less means more Lambda concurrency, more means some jobs could take longer. The jobs function `Timeout` must be lower than the `JobsQueue`'s `VisibilityTimeout` property. When the batch size is one, the queue's visibility is generally one second more.
|
166
|
+
- You must use `ReportBatchItemFailures` response types. Lambdakiq assumes we are [reporting batch item failures](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting). This is a new feature of SQS introduced in [November 2021](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting).
|
153
167
|
- The `Metadata`'s Docker properties must be the same as our web function except for the `DockerTag`. This is needed for the image to be shared. This works around a known [SAM issue](https://github.com/aws/aws-sam-cli/issues/2466) vs using the `ImageConfig` property.
|
154
|
-
- The jobs function `Timeout` must be lower than the `JobsQueue`'s `VisibilityTimeout` property. When the batch size is one, the queue's visibility is generally one second more.
|
155
168
|
|
156
169
|
🎉 Deploy your application and have fun with ActiveJob on SQS & Lambda.
|
157
170
|
|
@@ -181,12 +194,6 @@ end
|
|
181
194
|
|
182
195
|
- `retry` - Overrides the default Lambdakiq `max_retries` for this one job.
|
183
196
|
|
184
|
-
## Concurrency & Limits
|
185
|
-
|
186
|
-
AWS SQS is highly scalable with [few limits](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-quotas.html). As your jobs in SQS increases so should your concurrent functions to process that work. However, as this article, ["Why isn't my Lambda function with an Amazon SQS event source scaling optimally?"](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sqs-scaling/) describes it is possible that errors will effect your concurrency.
|
187
|
-
|
188
|
-
To help keep your queue and workers scalable, reduce the errors raised by your jobs. You an also reduce the retry count.
|
189
|
-
|
190
197
|
## Observability with CloudWatch
|
191
198
|
|
192
199
|
Get ready to gain way more insights into your ActiveJobs using AWS' [CloudWatch](https://aws.amazon.com/cloudwatch/) service. Every AWS service, including SQS & Lambda, publishes detailed [CloudWatch Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html). This gem leverages [CloudWatch Embedded Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html) to add detailed ActiveJob metrics to that system. You can mix and match these data points to build your own [CloudWatch Dashboards](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html). If needed, any combination can be used to trigger [CloudWatch Alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html). Much like Sumo Logic, you can search & query for data using [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html).
|
data/lib/lambdakiq/error.rb
CHANGED
@@ -2,16 +2,6 @@ module Lambdakiq
|
|
2
2
|
class Error < StandardError
|
3
3
|
end
|
4
4
|
|
5
|
-
class JobError < Error
|
6
|
-
attr_reader :original_exception, :job
|
7
|
-
|
8
|
-
def initialize(error)
|
9
|
-
@original_exception = error
|
10
|
-
super(error.message)
|
11
|
-
set_backtrace Rails.backtrace_cleaner.clean(error.backtrace)
|
12
|
-
end
|
13
|
-
end
|
14
|
-
|
15
5
|
class FifoDelayError < Error
|
16
6
|
def initialize(error)
|
17
7
|
super
|
data/lib/lambdakiq/job.rb
CHANGED
@@ -9,9 +9,9 @@ module Lambdakiq
|
|
9
9
|
records = Event.records(event)
|
10
10
|
jobs = records.map { |record| new(record) }
|
11
11
|
jobs.each(&:perform)
|
12
|
-
|
13
|
-
|
14
|
-
|
12
|
+
failed_jobs = jobs.select { |j| j.error }
|
13
|
+
item_failures = failed_jobs.map { |j| { itemIdentifier: j.provider_job_id } }
|
14
|
+
{ batchItemFailures: item_failures }
|
15
15
|
end
|
16
16
|
|
17
17
|
end
|
@@ -40,10 +40,14 @@ module Lambdakiq
|
|
40
40
|
active_job.executions
|
41
41
|
end
|
42
42
|
|
43
|
+
def provider_job_id
|
44
|
+
active_job.provider_job_id
|
45
|
+
end
|
46
|
+
|
43
47
|
def perform
|
44
48
|
if fifo_delay?
|
45
49
|
fifo_delay
|
46
|
-
|
50
|
+
return
|
47
51
|
end
|
48
52
|
execute
|
49
53
|
end
|
@@ -104,6 +108,7 @@ module Lambdakiq
|
|
104
108
|
end
|
105
109
|
|
106
110
|
def fifo_delay
|
111
|
+
@error = FifoDelayError.new(active_job.job_id)
|
107
112
|
params = client_params.merge visibility_timeout: record.fifo_delay_visibility_timeout
|
108
113
|
client.change_message_visibility(params)
|
109
114
|
end
|
data/lib/lambdakiq/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: lambdakiq
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ken Collins
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-11-26 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|