lambdakiq 1.0.4 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f7c135b6e0287693cd9578f31b12517e005d534d26fe6e97018a7de1860b894a
-  data.tar.gz: b0c89e6ed8bc0686700ddfa20591cf1d5b61253ff3f6539393bab9e293f3e9ed
+  metadata.gz: 94804599e9c2e4f8f701c34cee61a7d18db8f9a20fee85a3e24681e2bd555076
+  data.tar.gz: 7550236dacfc649a36468bbc4b3a7116a03b90d5943df4cdb0c08f7521ad505f
 SHA512:
-  metadata.gz: 72dd47f3e94b10f3311cffe28a42d6fbc9df564f3767bae47f2b85ad13bab5671fae199df5e3ac054d68e1a887dc076a9a2024a35ac9ac91ad76a01184bdb028
-  data.tar.gz: fa0f5c5087f28e19ee204963237798a610257ecaf410e5efb59eda65513390b29e73a2939d3f826bbdb10aaf2a8aefba2efe00109b63d6010c30ef16148051b2
+  metadata.gz: e7592aaa326a0ed6fa6202c0877b61ed67f00d156078d37840dda13ee4a455ff4de1ac35a377dd95230288fdc6ae054b974ef990190c0a46b59a45eb81c10ee5
+  data.tar.gz: 2f8efaa743a20710fd57ad2aa1b0cf8983828100b6ec5880f0815320f186216a5420e362d24b1b21b4e0ae78187c75e517afc4e27ddf5f9d3c3b295803101c16

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 See this http://keepachangelog.com link for information on how we want this documented formatted.
 
+## v2.0.0
+
+#### Changed
+
+- Leverage new `ReportBatchItemFailures` feature of SQS.
+
 ## v1.0.2, v1.0.3, v1.0.4
 
 #### Fixed

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    lambdakiq (1.0.4)
+    lambdakiq (2.0.0)
       activejob
       aws-sdk-sqs
       concurrent-ruby

data/README.md

@@ -30,7 +30,7 @@ Add the Lambdakiq gem to your `Gemfile`.
 gem 'lambdakiq'
 ```
 
-Open `config/initializers/production.rb` and set Lambdakiq as your ActiveJob queue adapter.
+Open `config/environments/production.rb` and set Lambdakiq as your ActiveJob queue adapter.
 
 ```ruby
 config.active_job.queue_adapter = :lambdakiq
@@ -45,6 +45,17 @@ class ApplicationJob < ActiveJob::Base
 end
 ```
 
+Using ActionMailer's built-in deliver job with ActiveJob? Make sure to include the Lambdakiq worker and set the queue name depending on your Rails version. You can do this in a newly created `config/initializers//action_mailer.rb` or another initializer of your choice.
+
+```ruby
+# Rails 5.x
+ActionMailer::DeliveryJob.include Lambdakiq::Worker
+ActionMailer::DeliveryJob.queue_as ENV['JOBS_QUEUE_NAME']
+# Rails 6.x
+ActionMailer::MailDeliveryJob.include Lambdakiq::Worker
+ActionMailer::MailDeliveryJob.queue_as ENV['JOBS_QUEUE_NAME']
+```
+
 The same Docker image will be used for both your `web` and `jobs` functions (example setup in following sections) which means the same `app.rb` handler would be used. The [Lamby](https://lamby.custominktech.com) gem automatically detects if Lambdakiq is being used so the following handler works as is.
 
 ```ruby
@@ -133,6 +144,8 @@ JobsLambda:
         Properties:
           Queue: !GetAtt JobsQueue.Arn
           BatchSize: 1
+          FunctionResponseTypes:
+            - ReportBatchItemFailures
     MemorySize: 1792
     PackageType: Image
     Policies:
@@ -149,9 +162,9 @@ JobsLambda:
 Here are some key aspects of our `JobsLambda` resource above:
 
 - The `Events` property uses the [SQS Type](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html).
-- Our [BatchSize](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html#sam-function-sqs-batchsize) is set to one so we can handle retrys more easily without worrying about idempotency in larger batches.
+- The [BatchSize](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html#sam-function-sqs-batchsize) can be any number you like. Less means more Lambda concurrency, more means some jobs could take longer. The jobs function `Timeout` must be lower than the `JobsQueue`'s `VisibilityTimeout` property. When the batch size is one, the queue's visibility is generally one second more.
+- You must use `ReportBatchItemFailures` response types. Lambdakiq assumes we are [reporting batch item failures](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting). This is a new feature of SQS introduced in [November 2021](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting).
 - The `Metadata`'s Docker properties must be the same as our web function except for the `DockerTag`. This is needed for the image to be shared. This works around a known [SAM issue](https://github.com/aws/aws-sam-cli/issues/2466) vs using the `ImageConfig` property.
-- The jobs function `Timeout` must be lower than the `JobsQueue`'s `VisibilityTimeout` property. When the batch size is one, the queue's visibility is generally one second more.
 
 🎉 Deploy your application and have fun with ActiveJob on SQS & Lambda.
 
@@ -181,12 +194,6 @@ end
 
 - `retry` - Overrides the default Lambdakiq `max_retries` for this one job.
 
-## Concurrency & Limits
-
-AWS SQS is highly scalable with [few limits](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-quotas.html). As your jobs in SQS increases so should your concurrent functions to process that work. However, as this article, ["Why isn't my Lambda function with an Amazon SQS event source scaling optimally?"](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sqs-scaling/) describes it is possible that errors will effect your concurrency.
-
-To help keep your queue and workers scalable, reduce the errors raised by your jobs. You an also reduce the retry count.
-
 ## Observability with CloudWatch
 
 Get ready to gain way more insights into your ActiveJobs using AWS' [CloudWatch](https://aws.amazon.com/cloudwatch/) service. Every AWS service, including SQS & Lambda, publishes detailed [CloudWatch Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html). This gem leverages [CloudWatch Embedded Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html) to add detailed ActiveJob metrics to that system. You can mix and match these data points to build your own [CloudWatch Dashboards](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html). If needed, any combination can be used to trigger [CloudWatch Alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html). Much like Sumo Logic, you can search & query for data using [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html).

data/lib/lambdakiq/error.rb

@@ -2,16 +2,6 @@ module Lambdakiq
   class Error < StandardError
   end
 
-  class JobError < Error
-    attr_reader :original_exception, :job
-
-    def initialize(error)
-      @original_exception = error
-      super(error.message)
-      set_backtrace Rails.backtrace_cleaner.clean(error.backtrace)
-    end
-  end
-
   class FifoDelayError < Error
     def initialize(error)
       super

data/lib/lambdakiq/job.rb

@@ -9,9 +9,9 @@ module Lambdakiq
         records = Event.records(event)
         jobs = records.map { |record| new(record) }
         jobs.each(&:perform)
-        jwerror = jobs.detect{ |j| j.error }
-        return unless jwerror
-        raise JobError.new(jwerror.error)
+        failed_jobs = jobs.select { |j| j.error }
+        item_failures = failed_jobs.map { |j| { itemIdentifier: j.provider_job_id } }
+        { batchItemFailures: item_failures }
       end
 
     end
@@ -40,10 +40,14 @@ module Lambdakiq
       active_job.executions
     end
 
+    def provider_job_id
+      active_job.provider_job_id
+    end
+
     def perform
       if fifo_delay?
         fifo_delay
-        raise FifoDelayError, active_job.job_id
+        return
       end
       execute
     end
@@ -104,6 +108,7 @@ module Lambdakiq
     end
 
     def fifo_delay
+      @error = FifoDelayError.new(active_job.job_id)
       params = client_params.merge visibility_timeout: record.fifo_delay_visibility_timeout
       client.change_message_visibility(params)
     end

data/lib/lambdakiq/version.rb

@@ -1,3 +1,3 @@
 module Lambdakiq
-  VERSION = '1.0.4'
+  VERSION = '2.0.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: lambdakiq
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 2.0.0
 platform: ruby
 authors:
 - Ken Collins
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-10-05 00:00:00.000000000 Z
+date: 2021-11-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob