sqs_simplify 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: beb8c3464ca1465fe0dba02c9335ea205384e023648a569b7f896cd763630121
+  data.tar.gz: 571949adab26d1cf955941bd62b1a094b6919c6fdf7d4e32542d27d78d09569b
+SHA512:
+  metadata.gz: c4652f7a1fc21ea9e865335154d3f9b8a69fdfe3258d31ba202c71f89befca0f6d04a3b74e98a401fa9222981c0f232f60f36a2ddd5e9f31ab785d2c57921399
+  data.tar.gz: c4e58d330eadb2d0c3468f4e44e5f2ae4a932a9a12af0846494f7fdc11e72ba5cd3c49b1628b57d5a5c956b4309810f49549a69d2a86e7580329c81b1a839da2

data/.gitignore

@@ -0,0 +1,19 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/log
+/pkg/
+/spec/reports/
+/tmp/
+
+/.idea/*
+
+# rspec failure tracking
+.rspec_status
+/.rakeTasks
+/*.gem
+
+.ruby-version
+/Gemfile.lock

data/.rspec

@@ -0,0 +1,3 @@
+--format progress
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,6 @@
+---
+language: ruby
+cache: bundler
+rvm:
+  - 2.5.1
+before_install: gem install bundler -v 2.1.4

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## 0.2.0
+
+- Consumer interrupts a message that runs out of time with `SqsSimplify::Errors::ExecutionExpired` instead of `throw`, so open database transactions roll back instead of being committed halfway.
+- Consumer keeps the message (no delete) when the perform runs past its deadline, even if the application swallowed the timeout.
+- `group_id` support for FIFO queues in `Scheduler` and `Message`.
+- Worker options `worker_size` and `parallel_type` passed through the command line.
+- `SqsSimplify::Command#daemonize` renamed to `#run`; the `daemons` dependency was removed.
+- Dependencies relaxed to `aws-sdk-sqs ~> 1.0` and `parallel >= 1.20, < 3`; Ruby >= 3.0.
+
+## 0.1.0
+
+- Initial version.

data/Gemfile

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in sqs_simplify.gemspec
+gemspec
+
+gem 'rake', '~> 13.0'
+gem 'rspec', '~> 3.0'
+gem 'rubocop', require: false
+
+group :test do
+  gem 'simplecov', require: false
+end

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2021 ralph baesso
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,383 @@
+# SqsSimplify
+
+[![Gem Version](https://badge.fury.io/rb/sqs_simplify.svg)](https://badge.fury.io/rb/sqs_simplify)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.0-red.svg)](https://www.ruby-lang.org)
+
+A high-level Ruby DSL on top of `aws-sdk-sqs` for producing and consuming AWS SQS messages with minimal boilerplate.
+
+It has 3 main roles:
+* **SqsSimplify::Scheduler**: Sends messages to a queue.
+* **SqsSimplify::Consumer**: Consumes messages from a queue.
+* **SqsSimplify::Job**: Sends and consumes messages from a queue.
+
+> 🇧🇷 Uma versão deste documento em português está disponível em [README.pt-br.md](README.pt-br.md).
+
+## Table of Contents
+
+* [Requirements](#requirements)
+* [Installation](#installation)
+* [How to use](#how-to-use)
+  * [Initial Configuration](#1-initial-configuration)
+  * [Scheduler](#2-scheduler)
+  * [Consumer](#3-consumer)
+  * [Job](#4-job)
+* [Configuration](#configuration)
+  * [Global Configuration](#1-global-configuration)
+  * [Hooks](#2-hooks)
+* [Background Process](#background-process)
+* [Advanced features](#advanced-features)
+* [Development](#development)
+* [Contributing](#contributing)
+* [License](#license)
+
+## Requirements
+
+* Ruby `>= 3.0.0`
+* Runtime dependencies (installed automatically with the gem):
+  * [`aws-sdk-sqs`](https://rubygems.org/gems/aws-sdk-sqs) `~> 1.116`
+  * [`parallel`](https://rubygems.org/gems/parallel) `~> 2.1`
+
+You also need valid AWS credentials with access to SQS.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'sqs_simplify'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install sqs_simplify
+___
+
+## How to use
+### 1. Initial Configuration
+
+Create a configuration file to be loaded when the application starts.
+
+Example: *sqs_simplify.rb*
+
+If it is a **Rails** application, create it at *config/initializers/sqs_simplify.rb*.
+
+In this file you can configure your AWS credentials and other customizations.
+
+```ruby
+# sqs_simplify.rb
+
+SqsSimplify.configure do |config|
+  config.access_key_id = ENV['AWS_ACCESS_KEY_ID']
+  config.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+  config.region = 'us-east-1'
+
+  config.queue_prefix = Rails.env # optional
+  config.queue_suffix = 'my_application_name' # optional
+end
+```
+___
+
+
+### 2. Scheduler
+
+The Scheduler component is responsible for sending messages to the SQS queue.
+
+Its main focus is using a queue as a bus between two distinct applications.
+
+For example: application **A** has a Scheduler that sends a message to the SQS queue, but the one that will consume this message is application **B**.
+
+```ruby
+# app/jobs/my_scheduler
+
+class MyScheduler < SqsSimplify::Scheduler
+end
+
+
+# app/model/wheel_factory.rb
+
+class WheelFactory
+  def send_now
+    message = { wheels: ['back_wheel', 'front_wheel'], type: 'motorcycle' }
+    MyScheduler.send_message(message: message)
+  end
+  
+  def send_later(delay)
+    message = { wheels: ['back_wheel', 'front_wheel'], type: 'motorcycle' }
+    MyScheduler.send_message(message: message, after: delay)
+  end
+end
+
+
+wheel_factory = WheelFactory.new
+
+# run now
+wheel_factory.send_now # "8685d169-f4a0-476b-b970-39ee055f957b"
+
+# run after 2 minutes
+wheel_factory.send_later(120) # "5ebd6a74-8571-43e2-a9c8-7866b7598765"
+
+```
+
+#### `group_id`
+
+The `group_id` parameter is sent to SQS as `message_group_id`. Provide it in
+`send_message`:
+
+```ruby
+MyScheduler.send_message(message: message, group_id: 'tenant-123')
+```
+
+Its meaning depends on the queue type:
+
+* **FIFO queues**: SQS requires `message_group_id`. Messages with the same
+  `group_id` are processed in order (FIFO).
+* **Standard queues**: the `message_group_id` is used as a *tenant* identifier
+  for the [fair queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html)
+  feature, which mitigates the *noisy neighbor* impact in multi-tenant queues.
+  Here it does **not** guarantee ordering — it only groups messages by tenant.
+
+If omitted (`nil`), no `message_group_id` is sent.
+
+### 3. Consumer
+
+The Consumer component is responsible for consuming messages from the SQS queue.
+
+Its main focus is also using a queue as a bus between two distinct applications.
+
+For example: application **B** has a Consumer that requests messages from the SQS queue that were sent by application **A**.
+
+```ruby
+# app/jobs/motorcycle_assembler.rb
+
+class MotorcycleAssembler < SqsSimplify::Consumer
+  
+  def perform
+    # your logic here
+    # your object has "message" method with data of sqs_message
+    p message # {:wheels=>["back_wheel", "front_wheel"], :type=>"motorcycle"}
+  end
+  
+end
+
+```
+
+
+### 4. Job
+
+The Job component is responsible for sending and consuming messages from the SQS queue within the same application.
+
+Unlike the other components, its focus is **not** on using a queue as a bus.
+
+For example: your application has a **Report** class that generates a report.
+This processing takes a long time to run.
+So you can schedule its execution for later.
+
+The business method **must** be named `perform`.
+
+```ruby
+# app/jobs/report.rb
+
+class Report < SqsSimplify::Job
+  
+  def perform(list)
+    # your logic here
+    PersistReport.save(list)
+  end
+  
+end
+
+list = # many data
+
+# enqueue for later execution
+Report.perform_later(list) # "76107a55-43d9-4f2e-b449-02c329a51692"
+
+# enqueue with a 180 second delay
+Report.new_job(after: 180).perform_later(list) # "be6837d5-c11f-495c-a03e-cb093011f1d0"
+
+# run inline, without enqueuing
+# note: it will not be scheduled
+Report.perform(list) # :executed
+```
+
+___
+
+## Configuration
+
+### 1. Global Configuration
+
+#### SqsSimplify
+Every configuration made on the SqsSimplify class will be applied to all components.
+
+Example:
+
+```ruby
+# sqs_simplify.rb
+
+SqsSimplify.configure do |config|
+  config.access_key_id = ENV['AWS_ACCESS_KEY_ID']
+  config.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+  config.region = 'us-east-2'
+
+  config.queue_prefix = 'production'
+end
+```
+
+With this configuration all components — Scheduler, Consumer and Job — will share the same configuration.
+
+All of them will have access to the SQS queues in *us-east-2*
+
+All of them will have the *production* prefix
+
+
+```ruby
+
+class MyScheduler < SqsSimplify::Scheduler
+  
+end
+
+class MyConsumer < SqsSimplify::Consumer
+
+end
+
+class MyJob < SqsSimplify::Job
+
+end
+
+# same prefix
+MyScheduler.queue_name # "production_my_scheduler"
+MyConsumer.queue_name # "production_my_consumer"
+MyJob.queue_name # "production_my_job"
+
+```
+
+### 2. Hooks
+
+**resolver_exception:** Invoked whenever an **Exception** occurs in your application.
+It provides two parameters:
+* the **first parameter** is an **Exception**.
+* the **second parameter** may vary depending on the component and where the **Exception** occurred.
+
+**message_not_deleted:** Invoked when a Job or Consumer picks up a message from the SQS queue but is unable to delete it.
+There are two main reasons for this to happen:
+* **Exception**: when an **Exception** occurs. Note: the **resolver_exception** hook will also be invoked.
+* **Default visibility timeout**: the message was not processed within the defined time.
+
+```ruby
+# sqs_simplify.rb
+
+SqsSimplify.configure do |config|
+  config.hooks.resolver_exception do |exception, args|
+    logger.info "Exception => #{exception.message}, args => #{args}"
+  end
+
+  config.hooks.message_not_deleted do |consumer|
+    logger.info "Consumer => #{consumer}"
+  end
+end
+```
+
+___
+
+## Background Process
+### 1. Setup
+To run the consuming process you must create a script file.
+The `run` method runs a foreground loop that consumes the queues until it
+receives `SIGINT` (Ctrl+C) — it does not daemonize the process.
+Example of a script file named *sqs_simplify*
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'sqs_simplify/command'
+SqsSimplify::Command.new(ARGV).run
+```
+
+For a Rails project you can load the application before invoking the GEM.
+Example of a script file named *bin/sqs_simplify*
+```ruby
+#!/usr/bin/env ruby
+
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
+require 'sqs_simplify/command'
+SqsSimplify::Command.new(ARGV).run
+```
+
+And you must grant execution permission.
+````bash
+$ chmod +x sqs_simplify
+````
+
+### 2. Commands
+To see the command options, run it in the file's directory:
+
+````bash
+$ sqs_simplify -h
+Usage: sqs_simplify [options]
+    -h, --help                       Show help
+    -n, --number_of_workers=workers  Number of unique workers to spawn
+    -e, --environment=environment    Environment
+        --queues=queues              queues that will be consumed
+        --priority                   with priority in the queues
+    -f, --fork                       parallel in processes
+    -t, --thread                     parallel in threads
+
+
+````
+
+___
+
+## Advanced features
+
+Beyond the basics above, the gem also supports:
+
+* **`map_queue(nickname, &block)`** — route a single Scheduler to alternate queues
+  dynamically (`SqsSimplify::Scheduler`).
+* **Per-call destination override** — pass `queue_url:` to `send_message` to send a
+  message to a specific queue at call time.
+* **Class-level `set` DSL** — override per-class settings such as the queue name,
+  visibility timeout, serialization (`dump_message`/`load_message`) and more, e.g.
+  `set :queue_name, 'custom_name'`.
+* **Automatic dead-letter queues** — every queue gets a paired `<name>_dead`
+  dead-letter queue.
+* **Additional hooks** — besides `resolver_exception` and `message_not_deleted`, the
+  pipeline also supports `before`/`after` (`:each` / `:all`) and `around` hooks.
+* **Testing without AWS** — set `config.faker = true` to use the in-memory
+  `FakerClient`, or `config.stub_responses = true` to stub the `aws-sdk-sqs` client.
+
+## Development
+
+After checking out the repo, install dependencies and run the test suite:
+
+```bash
+bin/setup                  # install dependencies
+bundle exec rake spec      # run the full test suite
+bundle exec rubocop        # lint
+bin/console                # interactive prompt to experiment
+```
+
+Tests run against an in-memory fake SQS client, so no real AWS access is required.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To
+release a new version, update the version number in `lib/sqs_simplify/version.rb`,
+then run `bundle exec rake release`.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/ralphsbaesso/sqs_simplify. To contribute:
+
+1. Fork the repository.
+2. Create a feature branch (`git checkout -b my-feature`).
+3. Commit your changes and make sure the tests (`bundle exec rake spec`) and linter
+   (`bundle exec rubocop`) pass.
+4. Open a pull request.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).