shoryuken-waiter 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1d56266965274c49921dfd4ea1f07a9d98efd639
-  data.tar.gz: 2688764475bc65913eb61258cc12b9e18cfe8edd
+  metadata.gz: 47dca392390acc823bec52e81a7e30dc0ea1efc2
+  data.tar.gz: 6ad3144086dff7404d74e0a0742ca4b6c4537723
 SHA512:
-  metadata.gz: 6c165785551d2b79e5856020e3da32eb8c6f2f6ee214b7e384d432c65608edf7781eb9c880a8045d3f1dbbe91e6e7f84f2de22d61d57c7286231bd6a0c287a1d
-  data.tar.gz: 42ee5dfc75de31630864ae601252c2f0e66709c4102e251ec130001fa8f4a138628fd87cab987551f8924c0a772cce6a5f2a38306a200ba7e7a3ec1a75f8508a
+  metadata.gz: 888f44672933d04b697606832924612d37c9377230a8544929138164b4f0983e5f07b5c3f1fc82508fcab1ed1d2a7962b243bf42d907d746f1a39661a2f59cc2
+  data.tar.gz: 0f6221228bba8f6cdc9af999bc9e5f93b97f054a5e9c4ef32ba592aac92f50406bc1a906aade0d7ad95908bb2fb6dc8c2a85dbd5b6c6aabcb4a7a3fed546f1ae

data/.gitignore

@@ -9,3 +9,4 @@
 /spec/reports/
 /tmp/
 /vendor/bundle/
+/*.gem

data/README.md

@@ -4,18 +4,72 @@
 [![Dependency Status](https://gemnasium.com/farski/shoryuken-waiter.svg)](https://gemnasium.com/farski/shoryuken-waiter)
 [![Build Status](https://travis-ci.org/farski/shoryuken-waiter.svg)](https://travis-ci.org/farski/shoryuken-waiter)
 [![Code Climate](https://codeclimate.com/github/farski/shoryuken-waiter/badges/gpa.svg)](https://codeclimate.com/github/farski/shoryuken-waiter)
-[![Coverage Status](https://coveralls.io/repos/farski/shoryuken-waiter/badge.svg?branch=master)](https://coveralls.io/r/farski/shoryuken-waiter?branch=master)
+[![Coverage Status](https://coveralls.io/repos/farski/shoryuken-waiter/badge.svg?branch=master&service=github)](https://coveralls.io/github/farski/shoryuken-waiter?branch=master)
 
-Based heavily on [`shoryuken-later`](https://github.com/joekhoobyar/shoryuken-later).
+Based heavily on the concept of [`shoryuken-later`](https://github.com/joekhoobyar/shoryuken-later), `Shoryuken::Waiter` allows jobs to be scheduled greater that 15 minutes into the future when using [Shoryuken](https://github.com/phstc/shoryuken).
 
-Version 0.x is tightly coupled Rails and Shoryuken SQS queues. 1.x should add support for Shoryuken workers (currently only Active Job is supported), and more configurable DynamoDB tables.
+_**Notice:** Version 0.x is tightly coupled Rails and Shoryuken SQS queues. 1.x should add support for Shoryuken workers (currently only Active Job is supported), and more configurable DynamoDB tables._
 
-Waiter runs whenever Shoryuken is running. It watches DynamoDB tables that correspond with every Shoryuken queue defined, _if_ that DynamoDB table exists when Waiter starts up. If there are queues in Shoryuken called `queue_one` and `queue_two`, Waiter will look for `queue_one_waiter` and `queue_two_waiter` tables. It will silently ignore any missing tables.
+## Usage
 
-When a job is scheduled with a delay of over 15 minutes, the Active Job adpater will store a record of the job (the SQS message that Shoryuken would have queued) in the table that corresponds to the queue the job would have been placed in.
+### Integration with ActiveJob
+
+Because **[SQS](https://aws.amazon.com/sqs/)** only allows messages to be [delayed up to 15 minutes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html), the [`Active Job`](http://guides.rubyonrails.org/active_job_basics.html) adapter that ships with `Shoryuken` does not allow jobs to be scheduled further than 15 minutes out. `Shoryuken::Waiter` provides a replacement adapter that wraps the `Shoryuken` adapter, and allows for jobs to be scheduled arbitrarily far into the future.
+```
+# config/application.rb
+config.active_job.queue_adapter = :shoryuken_waiter
+```
+
+If a job is created with a delay of more than 15 minutes, **and** `Shoryuken::Waiter` finds a **[DynamoDB](https://aws.amazon.com/dynamodb/)** table whose name matches the name of the queue for which the job was intended, the properties of the job are captured to that table, allowing it to be rescheduled later on. If a matching table can't be found **or** the job's is 15 minutes or less the native `Shoryuken` adapters handles it.
+
+### The schedule poller
+
+There is no separate poller process that is run to watch for scheduled jobs. Whenever the `Shoryuken` message processor starts up, a Celluloid supervisor maintains a single `Shoryuken::Waiter::Querier` that polls **DynamoDB** for jobs. The poller shuts down when `Shoryuken` stops.
+
+The frequency that tables are queried for scheduleable jobs is `5.minutes` by default. It can be customized by setting the `delay` property in the `Shoryuken` configuration.
+
+```
+# config/shoryuken.yml
+waiter:
+  delay: 30 # seconds
+```
+
+### Queues & Tables
+
+When `Shoryuken::Waiter` loads it tries to find a **DynamoDB** table for each **SQS** `Shoryuken` queue that is configured. When a table with a matching name can't be found, `Shoryuken::Waiter` will not poll for jobs intended for that queue, and creating jobs intended for that queue is handled by the the `Shoryuken` `Active Job` adapter.
+
+_**Note:** There is no technical reason that each queue needs it's own table. It would be more cost effective to put all jobs into a single table, so that may be a change that is made in the future._
+
+#### Creating DynamoDB Tables
+
+Tables being used with `Shoryuken::Waiter` must be created with certain properties to work correctly.
+
+* **Table name**: The name of the table must match the name of an **SQS** queue registered with `Shoryuken`
+* **Primary key**: The `item key` must be a `String` with the value `scheduler`, and the `sort key` must be a `String` with the value `job_id`
+* **Secondary index**: An index (generally a *Local Secondary Index*) must be added to the table. The `item key` of the index's **primary key** must be a `String` with the value `scheduler`, and the `sort key` of the **primary key** must be a `Number` with the value `perform_at`. The **index name** must be `scheduler-perform_at-index`. **Projected attributes** generally can be set to `all`.
+
+Other properties of the table, such as the **provisioned capacity** will be application dependent.
+
+## Internals
+
+When a job is scheduled normally with `Shoryuken`, the message is sent to **SQS** with four properties `queue_url`, `message_body`, `delay_seconds`, and `message_attributes`.
+
+When the job must be delayed, `Shoryuken::Waiter` captures enough information to recreate an identical **SQS** message later on. The `message_body` and `message_attributes` are stored unaltered. `delay_seconds` is tranformed into a timestamp, relative to when the job was created, which can be used to query items from a **DynamoDB** table. The message's `queue_url` is discarded, since it can be recreated from the job's queue name, which is already captured as part of the `message_body`.
 
 ### Query
 
-To efficiently query tables for items that contain jobs ready to be handed back to SQS, there must be a common hash key across all items. This key is effectively meaningless in the implementation, and is not returned with the items. A job's `job_id` is used as the item's range key, to ensure each item's primary key is unique.
+After a job is delayed and stored in a **DynamoDB** table, it must be retrieved at the appropriate time, and sent to **SQS** like was originally intended.
+
+The poller needs a way of efficiently finding jobs in tables, and the **DynamoDB** API provides the `query` operation for that purpose.
+
+The `query` operation can either search based on just a **hash key**, or a **hash key** and a **range key**. `Shoryuken::Waiter` is looking for all jobs that are scheduled before a certain point in time (15 minutes from the time of the query). It's only possible to make comparison queries with a **range key**, so using just a hash key would not work. Since the hash key is required for all query operations, but there is no meaningful key in this case, it's value can be arbitrary, as long as it is consistent.
+
+Using the job's timestamp as the **range key** would be reasonable, except that more than one job may be scheduled for the same time. Since all items are being given the same **hash key**, identical **range keys** would violate the primary key uniqueness constraint. Instead of using a value from the table's primary key for the comparison query, a **secondary index** can be added to the table, which also allows for **range keys** to be queried.
+
+The **range key** of the table's primary key simply needs to provide uniqueness, so the job's unique ID can be used for that. The **hash key** of the secondary index must match the hash key of the table, so it will also be the arbitrary value that is selected. The **range key** will be the `perform_at` timestamp, and since secondary indexes do not require primary keys to be unique, there is no risk of collision between jobs with the same timestamp.
+
+A query can now be performed to find any item in the table with a given **hash key** (the arbitrary value shared by all items in the table) and a **range key** that only returns jobs scheduled no later than 15 minutes from now.
+
+If the query returns any items, they must be turned back into **SQS** messages and sent to their intended queue. The stored values are pulled back out of the item, and transformed if necessary (e.g. turning `perform_at` back to `delay_seconds`).
 
-A local secondary index is used to support querying based on the time jobs are scheduled. An LSI that shares hash keys with the table and uses the `perform_at` as the range key makes this possible. Index primary keys do not need to be unique, which prevents any issues where multiple jobs have the same `perform_at` value.
+The query returns sets of items, and **SQS** can be sent sets of messages (at most 10). The batches of items are processed into batches of messages, which get sent back to `Shoryuken`, and ultimately to **SQS**.

data/lib/shoryuken/waiter.rb

@@ -44,9 +44,11 @@ end
 
 require "shoryuken/waiter/extensions/active_job_adapter" if defined? ::ActiveJob
 
-Shoryuken.on_start do
-  tables = Shoryuken::Waiter.tables
-  queues = Shoryuken.queues.uniq
-  Shoryuken.logger.info { "[Shoryuken::Waiter] Starting. Polling #{tables.count} tables for #{queues.count} queues" }
-  Shoryuken::Waiter::Querier.supervise_as :shoryuken_waiter_querier
+Shoryuken.configure_server do |config|
+  config.on(:startup) do
+    tables = Shoryuken::Waiter.tables
+    queues = Shoryuken.queues.uniq
+    Shoryuken.logger.info { "[Shoryuken::Waiter] Starting. Polling #{tables.count} tables for #{queues.count} queues" }
+    Shoryuken::Waiter::Querier.supervise_as :shoryuken_waiter_querier
+  end
 end

data/lib/shoryuken/waiter/version.rb

@@ -1,5 +1,5 @@
 module Shoryuken
   module Waiter
-    VERSION = "0.0.1"
+    VERSION = "0.0.2"
   end
 end

data/shoryuken-waiter.gemspec

@@ -21,12 +21,12 @@ Gem::Specification.new do |spec|
 
   spec.add_development_dependency "bundler", "~> 1.10"
   spec.add_development_dependency "rake", "~> 10.0"
-  spec.add_development_dependency "minitest"
+  spec.add_development_dependency "minitest", "~> 5.8"
   spec.add_development_dependency "coveralls", "~> 0"
-  spec.add_development_dependency "pry"
-  spec.add_development_dependency "dotenv"
-  spec.add_development_dependency "rubocop"
+  spec.add_development_dependency "pry", "~> 0.10"
+  spec.add_development_dependency "dotenv", "~> 2.0"
+  spec.add_development_dependency "rubocop", "~> 0.34"
 
-  spec.add_dependency "shoryuken", "~> 2.0.0"
+  spec.add_dependency "shoryuken", "~> 2"
   spec.add_dependency "aws-sdk", "~> 2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: shoryuken-waiter
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
 platform: ruby
 authors:
 - Chris Kalafarski
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-10-10 00:00:00.000000000 Z
+date: 2015-10-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -42,16 +42,16 @@ dependencies:
   name: minitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '5.8'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '5.8'
 - !ruby/object:Gem::Dependency
   name: coveralls
   requirement: !ruby/object:Gem::Requirement
@@ -70,58 +70,58 @@ dependencies:
   name: pry
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.10'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.10'
 - !ruby/object:Gem::Dependency
   name: dotenv
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '2.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: rubocop
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.34'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.34'
 - !ruby/object:Gem::Dependency
   name: shoryuken
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: '2'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: '2'
 - !ruby/object:Gem::Dependency
   name: aws-sdk
   requirement: !ruby/object:Gem::Requirement