skyrunner 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 49176f855293135111d70129a79f8f6099d775f6
-  data.tar.gz: fe7cb68a221e708c668e49f950161f4db9d67542
+  metadata.gz: b828d1a9dbc216ff8aca55ce2e8007f07ff1c47d
+  data.tar.gz: 21ed7eb84af89b74578726cb69faadc981c6fdf6
 SHA512:
-  metadata.gz: ff7a55a80f9a5710ddddcec9b0df187d098557667e49db60c6a90386b7e006b9b08437cb02dd4791396169acc378d44dcf0bc633c582b11f6b62f751fd779538
-  data.tar.gz: 1df498dd88128acb06800004a76208292881c10e372d180cb2d7f38ef28cb7d8860821aa1f30ee11eaf28971bf2f35159fc7471b0734512d15eaf44cda0099b0
+  metadata.gz: f78e396093a0dcbe578b1976911b29e4eab5fdf01ecaffff0405ebbc7394b642923c5b3b47da2da6c4419168cbb8b5932646eddc3391a79c319e43b5b6e44e2a
+  data.tar.gz: 43e38c229e0f32c9cddc66ffa375657f96f1e2b8de18e26cf030c14e96a16d065cd813f895281b8eb7ab4cd95fc0807a778117b8f75a8dc8d1766fa0c3c9e25a

data/README.md

@@ -1,9 +1,9 @@
 SkyRunner
 =========
 
-SkyRunner is a simple job execution framework that lets you run jobs which are composed of small tasks using AWS SQS and DynamoDB. 
+SkyRunner is a simple job execution framework that lets you run jobs which are composed of small tasks. SkyRunner uses AWS SQS and DynamoDB for queueing and job coordination, so you don't need any servers other than your task consumers to run jobs. 
 
-The key feature provided is that once a job's last task completes, a completion method can be called so you can perform a post-processing step. For example, you may run a job that has 100 tasks that process image frames in parallel, and then when these tasks are completed you compose the frames into a video and upload it to S3.
+The key feature provided is that once a job's last task completes, a completion method can be called so you can perform a post-processing step. For example, you may run a job that has 100 tasks that process image frames in parallel, and then when these tasks are completed you compose the frames into a video and upload it to S3. Of course, you can also execute another job in this method, so you can easily chain jobs together.
 
 Add to your Gemfile:
 
@@ -19,21 +19,27 @@ bundle exec rails g sky_runner:install
 
 Customize `config/initializers/skyrunner.rb`. Update `lib/tasks/skyrunner.rake` to do what you want if there are exceptions during a task.
 
+Put your AWS keys in the usual environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. If you are running on EC2, you don't need to do this and SkyRunner will run within the machine's IAM role.
+
+Be sure the IAM permissions are granted to create DynamoDB tables and SQS queues, and to perform all read and write operations to the table and queue you've configured for SkyRunner.
+
 To initialize DynamoDB & SQS:
 
 ``
 bundle exec rake skyrunner:init
 ``
 
-To start a consumer
+To start a consumer:
 
 ``
 bundle exec rake skyrunner:consume
 ``
 
-To gracefully shut down a consumer, send it SIGINT. It will finish up processing the messages it has de-queued before terminating.
+To gracefully shut down a consumer, send it SIGINT (or hit Ctrl-C if you have a console.) It will finish up processing the tasks it has de-queued before terminating.
 
-See `jobs/example_job.rb` for an example job. To run a job, just call `execute!` on the job, passing any named job arguments you want. The job class should implement the method `run`. This method will get passed the job arguments. For each task you want consumers to run, `run` should yield an array of two elements, the first being the name of the method on the job class to run for the task, and the second a Hash of method arguments.
+See `jobs/example_job.rb` for an example job. To run a job, just call `execute!` on the job, passing any named job arguments you want. The job class should implement the method `run`. This method will get passed the job arguments. For each task you want consumers to run, `run` should yield an array of two elements, the first being the name of the method on the job class to run for the task, and the second a Hash of method arguments. Note that the consumer is (by default) multi-threaded, so please be sure your task methods are thread-safe.
 
 You can specify `on_complete` and `on_failure` method(s) to call when the tasks are all completed, or if any of them fail, respectively. These methods will also be passed the original job arguments. Importantly, the completion method is guaranteed to be called once and only once, when the final task has been completed.
 
+If any of your tasks fail in a job, consumers will stop consuming tasks for that job and deplete any queued tasks on SQS. SkyRunner does not currently retry failed tasks, for now you'll need to implement your own retry logic in your task method instead.
+

data/lib/skyrunner/version.rb

@@ -1,3 +1,3 @@
 module Skyrunner
-  VERSION = "0.1.2"
+  VERSION = "0.1.3"
 end

data/lib/skyrunner.rb

@@ -90,6 +90,7 @@ module SkyRunner
               job.consume!(task_args)
               message.delete
             rescue Exception => e
+              message.delete rescue nil
               error_queue.push(e)
               SkyRunner::log :error, "Task Failed: #{task_args} Job: #{job_id} #{e.message} #{e.backtrace.join("\n")}"
             end
@@ -135,9 +136,12 @@ module SkyRunner
               task_args = message_data["task_args"]
               local_queue.push([klass, job_id, task_args, message])
             rescue NameError => e
+              message.delete rescue nil
               log :error, "Task Failed: No such class #{record["class"]} #{e.message}"
               yield e if block_given?
             end
+          else
+            message.delete
           end
         end
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: skyrunner
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - Greg Fodor