iron_response 0.0.1 → 0.0.2

data/README.md

@@ -1,3 +1,5 @@
+[![Code Climate](https://codeclimate.com/github/adelevie/iron_response.png)](https://codeclimate.com/github/adelevie/iron_response) [![Gem Version](https://badge.fury.io/rb/iron_response.png)](http://badge.fury.io/rb/iron_response)
+
 # iron_response
 
 Provides a response object for IronWorkers.
@@ -5,40 +7,140 @@ Provides a response object for IronWorkers.
 ```ruby
 require "iron_response"
 
+config = {...}
 batch = IronResponse::Batch.new
-batch.config[:aws_s3] = {
-  access_key_id: 123, 
-  secret_access_key: 456, 
-  bucket: "iron_worker"
-}
-batch.config[:iron_io] = {
-  project_id: "abc", 
-  token: "defg"
-}
-batch.worker       = "path/to/worker/is_prime"
-batch.params_array = [1..1000].map {|i| {number: i}}
-results            = batch.run!
+
+batch.auto_update_worker = true
+batch.config[:iron_io]   = config[:iron_io]
+batch.config[:aws_s3]    = config[:aws_s3]
+batch.worker             = "test/workers/is_prime.rb"
+batch.params_array       = Array(1..10).map {|i| {number: i}}
+
+results                  = batch.run!
 
 p results
-#=> [true, true, true, false, true, false...]
+#=> [{"result"=>false}, {"result"=>true}, {"result"=>true}...]
 ```
 
 Assumes you have a worker file called `is_prime.rb`:
 ```ruby
 require "iron_response"
 
-IronResponse::Responder do
+IronResponse::Responder.new(binding) do
   def is_prime?(n)
     ("1" * n =~ /^1?$|^(11+?)\1+$/) == 0 ? false : true
   end
 
-  is_prime? params[:number]
+  {
+   result: is_prime?(params[:number])
+  }
 end
 ```
 
-## Installation
+## Rationale
+
+Iron.io's IronWorker is a great product that provides a lot of powerful concurrency options. With IronWorker, you can scale tasks to hundreds and even thousands of workers. However, IronWorker was missing one useful feature for me: responses. What do I mean? In the typical IronWorker setup, worker files are just one-off scripts that run independently of the client that queues them up. For example:
+
+```ruby
+client = IronWorkerNG::Client.new
+100.times do |i|
+  client.tasks.create("do_something", number: i)
+end
+```
+
+For many use cases, this is fine. But what if I want to know the result of `do_something`? A simple way to get the result would be for your worker to POST the final result somewhere, then have the client it. This gem simply abstracts that process away, allowing the developer to avoid boilerplate and to keep worker code elegant.
+
+Under the hood, `iron_response` uses some functional and meta-programming to capture the final expression of a worker file, convert it to JSON, and then POST it to Amazon S3. When all the workers in an `IronResponse::Batch` have finished, the gem retrieves the file and converts the JSON string back to Ruby.
 
-Don't install this yet. It's just a code sketch so far. Once this library works, these will be the installation instructions:
+This process means there a few important implications:
+
+- Response objects "sent" from workers should be JSON-parseable. This means sticking to basic Ruby objects and data structures such as `String`, `Fixnum`, `Hash`, and `Array`.
+- Though these objects must be parseable, they can be fairly large. Though I haven't figured out exactly what the limit is, you really are only constrained by IronWorkers RAM and HD limits, and the bandwidth between the client (your computer/server), IronWorker servers, and S3. In an ideal setup, all three of these components are inside of Amazon's cloud, allowing you to get very fast throughput.
+
+One final gotcha: I haven't yet implemented a clean way to use the `iron_worker_ng` gem to declare dependencies. This should be done soon, though.
+
+## Usage
+
+This gem requires a basic understanding of how to use [IronWorker with Ruby](https://github.com/iron-io/iron_worker_ruby_ng).
+
+Assuming you have an empy directory, called `foo`:
+
+```sh
+$ mkdir workers
+$ cd workers
+$ touch my_worker.rb
+```
+
+`my_worker.rb` should look like this:
+
+```ruby
+require "iron_response"
+
+IronResponse::Responder.new(binding) do
+  # your code here
+end
+```
+
+To run this worker, create at the top-level of `foo` files called `configuration.rb` and `enqueue.rb`:
+
+`configuration.rb`:
+```ruby
+class Configuration
+  def self.keys
+    {
+      iron_io: {
+        token:      "123",
+        project_id: "123"
+      },
+      aws_s3: {
+        access_key_id:     "123",
+        secret_access_key: "123",
+        bucket:            "iron_response"
+      }
+    }
+  end
+end
+```
+
+Obviously, fill in the appropriate API keys. It is highly recommended that you do not use your AWS master keys. Instead, go to the AWS Console, click on "IAM", and create a user with a policy that allows it to edit the bucket named in the configuration file. Here's an example policy:
+
+```json
+{
+  "Statement": [
+    {
+      "Action": "s3:*",
+      "Effect": "Allow",
+      "Resource": [
+        "arn:aws:s3:::iron_response",
+        "arn:aws:s3:::iron_response/*"
+      ]
+    }
+  ]
+}
+```
+
+Now, write your queueing script:
+
+`enqueue.rb`:
+```ruby
+require_relative "configuration"
+require "iron_response"
+
+config = Configuration.keys
+batch = IronResponse::Batch.new
+
+batch.auto_update_worker = true
+batch.config             = config
+batch.worker             = "workers/my_worker.rb"
+
+# The `params_array` is an Array of Hashes 
+# that get sent as the payload to IronWorker scripts.
+batch.params_array       = Array ("a".."z").map {|i| {letter: i}}
+
+results                  = batch.run!
+```
+
+## Installation
 
 Add this line to your application's Gemfile:
 

data/iron_response.gemspec

@@ -16,6 +16,8 @@ Gem::Specification.new do |gem|
   gem.version       = IronResponse::VERSION
 
   gem.add_dependency "iron_worker_ng"
+  gem.add_dependency "aws-s3"
   gem.add_development_dependency "minitest", "~> 4.7.3"
   gem.add_development_dependency "pry"
+  gem.add_development_dependency "pry-rescue"
 end

data/lib/iron_response/version.rb

@@ -1,3 +1,3 @@
 module IronResponse
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/lib/iron_response.rb

@@ -1,14 +1,31 @@
 require "iron_response/version"
 require "iron_worker_ng"
+require "aws/s3"
+require "json"
 
 module IronResponse
+  module Protocol
+    S3_PATH = "tasks"
+
+    def Protocol.s3_path(task_id)
+      "#{S3_PATH}/#{task_id}.json"
+    end
+  end
+
   class Responder
-    def initialize(&block)
-      send_data_to_s3(block.call)
+    def initialize(binding, &block)
+      task_id = eval("@iron_task_id", binding)
+      params  = eval("params", binding)
+      send_data_to_s3(params, task_id, block.call)
     end
 
-    def send_data_to_s3(data)
-      p "pretending to send #{data} to s3"
+    def send_data_to_s3(params, task_id, data)
+      aws_s3 = params[:aws_s3]
+      AWS::S3::Base.establish_connection! access_key_id:     aws_s3[:access_key_id],
+                                          secret_access_key: aws_s3[:secret_access_key]
+      path = IronResponse::Protocol.s3_path(task_id)
+      bucket_name = params[:aws_s3][:bucket]
+      AWS::S3::S3Object.store(path, data.to_json, bucket_name)
     end
   end
 
@@ -23,7 +40,7 @@ module IronResponse
     end
 
     def worker_name
-      @worker.split("/").last
+      @worker.split("/").last.split(".rb").first
     end
 
     def run!
@@ -33,18 +50,27 @@ module IronResponse
         create_code!
       end
 
-      pids = params_array.map do |params|
+      task_ids = params_array.map do |params|
         params[:aws_s3] = @config[:aws_s3]
         @client.tasks.create(worker_name, params)._id
       end
 
-      pids.map do |pid|
-        get_response_from_pid(@client.tasks.wait_for(pid))
+      task_ids.map do |task_id|
+        get_response_from_task_id(@client.tasks.wait_for(task_id)._id)
       end
     end
 
-    def get_response_from_pid(pid)
-      "FakeResponseFromPid:#{pid}"
+    def get_response_from_task_id(task_id)
+      aws_s3 = @config[:aws_s3]
+      AWS::S3::Base.establish_connection! access_key_id:     aws_s3[:access_key_id],
+                                          secret_access_key: aws_s3[:secret_access_key]
+
+      bucket_name = @config[:aws_s3][:bucket]
+      bucket      = AWS::S3::Bucket.find(bucket_name)
+      path        = IronResponse::Protocol.s3_path(task_id)
+      response    = bucket[path].value
+
+      JSON.parse(response)
     end
 
     def create_code!

data/test/{rdd_test.rb → synopsis_test.rb}

@@ -1,7 +1,7 @@
 require "test_helper"
 
-class RddTest < MiniTest::Unit::TestCase
-  def test_readme
+class SynopsisTest < MiniTest::Unit::TestCase
+  def test_synopsis
 
     config = Configuration.keys
     batch = IronResponse::Batch.new
@@ -10,11 +10,11 @@ class RddTest < MiniTest::Unit::TestCase
     batch.config[:iron_io]   = config[:iron_io]
     batch.config[:aws_s3]    = config[:aws_s3]
     batch.worker             = "test/workers/is_prime.rb"
-    batch.params_array       = Array(1..10).map {|i| {number: i}}
+    batch.params_array       = Array(1..20).map {|i| {number: i}}
     
     results                  = batch.run!
 
-    p results
     assert_equal Array, results.class
+    assert_equal batch.params_array.length, results.length
   end
 end

data/test/test_helper.rb

@@ -1,6 +1,7 @@
 require_relative "configuration"
 require "minitest/unit"
 require "minitest/autorun"
+require "pry-rescue/minitest"
 require "minitest/pride"
 require "pry"
 require "iron_response"

data/test/workers/is_prime.rb

@@ -1,12 +1,11 @@
 require "iron_response"
 
-IronResponse::Responder.new do
+IronResponse::Responder.new(binding) do
   def is_prime?(n)
     ("1" * n =~ /^1?$|^(11+?)\1+$/) == 0 ? false : true
   end
 
-  result = is_prime?(params[:number])
-  p result
-
-  return result
+  {
+   result: is_prime?(params[:number])
+  }
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: iron_response
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -27,6 +27,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: aws-s3
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
@@ -59,6 +75,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry-rescue
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Provides a response object for IronWorker tasks.
 email:
 - adelevie@gmail.com
@@ -75,7 +107,7 @@ files:
 - lib/iron_response.rb
 - lib/iron_response/version.rb
 - test/configuration.rb.example
-- test/rdd_test.rb
+- test/synopsis_test.rb
 - test/test_helper.rb
 - test/workers/is_prime.rb
 homepage: ''
@@ -104,7 +136,7 @@ specification_version: 3
 summary: Provides a response object for IronWorker tasks.
 test_files:
 - test/configuration.rb.example
-- test/rdd_test.rb
+- test/synopsis_test.rb
 - test/test_helper.rb
 - test/workers/is_prime.rb
 has_rdoc: