iron_response 0.0.3 → 0.1.0

data/README.md

@@ -2,17 +2,16 @@
 
 # iron_response
 
-Glues together IronWorker and AWS S3 to provide a response object to remote worker scripts. This allows you to write massively concurrent Ruby programs without worrying about threads.
+Provides a response object to remote worker scripts. This allows you to write massively concurrent Ruby programs without worrying about threads.
 
 ```ruby
 require "iron_response"
 
-config = {...}
+config = {token: "123", project_id: "456"}
 batch = IronResponse::Batch.new
 
 batch.auto_update_worker = true
 batch.config[:iron_io]   = config[:iron_io]
-batch.config[:aws_s3]    = config[:aws_s3]
 batch.worker             = "test/workers/is_prime.rb"
 batch.params_array       = Array(1..10).map {|i| {number: i}}
 
@@ -48,16 +47,16 @@ client = IronWorkerNG::Client.new
 end
 ```
 
-For many use cases, this is fine. But what if I want to know the result of `do_something`? A simple way to get the result would be for your worker to POST the final result somewhere, then have the client it. This gem simply abstracts that process away, allowing the developer to avoid boilerplate and to keep worker code elegant.
+For many use cases, this is fine. But what if I want to know the result of `do_something`? A simple way to get the result would be for your worker to POST the final result somewhere, then have the client retrieve it. This gem simply abstracts that process away, allowing the developer to avoid boilerplate and to keep worker code elegant.
 
 On top of all this, another benefit to using this gem is that it makes it much easier to test workers.
 
-Under the hood, `iron_response` uses some functional and meta-programming to capture the final expression of a worker file, convert it to JSON, and then POST it to Amazon S3. When all the workers in an `IronResponse::Batch` have finished, the gem retrieves the file and converts the JSON string back to Ruby.
+Under the hood, `iron_response` uses some functional and meta-programming to capture the final expression of a worker file, convert it to JSON, and then POST it to either IronCache or Amazon S3. When all the workers in an `IronResponse::Batch` have finished, the gem retrieves the file and converts the JSON string back to Ruby.
 
 This process means there a few important implications:
 
 - Response objects "sent" from workers should be JSON-parseable. This means sticking to basic Ruby objects and data structures such as `String`, `Fixnum`, `Hash`, and `Array`.
-- Though these objects must be parseable, they can be fairly large. Though I haven't figured out exactly what the limit is, you really are only constrained by IronWorkers RAM and HD limits, and the bandwidth between the client (your computer/server), IronWorker servers, and S3. In an ideal setup, all three of these components are inside of Amazon's cloud, allowing you to get very fast throughput.
+- If you're using IronCache (and not S3) for storage, response objects must be 1 MB or small.
 
 ## Usage
 
@@ -102,6 +101,8 @@ class Configuration
 end
 ```
 
+Of course, if you don't want to use S3, just leave that part out of the `Hash`. You can specify `:bucket` for S3 and `:cache` for for IronCache. Otherwise they both default to `"iron_response".`
+
 Obviously, fill in the appropriate API keys. It is highly recommended that you do not use your AWS master keys. Instead, go to the AWS Console, click on "IAM", and create a user with a policy that allows it to edit the bucket named in the configuration file. Here's an example policy:
 
 ```json

data/iron_response.gemspec

@@ -17,6 +17,7 @@ Gem::Specification.new do |gem|
 
   gem.add_dependency "iron_worker_ng"
   gem.add_dependency "aws-s3"
+  gem.add_dependency "iron_cache"
   gem.add_development_dependency "minitest", "~> 4.7.3"
   gem.add_development_dependency "pry"
   gem.add_development_dependency "pry-rescue"

data/lib/iron_response.rb

@@ -1,31 +1,72 @@
 require "iron_response/version"
 require "iron_worker_ng"
 require "aws/s3"
+require "iron_cache"
 require "json"
 
 module IronResponse
-  module Protocol
+  module Common
     S3_PATH = "tasks"
 
-    def Protocol.s3_path(task_id)
+    DEFAULT_S3_BUCKET = "iron_response"
+    DEFAULT_IRON_CACHE_CACHE_NAME = "iron_response"
+
+    def Common.s3_path(task_id)
       "#{S3_PATH}/#{task_id}.json"
     end
+
+    def Common.s3_bucket_name(config)
+      config[:aws_s3][:bucket].nil? ? DEFAULT_S3_BUCKET : @config[:aws_s3][:bucket]
+    end
+
+    def Common.iron_cache_key(task_id)
+      task_id
+    end
+
+    def Common.iron_cache_cache_name(config)
+      config[:iron_io][:cache].nil? ? DEFAULT_IRON_CACHE_CACHE_NAME : @config[:iron_io][:cache]
+    end
+
+    def Common.response_provider(config)
+      config[:aws_s3].nil? ? :iron_cache : :aws_s3
+    end
   end
 
-  class Responder
+  class Worker
     def initialize(binding, &block)
       task_id = eval("@iron_task_id", binding)
       params  = eval("params", binding)
-      send_data_to_s3(params, task_id, block.call)
+      @config = params[:config]
+
+      case IronResponse::Common.response_provider(@config)
+      when :iron_cache
+        send_data_to_iron_cache(params, task_id, block.call)
+      when :aws_s3
+        send_data_to_s3(params, task_id, block.call)
+      end
+    end
+
+    def send_data_to_iron_cache(params, task_id, data)
+      cache_client = IronCache::Client.new(@config[:iron_io])
+      cache_name   = IronResponse::Common.iron_cache_cache_name(@config)
+      cache        = cache_client.cache(cache_name)
+
+      key   = IronResponse::Common.iron_cache_key(task_id)
+      value = data.to_json
+
+      cache.put(key, value)
     end
 
     def send_data_to_s3(params, task_id, data)
-      aws_s3 = params[:aws_s3]
+      aws_s3 = @config[:aws_s3]
       AWS::S3::Base.establish_connection! access_key_id:     aws_s3[:access_key_id],
                                           secret_access_key: aws_s3[:secret_access_key]
-      path = IronResponse::Protocol.s3_path(task_id)
-      bucket_name = params[:aws_s3][:bucket]
-      AWS::S3::S3Object.store(path, data.to_json, bucket_name)
+      
+      path        = IronResponse::Common.s3_path(task_id)
+      bucket_name = IronResponse::Common.s3_bucket_name(@config)
+      value       = data.to_json
+
+      AWS::S3::S3Object.store(path, value, bucket_name)
     end
   end
 
@@ -51,7 +92,7 @@ module IronResponse
       end
 
       task_ids = params_array.map do |params|
-        params[:aws_s3] = @config[:aws_s3]
+        params[:config] = @config
         @client.tasks.create(worker_name, params)._id
       end
 
@@ -61,23 +102,44 @@ module IronResponse
     end
 
     def get_response_from_task_id(task_id)
+      case IronResponse::Common.response_provider(@config)
+      when :iron_cache
+        get_iron_cache_response(task_id)
+      when :aws_s3
+        get_aws_s3_response(task_id)
+      end
+    end
+
+    def get_aws_s3_response(task_id)
       aws_s3 = @config[:aws_s3]
       AWS::S3::Base.establish_connection! access_key_id:     aws_s3[:access_key_id],
                                           secret_access_key: aws_s3[:secret_access_key]
 
-      bucket_name = @config[:aws_s3][:bucket]
+      bucket_name = IronResponse::Common.s3_bucket_name(@config)
       bucket      = AWS::S3::Bucket.find(bucket_name)
-      path        = IronResponse::Protocol.s3_path(task_id)
+      path        = IronResponse::Common.s3_path(task_id)
       response    = bucket[path].value
 
       JSON.parse(response)
     end
 
+    def get_iron_cache_response(task_id)
+      cache_client = IronCache::Client.new(@config[:iron_io])
+      cache_name   = IronResponse::Common.iron_cache_cache_name(@config)
+      cache        = cache_client.cache(cache_name)
+
+      key   = IronResponse::Common.iron_cache_key(task_id)
+      value = cache.get(key).value
+
+      JSON.parse(value)
+    end
+
     def code
-      @code ||= IronWorkerNG::Code::Ruby.new(exec: @worker).tap do |c|
-        c.name = worker_name
-        c.merge_gem("iron_response")
-        c.runtime = "ruby"
+      if @code.nil?
+        @code = IronWorkerNG::Code::Ruby.new(exec: @worker)
+        @code.name = worker_name
+        @code.merge_gem("iron_response")
+        @code.runtime = "ruby"
       end
 
       @code

data/lib/iron_response/version.rb

@@ -1,3 +1,3 @@
 module IronResponse
-  VERSION = "0.0.3"
+  VERSION = "0.1.0"
 end

data/test/gem_dependency_test.rb

@@ -8,8 +8,10 @@ class GemDependencyTest < MiniTest::Unit::TestCase
     
     batch.auto_update_worker = true
     
-    batch.config[:iron_io]   = config[:iron_io]
-    batch.config[:aws_s3]    = config[:aws_s3]
+    batch.config = config
+
+    #batch.config[:iron_io]   = config[:iron_io]
+    #batch.config[:aws_s3]    = config[:aws_s3]
     batch.worker             = "test/workers/fcc_filings_counter.rb"
 
     batch.code.merge_gem("nokogiri", "< 1.6.0") # keeps the build times low

data/test/synopsis_test.rb

@@ -1,20 +1,36 @@
 require "test_helper"
 
 class SynopsisTest < MiniTest::Unit::TestCase
-  def test_synopsis
-
+  def test_synopsis_with_s3
     config = Configuration.keys
-    batch = IronResponse::Batch.new
+    batch  = IronResponse::Batch.new
     
     batch.auto_update_worker = true
     batch.config[:iron_io]   = config[:iron_io]
     batch.config[:aws_s3]    = config[:aws_s3]
-    batch.worker             = "test/workers/is_prime.rb"
-    batch.params_array       = Array(1..20).map {|i| {number: i}}
+    batch.worker             = "test/workers/hello.rb"
+    batch.params_array       = Array(1..4).map {|i| {number: i}}
+    
+    results                  = batch.run!
+
+    assert_equal Array, results.class
+    assert_equal batch.params_array.length, results.length
+    assert_equal true, results.select {|r| r.nil?}.length == 0
+  end
+
+  def test_synopsis_with_iron_cache
+    config = Configuration.keys
+    batch  = IronResponse::Batch.new
+    
+    batch.auto_update_worker = true
+    batch.config[:iron_io]   = config[:iron_io]
+    batch.worker             = "test/workers/hello.rb"
+    batch.params_array       = Array(1..4).map {|i| {number: i}}
     
     results                  = batch.run!
 
     assert_equal Array, results.class
     assert_equal batch.params_array.length, results.length
+    assert_equal true, results.select {|r| r.nil?}.length == 0
   end
 end

data/test/workers/hello.rb

@@ -0,0 +1,7 @@
+require "iron_response"
+
+IronResponse::Worker.new(binding) do
+  {
+    "result" => params
+  }
+end

data/test/workers/is_prime.rb

@@ -1,6 +1,6 @@
 require "iron_response"
 
-IronResponse::Responder.new(binding) do
+IronResponse::Worker.new(binding) do
   def is_prime?(n)
     ("1" * n =~ /^1?$|^(11+?)\1+$/) == 0 ? false : true
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: iron_response
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-10 00:00:00.000000000 Z
+date: 2013-08-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: iron_worker_ng
@@ -43,6 +43,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: iron_cache
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
@@ -111,6 +127,7 @@ files:
 - test/synopsis_test.rb
 - test/test_helper.rb
 - test/workers/fcc_filings_counter.rb
+- test/workers/hello.rb
 - test/workers/is_prime.rb
 homepage: ''
 licenses: []
@@ -142,5 +159,6 @@ test_files:
 - test/synopsis_test.rb
 - test/test_helper.rb
 - test/workers/fcc_filings_counter.rb
+- test/workers/hello.rb
 - test/workers/is_prime.rb
 has_rdoc: