girl_friday 0.9.1 → 0.9.2

data/.gitignore

@@ -2,3 +2,4 @@
 .bundle
 Gemfile.lock
 pkg/*
+.rbxdb/

data/.rvmrc

@@ -1,2 +1,3 @@
-rvm use rbx
+rvm use jruby@gf --create
 export RBXOPT=-Xrbc.db=~/.rbxdb
+export JRUBY_OPTS='--1.9 -X+O -J-Djruby.launch.inproc=false'

data/.travis.yml

@@ -0,0 +1,8 @@
+rvm:
+  - 1.9.2
+  - jruby
+  - rbx-2.0
+branches:
+  only:
+    - master
+env: JRUBY_OPTS='--1.9 -X+O -J-Djruby.launch.inproc=false'

data/Gemfile

@@ -6,3 +6,4 @@ gemspec
 # Needed for testing only!
 gem 'minitest'
 gem 'redis'
+gem 'flexmock-minitest'

data/History.md

@@ -1,14 +1,24 @@
 Changes
 ================
 
+0.9.2
+---------
+
+* Remove use of weakrefs to track queue instances, use ObjectSpace
+  instead.
+* Add support for Batch operations, providing an easy way to fan out
+  operations and then collect results when completed.
+* Added WorkQueue.immediate! and WorkQueue.queue! to switch background processing off and back on respectively. Nice to use when testing. (jc00ke, ryanlecompte)
+* Added some ajax updates to the girl\_friday status server. (jc00ke)
+
 0.9.1
 ---------
 
 * Lazy initialize the worker actors to avoid dead thread problems with Unicorn forking processes.
-* Add initial pass at girl_friday Rack server (see wiki).  It's awful looking, trust me, help wanted.
+* Add initial pass at girl\_friday Rack server (see wiki).  It's awful looking, trust me, help wanted.
 
 
 0.9.0
 ---------
 
-* Initial release
+* Initial release

data/README.md

@@ -1,29 +1,29 @@
-girl_friday
+girl\_friday
 ====================
 
-Have a task you want to get done sometime soon but don't want to do it yourself?  Give it to girl_friday!  From wikipedia:
+Have a task you want to get done sometime soon but don't want to do it yourself?  Give it to girl\_friday!  From wikipedia:
 
 > The term Man Friday has become an idiom, still in mainstream usage, to describe an especially faithful servant or
 > one's best servant or right-hand man. The female equivalent is Girl Friday. The title of the movie His Girl Friday
 > alludes to it and may have popularized it.
 
-girl_friday is a Ruby library for performing asynchronous tasks.  Often times you don't want to block a web response by performing some task, like sending an email, so you can just use this gem to perform it in the background.  It works with any Ruby application, including Rails 3 applications.
+girl\_friday is a Ruby library for performing asynchronous tasks.  Often times you don't want to block a web response by performing some task, like sending an email, so you can just use this gem to perform it in the background.  It works with any Ruby application, including Rails 3 applications.
 
 
 Installation
 ------------------
 
-We recommend using [JRuby 1.6+](http://jruby.org) or [Rubinius 2.0+](http://rubini.us) with girl_friday.  Both are excellent options for executing Ruby these days.
+We recommend using [JRuby 1.6+](http://jruby.org) or [Rubinius 2.0+](http://rubini.us) with girl\_friday.  Both are excellent options for executing Ruby these days.
 
     gem install girl_friday
 
-girl_friday does not support Ruby 1.8 (MRI) because of its poor threading support.  Ruby 1.9 will work reasonably well if you use gems that release the GIL for network I/O (mysql2 is a good example of this, do **not** use the original mysql gem).
+girl\_friday does not support Ruby 1.8 (MRI) because of its poor threading support.  Ruby 1.9 will work reasonably well if you use gems that release the GIL for network I/O (mysql2 is a good example of this, do **not** use the original mysql gem).
 
 
 Usage
 --------------------
 
-Put girl_friday in your Gemfile:
+Put girl\_friday in your Gemfile:
 
     gem 'girl_friday'
 
@@ -46,17 +46,19 @@ The msg parameter to push is just a Hash whose contents are completely up to you
 
 Your message processing block should **not** access any instance data or variables outside of the block.  That's shared mutable state and dangerous to touch!  I also strongly recommend your queue processor block be **VERY** short, ideally just a method call or two.  You can unit test those methods easily but not the processor block itself.
 
+You can call `GirlFriday::WorkQueue.immediate!` to process jobs immediately, which is helpful when testing. `GirlFriday::WorkQueue.queue!` will revert this & jobs will be processed by actors.
+
 
 More Detail
 --------------------
 
-Please see the [girl_friday wiki](https://github.com/mperham/girl_friday/wiki) for more detail and advanced options and tuning.  You'll find details on queue persistence with Redis, implementing clean shutdown, querying runtime metrics and SO MUCH MORE!
+Please see the [girl\_friday wiki](https://github.com/mperham/girl_friday/wiki) for more detail and advanced options and tuning.  You'll find details on queue persistence with Redis, implementing clean shutdown, querying runtime metrics and SO MUCH MORE!
 
 
 Thanks
 --------------------
 
-[Carbon Five](http://carbonfive.com), I write and maintain girl_friday on their clock.
+[Carbon Five](http://carbonfive.com), I write and maintain girl\_friday on their clock.
 
 This gem contains a copy of the Rubinius Actor API, modified to work on any Ruby VM.  Thanks to Evan Phoenix, MenTaLguY and the Rubinius project for permission to use and distribute this code.
 
@@ -64,4 +66,4 @@ This gem contains a copy of the Rubinius Actor API, modified to work on any Ruby
 Author
 --------------------
 
-Mike Perham, [@mperham](https://twitter.com/mperham), [mikeperham.com](http://mikeperham.com)
+Mike Perham, [@mperham](https://twitter.com/mperham), [mikeperham.com](http://mikeperham.com)

data/TODO.md

@@ -3,4 +3,3 @@ TODO
 
  - rufus-scheduler integration for scheduled tasks
  - web admin UI to surface status() metrics
- - nicer project homepage

data/examples/batch.rb

@@ -0,0 +1,42 @@
+require 'girl_friday'
+require 'open-uri'
+require 'benchmark'
+require 'nokogiri'
+
+class UrlProcessor
+  URLS = %w(http://www.bing.com http://www.google.com http://www.yahoo.com)
+
+  def parallel
+    batch = GirlFriday::Batch.new(URLS, :size => 3) do |url|
+      html = open(url)
+      doc = Nokogiri::HTML(html.read)
+      doc.css('span').count
+    end
+    p URLS.zip(batch.results)
+  end
+
+  def serial
+    results = URLS.map do |url|
+      html = open(url)
+      doc = Nokogiri::HTML(html.read)
+      doc.css('span').count
+    end
+    p URLS.zip(results)
+  end
+end
+
+# Expected output:
+# [["http://www.bing.com", 24], ["http://www.google.com", 8], ["http://www.yahoo.com", 172]]
+#
+# Benchmark results:
+# serial                   1.231000   0.000000   1.231000 (  1.231000)
+# parallel                 0.447000   0.000000   0.447000 (  0.447000)
+
+processor = UrlProcessor.new
+Benchmark.bm(25) do |x|
+  %w(serial parallel).each do |op|
+    x.report(op) do
+      processor.send(op.to_sym)
+    end
+  end
+end

data/examples/pipeline.rb

@@ -0,0 +1,89 @@
+require 'open-uri'
+require 'nokogiri'
+require 'girl_friday'
+
+
+##
+# In this example, we use girl_friday to implement a processing pipeline
+# for scraping large images from a website.  Given a URL, we want to fetch
+# the HTML for that URL, find all the images, download those images, discard images
+# which do not meet a size heuristic and save the ones that match.  This
+# processing is I/O-heavy and perfect for breaking into many threads.
+#
+# A processing pipeline is just a series of linked processing steps.
+# We create a girl_friday queue for each step, sized appropriately for how few/many parallel worker threads we want for that step.
+# The process_xxx methods implement the actual logic for the step.
+# The finish_xxx methods pass the result to the next step in the pipeline.
+#
+class ImagePipeline
+  def initialize
+    @download_html = GirlFriday::Queue.new(:download_html, :size => 5, &method(:process_html))
+    @extract_imgs = GirlFriday::Queue.new(:extract, :size => 2, &method(:process_extract))
+    @download_imgs = GirlFriday::Queue.new(:download_imgs, :size => 10, &method(:process_imgs))
+    @thumb = GirlFriday::Queue.new(:thumb_imgs, :size => 5, &method(:process_thumb))
+  end
+
+  def process(url)
+    log "Pushing #{url}"
+    @download_html.push({ :url => url }, &method(:finish_html))
+  end
+
+  private
+
+  def process_html(msg)
+    msg.merge(:htmlfile => open(msg[:url]))
+  end
+
+  def finish_html(result)
+    @extract_imgs.push(result, &method(:finish_extract))
+  end
+
+  def process_extract(msg)
+    doc = Nokogiri::HTML(msg[:htmlfile].read)
+    doc.css('img[src]').map{|n| n['src']}.select { |url| url =~ /^#{msg[:url]}/ }
+  end
+
+  def finish_extract(result)
+    result.each do |imgurl|
+      @download_imgs.push(imgurl, &method(:finish_imgs))
+    end
+  end
+
+  def process_imgs(msg)
+    log "Fetching image: #{msg}"
+    imgfile = open msg
+    return if imgfile.size < 20_000 # ignore images less than 20k
+    result = `identify #{imgfile.path}`
+    log "Image: #{result}"
+    return unless result =~ /(\d+)x(\d+)\+0\+0/
+    return if Integer($1) + Integer($2) < 500
+    # Passed all our heuristics, pass it on!
+    imgfile
+  end
+
+  def finish_imgs(result)
+    return unless result
+    @thumb.push(result, &method(:finish_thumb))
+  end
+
+  def process_thumb(msg)
+    FileUtils.cp msg.path, Time.now.to_f.to_s
+    msg.path
+  end
+
+  def finish_thumb(result)
+    log "Finished image at #{result}"
+  end
+
+  def log(msg)
+    print "#{Thread.current}: #{msg}\n"
+  end
+end
+
+
+pipeline = ImagePipeline.new
+pipeline.process 'http://blog.carbonfive.com'
+
+loop do
+  sleep 1
+end

data/girl_friday.gemspec

@@ -17,4 +17,5 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
   s.add_development_dependency 'sinatra', '~> 1.0'
+  s.add_development_dependency 'rake'
 end

data/lib/girl_friday/actor.rb

@@ -53,7 +53,7 @@ class Actor
     @@registered_lock = Queue.new
     @@registered = {}
     @@registered_lock << nil
-  
+
     def current
       Thread.current[:__current_actor__] ||= private_new
     end
@@ -113,7 +113,7 @@ class Actor
       recipient.notify_exited(current, reason)
       self
     end
-    
+
     # Link the current Actor to another one.
     def link(actor)
       current = self.current
@@ -121,7 +121,7 @@ class Actor
       actor.notify_link current
       self
     end
-    
+
     # Unlink the current Actor from another one
     def unlink(actor)
       current = self.current
@@ -259,12 +259,19 @@ class Actor
     begin
       raise @interrupts.shift unless @interrupts.empty?
 
-      for i in 0...(@mailbox.size)
-        message = @mailbox[i]
+      if @mailbox.size > 0
+        message = @mailbox.shift
         action = filter.action_for(message)
-        if action
-          @mailbox.delete_at(i)
-          break
+        unless action
+          @mailbox << message
+          for i in 1...(@mailbox.size)
+            message = @mailbox[i]
+            action = filter.action_for(message)
+            if action
+              @mailbox.delete_at(i)
+              break
+            end
+          end
         end
       end
 
@@ -303,7 +310,7 @@ class Actor
       action.call message
     end
   end
- 
+
   # Notify this actor that it's now linked to the given one; this is not
   # intended to be used directly except by actor implementations.  Most
   # users will want to use Actor.link instead.
@@ -322,7 +329,7 @@ class Actor
     actor.notify_exited(self, exit_reason) unless alive
     self
   end
-  
+
   # Notify this actor that it's now unlinked from the given one; this is
   # not intended to be used directly except by actor implementations.  Most
   # users will want to use Actor.unlink instead.
@@ -337,7 +344,7 @@ class Actor
     end
     self
   end
-  
+
   # Notify this actor that one of the Actors it's linked to has exited;
   # this is not intended to be used directly except by actor implementations.
   # Most users will want to use Actor.send_exit instead.
@@ -399,7 +406,7 @@ class Actor
     end
   end
   private :check_thread
-  
+
   def _trap_exit=(value) #:nodoc:
     check_thread
     @lock.pop
@@ -410,7 +417,7 @@ class Actor
       @lock << nil
     end
   end
-  
+
   def _trap_exit #:nodoc:
     check_thread
     @lock.pop

data/lib/girl_friday/batch.rb

@@ -0,0 +1,47 @@
+module GirlFriday
+
+  ##
+  # Batch represents a set of operations which can be processed
+  # concurrently.  Asking for the results of the batch acts as a barrier:
+  # the calling thread will block until all operations have completed.
+  # Results are guaranteed to be returned in the
+  # same order as the operations are given.
+  #
+  # Internally a girl_friday queue is created which limits the
+  # number of concurrent operations based on the :size option.
+  #
+  # TODO Errors are not handled well at all.
+  class Batch
+    def initialize(enumerable, options, &block)
+      @queue = GirlFriday::Queue.new(:batch, options, &block)
+      @complete = 0
+      @size = enumerable.count
+      @results = Array.new(@size)
+      @lock = Mutex.new
+      @condition = ConditionVariable.new
+      start(enumerable)
+    end
+
+    def results(timeout=nil)
+      @lock.synchronize do
+        @condition.wait(@lock, timeout) if @complete != @size
+        @results
+      end
+    end
+
+    private
+
+    def start(operations)
+      operations.each_with_index do |packet, index|
+        @queue.push(packet) do |result|
+          @lock.synchronize do
+            @complete += 1
+            @results[index] = result
+            @condition.signal if @complete == @size
+          end
+        end
+      end
+    end
+
+  end
+end

data/lib/girl_friday/error_handler.rb

@@ -15,7 +15,7 @@ module GirlFriday
   class ErrorHandler
     class Hoptoad
       def handle(ex)
-        HoptoadNotifier.notify(ex)
+        HoptoadNotifier.notify_or_ignore(ex)
       end
     end
   end

data/lib/girl_friday/persistence.rb

@@ -10,16 +10,16 @@ module GirlFriday
         @backlog << work
       end
       alias_method :<<, :push
-      
+
       def pop
-        @backlog.pop
+        @backlog.shift
       end
-      
+
       def size
         @backlog.size
       end
     end
-    
+
     class Redis
       def initialize(name, options)
         @opts = options
@@ -53,4 +53,3 @@ module GirlFriday
     end
   end
 end
-    

data/lib/girl_friday/server.rb

@@ -16,24 +16,23 @@ module GirlFriday
     set :views,  "#{basedir}/views"
     set :public, "#{basedir}/public"
     set :static, true
-    
+
     helpers do
       include Rack::Utils
       alias_method :h, :escape_html
 
-      def dashboard(stats)
-        if stats[:busy] == stats[:pool_size] && stats[:backlog] < stats[:pool_size]
-          ['#ffc', 'Busy']
-        elsif stats[:busy] == stats[:pool_size] && stats[:backlog] >= stats[:pool_size]
-          ['#fcc', 'Busy and Backlogged']
-        else
-          ['white', 'OK']
-        end
+      def url_path(*path_parts)
+        [path_prefix, path_parts].join('/').squeeze('/')
+      end
+      alias_method :u, :url_path
+
+      def path_prefix
+        request.env['SCRIPT_NAME']
       end
     end
-    
-    get '/' do
-      redirect "#{request.env['REQUEST_URI']}/status"
+
+    get '/?' do
+      redirect url_path('status')
     end
 
     get '/status' do
@@ -46,4 +45,4 @@ module GirlFriday
       GirlFriday.status.to_json
     end
   end
-end
+end