aggregator 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: cb22458b343af2f63b3c053075447a73c12b3da1
+  data.tar.gz: e3ea269f9c95a9d02b713449afe14e110d7c0145
+SHA512:
+  metadata.gz: 07e8a50c88287282d418cc0913d6f43ff7791b49c2437633037bbb4e1bb12e78ef1d7fb83f78dfe50b489082cd4aae416b876350c4226fb6773480626ec3108e
+  data.tar.gz: 64922c7189f36b8fbbfc6bdb9da4e24591c76b023b553cf9b6ddb0d8a74e8053b367600a439423d4d8828261055a4a9a0b1b15b01c0e96090dad71a42ed63de3

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/.travis.yml

@@ -0,0 +1,7 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+  - 2.1.0-preview2
+  - rbx
+  - jruby

data/Gemfile

@@ -0,0 +1,2 @@
+source "https://rubygems.org"
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Adtile, Inc.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,101 @@
+# Aggregator
+
+[![Build Status](https://travis-ci.org/adtile/aggregator.png?branch=master)](https://travis-ci.org/adtile/aggregator)
+[![Code Climate](https://codeclimate.com/github/adtile/aggregator.png)](https://codeclimate.com/github/adtile/aggregator)
+
+Aggregator is a Ruby gem that allows you to easily run aggregation work on a separate thread so that you can save yourself from doing too many expensive operations when you can do a batch operation less frequently.
+
+## Installation
+
+    $ gem install aggregator
+
+Or add it to your Gemfile.
+
+## Usage
+
+Let's create a sample aggregator for a Rails application to keep track of pageviews:
+
+``` ruby
+class PageviewAggregator < Aggregator
+  def process(collection, item)
+    collection ||= {}
+    collection[item] = collection.fetch(item, 0) + 1
+    collection
+  end
+
+  def finish(collection)
+    # Update the database based on your aggregated data:
+    # { "/" => 471, "/about" => 127, ... }
+  end
+end
+```
+
+Then, in a Rails controller action you could push the current page path:
+
+``` ruby
+PageviewAggregator.push(request.path)
+```
+
+That's it! Let's go through what happens in more detail.
+
+Every time a new item is pushed, the `#perform` method is called. For each new batch, `collection` will be `nil` and it is your responsibility to manage it. This way it can be any object you want (Hash, Array, etc.). You must also always return the collection object from this method. Since this method is called for each pushed item, you'll want to keep it fast.
+
+Whenever a batch is ready, `#finish` is called and the final collection is passed. In here you can do whatever you want with it. Most likely you'll be doing something like saving it to a database.
+
+A batch is considered ready whenever the one of two things happens:
+
+- A configured number of items has been processed.
+- A configured amount of time has passed since the batch started.
+
+See the configuration options below to see how to set these values.
+
+### Configuration options
+
+Configuration options are defined for each `Aggregator` subclass and are class methods that must be explicitly called on `self`:
+
+```ruby
+class MyAggregator < Aggregator
+  self.option_name = <value>
+end
+```
+
+The available options are:
+
+- `.max_batch_size=`: maximum number of items to process before a batch is considered ready and `#finish` is called. Defaults to 1000.
+
+- `.max_wait_time=`: maximum number of seconds given to the batch to process before it's considered ready. Defaults to 1.
+
+- `.logger=`: logger to use. In a Rails application you probably want to set it to `Rails.logger`. Defaults to `Logger.new(STDOUT)`.
+
+### Testing
+
+When you're writing tests for your application, you might need to wait until the aggregations run before you can assert something. In that case, you can just call `.drain` on your Aggregator subclass, which will block until all items have been processed and finished:
+
+``` ruby
+it "saves all aggregations to the database" do
+  5.times { get "/page" }
+  PageviewAggregator.drain
+  pageviews = Pageview.find("/page").total
+  expect(pageviews).to eq(5)
+end
+```
+
+## Guarantees and gotchas
+
+All background threads are handled for you and can recover from crashes. However, if a thread crashes due to an exception raised in the `#perform` method, that item may be lost forever. Similarly, if there is an uncaught exception in `#finish` the entire collection will be lost. It is up to you to rescue and retry based on your needs.
+
+One and only one background thread is started for each Aggregator subclass.
+
+When the process exits gracefully (e.g. web server shutdown), running aggregators will finish processing all items.
+
+## License
+
+MIT License.
+
+## Contributing
+
+1. Fork it ( http://github.com/<my-github-username>/aggregator/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,8 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+end
+
+task :default => :test

data/aggregator.gemspec

@@ -0,0 +1,24 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'aggregator/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "aggregator"
+  spec.version       = Aggregator::VERSION
+  spec.authors       = ["Joao Carlos"]
+  spec.email         = ["joao@adtile.me"]
+  spec.summary       = %q{Aggregate items on a separate thread.}
+  spec.description   = %q{Define aggregators that run on a separate thread so that you can do more, faster.}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "minitest", ">= 5.0.0"
+  spec.add_development_dependency "rubysl", "~> 2.0" if RUBY_ENGINE == "rbx"
+end

data/lib/aggregator.rb

@@ -0,0 +1,144 @@
+require "thread"
+require "singleton"
+require "logger"
+
+class Aggregator
+  include Singleton
+
+  attr_accessor :max_batch_size, :max_wait_time, :logger
+
+  def self.push(data)
+    self.instance.push(data)
+  end
+
+  def self.max_batch_size=(value)
+    self.instance.max_batch_size = value
+  end
+
+  def self.max_wait_time=(value)
+    self.instance.max_wait_time = value
+  end
+
+  def self.logger=(logger)
+    self.instance.logger = logger
+  end
+
+  def self.drain
+    self.instance.drain
+  end
+
+  def initialize
+    @queue = Queue.new
+    @mutex = Mutex.new
+    @thread = nil
+
+    at_exit { stop }
+  end
+
+  def push(data)
+    @queue.push(data)
+    start unless running?
+  end
+
+  def drain
+    if running?
+      if ! @queue.empty?
+        log :info, "joining thread #{@thread.inspect} (queue length = #{@queue.length})"
+        @drain = true
+        @thread.join if running?
+      end
+
+      log :info, "stopping thread #{@thread.inspect} (queue length = #{@queue.length})"
+      @thread = nil
+    elsif ! @queue.empty?
+      start and drain
+    end
+
+    true
+  end
+
+  private
+
+  def max_batch_size
+    @max_batch_size || 1000
+  end
+
+  def max_wait_time
+    @max_wait_time || 1
+  end
+
+  def process(collection, item)
+    raise NoMethodError,
+      "#{self.class.name}#process(collection, item) must be implemented"
+  end
+
+  def finish(collection)
+    raise NoMethodError,
+      "#{self.class.name}#finish(collection) must be implemented"
+  end
+
+  def running?
+    @thread && @thread.alive?
+  end
+
+  def logger
+    @logger ||= Logger.new(STDOUT)
+  end
+
+  def log(level, message)
+    logger.send(level, "[#{self.class.name}] #{message}")
+  end
+
+  def process_queue
+    raise StopIteration if @queue.empty? && @drain
+
+    processed_items = 0
+    start_time = Time.now
+
+    while processed_items < max_batch_size && (Time.now - start_time) < max_wait_time
+      raise StopIteration if @queue.empty? && @drain
+      if @queue.empty?
+        sleep 0.1
+      else
+        collection = process(collection, @queue.pop(true))
+        processed_items += 1
+      end
+    end
+  ensure
+    finish(collection) if collection
+  end
+
+  def start
+    @mutex.synchronize do
+      return false if running?
+
+      @drain = false
+
+      @thread = Thread.new do
+        begin
+          log :info, "starting thread #{Thread.current}"
+
+          loop do
+            process_queue
+          end
+        rescue Exception => e
+          log :warn, "thread crashed with exception: #{e.inspect}"
+        end
+      end
+
+      @thread.priority = 2
+
+      @thread
+    end
+  end
+
+  def stop
+    if running?
+      drain
+    else
+      log :info, "thread not running - nothing to stop"
+      return false
+    end
+  end
+
+end

data/lib/aggregator/version.rb

@@ -0,0 +1,3 @@
+class Aggregator
+  VERSION = "1.0.0"
+end

data/test/minitest_helper.rb

@@ -0,0 +1,5 @@
+$LOAD_PATH.unshift File.expand_path("../../lib", __FILE__)
+require "aggregator"
+
+require "minitest"
+require "minitest/autorun"

data/test/test_aggregator.rb

@@ -0,0 +1,79 @@
+require "minitest_helper"
+require "stringio"
+
+class TestAggregator < Minitest::Test
+
+  class EventAggregator < Aggregator
+    attr_reader :process_counter, :finish_counter
+
+    self.max_wait_time = 2
+    self.max_batch_size = 25
+    self.logger = Logger.new(StringIO.new)
+
+    def reset_counters
+      @process_counter = 0
+      @finish_counter = 0
+    end
+
+    def process(collection, item)
+      fail if item.nil?
+      collection ||= []
+      collection << item
+      collection
+    end
+
+    def finish(collection)
+      @process_counter ||= 0
+      @process_counter += collection.count
+
+      @finish_counter ||= 0
+      @finish_counter += 1
+    end
+  end
+
+  def setup
+    EventAggregator.instance.reset_counters
+    EventAggregator.instance.instance_variable_get(:@queue).clear
+  end
+
+  def test_it_processes_pushed_items
+    100.times { EventAggregator.push({}) }
+    EventAggregator.drain
+    assert_equal 100, EventAggregator.instance.process_counter
+  end
+
+  def test_it_processes_in_batches
+    100.times { EventAggregator.push({}) }
+    EventAggregator.drain
+    assert_equal 4, EventAggregator.instance.finish_counter
+  end
+
+  def test_it_can_drain_multiple_times
+    100.times do
+      EventAggregator.instance.reset_counters
+      90.times { EventAggregator.push({}) }
+      EventAggregator.drain
+      assert_equal 90, EventAggregator.instance.process_counter
+    end
+  end
+
+  def test_it_can_recover_from_a_thread_crash
+    100.times do |i|
+      item = (i == 35) ? nil : {}
+      EventAggregator.push(item)
+    end
+
+    EventAggregator.drain
+    EventAggregator.push({})
+    EventAggregator.drain
+
+    assert_equal 100, EventAggregator.instance.process_counter
+  end
+
+  def test_it_drains_the_queue_even_if_the_thread_was_not_running
+    EventAggregator.instance.instance_variable_get(:@queue).push({})
+    EventAggregator.drain
+    assert_equal 1, EventAggregator.instance.process_counter
+  end
+
+end

metadata

@@ -0,0 +1,87 @@
+--- !ruby/object:Gem::Specification
+name: aggregator
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Joao Carlos
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-12-20 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+description: Define aggregators that run on a separate thread so that you can do more,
+  faster.
+email:
+- joao@adtile.me
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .travis.yml
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- aggregator.gemspec
+- lib/aggregator.rb
+- lib/aggregator/version.rb
+- test/minitest_helper.rb
+- test/test_aggregator.rb
+homepage: ''
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.3
+signing_key: 
+specification_version: 4
+summary: Aggregate items on a separate thread.
+test_files:
+- test/minitest_helper.rb
+- test/test_aggregator.rb
+has_rdoc: