batsir 0.1.0

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/.rspec

@@ -0,0 +1,3 @@
+--colour
+--backtrace
+--format documentation

data/.travis.yml

@@ -0,0 +1,10 @@
+language: ruby
+
+jdk:
+  - oraclejdk7
+
+rvm:
+  - 1.9.2
+  - 1.9.3
+  - jruby-19mode
+  - rbx-19mode

data/Gemfile

@@ -0,0 +1,22 @@
+source "http://rubygems.org"
+# Add dependencies required to use your gem here.
+# Example:
+#   gem "activesupport", ">= 2.3.5"
+
+# Add dependencies to develop your gem here.
+# Include everything needed to run rake, tests, features, etc.
+gem "bundler", "> 1.0.0"
+gem "jeweler"
+gem "blockenspiel", "~> 0.4.3"
+gem "celluloid"
+gem "sidekiq"
+gem "bunny", :git => "https://github.com/ruby-amqp/bunny.git"
+gem "json"
+#gem "hot_bunnies", :git => "https://github.com/ruby-amqp/hot_bunnies.git"
+
+group :development do
+end
+
+group :test do
+  gem "rspec"
+end

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2012 J.W. Koelewijn
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,79 @@
+[![Build Status](https://secure.travis-ci.org/jwkoelewijn/batsir.png?branch=master)](http://travis-ci.org/jwkoelewijn/batsir)
+
+
+![Batsir Logo](/jwkoelewijn/batsir/raw/master/batsir.png)
+
+# Batsir
+Batsir is an execution platform for stage based filter queue execution.
+It is based on the Pipes and Filters patterns, and makes it possible to create
+multiples Pipes and Filters (called Stages) that can be invoked asynchronously using so called
+inbound Acceptors. Acceptors are started automatically and will send a payload
+into the filter chain, after which the (possibly) transformed message will be
+processed by the so called outbound Notifiers.
+Notifiers can be used to asynchronously send a message to another stage, as long
+as corresponding inbound Acceptors have been configured.
+
+This makes it possible to create chains of stages to perform tasks that depend on each
+other, but otherwise have a low coupling.
+
+The usage of the Pipes and Filter pattern make it possible to thoroughly test each
+filter in isolation, thus promoting fast test cycles.
+
+# Example
+
+```ruby
+Batsir.create_and_start do
+  stage "stage 1" do
+    inbound do
+      acceptor AMQPAcceptor, :queue => 'some_queue', :host => 'localhost'
+    end
+    filter SumFilter
+    filter AverageFilter
+    outbound do
+      notifier AMQPNotifier, :queue => 'queue_2'
+    end
+  end
+
+  stage "stage 2" do
+    inbound do
+      acceptor AMQPAcceptor, :queue => 'queue_2'
+    end
+    filter PrintFilter
+  end
+end
+```
+
+This example creates 2 stages, 'stage 1' and 'stage 2'. The first stage creates and AMQPAcceptor, 
+that will connect to a AMQP Broker on localhost and will listen for messages on the 'some_queue' queue.
+When a message is received, the message will be offered to the SumFilter first. The result of the
+SumFilter is then sent to the #execute method of the AverageFilter. The result of this filter will
+than be sent as an AMQP message on the 'queue_2' queue.
+
+The inbound AMQPAcceptor of the second stage will then receive the message and its filters will be
+invoked (the PrintFilter in this example).
+
+# Sidekiq & Celluloid
+Batsir acts as both a Sidekiq server and client at the same time. When Batsir#create_and_start is invoked,
+Batsir will create Sidekiq workers on the fly, with instantiated filters on the workers. These workers will
+be deployed in the Sidekiq server, so that they will be available for processing. The workers also register
+themselves in a registry, where they can be requested using the stage name.
+
+The inbound acceptors will listen as a Celluloid Actor on the client side of Sidekiq. When a message is 
+received, it will look up the corresponding StageWorker in the registry and it will invoke the
+StageWorker asynchronously using Sidekiq.
+
+## Contributing to batsir
+ 
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
+* Fork the project
+* Start a feature/bugfix branch
+* Commit and push until you are happy with your contribution
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+## Copyright
+
+Copyright (c) 2012 J.W. Koelewijn. See LICENSE.txt for
+further details.
+

data/Rakefile

@@ -0,0 +1,55 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "batsir"
+  gem.homepage = "http://github.com/jwkoelewijn/batsir"
+  gem.license = "MIT"
+  gem.summary = %Q{Batsir is an execution platform for stage based operation queue execution}
+  gem.description = %Q{Batsir uses so called stages to define operation queues. These operation queus
+ consist of several operations that will be executed one after the other. Each stage
+ is defined by its name and the queue on which it will listen. Once a message is received
+ on the queue, it is dispatched to a worker in a seperate thread that will pass the message
+ to each operation in the operation queue.
+ Operation queues can have 4 different operations, 1 common operation type, and 3 special 
+ purpose operations: retrieval operations (which are always executed before all other operations),
+ persistence operations (which are always executed after the common operations, but before the
+ notification operations) and notification operations (which will always be executed last)
+ This makes it possible to create chains of stages to perform tasks that depend on each
+ other, but otherwise have a low coupling}
+  gem.email = "jwkoelewijn@gmail.com"
+  gem.authors = ["J.W. Koelewijn"]
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rspec/core/rake_task'
+
+desc "Run specs"
+RSpec::Core::RakeTask.new do |t|
+  t.pattern = 'spec/**/*_spec.rb'
+end
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "batsir #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/batsir.gemspec

@@ -0,0 +1,104 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = "batsir"
+  s.version = "0.1.0"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["J.W. Koelewijn"]
+  s.date = "2012-06-28"
+  s.description = "Batsir uses so called stages to define operation queues. These operation queus\n consist of several operations that will be executed one after the other. Each stage\n is defined by its name and the queue on which it will listen. Once a message is received\n on the queue, it is dispatched to a worker in a seperate thread that will pass the message\n to each operation in the operation queue.\n Operation queues can have 4 different operations, 1 common operation type, and 3 special \n purpose operations: retrieval operations (which are always executed before all other operations),\n persistence operations (which are always executed after the common operations, but before the\n notification operations) and notification operations (which will always be executed last)\n This makes it possible to create chains of stages to perform tasks that depend on each\n other, but otherwise have a low coupling"
+  s.email = "jwkoelewijn@gmail.com"
+  s.extra_rdoc_files = [
+    "LICENSE.txt",
+    "README.md"
+  ]
+  s.files = [
+    ".document",
+    ".rspec",
+    ".travis.yml",
+    "Gemfile",
+    "LICENSE.txt",
+    "README.md",
+    "Rakefile",
+    "VERSION",
+    "batsir.gemspec",
+    "batsir.png",
+    "lib/batsir.rb",
+    "lib/batsir/acceptors/acceptor.rb",
+    "lib/batsir/acceptors/amqp_acceptor.rb",
+    "lib/batsir/amqp.rb",
+    "lib/batsir/chain.rb",
+    "lib/batsir/config.rb",
+    "lib/batsir/dsl/dsl_mappings.rb",
+    "lib/batsir/filter.rb",
+    "lib/batsir/filter_queue.rb",
+    "lib/batsir/logo.rb",
+    "lib/batsir/notifiers/amqp_notifier.rb",
+    "lib/batsir/notifiers/notifier.rb",
+    "lib/batsir/registry.rb",
+    "lib/batsir/stage.rb",
+    "lib/batsir/stage_worker.rb",
+    "lib/batsir/transformers/field_transformer.rb",
+    "lib/batsir/transformers/json_input_transformer.rb",
+    "lib/batsir/transformers/json_output_transformer.rb",
+    "lib/batsir/transformers/transformer.rb",
+    "spec/batsir/acceptors/acceptor_spec.rb",
+    "spec/batsir/acceptors/amqp_acceptor_spec.rb",
+    "spec/batsir/chain_spec.rb",
+    "spec/batsir/dsl/chain_mapping_spec.rb",
+    "spec/batsir/dsl/stage_mapping_spec.rb",
+    "spec/batsir/filter_queue_spec.rb",
+    "spec/batsir/filter_spec.rb",
+    "spec/batsir/notifiers/amqp_notifier_spec.rb",
+    "spec/batsir/notifiers/notifier_spec.rb",
+    "spec/batsir/stage_spec.rb",
+    "spec/batsir/stage_worker_spec.rb",
+    "spec/batsir/support/bunny_mocks.rb",
+    "spec/batsir/support/mock_filters.rb",
+    "spec/batsir/transformers/field_transformer_spec.rb",
+    "spec/batsir/transformers/json_input_transformer_spec.rb",
+    "spec/batsir/transformers/json_output_transformer_spec.rb",
+    "spec/batsir/transformers/transformer_spec.rb",
+    "spec/spec_helper.rb"
+  ]
+  s.homepage = "http://github.com/jwkoelewijn/batsir"
+  s.licenses = ["MIT"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = "1.8.22"
+  s.summary = "Batsir is an execution platform for stage based operation queue execution"
+
+  if s.respond_to? :specification_version then
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<bundler>, ["> 1.0.0"])
+      s.add_runtime_dependency(%q<jeweler>, [">= 0"])
+      s.add_runtime_dependency(%q<blockenspiel>, ["~> 0.4.3"])
+      s.add_runtime_dependency(%q<celluloid>, [">= 0"])
+      s.add_runtime_dependency(%q<sidekiq>, [">= 0"])
+      s.add_runtime_dependency(%q<bunny>, [">= 0"])
+      s.add_runtime_dependency(%q<json>, [">= 0"])
+    else
+      s.add_dependency(%q<bundler>, ["> 1.0.0"])
+      s.add_dependency(%q<jeweler>, [">= 0"])
+      s.add_dependency(%q<blockenspiel>, ["~> 0.4.3"])
+      s.add_dependency(%q<celluloid>, [">= 0"])
+      s.add_dependency(%q<sidekiq>, [">= 0"])
+      s.add_dependency(%q<bunny>, [">= 0"])
+      s.add_dependency(%q<json>, [">= 0"])
+    end
+  else
+    s.add_dependency(%q<bundler>, ["> 1.0.0"])
+    s.add_dependency(%q<jeweler>, [">= 0"])
+    s.add_dependency(%q<blockenspiel>, ["~> 0.4.3"])
+    s.add_dependency(%q<celluloid>, [">= 0"])
+    s.add_dependency(%q<sidekiq>, [">= 0"])
+    s.add_dependency(%q<bunny>, [">= 0"])
+    s.add_dependency(%q<json>, [">= 0"])
+  end
+end
+

data/lib/batsir.rb

@@ -0,0 +1,73 @@
+require 'blockenspiel'
+require 'celluloid'
+require 'json'
+require 'bunny'
+require 'sidekiq'
+require 'sidekiq/cli'
+require 'batsir/registry'
+require 'batsir/config'
+require 'batsir/chain'
+require 'batsir/filter'
+require 'batsir/filter_queue'
+require 'batsir/stage'
+require 'batsir/stage_worker'
+require 'batsir/dsl/dsl_mappings'
+require 'batsir/acceptors/acceptor'
+require 'batsir/acceptors/amqp_acceptor'
+require 'batsir/notifiers/notifier'
+require 'batsir/notifiers/amqp_notifier'
+require 'batsir/transformers/transformer'
+require 'batsir/transformers/field_transformer'
+require 'batsir/transformers/json_input_transformer'
+require 'batsir/transformers/json_output_transformer'
+require 'batsir/logo'
+
+module Batsir
+  def self.config
+    @config ||= Batsir::Config.new(config_defaults)
+  end
+
+  def self.config_defaults
+    {
+      :redis_url => "redis://localhost:6379/0"
+    }
+  end
+
+  def self.create(&block)
+    puts logo
+    new_block = ::Proc.new do
+      aggregator_chain &block
+    end
+    @chain = ::Blockenspiel.invoke(new_block, Batsir::DSL::ChainMapping.new)
+  end
+
+  def self.start
+    return unless @chain
+
+    sidekiq_cli = Sidekiq::CLI.instance
+    Sidekiq.options[:queues] << 'default'
+
+    initialize_sidekiq
+
+    generated_code = @chain.compile
+
+    eval(generated_code)
+
+    @chain.start
+    sidekiq_cli.run
+  end
+
+  def self.initialize_sidekiq
+    Sidekiq.configure_server do |config|
+      config.redis = {:url => Batsir.config.redis_url}
+    end
+    Sidekiq.configure_client do |config|
+      config.redis = {:url => Batsir.config.redis_url}
+    end
+  end
+
+  def self.create_and_start(&block)
+    create(&block)
+    start
+  end
+end

data/lib/batsir/acceptors/acceptor.rb

@@ -0,0 +1,45 @@
+module Batsir
+  module Acceptors
+    class Acceptor
+      include Celluloid
+
+      attr_accessor :stage_name
+      attr_accessor :transformer_queue
+      attr_accessor :cancellator
+
+      def initialize(options = {})
+        options.each do |option, value|
+          self.send("#{option}=", value)
+        end
+        @transformer_queue = []
+      end
+
+      def add_transformer(transformer)
+        @transformer_queue << transformer
+      end
+
+      # This method is called automatically when the stage is
+      # started, it is here that you set up the accepting
+      # logic. Make sure that somewhere within this logic
+      # the #start_filter_chain(msg) is called to start
+      # actual processing
+      #
+      # Note that this method will be invoked asynchronously
+      # using the Celluloid actor semantics.
+      def start
+
+      end
+
+      # When a message is accepted by an Acceptor, this method
+      # should be invoked with the received payload to start
+      # processing of the filter chain
+      def start_filter_chain(message)
+        klazz = Batsir::Registry.get(stage_name)
+        transformer_queue.each do |transformer|
+          message = transformer.transform(message)
+        end
+        klazz.perform_async(message) if klazz
+      end
+    end
+  end
+end

data/lib/batsir/acceptors/amqp_acceptor.rb

@@ -0,0 +1,19 @@
+require 'batsir/amqp'
+
+module Batsir
+  module Acceptors
+    class AMQPAcceptor < Acceptor
+      include Batsir::AMQP
+      def start
+        Bunny.run( bunny_options ) do |bunny|
+          q   = bunny.queue( queue )
+          exc = bunny.exchange( exchange )
+          q.bind( exc, :key => queue)
+          q.subscribe(:cancellator => cancellator) do |msg|
+            start_filter_chain(msg[:payload])
+          end
+        end
+      end
+    end
+  end
+end