qswarm 0.0.21 → 1.0.0

data/README.md

@@ -1 +1,264 @@
-Work in progress
+# Qswarm - stream processing for Ruby #
+
+Qswarm is a Ruby DSL for manipulating real-time streams of messages. It defines three basic concepts - [connections](#connections), [sources](#sources), and [sinks](#sinks). Connections emit messages which sources can catch, sinking them to other connections. In this way you can construct data flows between systems and transform messages in-flight with Ruby.
+
+Install with:
+
+```sh
+gem install qswarm
+```
+
+## Agent ##
+
+Use the `agent` command to wrap a set of DSL commands.
+
+```ruby
+agent :bob do
+  ...
+end
+```
+
+Alternatively you could save each agent in a separate file and use a process manager such as [supervisord][], [god][], [bluepill][] to manage the swarm.
+
+## Connections ##
+
+
+Use `connect` to setup connections to services. Currently [Logger](#logger), [AMQP](#amqp), [XMPP](#xmpp), and [Twitter](#twitter) are supported. You can also pass an optional block which will be executed once the connection is set up.
+
+### Logger ###
+
+```ruby
+  connect :mylog,
+          :type            => :logger,
+          :filename        => 'foo.log'
+```
+
+Logger is a very simple connection type which can be used to append a stream of messages to a file. It can only `sink` messages (i.e. doesn't not emit any data) and it provides no arguments to `sink`.
+
+### AMQP ###
+
+```ruby
+  connect :messages,
+          :type            => :amqp,
+          :uri             => 'guest:guest@localhost:5672/',
+          :exchange_type   => :headers,
+          :exchange_name   => 'myexchange',
+          :exchange_args   => { :durable => true },
+          :queue_args      => { :auto_delete => true, :durable => true, :exclusive => true },
+          :subscribe_args  => { :exclusive => false, :ack => false },
+          :bind_args       => {},
+          :prefetch        => 0,
+          :bind            => 'foo.bar.#',
+          :format          => :json
+```
+
+This sets up an AMQP connection called `:messages` using the credentials in `:uri` (user:pass@host:port/vhost) and creates the exchange if it doesn't exist already (using `:exchange_args`). If a routing key is passed with `:bind` then a queue will be created with the dotted concatenation of the agent name and the connection name, e.g. bob.messages, and bound to the exchange specified (you can pass `:uniq => true` if you want a UUID appended to the queue name to make it unique for situations such as load balancing). At the moment you can't bind a queue to an exchange without specifying a routing key in `:bind`. You can pass configuration to the binding with `:bind_args`. Similarly `:queue_args` allow you to pass configuration options to the queue creation. Defaults for *_args are as in the example.
+
+The agent is automatically subscribed to the created queue and you can pass `:subscribe_args` to configure the subscription. If you specified `:ack` to be true then you can use `:prefetch` to specify how many messages you want to have from the queue at a time.
+
+The `:format` argument determines what Qswarm does with the payloads it receives and how it transforms messages to be sent, see section [Payload](#payload).
+
+AMQP sets the following headers for `source` to use as [guards](#filters-and-guards):
+
+* `:routing_key`
+* Any headers from a headers exchange will be passed verbatim
+
+AMQP supports the following arguments to `sink`:
+
+* `:routing_key` - the key to post the message under
+* `:headers` - a Hash that will be used instead of the payload.headers for posting to headers exchanges
+
+### XMPP ###
+
+```ruby
+  connect :hipchat,
+          :type            => :xmpp,
+          :jid             => '54321_123456@chat.hipchat.com',
+          :real_name       => 'My bot',
+          :channel         => ['54321_lounge@conf.hipchat.com', '54321_chat@conf.hipchat.com'],
+          :password        => 'foobar'
+```
+
+The above example connects to an XMPP service called `:hipchat` using the JID and password provided. The `:real_name` will be used when joining groupchat rooms and for some services (like Hipchat) needs to match exactly your registered name including case. The script will automatically join the groupchat channel(s) specified in `:channel` and will use these channels list for sinks which don't specify a channel destination. XMPP support is provided using the [Blather][] library, which means that you can include Blather DSL in connect's block to implement bot behaviours. This block will execute once the connection to the XMPP server has been established (when_ready).
+
+Currently there is no support to `source` messages from an XMPP connection (i.e. you can only talk not listen) so the Blather DSL is your only option if you want interactivity at the moment.
+
+XMPP supports the following arguments to `sink`:
+
+* `:channel` - an Array or String of the groupchat channel(s) to sink the message to (will join if not already present) 
+
+### Twitter ###
+
+```ruby
+  connect :tweetstream,
+          :type            => :twitter,
+          :consumer_key    => 'YOURKEYHERE',
+          :consumer_secret => 'YOURSECRETHERE',
+          :oauth_token     => 'YOURTOKENHERE',
+          :oauth_token_secret => 'YOURSECRETHERE',
+          :track           => {
+            :colours         => ['red', 'green', 'blue'],
+            :feelings        => ['happy', 'sad'],
+            :tech            => ['ruby', 'python'],
+          },
+          :follow          => {
+            :tech            => [11987892]
+          },
+          :list            => {
+            :flibbertigibbets     => { 'Scobleizer' => 'most-influential-in-tech' }
+          }
+```
+
+A Twitter connection uses the [Tweetstream][Tweetstream gem] and [Twitter][Twitter gem] gems and requires oAuth credentials which you can get from [dev.twitter.com][Twitter auth]. There are three options that the Twitter API gives you - you can `:track` keywords in the global tweet stream using track, you can `:follow` the full stream of particular users (by twitter ID as Tweetstream doesn't let you use handles), or you can get updates from everyone included on a `:list` (this uses the REST API and the list is polled every minute).
+
+You specify groups (:colours/:feelings/:tech/:flibbertigibbets above) to allow for easy filtering later on with [guards](#filters-and-guards). Twitter messages are always JSON so there is no `:format` option for connect.
+
+Twitter would set the following example headers for `source` to use depending on the `:type` that generated the message.
+
+* **:type** => 'track', **:group** => 'colours|feelings|tech', **:matches** => [red|green|blue|happy|sad|ruby|python]
+* **:type** => 'follow', **:group** => 'tech', **:user_id** => 11987892
+* **:type** => 'list', **:group** => 'flibbertigibbets', **:user_id** => 'Scobleizer', **:slug** => 'most-influential-in-tech'
+
+There is currently no support to `sink` messages to a Twitter connection - i.e. you cannot Tweet.
+
+
+## Payload ##
+
+Payload is a Hash containing the following data by default:
+
+* payload.raw
+* payload.data
+* payload.format (set by arguments to the originating connect)
+* payload.headers
+
+When a message is received from a connection, it is accessible in the DSL with `payload`. The `:format` option in a `connect` declares the format of messages emitted by this connection and determines processing that will be applied to the raw payload. What this means is that if the `:format` is set to JSON, `payload.data` will be set to a Ruby Hash created by `JSON.parse(payload.raw, :symbolize_names => true)`. If `:format` is :xml then `payload.data` will be set to `Nokogiri::XML(payload.raw)`. If :raw then `payload.data` will equal `payload.raw`.
+
+Sinks can also set `:format` to define the reverse as messages are converted back from their Ruby objects for transmission. If no argument is supplied a `sink` will assume the `connect` specified value as a default.
+
+Some connection types add `payload.headers` which will contain a Ruby Hash of relevant data.
+
+## Filters and Guards ##
+
+You can use `before` and `after` as filters which will execute on receipt of a message from a specified connection. They will execute before or after your `source` commands. This example creates a plain text format of a tweet, expanding twitter handles, which can then be used in all `source` blocks.
+
+```ruby
+before :tweetstream do
+    @pp = "<#{payload.data[:user][:name]}/#{payload.data[:user][:screen_name]}> #{payload.data[:text]}"
+    payload.data[:entities][:user_mentions].each do |u|
+      @pp.gsub!(/@#{u[:screen_name]}/,"<#{u[:name]}/#{u[:screen_name]}>")
+    end
+  end
+```
+
+Guards (shamelessly copied from [Blather][]) allow you to put conditional execution on `before`, `after`, and `source` blocks by filtering on data passed in `payload.headers`. Please note that header values are always Strings not Symbols.
+
+The types of guards are:
+
+```ruby
+# Hash with any value
+#   Equivalent to payload.headers[:body] == 'exit'
+source :messages, :body => 'exit'
+
+# Hash with regular expression
+#   Equivalent to payload.headers[:body].match /exit/
+source :messages, :body => /exit/
+
+# Hash with array
+#   Equivalent to ['gone', 'forbidden'].include?(payload.headers[:name])
+source :messages, :name => ['gone', 'forbidden']
+
+# Proc
+#   Calls the proc passing in payload.headers
+#   Checks that the ID is modulo 3
+source :messages, Proc { |header| header[:id] % 3 == 0 }
+
+# Array
+#   Use arrays with the previous types effectively turns the guard into
+#   an OR statement.
+#   Equivalent to payload.headers[:body] == 'foo' || payload.headers[:body] == 'baz'
+source :messages, [{:body => 'foo'}, {:body => 'baz'}]
+```
+
+## Sources ##
+
+Sources listen to messages from connections and process them using their blocks which are executed on receipt of a message. The headers available for [guards][Filters and Guards] will be dependant on the connection that sent the message. All sources that match will receive the message and execute.
+
+```ruby
+  source  :tweetstream, :type => 'follow', :user_id => 224662544 do
+    if payload.data[:text].match(/A14/)
+      ...
+    end
+  end
+```
+
+The above will listen to messages from the `:tweetstream` connection. The [guards](#filters-and-guards) will eliminate any tweet which doesn't come from a `:follow` (rather than `:track` or `:list`) and where the user doesn't match the provided ID which happens to be the Highways Agency twitter account for East of England travel news. The pattern match for A14 is done in the block because the tweet text isn't available in the headers.
+
+## Sinks ##
+
+Sinks publish to connections the output of their blocks. Here's an example of sinking a text message generated from a connection sending XML messages.
+
+```ruby
+    sink  :hipchat,
+          :format          => :xml,
+          :channels        => ['12345_errors@conf.hipchat.com'] do
+
+      message = payload.data.at_xpath('error')['message']
+      "*** ERROR: " + message[0..140] + (message.size > 140 ? ' ... ' : ' ' ) + payload.headers.to_s
+    end
+```
+
+In this case the `:format` argument isn't really needed because a return payload is specified by the block, but if the block was absent Qswarm would use it to know it needed to do a `payload.data.to_xml` before sending to the connection. You can have multiple sinks in a single source block that will all process the same payload.
+
+## Full Example ##
+
+```ruby
+agent :bob do
+  connect :hipchat,
+          :type            => :xmpp,
+          :jid             => '54321_123456@chat.hipchat.com',
+          :channel         => ['54321_lounge@conf.hipchat.com', '54321_chat@conf.hipchat.com'],
+          :password        => 'foobar'
+
+  connect :tweetstream,
+          :type            => :twitter,
+          :consumer_key    => 'YOURKEYHERE',
+          :consumer_secret => 'YOURSECRETHERE',
+          :oauth_token     => 'YOURTOKENHERE',
+          :oauth_token_secret => 'YOURSECRETHERE',
+          :track           => {
+            :colours         => ['red', 'green', 'blue'],
+            :feelings        => ['happy', 'sad'],
+            :tech            => ['ruby', 'python'],
+          },
+          :follow          => {
+            :tech            => [11987892]
+          },
+          :list            => {
+            :flibbertigibbets     => { 'Scobleizer' => 'most-influential-in-tech' }
+          }
+
+  source  :tweetstream, :type => %w( follow list ) do
+    sink  :hipchat,
+          :channel => '54321_influencers@conf.hipchat.com'
+  end
+
+  source  :tweetstream, :group => 'tech' do
+    sink  :hipchat,
+          :channel => '54321_cool_stuff@conf.hipchat.com'
+  end
+end
+```
+
+More examples can be found in these blog posts:
+
+* [Stream processing in Ruby](http://ecafe.org/blog/2013/12/13/stream-processing-in-ruby.html)
+
+----
+
+[supervisord]: http://supervisord.org
+[god]: http://godrb.com
+[bluepill]: https://github.com/bluepill-rb/bluepill
+[Blather]: https://github.com/adhearsion/blather
+[Tweetstream gem]: https://github.com/tweetstream/tweetstream
+[Twitter gem]: https://github.com/sferik/twitter
+[Twitter auth]: https://dev.twitter.com/docs/auth/tokens-devtwittercom

data/bin/qswarm

@@ -3,14 +3,17 @@
 $stdout.sync = true
 
 require 'qswarm'
+require 'trollop'
 
-usage = "#{$0} <configuration file>"
-abort usage unless config = ARGV.shift
+opts = Trollop::options do
+  opt :debug, "Turn log level up to DEBUG"
+end
 
-$graylog2_facility = config.split('.')[0]
-$graylog2_host = ARGV.shift || 'localhost'
+abort "Usage: #{$0} <configuration file>" unless config = ARGV.shift
 
+if opts[:debug]
+  Qswarm.logger.level = Logger::DEBUG
+end
 
-swarm = Qswarm::Swarm.load config
-swarm.log.level = Logger::DEBUG
+swarm = Qswarm::Swarm.new(config)
 swarm.run

data/lib/qswarm.rb

@@ -1,3 +1,17 @@
-require "qswarm/version"
-require 'qswarm/loggable'
-require 'qswarm/swarm'
+%w[
+  qswarm/version
+  qswarm/dsl
+  qswarm/swarm
+  qswarm/agent
+  qswarm/connection
+  logger
+].each { |r| require r }
+
+module Qswarm
+  @@logger = nil
+  class << self
+    def logger
+      @@logger ||= Logger.new($stdout).tap {|logger| logger.level = Logger::INFO }
+    end
+  end
+end

data/lib/qswarm/agent.rb

@@ -1,65 +1,185 @@
-require 'qswarm/broker'
-require 'qswarm/listener'
+require 'ostruct'
+require 'json'
+require 'nokogiri'
 
 module Qswarm
   class Agent
-    include Qswarm::Loggable
+    include Qswarm::DSL
 
     attr_reader :swarm, :name
+    dsl :connect, :before, :after, :source, :sink, :emit, :agent, :payload
 
-    def initialize(swarm, name, args, &block)
+    def initialize(swarm, name, args = nil, &block)
       @swarm              = swarm
-      @name               = name.to_s
-      @brokers            = {}
-      @listeners          = []
+      @name               = name
+      @clients            = {}
       @args               = args
+      @filters            = {}
+      @handlers           = {}
+      @payload            = nil
 
-      unless args.nil?
-        case args.delete :type
-        when :esper
-          require 'java'
-
-          require 'esper-4.5.0/esper-4.5.0.jar'
-          require 'esper-4.5.0/esper/lib/commons-logging-1.1.1.jar'
-          require 'esper-4.5.0/esper/lib/antlr-runtime-3.2.jar'
-          require 'esper-4.5.0/esper/lib/cglib-nodep-2.2.jar'
-          require 'esper-4.5.0/esper/lib/log4j-1.2.16.jar'
-
-          include_class 'com.espertech.esper.client.EPRuntime'
-          include_class 'com.espertech.esper.client.EPServiceProviderManager'
-          include_class 'com.espertech.esper.client.EPServiceProvider'
-          include_class 'com.espertech.esper.client.EPStatement'
-
-          include_class 'com.espertech.esper.client.UpdateListener'
-          include_class 'com.espertech.esper.client.EventBean'
-          include_class 'org.apache.commons.logging.Log'
-          include_class 'org.apache.commons.logging.LogFactory'
-        end
+      dsl_call(&block)
+    end
+
+    def agent
+      self
+    end
+
+    def payload
+      @payload
+    end
+
+    # Connects to a data stream
+    #
+    # @param name [String] the name of the connection
+    # @param args [Hash] arguments for the connection
+    # @param &block [Proc] a block which is passed to the client constructor
+    def connect(name, args = nil, &block)
+      raise "Connection '#{name.inspect}' is already registered" if @clients[name]
+
+      if !args.nil? && !args[:type].nil?
+        Qswarm.logger.info "[#{@name.inspect}] Registering #{args[:type].inspect} connection #{name.inspect}"
+        require "qswarm/connections/#{args[:type].downcase}"
+        @clients[name] = eval("Qswarm::Connections::#{args[:type].capitalize}").new(self, name, args, &block)
+      else
+        Qswarm.logger.info "[#{@name.inspect}] Registering default connection #{name.inspect}"
+        @clients[name] = Qswarm::Connection.new(self, name, args, &block)
       end
+    end
+
+    def before(connection, *guards, &block)
+      Qswarm.logger.info "[#{@name.inspect}] Registering :before filter for #{connection.inspect}/#{guards.inspect}"
 
-      self.instance_eval(&block)
+      [*connection].each do |c|
+        register_filter :before, c, *guards, &block
+      end
     end
 
-    def listen(name, args = nil, &block)
-      logger.info "Registering listener: #{name}"
-      @listeners << Qswarm::Listener.new(self, name, args, &block)
+    def after(connection, *guards, &block)
+      Qswarm.logger.info "[#{@name.inspect}] Registering :after filter for #{connection.inspect}/#{guards.inspect}"
+
+      [*connection].each do |c|
+        register_filter :after, c, *guards, &block
+      end
     end
 
-    def broker(name, &block)
-      logger.info "Registering broker: #{name}"
-      @brokers[name] = Qswarm::Broker.new(name, &block)
+    def source(connection, *guards, &block)
+      Qswarm.logger.info "[#{@name.inspect}] Registering handler for #{connection.inspect}/#{guards.inspect}"
+
+      [*connection].each do |c|
+        register_handler c, *guards, &block
+      end
     end
 
-    def get_broker(name)
-      @brokers[name] || @swarm.get_broker(name)
+    def sink(connection, args = nil, &block)
+      Qswarm.logger.debug "[#{@name.inspect}] Sink #{connection.inspect} received #{@payload.inspect}"
+
+      # Payload from DSL parent context overidden by arguments and block locally to sink
+      p = @payload.dup
+      p.data = args[:data] unless args.nil? || args[:data].nil?
+      p.data = dsl_call(&block) if block_given?
+
+      [*connection].each do |c|
+        # Update raw from the current data
+        p.raw = case args[:format].nil? ? @clients[c].format : args[:format]
+                when :json
+                  JSON.generate(p.data)
+                when :xml
+                  p.data.to_xml
+                else # raw
+                  p.data
+                end unless args.nil?
+
+        @clients[c].sink(args, p)
+      end
     end
 
-    def bind
-      logger.info "Binding to exchange"
+    def emit(connection, args = nil, &block)
+      # Need to set @payload for access by the dsl_call when handlers are run
+      # Overwriting global parent this will break when nesting emit in source which will loose the payload originating from connection
+      @payload = args[:payload] unless args.nil? || args[:payload].nil?
+      @payload = dsl_call(&block) if block_given?
+
+      Qswarm.logger.debug "[#{@name.inspect}] Connection #{connection.inspect} emitting #{@payload.inspect}"
+
+      @payload.data = case payload.format
+                      when :json
+                        JSON.parse(@payload.raw, :symbolize_names => true)
+                      when :xml
+                        Nokogiri::XML(@payload.raw)
+                      else # :raw
+                       @payload.raw
+                      end
+
+      [*connection].each do |c|
+        run_filters :before, c
+        call_handlers c
+        run_filters :after, c
+      end
     end
 
     def run
-      @listeners.map { |l| l.run }
+      @clients.each { |name, client| client.run }
+    end
+
+    private
+
+    def run_filters(type, connection)
+      return if @filters[type].nil?
+      @filters[type].each do |guards, client, block|
+        next if client != connection
+        dsl_call(&block) unless guarded?(guards, OpenStruct.new(@payload.headers))
+      end
+    end
+
+    def call_handlers(connection)
+      return if !handlers = @handlers[connection]
+      handlers.each do |guards, block|
+        if !guarded?(guards, OpenStruct.new(@payload.headers))
+          Qswarm.logger.debug "[#{@name.inspect}] Source #{connection.inspect} received #{@payload.inspect}"
+          dsl_call(&block)
+        end
+      end
+    end
+
+    def register_filter(type, client, *guards, &block)
+      raise "Invalid filter: #{type}. Must be :before or :after" unless [:before, :after].include? type
+      @filters[type] ||= []
+      @filters[type] << [guards, client, block]
+    end
+
+    def register_handler(client, *guards, &block)
+      @handlers[client] ||= []
+      @handlers[client] << [guards, block]
     end
+
+    def guarded?(guards, data)
+      return false if guards.nil? || guards.empty?
+      guards.find do |guard|
+        case guard
+        when Symbol
+          !data.__send__(guard)
+        when Array
+          # return FALSE if any item is TRUE
+          !guard.detect { |condition| !guarded?([condition], data) }
+        when Hash
+          # return FALSE unless any inequality is found
+          guard.find do |method, test|
+            value = data.__send__(method)
+            # last_match is the only method found unique to Regexp classes
+            if test.class.respond_to?(:last_match)
+              !(test =~ value.to_s)
+            elsif test.is_a?(Array)
+              !test.include? value
+            else
+              test != value
+            end
+          end
+        when Proc
+          !guard.call(data)
+        end
+      end
+    end
+
   end
 end