redstorm 0.1.0

data/LICENSE.md

@@ -0,0 +1,13 @@
+# License
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,116 @@
+# RedStorm v0.1.0 - JRuby on Storm
+
+RedStorm provides the JRuby integration for the [Storm][storm] distributed realtime computation system.
+
+## disclaimer/limitations
+
+The current Ruby interface is **very** similar to the Java interface. A more idiomatic Ruby interface will be be addded, as I better understand the various usage patterns.
+
+## dependencies
+
+This has been tested on OSX 10.6.8 and Linux 10.04 using Storm 0.5.4 and JRuby 1.6.5
+
+## installation
+``` sh
+$ gem install redstorm
+```
+
+## usage
+
+The currently supported usage pattern is to start your new Storm project in an empty directory, install the RedStorm gem and follow the steps below. There is no layout constrains for your project. The `target/` directory will be created by RedStorm in the root of your project.
+
+### initial setup
+
+Install RedStom dependencies; from your project root directory execute:
+
+``` sh
+$ redstorm install
+```
+
+The `install` command will install all Java jars dependencies using [ruby-maven][ruby-maven] in `target/dependency` and generate & compile the Java bindings in `target/classes`
+
+### run in local mode
+
+Create a topology class that implements the `start` method. The *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
+
+``` sh
+$ redstorm topology_class_file_name.rb
+```
+
+**See examples below** to run examples in local mode or on a production cluster.
+
+### run on production cluster
+
+- generate `target/cluster-topology.jar`. This jar file will include everything in your project directory plus the required dependencies from the `target/` directory:
+
+``` sh
+$ redstorm jar
+```
+
+- submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+
+``` sh
+storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher topology_class_file_name.rb
+```
+
+Basically you must follow the [Storm instructions](https://github.com/nathanmarz/storm/wiki) to [setup a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster) and [submit your topology to the cluster](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
+
+
+## examples
+
+Install the example files into `examples/`:
+
+``` sh
+$ redstorm examples
+```
+
+### local mode
+
+``` sh
+$ redstorm examples/local_exclamation_topology.rb
+$ redstorm examples/local_exclamation_topology2.rb
+$ redstorm examples/local_word_count_topology.rb
+```
+
+This next example requires the use of a [Redis][redis] server on `localhost:6379`
+
+``` sh
+$ redstorm examples/local_redis_word_count_topology.rb
+```
+
+Using `redis-cli`, push words into the `test` list and watch Storm pick them up
+
+### production cluster
+
+The only example compatible with a production cluster is `examples/cluster_word_count_topology.rb`
+
+- genererate the `target/cluster-topology.jar`
+
+``` sh
+$ redstorm jar
+```
+
+- submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+
+``` sh
+storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher examples/cluster_word_count_topology.rb
+```
+
+Basically you must follow the [Storm instructions](https://github.com/nathanmarz/storm/wiki) to [setup a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster) and [submit your topology to the cluster](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
+
+
+## author
+Colin Surprenant, [@colinsurprenant][twitter], [colin.surprenant@needium.com][needium], [colin.surprenant@gmail.com][gmail], [http://github.com/colinsurprenant][github]
+
+## license
+Apache License, Version 2.0. See the LICENSE.md file.
+
+[needium]: colin.surprenant@needium.com
+[gmail]: colin.surprenant@gmail.com
+[twitter]: http://twitter.com/colinsurprenant
+[github]: http://github.com/colinsurprenant
+[rvm]: http://beginrescueend.com/
+[storm]: https://github.com/nathanmarz/storm
+[jruby]: http://jruby.org/
+[ruby-maven]: https://github.com/mkristian/ruby-maven
+[redis]: http://redis.io/

data/Rakefile

@@ -0,0 +1,127 @@
+require 'ant'
+
+begin
+  # will work from gem, since lib dir is in gem require_paths
+  require 'red_storm'
+rescue LoadError
+  # will work within RedStorm dev project
+  $:.unshift './lib'
+  require 'red_storm'
+end
+
+CWD = Dir.pwd
+TARGET_DIR = "#{CWD}/target"
+TARGET_SRC_DIR = "#{TARGET_DIR}/src"
+TARGET_CLASSES_DIR = "#{TARGET_DIR}/classes"  
+TARGET_DEPENDENCY_DIR = "#{TARGET_DIR}/dependency"
+TARGET_DEPENDENCY_UNPACKED_DIR = "#{TARGET_DIR}/dependency-unpacked"
+TARGET_CLUSTER_JAR = "#{TARGET_DIR}/cluster-topology.jar"
+
+JAVA_SRC_DIR = "#{RedStorm::REDSTORM_HOME}/src/main"
+JRUBY_SRC_DIR = "#{RedStorm::REDSTORM_HOME}/lib/red_storm"
+
+SRC_EXAMPLES = "#{RedStorm::REDSTORM_HOME}/examples"
+DST_EXAMPLES = "#{CWD}/examples"
+  
+task :default => [:clean, :build]
+
+task :launch, :class_file do |t, args|
+  system("java -cp \"#{TARGET_CLASSES_DIR}:#{TARGET_DEPENDENCY_DIR}/*\" redstorm.TopologyLauncher #{args[:class_file]}")
+end
+
+task :clean do
+  ant.delete :dir => TARGET_DIR
+end
+
+task :clean_jar do
+  ant.delete :dir => "#{TARGET_DIR}/cluster-topology.jar"
+end
+
+task :setup do  
+  ant.mkdir :dir => TARGET_DIR 
+  ant.mkdir :dir => TARGET_CLASSES_DIR 
+  ant.mkdir :dir => TARGET_SRC_DIR
+  ant.path :id => 'classpath' do  
+    fileset :dir => TARGET_DEPENDENCY_DIR  
+    fileset :dir => TARGET_CLASSES_DIR  
+  end  
+end
+
+task :install => [:deps, :build] do
+  puts("\nRedStorm install completed. All dependencies installed in #{TARGET_DIR}")
+end
+
+task :unpack do
+  system("rmvn dependency:unpack -f #{RedStorm::REDSTORM_HOME}/pom.xml -DoutputDirectory=#{TARGET_DEPENDENCY_UNPACKED_DIR}")
+end
+
+task :jar => [:unpack, :clean_jar] do
+  ant.jar :destfile => TARGET_CLUSTER_JAR do
+    fileset :dir => TARGET_CLASSES_DIR
+    fileset :dir => TARGET_DEPENDENCY_UNPACKED_DIR
+    fileset :dir => CWD do
+      exclude :name => "target/**/*"
+    end
+    manifest do
+      attribute :name => "Main-Class", :value => "redstorm.TopologyLauncher"
+    end
+  end
+  puts("\nRedStorm jar completed. Generated jar file #{TARGET_CLUSTER_JAR}")
+end
+
+task :examples do
+  if File.identical?(SRC_EXAMPLES, DST_EXAMPLES)
+    STDERR.puts("error: cannot copy examples into itself")
+    exit(1)
+  end
+  if File.exist?(DST_EXAMPLES)
+    STDERR.puts("error: directory #{DST_EXAMPLES} already exists")
+    exit(1)
+  end
+
+  puts("copying examples into #{DST_EXAMPLES}")
+  system("mkdir #{DST_EXAMPLES}")
+  system("cp -r #{SRC_EXAMPLES}/* #{DST_EXAMPLES}")
+  puts("\nRedStorm examples completed. All examples copied in #{DST_EXAMPLES}")
+end
+
+task :deps do
+  system("rmvn dependency:copy-dependencies -f #{RedStorm::REDSTORM_HOME}/pom.xml -DoutputDirectory=#{TARGET_DEPENDENCY_DIR}")
+end
+
+task :build => :setup do
+  # compile the JRuby proxy classes to Java
+  build_jruby("#{JRUBY_SRC_DIR}/proxy")
+
+  # compile the generated Java proxy classes
+  build_java_dir("#{TARGET_SRC_DIR}")
+
+  # generate the JRuby topology launcher
+  build_jruby("#{JRUBY_SRC_DIR}/topology_launcher.rb")
+
+  # compile the JRuby proxy classes
+  build_java_dir("#{JAVA_SRC_DIR}")
+
+  # compile the JRuby proxy classes
+  build_java_dir("#{TARGET_SRC_DIR}")
+end 
+
+def build_java_dir(source_folder)
+  puts("\n--> Compiling Java")
+  ant.javac(
+    :srcdir => source_folder,
+    :destdir => TARGET_CLASSES_DIR,
+    :classpathref => 'classpath', 
+    :source => "1.6",
+    :target => "1.6",
+    :debug => "yes",
+    :includeantruntime => "no",
+    :verbose => false,
+    :listfiles => true
+  )
+end  
+
+def build_jruby(source_path)
+  puts("\n--> Compiling JRuby")
+  system("cd #{RedStorm::REDSTORM_HOME}; jrubyc -t #{TARGET_SRC_DIR} --verbose --java -c \"#{TARGET_DEPENDENCY_DIR}/storm-0.5.3.jar\" -c \"#{TARGET_CLASSES_DIR}\" #{source_path}")
+end

data/bin/redstorm

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require 'rubygems' 
+
+begin
+  # will work from gem, since lib dir is in gem require_paths
+  require 'red_storm'
+rescue LoadError
+  # will work within RedStorm dev project
+  $:.unshift './lib'
+  require 'red_storm'
+end
+
+RedStorm::Application.new.run(ARGV.dup)

data/examples/cluster_word_count_topology.rb

@@ -0,0 +1,18 @@
+require 'examples/random_sentence_spout'
+require 'examples/split_sentence_bolt'
+require 'examples/word_count_bolt'
+
+class ClusterWordCountTopology
+  def start(base_class_path)
+    builder = TopologyBuilder.new
+    builder.setSpout(1, JRubySpout.new(base_class_path, "RandomSentenceSpout"), 5)
+    builder.setBolt(2, JRubyBolt.new(base_class_path, "SplitSentenceBolt"), 4).shuffleGrouping(1)
+    builder.setBolt(3, JRubyBolt.new(base_class_path, "WordCountBolt"), 4).fieldsGrouping(2, Fields.new("word"))
+
+    conf = Config.new
+    conf.setDebug(true)
+    conf.setNumWorkers(20);
+    conf.setMaxSpoutPending(1000);
+    StormSubmitter.submitTopology("word-count", conf, builder.createTopology);
+  end
+end

data/examples/exclamation_bolt.rb

@@ -0,0 +1,14 @@
+class ExclamationBolt
+  def prepare(conf, context, collector)
+    @collector = collector
+  end
+
+  def execute(tuple)
+    @collector.emit(tuple, Values.new(tuple.getString(0) + "!!!"))
+    @collector.ack(tuple)
+  end
+
+  def declare_output_fields(declarer)
+    declarer.declare(Fields.new("word"))
+  end
+end

data/examples/local_exclamation_topology.rb

@@ -0,0 +1,23 @@
+java_import 'backtype.storm.testing.TestWordSpout'
+require 'examples/exclamation_bolt'
+
+# this example topology uses the Storm TestWordSpout and our own JRuby ExclamationBolt
+
+class LocalExclamationTopology
+  def start(base_class_path)
+    builder = TopologyBuilder.new
+    
+    builder.setSpout(1, TestWordSpout.new, 10)     
+    builder.setBolt(2, JRubyBolt.new(base_class_path, "ExclamationBolt"), 3).shuffleGrouping(1)
+    builder.setBolt(3, JRubyBolt.new(base_class_path, "ExclamationBolt"), 2).shuffleGrouping(2)
+            
+    conf = Config.new
+    conf.setDebug(true)
+    
+    cluster = LocalCluster.new
+    cluster.submitTopology("test", conf, builder.createTopology)
+    sleep(5)
+    cluster.killTopology("test")
+    cluster.shutdown
+  end
+end

data/examples/local_exclamation_topology2.rb

@@ -0,0 +1,37 @@
+java_import 'backtype.storm.testing.TestWordSpout'
+
+class ExclamationBolt2
+  def prepare(conf, context, collector)
+    @collector = collector
+  end
+
+  def execute(tuple)
+    @collector.emit(tuple, Values.new(tuple.getString(0) + "!!!"))
+    @collector.ack(tuple)
+  end
+
+  def declare_output_fields(declarer)
+    declarer.declare(Fields.new("word"))
+  end
+end
+
+# this example topology uses the Storm TestWordSpout and our own JRuby ExclamationBolt
+
+class LocalExclamationTopology2
+  def start(base_class_path)
+    builder = TopologyBuilder.new
+    
+    builder.setSpout(1, TestWordSpout.new, 10)     
+    builder.setBolt(2, JRubyBolt.new(base_class_path, "ExclamationBolt2"), 3).shuffleGrouping(1)
+    builder.setBolt(3, JRubyBolt.new(base_class_path, "ExclamationBolt2"), 2).shuffleGrouping(2)
+            
+    conf = Config.new
+    conf.setDebug(true)
+    
+    cluster = LocalCluster.new
+    cluster.submitTopology("test", conf, builder.createTopology)
+    sleep(5)
+    cluster.killTopology("test")
+    cluster.shutdown
+  end
+end

data/examples/local_redis_word_count_topology.rb

@@ -0,0 +1,58 @@
+require 'redis'
+require 'thread'
+require 'examples/word_count_bolt'
+
+# RedisWordSpout reads the Redis queue "test" on localhost:6379 
+# and emits each word items pop'ed from the queue.
+class RedisWordSpout
+  def open(conf, context, collector)
+    @collector = collector
+    @q = Queue.new
+    @redis_reader = detach_redis_reader
+  end
+  
+  def next_tuple
+    # per doc nextTuple should not block, and sleep a bit when there's no data to process.
+    if @q.size > 0
+      @collector.emit(Values.new(@q.pop))
+    else
+      sleep(0.1)
+    end
+  end
+
+  def declare_output_fields(declarer)
+    declarer.declare(Fields.new("word"))
+  end
+
+  private
+
+  def detach_redis_reader
+    Thread.new do
+      Thread.current.abort_on_exception = true
+
+      redis = Redis.new(:host => "localhost", :port => 6379)
+      loop do
+        if data = redis.blpop("test", 0)
+          @q << data[1]
+        end
+      end
+    end
+  end
+end
+
+class LocalRedisWordCountTopology
+  def start(base_class_path)
+    builder = TopologyBuilder.new
+    builder.setSpout(1, JRubySpout.new(base_class_path, "RedisWordSpout"), 1)
+    builder.setBolt(2, JRubyBolt.new(base_class_path, "WordCountBolt"), 3).fieldsGrouping(1, Fields.new("word"))
+
+    conf = Config.new
+    conf.setDebug(true)
+    conf.setMaxTaskParallelism(3)
+
+    cluster = LocalCluster.new
+    cluster.submitTopology("redis-word-count", conf, builder.createTopology)
+    sleep(600)
+    cluster.shutdown
+  end
+end

data/examples/local_word_count_topology.rb

@@ -0,0 +1,21 @@
+require 'examples/random_sentence_spout'
+require 'examples/split_sentence_bolt'
+require 'examples/word_count_bolt'
+
+class LocalWordCountTopology
+  def start(base_class_path)
+    builder = TopologyBuilder.new
+    builder.setSpout(1, JRubySpout.new(base_class_path, "RandomSentenceSpout"), 5)
+    builder.setBolt(2, JRubyBolt.new(base_class_path, "SplitSentenceBolt"), 8).shuffleGrouping(1)
+    builder.setBolt(3, JRubyBolt.new(base_class_path, "WordCountBolt"), 12).fieldsGrouping(2, Fields.new("word"))
+
+    conf = Config.new
+    conf.setDebug(true)
+    conf.setMaxTaskParallelism(3)
+
+    cluster = LocalCluster.new
+    cluster.submitTopology("word-count", conf, builder.createTopology)
+    sleep(5)
+    cluster.shutdown
+  end
+end

data/examples/random_sentence_spout.rb

@@ -0,0 +1,26 @@
+class RandomSentenceSpout
+  attr_reader :is_distributed
+
+  def initialize
+    @is_distributed = true
+    @sentences = [
+      "the cow jumped over the moon",
+      "an apple a day keeps the doctor away",
+      "four score and seven years ago",
+      "snow white and the seven dwarfs",
+      "i am at two with nature"
+    ]
+  end 
+
+  def open(conf, context, collector)
+    @collector = collector
+  end
+  
+  def next_tuple
+    @collector.emit(Values.new(@sentences[rand(@sentences.length)]))
+  end
+
+  def declare_output_fields(declarer)
+    declarer.declare(Fields.new("word"))
+  end
+end

data/examples/split_sentence_bolt.rb

@@ -0,0 +1,13 @@
+class SplitSentenceBolt
+  def prepare(conf, context, collector)
+    @collector = collector
+  end
+
+  def execute(tuple)
+    tuple.getString(0).split(" ").each {|w| @collector.emit(Values.new(w)) }
+  end
+
+  def declare_output_fields(declarer)
+    declarer.declare(Fields.new("word"))
+  end
+end

data/examples/word_count_bolt.rb

@@ -0,0 +1,19 @@
+class WordCountBolt
+  def initialize
+    @counts = Hash.new{|h, k| h[k] = 0}
+  end
+
+  def prepare(conf, context, collector)
+    @collector = collector
+  end
+
+  def execute(tuple)
+    word = tuple.getString(0)
+    @counts[word] += 1
+    @collector.emit(Values.new(word, @counts[word]))
+  end
+
+  def declare_output_fields(declarer)
+    declarer.declare(Fields.new("word", "count"))
+  end
+end

data/lib/red_storm.rb

@@ -0,0 +1,6 @@
+module RedStorm
+  REDSTORM_HOME = File.expand_path(File.dirname(__FILE__) + '/..') unless defined?(REDSTORM_HOME)
+end
+
+require 'red_storm/version'
+require 'red_storm/application'

data/lib/red_storm/application.rb

@@ -0,0 +1,20 @@
+require 'rake'
+
+class RedStorm::Application 
+
+  def run(args)
+    if args.size == 1 && File.exist?(args.first)
+      load("#{RedStorm::REDSTORM_HOME}/Rakefile")
+      Rake::Task['launch'].invoke(args)
+    else
+      task = args.shift
+      if ["install", "examples", "jar"].include?(task)
+        load("#{RedStorm::REDSTORM_HOME}/Rakefile")
+        Rake::Task[task].invoke(args)
+      else
+        puts("\nUsage: redstorm install|examples|jar|topology_class_file_name")
+        exit(1)
+      end
+    end
+  end
+end

data/lib/red_storm/proxy/bolt.rb

@@ -0,0 +1,55 @@
+require 'java'
+
+java_import 'backtype.storm.task.OutputCollector'
+java_import 'backtype.storm.task.TopologyContext'
+java_import 'backtype.storm.topology.IRichBolt'
+java_import 'backtype.storm.topology.OutputFieldsDeclarer'
+java_import 'backtype.storm.tuple.Tuple'
+java_import 'backtype.storm.tuple.Fields'
+java_import 'backtype.storm.tuple.Values'
+java_import 'java.util.Map'
+
+java_package 'redstorm.proxy'
+
+# the Bolt class is a proxy to the real bolt to avoid having to deal with all the
+# Java artifacts when creating a bolt.
+#
+# The real bolt class implementation must define these methods:
+# - prepare(conf, context, collector)
+# - execute(tuple)
+# - declare_output_fields
+#
+# and optionnaly:
+# - cleanup
+#
+class Bolt
+  java_implements IRichBolt
+
+  java_signature 'IRichBolt (String base_class_path, String real_bolt_class_name)'
+  def initialize(base_class_path, real_bolt_class_name)
+    @real_bolt = Object.module_eval(real_bolt_class_name).new
+  rescue NameError
+    require base_class_path
+    @real_bolt = Object.module_eval(real_bolt_class_name).new
+  end
+
+  java_signature 'void prepare(Map, TopologyContext, OutputCollector)'
+  def prepare(conf, context, collector)
+    @real_bolt.prepare(conf, context, collector)
+  end
+
+  java_signature 'void execute(Tuple)'
+  def execute(tuple)
+    @real_bolt.execute(tuple)
+  end
+
+  java_signature 'void cleanup()'
+  def cleanup
+    @real_bolt.cleanup if @real_bolt.respond_to?(:cleanup)
+  end
+
+  java_signature 'void declareOutputFields(OutputFieldsDeclarer)'
+  def declareOutputFields(declarer)
+    @real_bolt.declare_output_fields(declarer)
+  end
+end

data/lib/red_storm/proxy/spout.rb

@@ -0,0 +1,73 @@
+require 'java'
+
+java_import 'backtype.storm.spout.SpoutOutputCollector'
+java_import 'backtype.storm.task.TopologyContext'
+java_import 'backtype.storm.topology.IRichSpout'
+java_import 'backtype.storm.topology.OutputFieldsDeclarer'
+java_import 'backtype.storm.tuple.Tuple'
+java_import 'backtype.storm.tuple.Fields'
+java_import 'backtype.storm.tuple.Values'
+java_import 'java.util.Map'
+
+java_package 'redstorm.proxy'
+
+# the Spout class is a proxy to the real spout to avoid having to deal with all the
+# Java artifacts when creating a spout.
+#
+# The real spout class implementation must define these methods:
+# - open(conf, context, collector)
+# - next_tuple
+# - is_distributed
+# - declare_output_fields
+#
+# and optionnaly:
+# - ack(msg_id)
+# - fail(msg_id)
+# - close
+#
+class Spout
+  java_implements IRichSpout
+
+  java_signature 'IRichSpout (String base_class_path, String real_spout_class_name)'
+  def initialize(base_class_path, real_spout_class_name)
+    @real_spout = Object.module_eval(real_spout_class_name).new
+  rescue NameError
+    require base_class_path
+    @real_spout = Object.module_eval(real_spout_class_name).new
+  end
+
+  java_signature 'boolean isDistributed()'
+  def isDistributed
+    @real_spout.respond_to?(:is_distributed) ? @real_spout.is_distributed : false
+  end
+
+  java_signature 'void open(Map, TopologyContext, SpoutOutputCollector)'
+  def open(conf, context, collector)
+    @real_spout.open(conf, context, collector)
+  end
+
+  java_signature 'void close()'
+  def close
+    @real_spout.close if @real_spout.respond_to?(:close)
+  end
+
+  java_signature 'void nextTuple()'
+  def nextTuple
+    @real_spout.next_tuple
+  end
+
+  java_signature 'void ack(Object)'
+  def ack(msg_id)
+    @real_spout.ack(msg_id) if @real_spout.respond_to?(:ack)
+  end
+
+  java_signature 'void fail(Object)'
+  def fail(msg_id)
+    @real_spout.fail(msg_id) if @real_spout.respond_to?(:fail)
+  end
+
+  java_signature 'void declareOutputFields(OutputFieldsDeclarer)'
+  def declareOutputFields(declarer)
+    @real_spout.declare_output_fields(declarer)
+  end
+end

data/lib/red_storm/topology_launcher.rb

@@ -0,0 +1,49 @@
+require 'java'
+require 'rubygems'
+
+begin
+  # will work from gem, since lib dir is in gem require_paths
+  require 'red_storm/version'
+rescue LoadError
+  # will work within RedStorm dev project
+  $:.unshift './lib'
+  require 'red_storm/version'
+end
+
+
+java_import 'backtype.storm.Config'
+java_import 'backtype.storm.LocalCluster'
+java_import 'backtype.storm.StormSubmitter'
+java_import 'backtype.storm.topology.TopologyBuilder'
+java_import 'backtype.storm.tuple.Fields'
+java_import 'backtype.storm.tuple.Tuple'
+java_import 'backtype.storm.tuple.Values'
+
+java_import 'redstorm.storm.jruby.JRubyBolt'
+java_import 'redstorm.storm.jruby.JRubySpout'
+
+java_package 'redstorm'
+
+# TopologyLauncher is the application entry point when launching a topology. Basically it will 
+# call require on the specified Ruby topology/project class file path and call its start method
+class TopologyLauncher
+
+  java_signature 'void main(String[])'
+  def self.main(args)
+    unless args.size > 0 
+      puts("Usage: redstorm topology_class_file_name")
+      exit(1)
+    end
+    class_path = args[0]
+    clazz = camel_case(class_path.split('/').last.split('.').first)
+    puts("redstorm v#{RedStorm::VERSION} launching #{clazz}")
+    require class_path
+    Object.module_eval(clazz).new.start(class_path)
+  end
+
+  private 
+
+  def self.camel_case(s)
+    s.to_s.gsub(/\/(.?)/) { "::#{$1.upcase}" }.gsub(/(?:^|_)(.)/) { $1.upcase }
+  end
+end

data/lib/red_storm/version.rb

@@ -0,0 +1,3 @@
+module RedStorm
+  VERSION = '0.1.0'
+end

data/pom.xml

@@ -0,0 +1,69 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <groupId>redstorm</groupId>
+  <artifactId>redstorm</artifactId>
+  <version>0.1.0</version>
+  <name>RedStorm JRuby on Storm</name>
+
+  <properties>
+    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+  </properties>
+
+  <repositories>
+    <repository>
+      <id>clojars</id>
+      <url>http://clojars.org/repo/</url>
+    </repository>
+  
+    <repository>
+      <id>central</id>
+      <url>http://repo1.maven.org/maven2</url>
+    </repository>
+  </repositories>
+
+  <dependencies>
+    <dependency>
+      <groupId>storm</groupId>
+      <artifactId>storm</artifactId>
+      <version>0.5.4</version>
+    </dependency>
+  
+    <dependency>
+      <groupId>org.jruby</groupId>
+      <artifactId>jruby-complete</artifactId>
+      <version>1.6.5</version>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-dependency-plugin</artifactId>
+        <version>2.3</version>
+        <configuration>
+          <artifactItems>
+            <artifactItem>
+              <groupId>org.jruby</groupId>
+              <artifactId>jruby-complete</artifactId>
+              <version>1.6.5</version>
+              <type>jar</type>
+              <overWrite>false</overWrite>
+            </artifactItem>
+          </artifactItems>
+        </configuration>
+        <executions>
+          <execution>
+            <id>unpack</id>
+            <goals>
+              <goal>unpack</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>
+    </plugins>
+  </build>
+
+</project>

data/src/main/redstorm/storm/jruby/JRubyBolt.java

@@ -0,0 +1,70 @@
+package redstorm.storm.jruby;
+
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.IRichBolt;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.tuple.Tuple;
+import java.util.Map;
+
+/**
+ * the JRubyBolt class is a simple proxy class to the actual bolt implementation in JRuby.
+ * this proxy is required to bypass the serialization/deserialization process when dispatching
+ * the bolts to the workers. JRuby does not yet support serialization from Java
+ * (Java serialization call on a JRuby class). 
+ *
+ * Note that the JRuby bolt proxy class is instanciated in the prepare method which is called after 
+ * deserialization at the worker and in the declareOutputFields method which is called once before 
+ * serialization at topology creation. 
+ */
+public class JRubyBolt implements IRichBolt {
+  IRichBolt _proxyBolt;
+  String _realBoltClassName;
+  String _baseClassPath; 
+  /**
+   * create a new JRubyBolt
+   * 
+   * @param baseClassPath the topology/project base JRuby class file path 
+   * @param realBoltClassName the fully qualified JRuby bolt implementation class name
+   */
+  public JRubyBolt(String baseClassPath, String realBoltClassName) {
+    _baseClassPath = baseClassPath;
+    _realBoltClassName = realBoltClassName;
+  }
+
+  @Override
+  public void prepare(final Map stormConf, final TopologyContext context, final OutputCollector collector) {
+    // create instance of the jruby class here, after deserialization in the workers.
+    _proxyBolt = newProxyBolt(_baseClassPath, _realBoltClassName);
+    _proxyBolt.prepare(stormConf, context, collector);
+  }
+
+  @Override
+  public void execute(Tuple input) {
+    _proxyBolt.execute(input);
+  }
+
+  @Override
+  public void cleanup() {
+    _proxyBolt.cleanup();
+  }
+
+  @Override
+  public void declareOutputFields(OutputFieldsDeclarer declarer) {
+    // declareOutputFields is executed in the topology creation time, before serialisation.
+    // do not set the _proxyBolt instance variable here to avoid JRuby serialization
+    // issues. Just create tmp bolt instance to call declareOutputFields.
+    IRichBolt bolt = newProxyBolt(_baseClassPath, _realBoltClassName);
+    bolt.declareOutputFields(declarer);
+  }
+
+  private static IRichBolt newProxyBolt(String baseClassPath, String realBoltClassName) {
+    try {
+      redstorm.proxy.Bolt proxy = new redstorm.proxy.Bolt(baseClassPath, realBoltClassName);
+      return proxy;
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+}

data/src/main/redstorm/storm/jruby/JRubySpout.java

@@ -0,0 +1,90 @@
+package redstorm.storm.jruby;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.IRichSpout;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.tuple.Tuple;
+import java.util.Map;
+
+/**
+ * the JRubySpout class is a simple proxy class to the actual spout implementation in JRuby.
+ * this proxy is required to bypass the serialization/deserialization process when dispatching
+ * the spout to the workers. JRuby does not yet support serialization from Java
+ * (Java serialization call on a JRuby class). 
+ *
+ * Note that the JRuby spout proxy class is instanciated in the open method which is called after 
+ * deserialization at the worker and in both the declareOutputFields and isDistributed methods which 
+ * are called once before serialization at topology creation. 
+ */
+public class JRubySpout implements IRichSpout {
+  IRichSpout _proxySpout;
+  String _realSpoutClassName;
+  String _baseClassPath;
+  
+  /**
+   * create a new JRubySpout
+   * 
+   * @param baseClassPath the topology/project base JRuby class file path 
+   * @param realSpoutClassName the fully qualified JRuby spout implementation class name
+   */
+  public JRubySpout(String baseClassPath, String realSpoutClassName) {
+    _baseClassPath = baseClassPath;
+    _realSpoutClassName = realSpoutClassName;
+  }
+
+  @Override
+  public boolean isDistributed() {
+    // isDistributed is executed in the topology creation time before serialisation.
+    // do not set the _proxySpout instance variable here to avoid JRuby serialization
+    // issues. Just create tmp spout instance to call isDistributed.
+    IRichSpout spout = newProxySpout(_baseClassPath, _realSpoutClassName);
+    return spout.isDistributed();
+  }
+
+  @Override
+  public void open(final Map conf, final TopologyContext context, final SpoutOutputCollector collector) {
+    // create instance of the jruby proxy class here, after deserialization in the workers.
+    _proxySpout = newProxySpout(_baseClassPath, _realSpoutClassName);
+    _proxySpout.open(conf, context, collector);
+  }
+
+  @Override
+  public void close() {
+    _proxySpout.close();
+  }
+
+  @Override
+  public void nextTuple() {
+    _proxySpout.nextTuple();
+  }
+
+  @Override
+  public void ack(Object msgId) {
+    _proxySpout.ack(msgId);
+  }
+
+  @Override
+  public void fail(Object msgId) {
+    _proxySpout.fail(msgId);
+  }
+
+  @Override
+  public void declareOutputFields(OutputFieldsDeclarer declarer) {
+    // declareOutputFields is executed in the topology creation time before serialisation.
+    // do not set the _proxySpout instance variable here to avoid JRuby serialization
+    // issues. Just create tmp spout instance to call declareOutputFields.
+    IRichSpout spout = newProxySpout(_baseClassPath, _realSpoutClassName);
+    spout.declareOutputFields(declarer);
+  }  
+
+  private static IRichSpout newProxySpout(String baseClassPath, String realSpoutClassName) {
+    try {
+      redstorm.proxy.Spout proxy = new redstorm.proxy.Spout(baseClassPath, realSpoutClassName);
+      return proxy;
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+}

metadata

@@ -0,0 +1,109 @@
+--- !ruby/object:Gem::Specification 
+name: redstorm
+version: !ruby/object:Gem::Version 
+  prerelease: 
+  version: 0.1.0
+platform: ruby
+authors: 
+  - Colin Surprenant
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-11-07 00:00:00 Z
+dependencies: 
+  - !ruby/object:Gem::Dependency 
+    name: rubyforge
+    prerelease: false
+    requirement: &id001 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ">="
+          - !ruby/object:Gem::Version 
+            version: "0"
+    type: :development
+    version_requirements: *id001
+  - !ruby/object:Gem::Dependency 
+    name: rake
+    prerelease: false
+    requirement: &id002 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ~>
+          - !ruby/object:Gem::Version 
+            version: 0.9.2
+    type: :runtime
+    version_requirements: *id002
+  - !ruby/object:Gem::Dependency 
+    name: ruby-maven
+    prerelease: false
+    requirement: &id003 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ~>
+          - !ruby/object:Gem::Version 
+            version: 3.0.3.0.28.5
+    type: :runtime
+    version_requirements: *id003
+description: JRuby integration for the Storm distributed realtime computation system
+email: 
+  - colin.surprenant@gmail.com
+executables: 
+  - redstorm
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+  - lib/red_storm.rb
+  - lib/red_storm/application.rb
+  - lib/red_storm/topology_launcher.rb
+  - lib/red_storm/version.rb
+  - lib/red_storm/proxy/bolt.rb
+  - lib/red_storm/proxy/spout.rb
+  - examples/cluster_word_count_topology.rb
+  - examples/exclamation_bolt.rb
+  - examples/local_exclamation_topology.rb
+  - examples/local_exclamation_topology2.rb
+  - examples/local_redis_word_count_topology.rb
+  - examples/local_word_count_topology.rb
+  - examples/random_sentence_spout.rb
+  - examples/split_sentence_bolt.rb
+  - examples/word_count_bolt.rb
+  - src/main/redstorm/storm/jruby/JRubyBolt.java
+  - src/main/redstorm/storm/jruby/JRubySpout.java
+  - bin/redstorm
+  - Rakefile
+  - pom.xml
+  - README.md
+  - CHANGELOG.md
+  - LICENSE.md
+homepage: https://github.com/colinsurprenant/redstorm
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+  - lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.3.0
+requirements: []
+
+rubyforge_project: redstorm
+rubygems_version: 1.8.9
+signing_key: 
+specification_version: 3
+summary: JRuby on Storm
+test_files: []
+