redstorm 0.5.0 → 0.5.1

data/CHANGELOG.md

@@ -30,3 +30,8 @@
 - JRuby 1.6.7
 - DSL Support for per bolt/spout configuration and spout activate/deactivate in Storm 0.7.x
 - consistent workflow using the redstorm command in local dev or gem mode
+
+# 0.5.1, 06-05-2012
+- better handling of enviroments and paths
+- redstorm bundle command to install topology gems
+- issue #26, fixed examples/native for 0.5.1 compatibility

data/README.md

@@ -1,4 +1,4 @@
-# RedStorm v0.5.0 - JRuby on Storm
+# RedStorm v0.5.1 - JRuby on Storm
 
 [![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
 
@@ -8,9 +8,10 @@ RedStorm provides a Ruby DSL using JRuby integration for the [Storm][storm] dist
 
 Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary. This README reflects the current/master developement state. 
 
-### v0.4.x gem
+### Released gems
 
 - [RedStorm Gem v0.4.x Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.4.x-Documentation)
+- [RedStorm Gem v0.5.0 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.0-Documentation)
 
 ## Dependencies
 
@@ -28,7 +29,7 @@ For each of these contexts, 1.9 mode has to be explicitly specified to avoid any
 
 ## Installation
 
-### Released v0.4.0 gem
+### Latest released gem
 ``` sh
 $ gem install redstorm
 ```
@@ -69,12 +70,12 @@ $ gem install redstorm-x.y.z.gem
 RedStorm now support [Bundler](http://gembundler.com/) for using gems in your topology. Basically supply a `Gemfile` in the root of your project directory and execute this command to install the gems into the `target/gems` directory. **Note that if you change the Gemfile you must rerun this command**.
 
   ``` sh
-  $ redstorm --1.9 gems [--gemfile=GEMFILE]
+  $ redstorm --1.9 bundle [--gemfile=GEMFILE]
   ```
 
-All `bundle install` command options can be passed as options to `redstorm --1.9 gem` like `--gemfile=GEMFILE` to specify a Gemfile in an alternate path.
+All `bundle install` command options can be passed as options to `redstorm --1.9 bundle` like `--gemfile=GEMFILE` to specify a Gemfile in an alternate path.
 
-Basically, the `redstorm --1.9 gems` command installs the *Bundler* and *Rake* gems and all the gems specified in the Gemfile into the `target/gems` directory. The idea is that in order for the topology to run in a Storm cluster, everything, including the fully *installed* gems, must be packaged and self-contained into a single JAR file. This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the JAR file. This could possibly lead to problems if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have *native* C/FFI extensions.
+Basically, the `redstorm --1.9 bundle` command installs the *Bundler* and *Rake* gems and all the gems specified in the Gemfile into the `target/gems` directory. The idea is that in order for the topology to run in a Storm cluster, everything, including the fully *installed* gems, must be packaged and self-contained into a single JAR file. This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the JAR file. This could possibly lead to problems if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have *native* C/FFI extensions.
 
 ### Run in local mode
 
@@ -123,7 +124,7 @@ $ redstorm --1.9 local examples/simple/word_count_topology.rb
 To run `examples/simple/redis_word_count_topology.rb` you need a [Redis][redis] server running on `localhost:6379`
 
 ``` sh
-$ redstorm --1.9 gems --gemfile examples/simple/Gemfile
+$ redstorm --1.9 bundle --gemfile examples/simple/Gemfile
 ```
 
 Run the topology in local mode
@@ -155,7 +156,7 @@ All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/
 - to run `examples/simple/redis_word_count_topology.rb` you need a [Redis][redis] server running on `localhost:6379`
 
    ``` sh
-  $ redstorm --1.9 gems --gemfile examples/simple/Gemfile
+  $ redstorm --1.9 bundle --gemfile examples/simple/Gemfile
   $ redstorm --1.9 jar examples
   $ storm jar ./target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster examples/simple/redis_word_count_topology.rb
   ```
@@ -179,25 +180,34 @@ It is possible to fork the RedStorm project and run local and remote/cluster top
 ### Workflow
 
 - fork project and create branch
+
 - install required gems
 
   ```sh
   $ jruby --1.9 -S bundle install
   ```
 
+---
+
 - install dependencies in `target/dependencies`
 
   ```sh
   $ bin/redstorm --1.9 deps
   ```
 
+  **if you modify any of the RedStorm Ruby code** in `lib/red_storm`, you need to run this to refresh code in `target/`.
+
+---
+
 - generate and build Java source into `target/classes`
 
   ```sh
   $ bin/redstorm --1.9 build
   ```
 
-  if you modify any of the Java binding code, you need to run this to rebuild the bindings
+  **if you modify any of the Java binding code**, you need to run this to rebuild the bindings
+
+---
 
 - run topology in **local** Storm mode
 
@@ -207,16 +217,20 @@ It is possible to fork the RedStorm project and run local and remote/cluster top
 
   If you only make changes to your topology code, this is the only step you need to repeat to try your updated code.
 
+---
+
 - generate remote cluster topology jar into `target/cluster-topology.jar`, including the `mydir/` directory.
 
   ```sh
   $ bin/redstorm --1.9 jar mydir otherdir1 otherdir2 ...
   ```
 
+---
+
 - **if you add/change** Gemfile for your topology, install gems in `target/gems`. Alternate gemfile path can be specified using --gemfile=GEMFILE
 
   ```sh
-  $ bin/redstorm --1.9 gems [--gemfile=GEMFILE]
+  $ bin/redstorm --1.9 bundle [--gemfile=GEMFILE]
   ```
 
   **do not forget** to rerurn `bin/redstorm --1.9 jar ...` to pick up these gems, before submitting your topology on a remote cluster.

data/examples/native/Gemfile

@@ -0,0 +1,2 @@
+source :rubygems
+gem 'redis'

data/examples/native/cluster_word_count_topology.rb

@@ -14,7 +14,7 @@ module RedStorm
         builder.setBolt('SplitSentenceBolt', JRubyBolt.new(base_class_path, "RedStorm::Examples::SplitSentenceBolt"), 4).shuffleGrouping('RandomSentenceSpout')
         builder.setBolt('WordCountBolt', JRubyBolt.new(base_class_path, "RedStorm::Examples::WordCountBolt"), 4).fieldsGrouping('SplitSentenceBolt', Fields.new("word"))
 
-        conf = Config.new
+        conf = Backtype::Config.new
         conf.setDebug(true)
         conf.setNumWorkers(20);
         conf.setMaxSpoutPending(1000);

data/examples/native/exclamation_bolt.rb

@@ -10,6 +10,9 @@ module RedStorm
         @collector.ack(tuple)
       end
 
+      def get_component_configuration
+      end
+
       def declare_output_fields(declarer)
         declarer.declare(Fields.new("word"))
       end

data/examples/native/local_exclamation_topology.rb

@@ -17,7 +17,7 @@ module RedStorm
         builder.setBolt('ExclamationBolt1', JRubyBolt.new(base_class_path, 'RedStorm::Examples::ExclamationBolt'), 3).shuffleGrouping('TestWordSpout')
         builder.setBolt('ExclamationBolt2', JRubyBolt.new(base_class_path, 'RedStorm::Examples::ExclamationBolt'), 3).shuffleGrouping('ExclamationBolt1')
                 
-        conf = Config.new
+        conf = Backtype::Config.new
         conf.setDebug(true)
         
         cluster = LocalCluster.new

data/examples/native/local_exclamation_topology2.rb

@@ -14,6 +14,9 @@ module RedStorm
         @collector.ack(tuple)
       end
 
+      def get_component_configuration
+      end
+
       def declare_output_fields(declarer)
         declarer.declare(Fields.new("word"))
       end
@@ -31,7 +34,7 @@ module RedStorm
         builder.setBolt('ExclamationBolt21', JRubyBolt.new(base_class_path, "RedStorm::Examples::ExclamationBolt2"), 3).shuffleGrouping('TestWordSpout')
         builder.setBolt('ExclamationBolt22', JRubyBolt.new(base_class_path, "RedStorm::Examples::ExclamationBolt2"), 2).shuffleGrouping('ExclamationBolt21')
                 
-        conf = Config.new
+        conf = Backtype::Config.new
         conf.setDebug(true)
         
         cluster = LocalCluster.new

data/examples/native/local_redis_word_count_topology.rb

@@ -1,3 +1,4 @@
+require 'bundler/setup'
 require 'redis'
 require 'thread'
 require 'lib/red_storm'
@@ -23,6 +24,9 @@ module RedStorm
         end
       end
 
+      def get_component_configuration
+      end
+
       def declare_output_fields(declarer)
         declarer.declare(Fields.new("word"))
       end
@@ -51,7 +55,7 @@ module RedStorm
         builder.setSpout('RedisWordSpout', JRubySpout.new(base_class_path, "RedStorm::Examples::RedisWordSpout"), 1)
         builder.setBolt('WordCountBolt', JRubyBolt.new(base_class_path, "RedStorm::Examples::WordCountBolt"), 3).fieldsGrouping('RedisWordSpout', Fields.new("word"))
 
-        conf = Config.new
+        conf = Backtype::Config.new
         conf.setDebug(true)
         conf.setMaxTaskParallelism(3)
 

data/examples/native/local_word_count_topology.rb

@@ -14,7 +14,7 @@ module Examples
       builder.setBolt('SplitSentenceBolt', JRubyBolt.new(base_class_path, "RedStorm::Examples::SplitSentenceBolt"), 8).shuffleGrouping('RandomSentenceSpout')
       builder.setBolt('WordCountBolt', JRubyBolt.new(base_class_path, "RedStorm::Examples::WordCountBolt"), 12).fieldsGrouping('SplitSentenceBolt', Fields.new("word"))
 
-      conf = Config.new
+      conf = Backtype::Config.new
       conf.setDebug(true)
       conf.setMaxTaskParallelism(3)
 

data/examples/native/random_sentence_spout.rb

@@ -1,10 +1,7 @@
 module RedStorm
   module Examples
     class RandomSentenceSpout
-      attr_reader :is_distributed
-
       def initialize
-        @is_distributed = true
         @sentences = [
           "the cow jumped over the moon",
           "an apple a day keeps the doctor away",
@@ -22,6 +19,9 @@ module RedStorm
         @collector.emit(Values.new(@sentences[rand(@sentences.length)]))
       end
 
+      def get_component_configuration
+      end
+
       def declare_output_fields(declarer)
         declarer.declare(Fields.new("word"))
       end

data/examples/native/split_sentence_bolt.rb

@@ -9,6 +9,9 @@ module RedStorm
         tuple.getString(0).split(" ").each {|w| @collector.emit(Values.new(w)) }
       end
 
+      def get_component_configuration
+      end
+
       def declare_output_fields(declarer)
         declarer.declare(Fields.new("word"))
       end

data/examples/native/word_count_bolt.rb

@@ -15,6 +15,9 @@ module RedStorm
         @collector.emit(Values.new(word, @counts[word]))
       end
 
+      def get_component_configuration
+      end
+
       def declare_output_fields(declarer)
         declarer.declare(Fields.new("word", "count"))
       end

data/lib/red_storm.rb

@@ -1,34 +1,41 @@
+# this is the entry point for these two contexts:
+# - running red_storm.rake
+# - at remote cluster topology execution. Once topology_launcher.rb has submitted the topology
+#   the spouts and bolts classes will be instanciated and will require red_storm.rb
+
 # we depends on rubygems begings loaded at this point for setting up gem/bundle environments
 # explicitely requiring rubygems is required in remote cluster environment
 require 'rubygems'
 
+# setup some environment constants
+# this is required here and in topology_launcher.rb which are both 
+# entry points in redstorm. 
 module RedStorm
-  LAUNCH_PATH = File.expand_path(File.dirname(__FILE__))
-  JAR_CONTEXT = !!(LAUNCH_PATH =~ /\.jar!$/)
+  TOPOLOGY_LAUNCHED = defined?(LAUNCH_PATH)
 
-  if JAR_CONTEXT
-    REDSTORM_HOME = LAUNCH_PATH
-    TARGET_PATH = LAUNCH_PATH
-    BUNDLE_GEMFILE = "#{TARGET_PATH}/bundler/Gemfile"
-    BUNDLE_PATH = "#{TARGET_PATH}/bundler/#{Gem.ruby_engine}/#{Gem::ConfigMap[:ruby_version]}/"
-    GEM_PATH = "#{TARGET_PATH}/gems/"
-  else
-    REDSTORM_HOME = File.expand_path(LAUNCH_PATH + '/..')
-    TARGET_PATH = Dir.pwd
-    BUNDLE_GEMFILE = "#{TARGET_PATH}/target/gems/bundler/Gemfile"
-    BUNDLE_PATH = "#{TARGET_PATH}/target/gems/bundler/#{Gem.ruby_engine}/#{Gem::ConfigMap[:ruby_version]}/"
-    GEM_PATH = "#{TARGET_PATH}/target/gems/gems"
-  end
+  # do not redefine if already defined in topology_launcher.rb
+  unless TOPOLOGY_LAUNCHED
+    LAUNCH_PATH = File.expand_path(File.dirname(__FILE__))
+    JAR_CONTEXT = !!(LAUNCH_PATH =~ /\.jar!$/)
 
-  # setup bundler environment
-  ENV['BUNDLE_GEMFILE'] = RedStorm::BUNDLE_GEMFILE
-  ENV['BUNDLE_PATH'] = RedStorm::BUNDLE_PATH
-  ENV["GEM_PATH"] = RedStorm::GEM_PATH
-  ENV['BUNDLE_DISABLE_SHARED_GEMS'] = "1"
+    if JAR_CONTEXT
+      BASE_PATH = LAUNCH_PATH
+    else
+      BASE_PATH = Dir.pwd
+    end
+  end
 end
 
-$:.unshift RedStorm::TARGET_PATH
+unless RedStorm::JAR_CONTEXT
+  # in JAR context red_storm.rb and red_storm/* is in the JAR root.
+  # otherwise this is in lib/...
+  $:.unshift "#{RedStorm::BASE_PATH}/lib" 
+end
 
+unless RedStorm::TOPOLOGY_LAUNCHED
+  require 'red_storm/environment'
+  RedStorm.setup_gems if RedStorm::JAR_CONTEXT
+end
 
 require 'red_storm/version'
 require 'red_storm/configuration'
@@ -36,13 +43,3 @@ require 'red_storm/application'
 require 'red_storm/simple_bolt'
 require 'red_storm/simple_spout'
 require 'red_storm/simple_topology'
-
-# puts("************************ PWD=#{Dir.pwd}")
-# puts("************************ RedStorm::JAR_CONTEXT=#{RedStorm::JAR_CONTEXT}")
-# puts("************************ RedStorm::LAUNCH_PATH=#{RedStorm::LAUNCH_PATH}")
-# puts("************************ RedStorm::REDSTORM_HOME=#{RedStorm::REDSTORM_HOME}")
-# puts("************************ RedStorm::TARGET_PATH=#{RedStorm::TARGET_PATH}")
-# puts("************************ RedStorm::GEM_PATH=#{RedStorm::GEM_PATH}")
-# puts("************************ ENV['BUNDLE_GEMFILE']=#{ENV['BUNDLE_GEMFILE']}")
-# puts("************************ ENV['BUNDLE_PATH']=#{ENV['BUNDLE_PATH']}")
-# puts("************************ ENV['GEM_PATH']=#{ENV['GEM_PATH']}")

data/lib/red_storm/application.rb

@@ -7,7 +7,7 @@ module RedStorm
     TASKS_FILE = "#{RedStorm::REDSTORM_HOME}/lib/tasks/red_storm.rake" 
 
     def usage
-      puts("usage: redstorm [--1.8|--1.9]  install | deps | build | examples | gems [--gemfile GEMFILE_PATH] | jar DIR1, DIR2, ... | local TOPOLOGY_CLASS_PATH")
+      puts("usage: redstorm [--1.8|--1.9]  install | deps | build | examples | gems | bundle [--gemfile GEMFILE_PATH] | jar DIR1, DIR2, ... | local TOPOLOGY_CLASS_PATH")
       exit(1)
     end
 
@@ -16,7 +16,7 @@ module RedStorm
         version = args.delete("--1.8") || args.delete("--1.9") || DEFAULT_RUBY_VERSION
         RUNTIME['RUBY_VERSION'] = version
 
-        if ["install", "examples", "jar", "gems", "deps", "build"].include?(args[0])
+        if ["install", "examples", "jar", "gems", "bundle", "deps", "build"].include?(args[0])
           load(TASKS_FILE)
           Rake::Task[args.shift].invoke(args.join(":"))
           exit

data/lib/red_storm/environment.rb

@@ -0,0 +1,28 @@
+module RedStorm
+  # LAUNCH_PATH, BASE_PATH and JAR_CONTEXT must be set before requiring this module
+  # typically, they must be set in either red_storm.rb and topology_launcher.rd
+  # which are the 2 entry points
+
+  if JAR_CONTEXT
+    REDSTORM_HOME = LAUNCH_PATH
+    TARGET_PATH = LAUNCH_PATH
+    BUNDLE_GEMFILE = "#{TARGET_PATH}/bundler/Gemfile"
+    BUNDLE_PATH = "#{TARGET_PATH}/bundler/#{Gem.ruby_engine}/#{Gem::ConfigMap[:ruby_version]}/"
+    GEM_PATH = "#{TARGET_PATH}/gems/"
+  else
+    REDSTORM_HOME = File.expand_path(LAUNCH_PATH + '/..')
+    TARGET_PATH = "#{BASE_PATH}/target"
+    BUNDLE_GEMFILE = "#{TARGET_PATH}/gems/bundler/Gemfile"
+    BUNDLE_PATH = "#{TARGET_PATH}/gems/bundler/#{Gem.ruby_engine}/#{Gem::ConfigMap[:ruby_version]}/"
+    GEM_PATH = "#{TARGET_PATH}/gems/gems"
+  end
+
+  def setup_gems
+    ENV['BUNDLE_GEMFILE'] = RedStorm::BUNDLE_GEMFILE
+    ENV['BUNDLE_PATH'] = RedStorm::BUNDLE_PATH
+    ENV["GEM_PATH"] = RedStorm::GEM_PATH
+    ENV['BUNDLE_DISABLE_SHARED_GEMS'] = "1"
+  end
+
+  module_function :setup_gems
+end

data/lib/red_storm/topology_launcher.rb

@@ -1,14 +1,4 @@
 require 'java'
-require 'rubygems'
-
-begin
-  # will work from gem, since lib dir is in gem require_paths
-  require 'red_storm'
-rescue LoadError
-  # will work within RedStorm dev project
-  $:.unshift './lib'
-  require 'red_storm'
-end
 
 # see https://github.com/colinsurprenant/redstorm/issues/7
 module Backtype
@@ -27,20 +17,48 @@ java_import 'redstorm.storm.jruby.JRubySpout'
 
 java_package 'redstorm'
 
+# setup some environment constants
+# this is required here and in red_storm.rb which are both 
+# entry points in redstorm. 
+module RedStorm
+  LAUNCH_PATH = File.expand_path(File.dirname(__FILE__))
+  JAR_CONTEXT = !!(LAUNCH_PATH =~ /\.jar!$/)
+
+  if JAR_CONTEXT
+    BASE_PATH = LAUNCH_PATH
+    LIB_PATH = "#{BASE_PATH}/lib"
+  else
+    BASE_PATH = Dir.pwd
+    LIB_PATH = "#{BASE_PATH}/target/lib"
+  end
+end
+
 # TopologyLauncher is the application entry point when launching a topology. Basically it will 
 # call require on the specified Ruby topology class file path and call its start method
 class TopologyLauncher
 
   java_signature 'void main(String[])'
   def self.main(args)
-    unless args.size > 1 
+    # this is the entry point for these two contexts:
+    # - runnig a topology in local mode. the current Ruby env will stay the same at topology execution
+    # - submitting a topology in cluster mode. the current Ruby env will be valid only at topology submission. At topology execution
+    #   in the cluster, the new entry point will be the red_storm.rb, topology_launcher will not be called
+
+    unless args.size > 1
       puts("Usage: redstorm local|cluster topology_class_file_name")
       exit(1)
     end
+
     env = args[0].to_sym
     class_path = args[1]
 
-    require "./#{class_path}" # ./ for 1.9 compatibility
+    $:.unshift "#{RedStorm::BASE_PATH}"
+    $:.unshift "#{RedStorm::LIB_PATH}"
+
+    require 'red_storm/environment'
+    RedStorm.setup_gems
+
+    require "#{class_path}" 
 
     topology_name = RedStorm::Configuration.topology_class.respond_to?(:topology_name) ? "/#{RedStorm::Configuration.topology_class.topology_name}" : ''
     puts("RedStorm v#{RedStorm::VERSION} starting topology #{RedStorm::Configuration.topology_class.name}#{topology_name} in #{env.to_s} environment")

data/lib/red_storm/version.rb

@@ -1,3 +1,3 @@
 module RedStorm
-  VERSION = '0.5.0'
+  VERSION = '0.5.1'
 end

data/lib/tasks/red_storm.rake

@@ -1,20 +1,22 @@
 require 'ant'
-
-begin
-  # will work from gem, since lib dir is in gem require_paths
-  require 'red_storm'
-rescue LoadError
-  # will work within RedStorm dev project
-  $:.unshift './lib'
-  require 'red_storm'
-end
-
+require 'red_storm'
+
+# begin
+#   # will work from gem, since lib dir is in gem require_paths
+#   require 'red_storm'
+# rescue LoadError
+#   # will work within RedStorm dev project
+#   $:.unshift './lib'
+#   require 'red_storm'
+# end
+ 
 INSTALL_STORM_VERSION = "0.7.1"
 INSTALL_JRUBY_VERSION = "1.6.7"
 DEFAULT_GEMFILE = "Gemfile"
 
 CWD = Dir.pwd
 TARGET_DIR = "#{CWD}/target"
+TARGET_LIB_DIR = "#{TARGET_DIR}/lib"
 TARGET_SRC_DIR = "#{TARGET_DIR}/src"
 TARGET_CLASSES_DIR = "#{TARGET_DIR}/classes"  
 TARGET_DEPENDENCY_DIR = "#{TARGET_DIR}/dependency"
@@ -23,15 +25,16 @@ TARGET_MARKERS_DIR = "#{TARGET_DIR}/dependency-markers"
 TARGET_GEMS_DIR = "#{TARGET_DIR}/gems"
 TARGET_CLUSTER_JAR = "#{TARGET_DIR}/cluster-topology.jar"
 
-JAVA_SRC_DIR = "#{RedStorm::REDSTORM_HOME}/src/main"
-JRUBY_SRC_DIR = "#{RedStorm::REDSTORM_HOME}/lib"
+REDSTORM_JAVA_SRC_DIR = "#{RedStorm::REDSTORM_HOME}/src/main"
+REDSTORM_LIB_DIR = "#{RedStorm::REDSTORM_HOME}/lib"
 
 SRC_EXAMPLES = "#{RedStorm::REDSTORM_HOME}/examples"
 DST_EXAMPLES = "#{CWD}/examples"
 
 task :launch, :env, :class_file do |t, args|
   version_token = RedStorm::RUNTIME['RUBY_VERSION'] == "--1.9" ? "RUBY1_9" : "RUBY1_8"
-  gem_home = ENV["GEM_HOME"].to_s.empty? ? " -Djruby.gem.home=`gem env home`" : ""
+  # gem_home = ENV["GEM_HOME"].to_s.empty? ? " -Djruby.gem.home=`gem env home`" : ""
+  gem_home = " -Djruby.gem.home=#{RedStorm::GEM_PATH}" 
   command = "java -Djruby.compat.version=#{version_token} -cp \"#{TARGET_CLASSES_DIR}:#{TARGET_DEPENDENCY_DIR}/*\"#{gem_home} redstorm.TopologyLauncher #{args[:env]} #{args[:class_file]}"
   puts("launching #{command}")
   system(command)
@@ -64,15 +67,16 @@ task :install => [:deps, :build, :gems] do
 end
 
 task :unpack do
-  system("rmvn dependency:unpack \
-          -f #{RedStorm::REDSTORM_HOME}/pom.xml \
-          -DoutputDirectory=#{TARGET_DEPENDENCY_UNPACKED_DIR} \
-          -DmarkersDirectory=#{TARGET_MARKERS_DIR} \
-          -Dstorm-storm.version=#{INSTALL_STORM_VERSION} \
-          -Dorg.jruby-jruby-complete.version=#{INSTALL_JRUBY_VERSION}")
+  system("rmvn dependency:unpack " + \
+         "-f #{RedStorm::REDSTORM_HOME}/pom.xml " + \
+         "-DoutputDirectory=#{TARGET_DEPENDENCY_UNPACKED_DIR} " + \
+         "-DmarkersDirectory=#{TARGET_MARKERS_DIR} " + \
+         "-Dstorm-storm.version=#{INSTALL_STORM_VERSION} " + \
+         "-Dorg.jruby-jruby-complete.version=#{INSTALL_JRUBY_VERSION}")
 end
 
 task :jar, [:include_dir] => [:unpack, :clean_jar] do |t, args|
+  puts("\n--> Generating JAR file #{TARGET_CLUSTER_JAR}")
   ant.jar :destfile => TARGET_CLUSTER_JAR do
     fileset :dir => TARGET_CLASSES_DIR
     fileset :dir => TARGET_DEPENDENCY_UNPACKED_DIR
@@ -80,7 +84,9 @@ task :jar, [:include_dir] => [:unpack, :clean_jar] do |t, args|
       # remove bundler config dir to avoid setting BUNDLE_PATH
       exclude :name => "bundler/.bundle/**"
     end
-    fileset :dir => JRUBY_SRC_DIR do
+    # red_storm.rb and red_storm/* must be in root of jar so that "require 'red_storm'"
+    # in bolts/spouts works in jar context
+    fileset :dir => TARGET_LIB_DIR do
       exclude :name => "tasks/**"
     end
     fileset :dir => CWD do
@@ -90,7 +96,7 @@ task :jar, [:include_dir] => [:unpack, :clean_jar] do |t, args|
       attribute :name => "Main-Class", :value => "redstorm.TopologyLauncher"
     end
   end
-  puts("\nRedStorm generated jar file #{TARGET_CLUSTER_JAR}")
+  puts("\nRedStorm generated JAR file #{TARGET_CLUSTER_JAR}")
 end
 
 task :examples do
@@ -103,63 +109,82 @@ task :examples do
     exit(1)
   end
 
-  puts("copying examples into #{DST_EXAMPLES}")
+  puts("\n--> Installing examples into #{DST_EXAMPLES}")
   system("mkdir #{DST_EXAMPLES}")
   system("cp -r #{SRC_EXAMPLES}/* #{DST_EXAMPLES}")
-  puts("\nRedStorm examples completed. All examples copied in #{DST_EXAMPLES}")
 end
 
 task :deps => :setup do
   puts("\n--> Installing dependencies")
-  system("rmvn dependency:copy-dependencies \
-          -f #{RedStorm::REDSTORM_HOME}/pom.xml \
-          -DoutputDirectory=#{TARGET_DEPENDENCY_DIR} \
-          -DmarkersDirectory=#{TARGET_MARKERS_DIR} \
-          -Dstorm-storm.version=#{INSTALL_STORM_VERSION} \
-          -Dorg.jruby-jruby-complete.version=#{INSTALL_JRUBY_VERSION}")
+  # install maven dependencies in target
+  system("rmvn dependency:copy-dependencies " + \
+         "-f #{RedStorm::REDSTORM_HOME}/pom.xml " + \
+         "-DoutputDirectory=#{TARGET_DEPENDENCY_DIR} " + \
+         "-DmarkersDirectory=#{TARGET_MARKERS_DIR} " + \
+         "-Dstorm-storm.version=#{INSTALL_STORM_VERSION} " + \
+         "-Dorg.jruby-jruby-complete.version=#{INSTALL_JRUBY_VERSION}")
+
+  # copy RedStorm lib dir in target
+  system("cp -r #{REDSTORM_LIB_DIR} #{TARGET_DIR}")
 end
 
 task :build => :setup do
   # compile the JRuby proxy classes to Java
-  build_jruby("#{JRUBY_SRC_DIR}/red_storm/proxy")
+  build_jruby("#{REDSTORM_LIB_DIR}/red_storm/proxy")
 
   # compile the generated Java proxy classes
   build_java_dir("#{TARGET_SRC_DIR}")
 
   # generate the JRuby topology launcher
-  build_jruby("#{JRUBY_SRC_DIR}/red_storm/topology_launcher.rb")
+  build_jruby("#{REDSTORM_LIB_DIR}/red_storm/topology_launcher.rb")
 
   # compile the JRuby proxy classes
-  build_java_dir("#{JAVA_SRC_DIR}")
+  build_java_dir("#{REDSTORM_JAVA_SRC_DIR}")
 
   # compile the JRuby proxy classes
   build_java_dir("#{TARGET_SRC_DIR}")
 end
 
-task :gems, [:bundler_options] => :setup do |t, args|
+task :bundle, [:bundler_options] => :gems do |t, args|
   bundler_options = args[:bundler_options].to_s.split(":").join(" ")
 
-  # basically copy original Gemfile to target/gems/bundler and install into this dir
-  gemfile = bundler_options =~ /--gemfile\s+([^\s]+)/ ? $1 : DEFAULT_GEMFILE
-  if bundler_options =~ /--gemfile\s+[^\s]+/
-    bundler_options.gsub!(/--gemfile\s+[^\s]+/, "--gemfile #{TARGET_GEMS_DIR}/bundler/Gemfile")
+  gemfile = ""
+  if bundler_options.split(" ").size == 1
+    gemfile = bundler_options
+    bundler_options = ""
   else
-    bundler_options = bundler_options + " --gemfile #{TARGET_GEMS_DIR}/bundler/Gemfile"
+    gemfile = (bundler_options =~ /--gemfile\s+([^\s]+)/) ? $1 : DEFAULT_GEMFILE
   end
 
-  puts("\n--> Installing gems in #{TARGET_GEMS_DIR}/gems")
-  system("gem install bundler --install-dir #{TARGET_GEMS_DIR}/gems --no-ri --no-rdoc --quiet --no-verbose")
-  system("gem install rake --install-dir #{TARGET_GEMS_DIR}/gems --no-ri --no-rdoc --quiet --no-verbose")
+  if bundler_options =~ /--gemfile\s+[^\s]+/
+    bundler_options.gsub!(/--gemfile\s+[^\s]+/, "--gemfile ./Gemfile ")
+  else
+    bundler_options = bundler_options + " --gemfile ./Gemfile"
+  end
+  bundler_options = bundler_options + " --path ./"
 
   if File.exist?(gemfile)
     puts("\n--> Bundling gems in #{TARGET_GEMS_DIR}/bundler using #{gemfile}")
     system("cp #{gemfile} #{TARGET_GEMS_DIR}/bundler/Gemfile")
-    system("jruby #{RedStorm::RUNTIME['RUBY_VERSION']} -S bundle install #{bundler_options} --path #{TARGET_GEMS_DIR}/bundler/")
+    cmd = "(cd #{TARGET_GEMS_DIR}/bundler; " + \
+          "unset BUNDLE_GEMFILE; " + \
+          "unset BUNDLE_PATH; " + \
+          "unset RUBYOPT; " + \
+          "export GEM_PATH=#{RedStorm::GEM_PATH}; " + \
+          "export GEM_HOME=#{RedStorm::GEM_PATH}; " + \
+          "jruby #{RedStorm::RUNTIME['RUBY_VERSION']} -S bundle install #{bundler_options})"
+    system(cmd)
   elsif gemfile != DEFAULT_GEMFILE
     puts("WARNING: #{gemfile} not found, cannot bundle gems")
   end
 end
 
+task :gems => :setup do
+  puts("\n--> Installing base gems in #{TARGET_GEMS_DIR}/gems")
+  system("gem install bundler --install-dir #{TARGET_GEMS_DIR}/gems --no-ri --no-rdoc --quiet --no-verbose")
+  system("gem install rake --install-dir #{TARGET_GEMS_DIR}/gems --no-ri --no-rdoc --quiet --no-verbose")
+end
+
 def build_java_dir(source_folder)
   puts("\n--> Compiling Java")
   ant.javac(

data/pom.xml

@@ -5,7 +5,7 @@
 
   <groupId>redstorm</groupId>
   <artifactId>redstorm</artifactId>
-  <version>0.5.0</version>
+  <version>0.5.1</version>
   <name>RedStorm</name>
 
   <properties>

metadata

@@ -2,7 +2,7 @@
 name: redstorm
 version: !ruby/object:Gem::Version 
   prerelease: 
-  version: 0.5.0
+  version: 0.5.1
 platform: ruby
 authors: 
   - Colin Surprenant
@@ -10,7 +10,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2012-06-01 00:00:00 Z
+date: 2012-06-05 00:00:00 Z
 dependencies: 
   - !ruby/object:Gem::Dependency 
     name: rspec
@@ -59,6 +59,7 @@ files:
   - lib/red_storm/application.rb
   - lib/red_storm/configuration.rb
   - lib/red_storm/configurator.rb
+  - lib/red_storm/environment.rb
   - lib/red_storm/loggable.rb
   - lib/red_storm/simple_bolt.rb
   - lib/red_storm/simple_spout.rb
@@ -70,6 +71,7 @@ files:
   - lib/tasks/red_storm.rake
   - examples/native/cluster_word_count_topology.rb
   - examples/native/exclamation_bolt.rb
+  - examples/native/Gemfile
   - examples/native/local_exclamation_topology.rb
   - examples/native/local_exclamation_topology2.rb
   - examples/native/local_redis_word_count_topology.rb