redstorm 0.6.3 → 0.6.4

data/CHANGELOG.md

@@ -62,4 +62,13 @@
 - issue #28, allow specification of output_fields in topology DSL
 - issue #41, add support for ShellBolt and ShellSpout
 - issue #49, redstorm bundle not picking up default group in Gemfile
-- support constructor parameters for Java spout/bolt in topology DSL
+- support constructor parameters for Java spout/bolt in topology DSL
+
+# 0.6.4, 10-19-2012
+- Storm 0.8.1 and JRuby 1.6.8
+- improved and more flexible jar dependencies handling
+- issue #36, fix slf4j-api-1.6.3 and slf4j-log4j12-1.5.8 version conflict
+- issue #37, fix dependencies xml files naming
+- issue #47, Log4j Logger class conflict
+- issue #48, add support for external Jars
+- issue #50, update RedStorm to target Storm 0.8

data/README.md

@@ -1,8 +1,8 @@
-# RedStorm v0.6.3 - JRuby on Storm
+# RedStorm v0.6.4 - JRuby on Storm
 
 [![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
 
-RedStorm provides a Ruby DSL using JRuby integration for the [Storm][storm] distributed realtime computation system.
+RedStorm provides a Ruby DSL using JRuby integration for the [Storm](https://github.com/nathanmarz/storm/) distributed realtime computation system.
 
 ## Documentation
 
@@ -13,10 +13,11 @@ Chances are new versions of RedStorm will introduce changes that will break comp
 - [RedStorm Gem v0.4.x Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.4.x-Documentation)
 - [RedStorm Gem v0.5.0 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.0-Documentation)
 - [RedStorm Gem v0.5.1 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.1-Documentation)
+- [RedStorm Gem v0.6.3 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.6.3-Documentation)
 
 ## Dependencies
 
-Tested on OSX 10.6.8 and Linux 10.04 & 11.10 using Storm 0.7.4 and JRuby 1.6.7.2
+Tested on **OSX 10.8.2** and **Linux 12.04** using **Storm 0.8.1** and **JRuby 1.6.8** and **OpenJDK 7**
 
 ## Notes about 1.8/1.9 JRuby compatibility
 
@@ -58,7 +59,7 @@ $ redstorm local|cluster [--1.8|--1.9] ...
 
   ``` ruby
   source :rubygems
-  gem "redstorm", "~> 0.6.3"
+  gem "redstorm", "~> 0.6.4"
   ```
 
 ## Usage overview
@@ -81,7 +82,7 @@ or if your default JRuby mode is 1.8 but you want to use 1.9 for your topology d
 $ jruby --1.9 -S redstorm install
 ```
 
-This will basically install all Java jar dependencies in `target/dependency`, generate & compile the Java bindings in `target/classes`.
+This will basically install default Java jar dependencies in `target/dependency`, generate & compile the Java bindings in `target/classes`.
 
 ### Create a topology
 
@@ -113,6 +114,42 @@ Basically, the `redstorm bundle` command copy the gems specified in the Gemfile
 
 This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the jar file. This **could lead to problems** if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have **C or FFI** extensions.
 
+### Custom Jar dependencies in your topology
+
+By defaut, RedStorm installs Storm and JRuby jars dependencies. If you require custom dependencies, these can be specified by creating the `Dependencies` file in the root of your project. Note that this file overwrites the defaults dependencies so you must also include the Storm and JRuby dependencies. Here's an example of a `Dependencies` file which included the jars required to run the `KafkaTopology` in the examples.
+
+``` ruby
+{
+  :storm_artifacts => [
+    "storm:storm:0.8.1, transitive=true",
+  ],
+  :topology_artifacts => [
+    "org.jruby:jruby-complete:1.6.8, transitive=false",
+    "org.scala-lang:scala-library:2.8.0, transitive=false",
+    "storm:kafka:0.7.0-incubating, transitive=false",
+    "storm:storm-kafka:0.8.0-wip4, transitive=false",
+  ],
+}
+```
+
+Basically the dependendencies are speified as Maven artifacts. There are two sections, the `:storm_artifacts =>` contains the dependencies for running storm in local mode and the `:topology_artifacts =>` are the dependencies specific for your topology. The format is self explainatory and the attribute `transitive=[true|false]` controls the recursive dependencies resolution (using `true`).
+
+The jars repositories can be configured by adding the `ivy/setting.xml` file in the root of your project. For information on the Ivy settings format, see the [Ivy Settings Documentation](http://ant.apache.org/ivy/history/2.2.0/settings.html). I will try my best to eliminate all XML :) but for now I haven't figured how to get rid of this one. For an example Ivy settings file, RedStorm is using the following settings by default:
+
+``` xml
+<ivysettings>
+  <settings defaultResolver="repositories"/>
+  <resolvers>
+    <chain name="repositories">
+      <ibiblio name="ibiblio" m2compatible="true"/>
+      <ibiblio name="maven2" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
+      <ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
+      <ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/> 
+    </chain>
+  </resolvers>
+</ivysettings>
+```
+
 ### Run in local mode
 
 ``` sh
@@ -125,7 +162,7 @@ By defaut, a topology will be executed in the **same mode** as the interpreter r
 
 ### Run on production cluster
 
-1. download and unpack the [Storm 0.7.4 distribution](https://github.com/downloads/nathanmarz/storm/storm-0.7.4.zip) locally and **add** the Storm `bin/` directory to your path
+1. download and unpack the [Storm 0.8.1 distribution](https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip) locally and **add** the Storm `bin/` directory to your `$PATH`.
 
 2. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
 
@@ -158,14 +195,14 @@ All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/
 #### Example topologies without gems 
 
 ``` sh
-$ redstorm local --1.9 examples/simple/exclamation_topology.rb
-$ redstorm local --1.9 examples/simple/exclamation_topology2.rb
-$ redstorm local --1.9 examples/simple/word_count_topology.rb
+$ redstorm local examples/simple/exclamation_topology.rb
+$ redstorm local examples/simple/exclamation_topology2.rb
+$ redstorm local examples/simple/word_count_topology.rb
 ```
 
 #### Example topologies with gems 
 
-For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis][redis] server running on `localhost:6379`
+For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
 
 1. create a `Gemfile`
 
@@ -173,21 +210,21 @@ For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required a
   source :rubygems
 
   group :word_count do
-      gem "redis"
+    gem "redis"
   end
   ```
 
 2. install the topology gems
 
   ``` sh
-  $ jruby --1.9 -S bundle install
+  $ bundle install
   $ redstorm bundle word_count
   ```
 
 3. run the topology in local mode
 
   ``` sh
-  $ redstorm local --1.9 examples/simple/redis_word_count_topology.rb
+  $ redstorm local examples/simple/redis_word_count_topology.rb
   ```
 
 Using `redis-cli` push words into the `test` list and watch Storm pick them up
@@ -208,15 +245,15 @@ All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/
 2. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
 
   ``` sh
-  $ redstorm cluster --1.9 examples/simple/exclamation_topology.rb
-  $ redstorm cluster --1.9 examples/simple/exclamation_topology2.rb
-  $ redstorm cluster --1.9 examples/simple/word_count_topology.rb
+  $ redstorm cluster examples/simple/exclamation_topology.rb
+  $ redstorm cluster examples/simple/exclamation_topology2.rb
+  $ redstorm cluster examples/simple/word_count_topology.rb
   ```
 
 
 #### Topologies with gems 
 
-For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis][redis] server running on `localhost:6379`
+For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
 
 1. create a `Gemfile`
 
@@ -231,7 +268,7 @@ For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required a
 2. install the topology gems
 
   ``` sh
-  $ jruby --1.9 -S bundle install
+  $ bundle install
   $ redstorm bundle word_count
   ```
 
@@ -244,7 +281,7 @@ For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required a
 4. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
 
   ``` sh
-  $ redstorm cluster --1.9 examples/simple/redis_word_count_topology.rb
+  $ redstorm cluster examples/simple/redis_word_count_topology.rb
   ```
 
 Using `redis-cli` push words into the `test` list and watch Storm pick them up
@@ -280,7 +317,7 @@ It is possible to fork the RedStorm project and run local and remote/cluster top
 
 ### Requirements
 
-- JRuby 1.6.7.x
+- JRuby 1.6.8
 
 ### Workflow
 
@@ -289,7 +326,7 @@ It is possible to fork the RedStorm project and run local and remote/cluster top
 - install RedStorm required gems
 
   ```sh
-  $ jruby --1.9 -S bundle install
+  $ bundle install
   ```
 
 - install dependencies in `target/dependencies`
@@ -301,7 +338,7 @@ It is possible to fork the RedStorm project and run local and remote/cluster top
 - generate and build Java source into `target/classes`
 
   ```sh
-  $ jruby --1.9 -S bin/redstorm build
+  $ bin/redstorm build
   ```
 
   **if you modify any of the RedStorm Ruby code or Java binding code**, you need to run this to refresh code and rebuild the bindings
@@ -326,21 +363,13 @@ Some ways you can contribute:
 
 If you want to list your RedStorm project here, contact me.
 
-- [Tweigeist](https://github.com/colinsurprenant/tweitgeist) - realtime computation of the top trending hashtags on Twitter. Live Demo in search of a new home.
+- [Tweigeist](https://github.com/colinsurprenant/tweitgeist) - realtime computation of the top trending hashtags on Twitter. See [Live Demo](http://tweitgeist.colinsurprenant.com/).
 
 ## Author
-Colin Surprenant, [@colinsurprenant][twitter], [http://github.com/colinsurprenant][github], colin.surprenant@gmail.com
+***Colin Surprenant***, [@colinsurprenant](http://twitter.com/colinsurprenant/), [http://github.com/colinsurprenant/](http://github.com/colinsurprenant/), colin.surprenant@gmail.com, [http://colinsurprenant.com/](http://colinsurprenant.com/)
 
 ## Contributors
 Theo Hultberg, https://github.com/iconara
 
 ## License
 Apache License, Version 2.0. See the LICENSE.md file.
-
-[twitter]: http://twitter.com/colinsurprenant
-[github]: http://github.com/colinsurprenant
-[rvm]: http://beginrescueend.com/
-[storm]: https://github.com/nathanmarz/storm
-[jruby]: http://jruby.org/
-[ruby-maven]: https://github.com/mkristian/ruby-maven
-[redis]: http://redis.io/

data/examples/simple/kafka_topology.rb

@@ -0,0 +1,55 @@
+require 'red_storm'
+java_import 'storm.kafka.KafkaConfig'
+java_import 'storm.kafka.SpoutConfig'
+java_import 'storm.kafka.StringScheme'
+java_import 'storm.kafka.KafkaSpout'
+
+# the KafkaTopology obviously requires a Kafka server running, you can ajust the
+# host and port below. 
+#
+# custom dependencies are required for the Kafka and Scala jars. put the following
+# dependencies in the "Dependencies" file in the root of your RedStorm project:
+#
+# {
+#   :storm_artifacts => [
+#     "storm:storm:0.8.1, transitive=true",
+#   ],
+#   :topology_artifacts => [
+#     "org.jruby:jruby-complete:1.6.8, transitive=false",
+#     "org.scala-lang:scala-library:2.8.0, transitive=false",
+#     "storm:kafka:0.7.0-incubating, transitive=false",
+#     "storm:storm-kafka:0.8.0-wip4, transitive=false",
+#   ],
+# }
+
+class KafkaTopology < RedStorm::SimpleTopology
+  spout_config = SpoutConfig.new(
+    KafkaConfig::ZkHosts.new("localhost:2181", "/brokers"),
+    "words",        # topic to read from
+    "/kafkastorm",  # Zookeeper root path to store the consumer offsets
+    "discovery"     # Zookeeper consumer id to store the consumer offsets
+  )
+  spout_config.scheme = StringScheme.new
+
+  class SplitStringBolt < RedStorm::SimpleBolt
+    on_receive {|tuple| tuple.getString(0).split.map{|w| [w]}}
+  end
+
+  spout KafkaSpout, [spout_config]
+      
+  bolt SplitStringBolt do
+    output_fields :word
+    source KafkaSpout, :shuffle
+  end
+
+  configure do |env|
+    debug true
+  end
+
+  on_submit do |env|
+    if env == :local
+      sleep(10)
+      cluster.shutdown
+    end
+  end
+end

data/ivy/settings.xml

@@ -0,0 +1,11 @@
+<ivysettings>
+  <settings defaultResolver="repositories"/>
+  <resolvers>
+    <chain name="repositories">
+      <ibiblio name="ibiblio" m2compatible="true"/>
+      <ibiblio name="maven2" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
+      <ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
+      <ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/> 
+    </chain>
+  </resolvers>
+</ivysettings>

data/lib/red_storm/application.rb

@@ -18,6 +18,12 @@ REDSTORM_LIB_DIR = "#{RedStorm::REDSTORM_HOME}/lib"
 SRC_EXAMPLES = "#{RedStorm::REDSTORM_HOME}/examples"
 DST_EXAMPLES = "#{CWD}/examples"
 
+SRC_IVY_DIR = "#{RedStorm::REDSTORM_HOME}/ivy"
+DST_IVY_DIR = "#{CWD}/ivy"
+CUSTOM_DEPENDENCIES = "#{CWD}/Dependencies"
+DEFAULT_IVY_SETTINGS = "#{SRC_IVY_DIR}/settings.xml"
+CUSTOM_IVY_SETTINGS = "#{DST_IVY_DIR}/settings.xml"
+
 
 module RedStorm
   
@@ -26,7 +32,7 @@ module RedStorm
 
     def self.local_storm_command(class_file, ruby_mode = nil)
       src_dir = File.expand_path(File.dirname(class_file))
-      "java -Djruby.compat.version=#{RedStorm.jruby_mode_token(ruby_mode)} -cp \"#{TARGET_CLASSES_DIR}:#{TARGET_DEPENDENCY_DIR}/*:#{src_dir}/\" redstorm.TopologyLauncher local #{class_file}"
+      "java -Djruby.compat.version=#{RedStorm.jruby_mode_token(ruby_mode)} -cp \"#{TARGET_CLASSES_DIR}:#{TARGET_DEPENDENCY_DIR}/storm/default/*:#{TARGET_DEPENDENCY_DIR}/topology/default/*:#{src_dir}/\" redstorm.TopologyLauncher local #{class_file}"
     end
 
     def self.cluster_storm_command(class_file, ruby_mode = nil)
@@ -70,7 +76,7 @@ module RedStorm
     end
 
     def self.subshell(command)
-      out = IO.popen([command, {:err => [:child, :out]}]) {|io| io.read}
+      out = IO.popen(command, {:err => [:child, :out]}) {|io| io.read}
       [!!$?.success?, out]
     end
 

data/lib/red_storm/loggable.rb

@@ -1,8 +1,10 @@
+require 'java'
+
 module RedStorm
   module Loggable
 
     def self.log
-      @log ||= Logger.getLogger(self.name)
+      @log ||= Java::OrgApacheLog4j::Logger.getLogger(self.name)
     end
 
     def log

data/lib/red_storm/proxy/bolt.rb

@@ -8,7 +8,6 @@ java_import 'backtype.storm.tuple.Tuple'
 java_import 'backtype.storm.tuple.Fields'
 java_import 'backtype.storm.tuple.Values'
 java_import 'java.util.Map'
-java_import 'org.apache.log4j.Logger'
 module Backtype
   java_import 'backtype.storm.Config'
 end

data/lib/red_storm/proxy/spout.rb

@@ -8,7 +8,6 @@ java_import 'backtype.storm.tuple.Tuple'
 java_import 'backtype.storm.tuple.Fields'
 java_import 'backtype.storm.tuple.Values'
 java_import 'java.util.Map'
-java_import 'org.apache.log4j.Logger'
 module Backtype
   java_import 'backtype.storm.Config'
 end

data/lib/red_storm/simple_bolt.rb

@@ -1,3 +1,4 @@
+require 'java'
 require 'red_storm/configurator'
 
 module RedStorm
@@ -8,7 +9,7 @@ module RedStorm
     # DSL class methods
 
     def self.log
-      @log ||= Logger.getLogger(self.name)
+      @log ||= Java::OrgApacheLog4j::Logger.getLogger(self.name)
     end
 
     def self.output_fields(*fields)

data/lib/red_storm/simple_spout.rb

@@ -1,3 +1,4 @@
+require 'java'
 require 'red_storm/configurator'
 
 module RedStorm
@@ -12,7 +13,7 @@ module RedStorm
     end
 
     def self.log
-      @log ||= Logger.getLogger(self.name)
+      @log ||= Java::OrgApacheLog4j::Logger.getLogger(self.name)
     end
 
     def self.output_fields(*fields)

data/lib/red_storm/simple_topology.rb

@@ -101,7 +101,7 @@ module RedStorm
     end
 
     def self.log
-      @log ||= org.apache.log4j.Logger.getLogger(self.name)
+      @log ||= Java::OrgApacheLog4j::Logger.getLogger(self.name)
     end
 
     # def self.spout(spout_class, contructor_args = [], options = {}, &spout_block)

data/lib/red_storm/version.rb

@@ -1,3 +1,3 @@
 module RedStorm
-  VERSION = '0.6.3'
+  VERSION = '0.6.4'
 end

data/lib/tasks/red_storm.rake

@@ -1,15 +1,27 @@
-require 'ant'
+begin
+  require 'ant'
+rescue
+  puts("error: unable to load Ant, make sure Ant is installed and in your PATH")
+  puts("\nerror detail:\n#{$!}")
+  exit(1)
+end
+
 require 'jruby/jrubyc'
-require 'pompompom'
 require 'red_storm'
 require 'red_storm/application'
 
-INSTALL_STORM_VERSION = "0.7.4"
-INSTALL_JRUBY_VERSION = "1.6.7.2"
+DEP_STORM_VERSION = "0.8.1"
+DEP_JRUBY_VERSION = "1.6.8"
+INSTALL_IVY_VERSION = "2.2.0"
 
-module JavaZip
-  import 'java.util.zip.ZipFile'
-end
+DEFAULT_DEPENDENCIES = {
+  :storm_artifacts => [
+    "storm:storm:#{DEP_STORM_VERSION}, transitive=true",
+  ],
+  :topology_artifacts => [
+    "org.jruby:jruby-complete:#{DEP_JRUBY_VERSION}, transitive=false",
+  ],
+}
 
 task :launch, :env, :ruby_mode, :class_file do |t, args|
   # use ruby mode parameter or default to current interpreter version
@@ -20,7 +32,7 @@ task :launch, :env, :ruby_mode, :class_file do |t, args|
     RedStorm::Application.local_storm_command(args[:class_file], args[:ruby_mode])
   when "cluster"
     unless File.exist?(TARGET_CLUSTER_JAR)
-      puts("cluster jar file #{TARGET_CLUSTER_JAR} not found. Generate it using $redstorm jar DIR1 [DIR2, ...]")
+      puts("error: cluster jar file #{TARGET_CLUSTER_JAR} not found. Generate it usingw $redstorm jar DIR1 [DIR2, ...]")
       exit(1)
     end
     RedStorm::Application.cluster_storm_command(args[:class_file], args[:ruby_mode])
@@ -58,61 +70,6 @@ task :install => [:deps, :build] do
   puts("\nRedStorm install completed. All dependencies installed in #{TARGET_DIR}")
 end
 
-task :unpack do
-  unpack_artifacts = %w[jruby-complete]
-  unpack_glob = "#{TARGET_DEPENDENCY_DIR}/{#{unpack_artifacts.join(',')}}-*-jar.jar"
-  Dir[unpack_glob].each do |jar|
-    puts("Extracting #{jar}")
-    zf = JavaZip::ZipFile.new(jar)
-    zf.entries.each do |entry|
-      next if entry.directory?
-      destination = "#{TARGET_DEPENDENCY_UNPACKED_DIR}/#{entry.name}"
-      in_io = zf.get_input_stream(entry).to_io
-      FileUtils.mkdir_p(File.dirname(destination))
-      File.open(destination, 'w') { |out_io| out_io.write(in_io.read) }
-    end
-  end
-end
-
-task :jar, [:include_dir] => [:unpack, :clean_jar] do |t, args|
-  puts("\n--> Generating JAR file #{TARGET_CLUSTER_JAR}")
-  ant.jar :destfile => TARGET_CLUSTER_JAR do
-    fileset :dir => TARGET_DEPENDENCY_UNPACKED_DIR
-    fileset :dir => TARGET_DIR do
-      include :name => "gems/**"
-    end
-    fileset :dir => TARGET_CLASSES_DIR
-    # red_storm.rb and red_storm/* must be in root of jar so that "require 'red_storm'"
-    # in bolts/spouts works in jar context
-    fileset :dir => TARGET_LIB_DIR do
-      exclude :name => "tasks/**"
-    end
-    if args[:include_dir]
-      dirs = args[:include_dir].split(":")
-
-      # first add any resources/ dir in the tree in the jar root - requirement for ShellBolt multilang resources
-      dirs.each do |dir|
-        resources_dirs = Dir.glob("#{dir}/**/resources")
-        resources_dirs.each do |resources_dir|
-          resources_parent = resources_dir.gsub("/resources", "")
-          fileset :dir => resources_parent do
-            include :name => "resources/**/*"
-          end
-        end
-      end
-
-      # include complete source dir tree (note we don't care about potential duplicated resources dir)
-      fileset :dir => CWD do
-        dirs.each{|dir| include :name => "#{dir}/**/*"}
-      end
-    end
-    manifest do
-      attribute :name => "Main-Class", :value => "redstorm.TopologyLauncher"
-    end
-  end
-  puts("\nRedStorm generated JAR file #{TARGET_CLUSTER_JAR}")
-end
-
 task :examples do
   if File.identical?(SRC_EXAMPLES, DST_EXAMPLES)
     STDERR.puts("error: cannot copy examples into itself")
@@ -132,26 +89,6 @@ task :copy_red_storm do
   FileUtils.cp_r(REDSTORM_LIB_DIR, TARGET_DIR)
 end
 
-task :deps => :setup do
-  puts("\n--> Installing dependencies")
-
-  configuration = {
-    :repositories => {:clojars => 'http://clojars.org/repo/', :sonatype => "http://oss.sonatype.org/content/groups/public/"},
-    :dependencies => [
-      "storm:storm:#{INSTALL_STORM_VERSION}|type_filter=jar",
-      "org.slf4j:slf4j-log4j12:1.5.8|type_filter=jar",
-      "org.jruby:jruby-complete:#{INSTALL_JRUBY_VERSION}|transitive=false,type_filter=jar",
-    ],
-    :destination => TARGET_DEPENDENCY_DIR
-  }
-
-  installer = PomPomPom::Runner.new(configuration)
-  installer.run
-
-  # tmp hack to clenup a dependency weirdness (issue #36)
-  FileUtils.rm("#{TARGET_DEPENDENCY_DIR}/slf4j-api-1.6.3-jar.jar")
-end
-
 task :build => [:setup, :copy_red_storm] do
   # compile the JRuby proxy classes to Java
   build_jruby("#{REDSTORM_LIB_DIR}/red_storm/proxy")
@@ -189,6 +126,90 @@ task :bundle, [:groups] => :setup do |t, args|
   end
 end
 
+namespace :ivy do
+  task :download do
+    mkdir_p DST_IVY_DIR
+    ant.get({
+      :src => "http://repo1.maven.org/maven2/org/apache/ivy/ivy/#{INSTALL_IVY_VERSION}/ivy-#{INSTALL_IVY_VERSION}.jar",
+      :dest => "#{DST_IVY_DIR}/ivy-#{INSTALL_IVY_VERSION}.jar",
+      :usetimestamp => true,
+    })
+  end
+
+  task :install => :download do
+    ant.path :id => 'ivy.lib.path' do
+      fileset :dir => DST_IVY_DIR, :includes => '*.jar'
+    end
+
+    ant.taskdef({
+      :resource => "org/apache/ivy/ant/antlib.xml",
+      :classpathref => "ivy.lib.path",
+      #:uri => "antlib:org.apache.ivy.ant",
+    })
+  end
+end
+
+task :deps => "ivy:install" do
+  puts("\n--> Installing dependencies")
+
+  dependencies = File.exists?(CUSTOM_DEPENDENCIES) ? eval(File.read(CUSTOM_DEPENDENCIES)) : DEFAULT_DEPENDENCIES
+  ant.configure :file => File.exists?(CUSTOM_IVY_SETTINGS) ? CUSTOM_IVY_SETTINGS : DEFAULT_IVY_SETTINGS
+
+  dependencies[:storm_artifacts].each do |dependency|
+    artifact, transitive = dependency.split(/\s*,\s*/)
+    ivy_retrieve(*artifact.split(':').concat([transitive.split(/\s*=\s*/).last, "#{TARGET_DEPENDENCY_DIR}/storm", "default"]))
+  end
+
+  dependencies[:topology_artifacts].each do |dependency|
+    artifact, transitive = dependency.split(/\s*,\s*/)
+    ivy_retrieve(*artifact.split(':').concat([transitive.split(/\s*=\s*/).last, "#{TARGET_DEPENDENCY_DIR}/topology", "default"]))
+  end
+end  
+
+task :jar, [:include_dir] => [:clean_jar] do |t, args|
+  puts("\n--> Generating JAR file #{TARGET_CLUSTER_JAR}")
+
+  ant.jar :destfile => TARGET_CLUSTER_JAR do
+    # rejar all topology jars
+    Dir["target/dependency/topology/default/*.jar"].each do |jar|
+      puts("Extracting #{jar}")
+      zipfileset :src => jar, :includes => "**/*"
+    end
+    fileset :dir => TARGET_DIR do
+      include :name => "gems/**"
+    end
+    fileset :dir => TARGET_CLASSES_DIR
+    # red_storm.rb and red_storm/* must be in root of jar so that "require 'red_storm'"
+    # in bolts/spouts works in jar context
+    fileset :dir => TARGET_LIB_DIR do
+      exclude :name => "tasks/**"
+    end
+    if args[:include_dir]
+      dirs = args[:include_dir].split(":")
+
+      # first add any resources/ dir in the tree in the jar root - requirement for ShellBolt multilang resources
+      dirs.each do |dir|
+        resources_dirs = Dir.glob("#{dir}/**/resources")
+        resources_dirs.each do |resources_dir|
+          resources_parent = resources_dir.gsub("/resources", "")
+          fileset :dir => resources_parent do
+            include :name => "resources/**/*"
+          end
+        end
+      end
+
+      # include complete source dir tree (note we don't care about potential duplicated resources dir)
+      fileset :dir => CWD do
+        dirs.each{|dir| include :name => "#{dir}/**/*"}
+      end
+    end
+    manifest do
+      attribute :name => "Main-Class", :value => "redstorm.TopologyLauncher"
+    end
+  end
+  puts("\nRedStorm generated JAR file #{TARGET_CLUSTER_JAR}")
+end
+
 def build_java_dir(source_folder)
   puts("\n--> Compiling Java")
   ant.javac(
@@ -213,9 +234,38 @@ def build_jruby(source_path)
     argv << '-t' << TARGET_SRC_DIR
     argv << '--verbose'
     argv << '--java'
-    argv << '-c' << %("#{TARGET_DEPENDENCY_DIR}/storm-#{INSTALL_STORM_VERSION}.jar")
+    Dir["#{TARGET_DEPENDENCY_DIR}/storm/default/*.jar"].each do |jar|
+      argv << '-c' << %("#{jar}")
+    end
     argv << '-c' << %("#{TARGET_CLASSES_DIR}")
     argv << source_path
     status =  JRuby::Compiler::compile_argv(argv)
   end
 end
+
+def truefalse(s)
+  return true if s.to_s.downcase =~ /1|yes|true/
+  return false if s.to_s.downcase =~ /0|no|false/
+  nil
+end
+
+def ivy_retrieve(org, mod, rev, transitive, dir, conf)
+  ant.resolve({
+    :organisation => org,
+    :module => mod,
+    :revision => rev,
+    :inline => true,
+    :transitive => truefalse(transitive),
+    :conf => conf,
+  })
+
+  ant.retrieve({
+    :organisation => org,
+    :module => mod,
+    :revision => rev,
+    :pattern => "#{dir}/[conf]/[artifact]-[revision].[ext]",
+    :inline => true,
+    :transitive => truefalse(transitive),
+    :conf => conf,
+  })
+end

metadata

@@ -2,14 +2,14 @@
 name: redstorm
 version: !ruby/object:Gem::Version
   prerelease:
-  version: 0.6.3
+  version: 0.6.4
 platform: ruby
 authors:
 - Colin Surprenant
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2012-10-10 00:00:00.000000000 Z
+date: 2012-10-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -17,13 +17,13 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 2.8.0
+        version: 2.11.0
     none: false
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 2.8.0
+        version: 2.11.0
     none: false
   prerelease: false
   type: :development
@@ -43,22 +43,6 @@ dependencies:
     none: false
   prerelease: false
   type: :runtime
-- !ruby/object:Gem::Dependency
-  name: pompompom
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 2.0.0.b2
-    none: false
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 2.0.0.b2
-    none: false
-  prerelease: false
-  type: :runtime
 description: JRuby integration & DSL for the Storm distributed realtime computation system
 email:
 - colin.surprenant@gmail.com
@@ -81,6 +65,7 @@ files:
 - lib/red_storm/proxy/bolt.rb
 - lib/red_storm/proxy/spout.rb
 - lib/tasks/red_storm.rake
+- ivy/settings.xml
 - examples/native/cluster_word_count_topology.rb
 - examples/native/exclamation_bolt.rb
 - examples/native/Gemfile
@@ -97,7 +82,7 @@ files:
 - examples/simple/exclamation_bolt.rb
 - examples/simple/exclamation_topology.rb
 - examples/simple/exclamation_topology2.rb
-- examples/simple/Gemfile
+- examples/simple/kafka_topology.rb
 - examples/simple/random_sentence_spout.rb
 - examples/simple/redis_word_count_topology.rb
 - examples/simple/ruby_version_topology.rb

data/examples/simple/Gemfile

@@ -1,2 +0,0 @@
-source :rubygems
-gem 'redis'