redstorm 0.6.5 → 0.6.6.beta1

data/CHANGELOG.md

@@ -80,4 +80,15 @@
 - issue #56 - make topology class configure statement optional
 - issue #57 - DRPC topologies
 - issue #73, external jars dependencies Ivy configurations completely externalized and configurable
-- lots of other [bug fixes and improvements](https://github.com/colinsurprenant/redstorm/issues?milestone=9&page=1&state=closed)
+- lots of other [bug fixes and improvements](https://github.com/colinsurprenant/redstorm/issues?milestone=9&page=1&state=closed)
+
+# 0.6.6, WIP
+- Storm 0.9.0-wip16 and JRuby 1.7.4
+- [issue #79](https://github.com/colinsurprenant/redstorm/issues/79) - fix example Kafka topology dependency doc
+- [issue #82](https://github.com/colinsurprenant/redstorm/issues/82) - configurable javac compilation source/target
+- [issue #83](https://github.com/colinsurprenant/redstorm/issues/83) - syntactic sugar on the Tuple class
+- [issue #84](https://github.com/colinsurprenant/redstorm/issues/84) - DSL classes naming and namespace
+- improved DSL performance by refactoring closures invocation, see [redstorm-benchmark](https://github.com/colinsurprenant/redstorm-benchmark/)
+- fixed FFI support
+- exposed topology builder for specs
+- [issue #11](https://github.com/colinsurprenant/redstorm/issues/11), [#17](https://github.com/colinsurprenant/redstorm/issues/17) -  example specs using the Storm testing API see [redstorm-starter](https://github.com/colinsurprenant/redstorm-starter/)

data/README.md

@@ -1,24 +1,23 @@
-# RedStorm v0.6.5 - JRuby on Storm
+# RedStorm v0.6.6.beta1 - JRuby on Storm
 
 [![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
 
 RedStorm provides a Ruby DSL using JRuby integration for the [Storm](https://github.com/nathanmarz/storm/) distributed realtime computation system.
 
-## Documentation
+Like RedStorm? visit us on IRC at #redstorm on freenode
+
+Check also these related projects:
 
-Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary. 
+- [redstorm-starter](https://github.com/colinsurprenant/redstorm-starter/)
+- [redstorm-benchmark](https://github.com/colinsurprenant/redstorm-benchmark/)
 
-### Released gems
+## Documentation
 
-- [RedStorm Gem v0.4.x Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.4.x-Documentation)
-- [RedStorm Gem v0.5.0 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.0-Documentation)
-- [RedStorm Gem v0.5.1 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.1-Documentation)
-- [RedStorm Gem v0.6.3 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.6.3-Documentation)
-- [RedStorm Gem v0.6.4 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.6.4-Documentation)
+Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept [in the wiki](https://github.com/colinsurprenant/redstorm/wiki) when necessary.
 
 ## Dependencies
 
-Tested on **OSX 10.8.3** and **Ubuntu Linux 12.10** using **Storm 0.8.2** and **JRuby 1.7.3** and **OpenJDK 7**
+Tested on **OSX 10.8.3** and **Ubuntu Linux 12.10** using **Storm 0.9.0-wip16** and **JRuby 1.7.4** and **OpenJDK 7**
 
 ## Installation
 
@@ -32,7 +31,7 @@ Tested on **OSX 10.8.3** and **Ubuntu Linux 12.10** using **Storm 0.8.2** and **
 
   ``` ruby
   source "https://rubygems.org"
-  gem "redstorm", "~> 0.6.5"
+  gem "redstorm", "~> 0.6.6.beta1"
   ```
 
 ## Usage
@@ -65,30 +64,30 @@ This will install default Java jar dependencies in `target/dependency`, generate
 
 Create a subdirectory for your topology code and create your topology class **using this naming convention**: *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
 
-Here's an example [hello_world_topology.rb](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/hello_world_topology.rb)
+Here's an example [hello_world_topology.rb](https://github.com/colinsurprenant/redstorm/tree/master/examples/dsl/hello_world_topology.rb)
 
 ``` ruby
 require 'red_storm'
 
-class HelloWorldSpout < RedStorm::SimpleSpout
+class HelloWorldSpout < RedStorm::DSL::Spout
   on_init {@words = ["hello", "world"]}
   on_send {@words.shift unless @words.empty?}
 end
 
-class HelloWorldBolt < RedStorm::SimpleBolt
+class HelloWorldBolt < RedStorm::DSL::Bolt
   on_receive :emit => false do |tuple|
     log.info(tuple.getString(0))
   end
 end
 
-class HelloWorldTopology < RedStorm::SimpleTopology
+class HelloWorldTopology < RedStorm::DSL::Topology
   spout HelloWorldSpout do
     output_fields :word
   end
 
   bolt HelloWorldBolt do
     source HelloWorldSpout, :global
-  end        
+  end
 end
 ```
 
@@ -110,7 +109,7 @@ Note that bundler is only used to help package the gems **prior** to running a t
   $ redstorm bundle [BUNDLER_GROUP]
   ```
 
-3. make sure your topology class has `require "red_storm"` 
+3. make sure your topology class has `require "red_storm"`
 
   ```ruby
   require 'red_storm'
@@ -120,7 +119,7 @@ The `redstorm bundle` command copy the gems specified in the Gemfile (in a speci
 
 ### Custom Jar dependencies in your topology (XML Warning! :P)
 
-By defaut, RedStorm installs Storm and JRuby jars dependencies into `target/dependency`. RedStorm uses [Ivy](https://ant.apache.org/ivy/) 2.3 to manage dependencies. You can fully control and customize these dependencies. 
+By defaut, RedStorm installs Storm and JRuby jars dependencies into `target/dependency`. RedStorm uses [Ivy](https://ant.apache.org/ivy/) 2.3 to manage dependencies. You can fully control and customize these dependencies.
 
 There are two distinct sets of dependencies: the `storm` dependencies manages the requirements (Storm jars) for the Storm **local mode** runtime. The `topology` dependencies manages the requirements (JRuby jars) for the **topology** runtime.
 
@@ -133,7 +132,7 @@ You can supply custom `storm` and `topology` dependencies by creating `ivy/storm
   <ivy-module version="2.0">
     <info organisation="redstorm" module="storm-deps"/>
     <dependencies>
-      <dependency org="storm" name="storm" rev="0.8.2" conf="default" transitive="true" />
+      <dependency org="storm" name="storm" rev="0.9.0-wip16" conf="default" transitive="true" />
       <override org="org.slf4j" module="slf4j-log4j12" rev="1.6.3"/>
     </dependencies>
   </ivy-module>
@@ -146,7 +145,7 @@ You can supply custom `storm` and `topology` dependencies by creating `ivy/storm
   <ivy-module version="2.0">
     <info organisation="redstorm" module="topology-deps"/>
     <dependencies>
-      <dependency org="org.jruby" name="jruby-core" rev="1.7.3" conf="default" transitive="true"/>
+      <dependency org="org.jruby" name="jruby-core" rev="1.7.4" conf="default" transitive="true"/>
     </dependencies>
   </ivy-module>
   ```
@@ -162,9 +161,9 @@ The jars repositories can be configured by adding the `ivy/settings.xml` file in
     <resolvers>
       <chain name="repositories">
         <ibiblio name="ibiblio" m2compatible="true"/>
-        <ibiblio name="maven2" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
-        <ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
-        <ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/> 
+        <ibiblio name="maven2" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/>
+        <ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/>
+        <ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/>
       </chain>
     </resolvers>
   </ivysettings>
@@ -173,7 +172,13 @@ The jars repositories can be configured by adding the `ivy/settings.xml` file in
 ### Run in local mode
 
 ``` sh
-$ redstorm local <path/to/topology_class_file_name.rb>
+$ redstorm local <sources_directory_path/topology_class_file_name.rb>
+```
+
+note that the topology can also be launched with the following command:
+
+``` sh
+$ java -Djruby.compat.version=RUBY1_9 -cp "target/classes:target/dependency/storm/default/*:target/dependency/topology/default/*:<sources_directory_path>" redstorm.TopologyLauncher local <sources_directory_path/topology_class_file_name.rb>
 ```
 
 **See examples below** to run examples in local mode or on a production cluster.
@@ -182,20 +187,34 @@ $ redstorm local <path/to/topology_class_file_name.rb>
 
 The Storm distribution is currently required for the cluster topology submission.
 
-1. download and unpack the [Storm 0.8.2 distribution](https://dl.dropbox.com/u/133901206/storm-0.8.2.zip) locally and **add** the Storm `bin/` directory to your `$PATH`.
+1. download and unpack the [Storm 0.9.0-wip16 distribution](https://dl.dropbox.com/u/133901206/storm-0.9.0-wip16.zip) locally
+
+2. add the Storm `bin/` directory to your `$PATH`
 
-2. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
+3. create `~/.storm/storm.yaml` and add the following
+
+  ```yaml
+  nimbus.host: "host_name_or_ip"
+  ```
+
+4. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
 
   ``` sh
   $ redstorm jar <sources_directory1> <sources_directory2> ...
   ```
 
-3. submit the cluster topology jar file to the cluster
+5. submit the cluster topology jar file to the cluster
 
   ``` sh
   $ redstorm cluster <sources_directory/topology_class_file_name.rb>
   ```
 
+  note that the cluster topology jar can also be submitted using the storm command with:
+
+  ``` sh
+  $ storm jar target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster <sources_directory/topology_class_file_name.rb>
+  ```
+
 The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [setting up a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). You can also [manually submit your topology](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
 
 ## Examples
@@ -206,21 +225,21 @@ Install the [example files](https://github.com/colinsurprenant/redstorm/tree/mas
 $ redstorm examples
 ```
 
-All examples using the [DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) are located in `examples/simple`. Examples using the standard Java interface are in `examples/native`.
+All examples using the [DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) are located in `examples/dsl`. Examples using the standard Java interface are in `examples/native`.
 
 ### Local mode
 
-#### Example topologies without gems 
+#### Example topologies without gems
 
 ``` sh
-$ redstorm local examples/simple/exclamation_topology.rb
-$ redstorm local examples/simple/exclamation_topology2.rb
-$ redstorm local examples/simple/word_count_topology.rb
+$ redstorm local examples/dsl/exclamation_topology.rb
+$ redstorm local examples/dsl/exclamation_topology2.rb
+$ redstorm local examples/dsl/word_count_topology.rb
 ```
 
-#### Example topologies with gems 
+#### Example topologies with gems
 
-For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
+For `examples/dsl/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
 
 1. create a `Gemfile`
 
@@ -242,7 +261,7 @@ For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required a
 3. run the topology in local mode
 
   ``` sh
-  $ redstorm local examples/simple/redis_word_count_topology.rb
+  $ redstorm local examples/dsl/redis_word_count_topology.rb
   ```
 
 Using `redis-cli` push words into the `test` list and watch Storm pick them up
@@ -252,7 +271,7 @@ Using `redis-cli` push words into the `test` list and watch Storm pick them up
 All examples using the [DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) can run in both local or on a remote cluster. The only **native** example compatible with a remote cluster is `examples/native/cluster_word_count_topology.rb`.
 
 
-#### Topologies without gems 
+#### Topologies without gems
 
 1. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
 
@@ -263,15 +282,15 @@ All examples using the [DSL](https://github.com/colinsurprenant/redstorm/wiki/Ru
 2. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
 
   ``` sh
-  $ redstorm cluster examples/simple/exclamation_topology.rb
-  $ redstorm cluster examples/simple/exclamation_topology2.rb
-  $ redstorm cluster examples/simple/word_count_topology.rb
+  $ redstorm cluster examples/dsl/exclamation_topology.rb
+  $ redstorm cluster examples/dsl/exclamation_topology2.rb
+  $ redstorm cluster examples/dsl/word_count_topology.rb
   ```
 
 
-#### Topologies with gems 
+#### Topologies with gems
 
-For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
+For `examples/dsl/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
 
 1. create a `Gemfile`
 
@@ -299,7 +318,7 @@ For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required a
 4. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
 
   ``` sh
-  $ redstorm cluster examples/simple/redis_word_count_topology.rb
+  $ redstorm cluster examples/dsl/redis_word_count_topology.rb
   ```
 
 Using `redis-cli` push words into the `test` list and watch Storm pick them up
@@ -337,7 +356,7 @@ Despite the fact that both transactional and linear DRPC topologies are now [dep
 
 - SnakeYAML conflict between Storm and JRuby
 
-  See [issue](https://github.com/colinsurprenant/redstorm/issues/78). This is a classic Java world jar conflict. Storm 0.8.2 uses snakeyaml 1.9 and JRuby 1.7.3 uses snakeyaml 1.11. If you try to use YAML serialization in your topology it will crash with an exception. This problem is easy to solve when running topologies in **local** mode, simply override in the storm dependencies with the correct jar version. You can do this be creating a custom storm dependencies:
+  See [issue](https://github.com/colinsurprenant/redstorm/issues/78). This is a classic Java world jar conflict. Storm 0.9.0 uses snakeyaml 1.9 and JRuby 1.7.x uses snakeyaml 1.11. If you try to use YAML serialization in your topology it will crash with an exception. This problem is easy to solve when running topologies in **local** mode, simply override in the storm dependencies with the correct jar version. You can do this be creating a custom storm dependencies:
 
   - `ivy/storm_dependencies.xml`
 
@@ -346,7 +365,7 @@ Despite the fact that both transactional and linear DRPC topologies are now [dep
     <ivy-module version="2.0">
       <info organisation="redstorm" module="storm-deps"/>
       <dependencies>
-        <dependency org="storm" name="storm" rev="0.8.2" conf="default" transitive="true" />
+        <dependency org="storm" name="storm" rev="0.9.0-wip16" conf="default" transitive="true" />
         <override org="org.slf4j" module="slf4j-log4j12" rev="1.6.3"/>
         <override org="org.yaml" module="snakeyaml" rev="1.11"/>
       </dependencies>
@@ -354,14 +373,14 @@ Despite the fact that both transactional and linear DRPC topologies are now [dep
     ```
 
   In remote **cluster** mode you will have to update snakeyaml manually or with your favorite deployment/provisioning tool.
-            
+
 ## RedStorm Development
 
 It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.
 
 ### Requirements
 
-- JRuby 1.7.3
+- JRuby 1.7.4
 
 ### Workflow
 
@@ -412,7 +431,7 @@ If you require Ruby 1.8 support, there are two ways to have JRuby run in 1.8 run
   $ export JRUBY_OPTS=--1.8
   ```
 
-- by using the --1.8 option 
+- by using the --1.8 option
 
   ``` sh
   $ jruby --1.8 -S redstorm ...

data/Rakefile

@@ -1,6 +1,6 @@
 require 'rspec/core/rake_task'
 
-
+desc "run specs"
 RSpec::Core::RakeTask.new(:spec) do
   system("ruby -v")
   module RedStorm; SPECS_CONTEXT = true; end

data/examples/dsl/exclamation_bolt.rb

@@ -0,0 +1,10 @@
+require 'red_storm'
+
+module RedStorm
+  module Examples
+    class ExclamationBolt < DSL::Bolt
+      output_fields :word
+      on_receive(:ack => true, :anchor => true) {|tuple| tuple[0] + "!!!"} # tuple[:word] or tuple["word"] are also valid
+    end
+  end
+end

data/examples/{simple → dsl}/exclamation_topology.rb

@@ -1,21 +1,21 @@
 java_import 'backtype.storm.testing.TestWordSpout'
 
 require 'red_storm'
-require 'examples/simple/exclamation_bolt'
+require 'examples/dsl/exclamation_bolt'
 
 # this example topology uses the Storm TestWordSpout and our own JRuby ExclamationBolt
 
 module RedStorm
   module Examples
-    class ExclamationTopology < SimpleTopology
+    class ExclamationTopology < DSL::Topology
       spout TestWordSpout, :parallelism => 2 do
         debug true
       end
-      
+
       bolt ExclamationBolt, :parallelism => 2 do
         source TestWordSpout, :shuffle
       end
-      
+
       bolt ExclamationBolt, :id => :ExclamationBolt2, :parallelism => 2 do
         source ExclamationBolt, :shuffle
         debug true
@@ -32,7 +32,7 @@ module RedStorm
 
       on_submit do |env|
         if env == :local
-          sleep(5)
+          sleep(10)
           cluster.shutdown
         end
       end

data/examples/{simple → dsl}/exclamation_topology2.rb

@@ -7,18 +7,18 @@ require 'red_storm'
 
 module RedStorm
   module Examples
-    class ExclamationBolt < RedStorm::SimpleBolt
+    class ExclamationBolt < DSL::Bolt
       output_fields :word
-      on_receive(:ack => true, :anchor => true) {|tuple| "!#{tuple.getString(0)}!"}
+      on_receive(:ack => true, :anchor => true) {|tuple| "!#{tuple[0]}!"} # tuple[:word] or tuple["word"] are also valid
     end
 
-    class ExclamationTopology2 < RedStorm::SimpleTopology
+    class ExclamationTopology2 < DSL::Topology
       spout TestWordSpout, :parallelism => 2
-      
+
       bolt ExclamationBolt, :parallelism => 2 do
         source TestWordSpout, :shuffle
       end
-      
+
       bolt ExclamationBolt, :id => :ExclamationBolt2, :parallelism => 2 do
         source ExclamationBolt, :shuffle
       end

data/examples/{simple → dsl}/hello_world_topology.rb

@@ -1,17 +1,17 @@
 require 'red_storm'
 
-class HelloWorldSpout < RedStorm::SimpleSpout
+class HelloWorldSpout < RedStorm::DSL::Spout
   on_init {@words = ["hello", "world"]}
   on_send {@words.shift unless @words.empty?}
 end
 
-class HelloWorldBolt < RedStorm::SimpleBolt
+class HelloWorldBolt < RedStorm::DSL::Bolt
   on_receive :emit => false do |tuple|
-    log.info(tuple.getString(0))
+    log.info(tuple[0]) # tuple[:word] or tuple["word"] are also valid
   end
 end
 
-class HelloWorldTopology < RedStorm::SimpleTopology
+class HelloWorldTopology < RedStorm::DSL::Topology
   spout HelloWorldSpout do
     output_fields :word
   end

data/examples/{simple → dsl}/kafka_topology.rb

@@ -6,24 +6,23 @@ java_import 'storm.kafka.KafkaSpout'
 require 'red_storm'
 
 # the KafkaTopology obviously requires a Kafka server running, you can ajust the
-# host and port below. 
+# host and port below.
 #
 # custom dependencies are required for the Kafka and Scala jars. put the following
-# dependencies in the "Dependencies" file in the root of your RedStorm project:
+# dependencies in the "ivy/topology_dependencies.xml" file in the root of your RedStorm project:
 #
-# {
-#   :storm_artifacts => [
-#     "storm:storm:0.8.1, transitive=true",
-#   ],
-#   :topology_artifacts => [
-#     "org.jruby:jruby-complete:1.6.8, transitive=false",
-#     "org.scala-lang:scala-library:2.8.0, transitive=false",
-#     "storm:kafka:0.7.0-incubating, transitive=false",
-#     "storm:storm-kafka:0.8.0-wip4, transitive=false",
-#   ],
-# }
-
-class KafkaTopology < RedStorm::SimpleTopology
+# <?xml version="1.0"?>
+# <ivy-module version="2.0">
+#   <info organisation="redstorm" module="topology-deps"/>
+#   <dependencies>
+#     <dependency org="org.jruby" name="jruby-core" rev="1.7.4" conf="default" transitive="true"/>
+#     <dependency org="org.scala-lang" name="scala-library" rev="2.8.0" conf="default" transitive="false"/>
+#     <dependency org="storm" name="kafka" rev="0.7.0-incubating" conf="default" transitive="false"/>
+#     <dependency org="storm" name="storm-kafka" rev="0.8.0-wip4" conf="default" transitive="false"/>
+#   </dependencies>
+# </ivy-module>
+
+class KafkaTopology < RedStorm::DSL::Topology
   spout_config = SpoutConfig.new(
     KafkaConfig::ZkHosts.new("localhost:2181", "/brokers"),
     "words",        # topic to read from
@@ -32,12 +31,12 @@ class KafkaTopology < RedStorm::SimpleTopology
   )
   spout_config.scheme = StringScheme.new
 
-  class SplitStringBolt < RedStorm::SimpleBolt
-    on_receive {|tuple| tuple.getString(0).split.map{|w| [w]}}
+  class SplitStringBolt < RedStorm::DSL::Bolt
+    on_receive {|tuple| tuple[0].split.map{|w| [w]}}
   end
 
   spout KafkaSpout, [spout_config]
-      
+
   bolt SplitStringBolt do
     output_fields :word
     source KafkaSpout, :shuffle