redstorm 0.5.1 → 0.6.0

data/CHANGELOG.md

@@ -34,4 +34,14 @@
 # 0.5.1, 06-05-2012
 - better handling of enviroments and paths
 - redstorm bundle command to install topology gems
-- issue #26, fixed examples/native for 0.5.1 compatibility
+- issue #26, fixed examples/native for 0.5.1 compatibility
+
+# 0.6.0, 06-27-2012
+- issue #29, updated to Storm 0.7.3
+- issue #30, add redstorm cluster command for remote topology submission
+- issue #31, added support for localOrShuffleGrouping
+- issue #33, avoid forking or shelling out on redstorm commands
+- JRuby 1.6.7.2
+- better handling of JRuby 1.8/1.9 mode
+- topology gems are now specified using a Bundler group in the project Gemfile
+- refactored environment/paths handling for local vs cluster context

data/README.md

@@ -1,4 +1,4 @@
-# RedStorm v0.5.1 - JRuby on Storm
+# RedStorm v0.6.0 - JRuby on Storm
 
 [![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
 
@@ -6,102 +6,142 @@ RedStorm provides a Ruby DSL using JRuby integration for the [Storm][storm] dist
 
 ## Documentation
 
-Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary. This README reflects the current/master developement state. 
+Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary. 
 
 ### Released gems
 
 - [RedStorm Gem v0.4.x Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.4.x-Documentation)
 - [RedStorm Gem v0.5.0 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.0-Documentation)
+- [RedStorm Gem v0.5.1 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.1-Documentation)
 
 ## Dependencies
 
-Tested on OSX 10.6.8 and Linux 10.04 using Storm 0.6.2 and JRuby 1.6.7
+Tested on OSX 10.6.8 and Linux 10.04 & 11.10 using Storm 0.7.3 and JRuby 1.6.7.2
 
 ## Notes about 1.8/1.9 JRuby compatibility
 
-Up until the upcoming JRuby 1.7, JRuby runs in 1.8 Ruby compatibility mode by default. Unless you have a specific need to run topologies in 1.8 mode, you should use 1.9 mode, which will become the default in JRuby. Things are a bit tricky with Storm/RedStorm. There are 3 contexts where the Ruby compatibility mode has to be controlled. 
+Up until the upcoming JRuby 1.7, JRuby runs in 1.8 Ruby compatibility mode by default. Unless you have a specific need to run topologies in 1.8 mode, you should use 1.9 mode, which will become the default in JRuby. 
 
-- when installing the topology required gems. the installation path embeds the Ruby version
-- when running in local mode or for the submission phase in remote/cluster mode
-- when Storm runs the topology in remote/cluster mode
+There are two ways to have JRuby 1.6.x run in 1.9 mode by default:
+- by setting the JRUBY_OPTS env variable
 
-For each of these contexts, 1.9 mode has to be explicitly specified to avoid any problems. All commands/examples below will use the 1.9 compatibility mode. If you want to avoid the explicit --1.9 mode option, using [RVM][rvm] you can compile your JRuby to run in 1.9 mode by default. If you run your topology in remote/cluster mode, you will still need to include some bits of 1.9 options and configuration since in this case JRuby and your topology is run independently by Storm.
+  ``` sh
+  $ export JRUBY_OPTS=--1.9
+  ```
+- by installing JRuby using RVM with 1.9 mode by default
 
-## Installation
+  ``` sh
+  $ rvm install jruby --1.9
+  ```
+
+Otherwise, to manually choose the JRuby compatibility mode, this JRuby syntax can be used 
 
-### Latest released gem
 ``` sh
-$ gem install redstorm
+$ jruby --1.9 -S redstorm ...
 ```
 
-### From github master
-
-- clone/fork project
+By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
 
 ``` sh
-$ gem build redstorm.gemspec
-$ gem install redstorm-x.y.z.gem
+$ redstorm local|cluster [--1.8|--1.9] ...
 ```
 
+## Installation
+
+- RubyGems
+
+  ``` sh
+  $ gem install redstorm
+  ```
+
+- Bundler
+
+  ``` ruby
+  source :rubygems
+  gem "redstorm", "~> 0.6.0"
+  ```
+
 ## Usage overview
 
-- create a new empty project directory.
+- create a project directory.
 - install the [RedStorm gem](http://rubygems.org/gems/redstorm).
-- create a subdirectory which will contain your sources.
-- perform the initial setup as described below to install the dependencies in the `target/` subdir of your project directory.
-- run your topology in local mode and/or on a production cluster as described below.
+- create a subdirectory for your topology code.
+- perform the initial setup as described below to build and install dependencies.
+- run your topology in local mode and/or on a remote cluster as described below.
 
 ### Initial setup
 
-- install RedStom dependencies. From your project root directory execute:
+``` sh
+$ redstorm install
+```
 
-  ``` sh
-  $ redstorm --1.9 install
-  ```
+or if your default JRuby mode is 1.8 but you want to use 1.9 for your topology development, use
+
+``` sh
+$ jruby --1.9 -S redstorm install
+```
 
-  The `install` command will install all Java jars dependencies using [ruby-maven][ruby-maven] in `target/dependency`, generate & compile the Java bindings in `target/classes` and install gems in `target/gems`.
+This will basically install all Java jar dependencies in `target/dependency`, generate & compile the Java bindings in `target/classes`.
 
-  ***DON'T PANIC*** it's Maven. The first time you run `$ redstorm --1.9 install` Maven will take a few minutes resolving dependencies and in the end will download and install the dependency jar files.
+### Create a topology
 
-- create a topology class in your sources subdirectory. The *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
+Create a subdirectory for your topology code and create your topology class **using this naming convention**: *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
 
 ### Gems in your topology
 
-RedStorm now support [Bundler](http://gembundler.com/) for using gems in your topology. Basically supply a `Gemfile` in the root of your project directory and execute this command to install the gems into the `target/gems` directory. **Note that if you change the Gemfile you must rerun this command**.
+RedStorm requires [Bundler](http://gembundler.com/) **if gems are needed** in your topology. Basically supply a `Gemfile` in the root of your project directory with the gems required in your topology. If you are using Bundler for other gems **you should** group the topology gems in a Bunder group of your choice.
+
+1. have Bundler install the gems locally
+
+  ``` sh
+  $ bundle install
+  ```
+
+  or if your default JRuby mode is 1.8 but you want to use 1.9 for your topology development, use
+
+  ``` sh
+  $ jruby --1.9 -S bundle install
+  ```
+
+2. copy the topology gems into the `target/gems` directory
 
   ``` sh
-  $ redstorm --1.9 bundle [--gemfile=GEMFILE]
+  $ redstorm bundle [BUNDLER_GROUP]
   ```
 
-All `bundle install` command options can be passed as options to `redstorm --1.9 bundle` like `--gemfile=GEMFILE` to specify a Gemfile in an alternate path.
+Basically, the `redstorm bundle` command copy the gems specified in the Gemfile (in a specific group if specified) into the `target/gems` directory. In order for the topology to run in a Storm cluster, the fully *installed* gems must be packaged and self-contained into a single jar file. **Note** you should **NOT** `require 'bundler/setup'` in the topology. 
 
-Basically, the `redstorm --1.9 bundle` command installs the *Bundler* and *Rake* gems and all the gems specified in the Gemfile into the `target/gems` directory. The idea is that in order for the topology to run in a Storm cluster, everything, including the fully *installed* gems, must be packaged and self-contained into a single JAR file. This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the JAR file. This could possibly lead to problems if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have *native* C/FFI extensions.
+This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the jar file. This **could lead to problems** if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have **C or FFI** extensions.
 
 ### Run in local mode
 
 ``` sh
-$ redstorm --1.9 local <path/to/topology_class_file_name.rb>
+$ redstorm local [--1.8|--1.9]  <path/to/topology_class_file_name.rb>
 ```
 
+By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
+
 **See examples below** to run examples in local mode or on a production cluster.
 
 ### Run on production cluster
 
-- generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies from the `target/` directory:
+1. download and unpack the [Storm 0.7.3 distribution](https://github.com/downloads/nathanmarz/storm/storm-0.7.3.zip) locally and **add** the Storm `bin/` directory to your path
+
+2. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
 
   ``` sh
-  $ redstorm --1.9 jar <sources_directory1> <sources_directory2> ...
+  $ redstorm jar <sources_directory1> <sources_directory2> ...
   ```
 
-- submit the cluster topology jar file to the cluster. Assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+3. submit the cluster topology jar file to the cluster
 
   ``` sh
-  storm jar ./target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster <path/to/topology_class_file_name.rb>
+  $ redstorm cluster [--1.8|--1.9]  <path/to/topology_class_file_name.rb>
   ```
 
-  Note the **-Djruby.compat.version=RUBY1_9** parameter. 
+  By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
 
-Basically you must follow the [Storm instructions](https://github.com/nathanmarz/storm/wiki) to [setup a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster) and [submit your topology to the cluster](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
+The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [setting up a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). You can also [manually submit your topology](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
 
 ## Examples
 
@@ -115,126 +155,139 @@ All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/
 
 ### Local mode
 
-``` sh
-$ redstorm --1.9 local examples/simple/exclamation_topology.rb
-$ redstorm --1.9 local examples/simple/exclamation_topology2.rb
-$ redstorm --1.9 local examples/simple/word_count_topology.rb
-```
-
-To run `examples/simple/redis_word_count_topology.rb` you need a [Redis][redis] server running on `localhost:6379`
+#### Example topologies without gems 
 
 ``` sh
-$ redstorm --1.9 bundle --gemfile examples/simple/Gemfile
+$ redstorm local --1.9 examples/simple/exclamation_topology.rb
+$ redstorm local --1.9 examples/simple/exclamation_topology2.rb
+$ redstorm local --1.9 examples/simple/word_count_topology.rb
 ```
 
-Run the topology in local mode
+#### Example topologies with gems 
 
-``` sh
-$ redstorm --1.9 local examples/simple/redis_word_count_topology.rb
-```
+For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis][redis] server running on `localhost:6379`
 
-Using `redis-cli`, push words into the `test` list and watch Storm pick them up
+1. create a `Gemfile`
 
-### Production cluster
+  ``` ruby
+  source :rubygems
 
-All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) can also run on a productions cluster. The only **native** example compatible with a production cluster is the [ClusterWordCountTopology](https://github.com/colinsurprenant/redstorm/tree/master/examples/native/cluster_word_count_topology.rb)
+  group :word_count do
+      gem "redis"
+  end
+  ```
 
-- genererate the `target/cluster-topology.jar` and include the `examples/` directory.
+2. install the topology gems
 
   ``` sh
-  $ redstorm --1.9 jar examples
+  $ jruby --1.9 -S bundle install
+  $ redstorm bundle word_count
   ```
 
-- submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+3. run the topology in local mode
 
   ``` sh
-  $ storm jar ./target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster examples/simple/word_count_topology.rb
+  $ redstorm local --1.9 examples/simple/redis_word_count_topology.rb
   ```
 
-  Note the **-Djruby.compat.version=RUBY1_9** parameter.
+Using `redis-cli` push words into the `test` list and watch Storm pick them up
 
-- to run `examples/simple/redis_word_count_topology.rb` you need a [Redis][redis] server running on `localhost:6379`
+### Remote cluster
 
-   ``` sh
-  $ redstorm --1.9 bundle --gemfile examples/simple/Gemfile
-  $ redstorm --1.9 jar examples
-  $ storm jar ./target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster examples/simple/redis_word_count_topology.rb
-  ```
+All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) can run in both local or on a remote cluster. The only **native** example compatible with a remote cluster is `examples/native/cluster_word_count_topology.rb`.
 
-  - using `redis-cli`, push words into the `test` list and watch Storm pick them up
 
-Basically you must follow the [Storm instructions](https://github.com/nathanmarz/storm/wiki) to [setup a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster) and [submit your topology to the cluster](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
+#### Topologies without gems 
 
-## Ruby DSL
+1. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
 
-[Ruby DSL Documentation](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation)
+  ``` sh
+  $ redstorm jar examples
+  ```
 
-## RedStorm Development
+2. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
 
-It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.
+  ``` sh
+  $ redstorm cluster --1.9 examples/simple/exclamation_topology.rb
+  $ redstorm cluster --1.9 examples/simple/exclamation_topology2.rb
+  $ redstorm cluster --1.9 examples/simple/word_count_topology.rb
+  ```
 
-### Requirements
 
-- JRuby 1.6.7
+#### Topologies with gems 
 
-### Workflow
+For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis][redis] server running on `localhost:6379`
 
-- fork project and create branch
+1. create a `Gemfile`
 
-- install required gems
+  ``` ruby
+  source :rubygems
 
-  ```sh
+  group :word_count do
+      gem "redis"
+  end
+  ```
+
+2. install the topology gems
+
+  ``` sh
   $ jruby --1.9 -S bundle install
+  $ redstorm bundle word_count
   ```
 
----
+3. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
 
-- install dependencies in `target/dependencies`
+  ``` sh
+  $ redstorm jar examples
+  ```
 
-  ```sh
-  $ bin/redstorm --1.9 deps
+4. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+
+  ``` sh
+  $ redstorm cluster --1.9 examples/simple/redis_word_count_topology.rb
   ```
 
-  **if you modify any of the RedStorm Ruby code** in `lib/red_storm`, you need to run this to refresh code in `target/`.
+Using `redis-cli` push words into the `test` list and watch Storm pick them up
 
----
+The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [setting up a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). You can also [manually submit your topology](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
 
-- generate and build Java source into `target/classes`
+## Ruby DSL
 
-  ```sh
-  $ bin/redstorm --1.9 build
-  ```
+[Ruby DSL Documentation](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation)
 
-  **if you modify any of the Java binding code**, you need to run this to rebuild the bindings
+## RedStorm Development
 
----
+It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.
 
-- run topology in **local** Storm mode
+### Requirements
 
-  ```sh
-  $ bin/redstorm --1.9 local path/to/topology_class.rb
-  ```
+- JRuby 1.6.7.x
 
-  If you only make changes to your topology code, this is the only step you need to repeat to try your updated code.
+### Workflow
 
----
+- fork project and create branch
 
-- generate remote cluster topology jar into `target/cluster-topology.jar`, including the `mydir/` directory.
+- install RedStorm required gems
 
   ```sh
-  $ bin/redstorm --1.9 jar mydir otherdir1 otherdir2 ...
+  $ jruby --1.9 -S bundle install
   ```
 
----
+- install dependencies in `target/dependencies`
+
+  ```sh
+  $ bin/redstorm deps
+  ```
 
-- **if you add/change** Gemfile for your topology, install gems in `target/gems`. Alternate gemfile path can be specified using --gemfile=GEMFILE
+- generate and build Java source into `target/classes`
 
   ```sh
-  $ bin/redstorm --1.9 bundle [--gemfile=GEMFILE]
+  $ jruby --1.9 -S bin/redstorm build
   ```
 
-  **do not forget** to rerurn `bin/redstorm --1.9 jar ...` to pick up these gems, before submitting your topology on a remote cluster.
+  **if you modify any of the RedStorm Ruby code or Java binding code**, you need to run this to refresh code and rebuild the bindings
 
+- follow the normal usage patterns to run the topology in local or remote cluster.
 
 ### How to Contribute
 
@@ -250,9 +303,18 @@ Some ways you can contribute:
 - by refactoring code
 - ...
 
+## Projects using RedStorm
+
+If you want to list your RedStorm project here, contact me.
+
+- [Tweigeist](https://github.com/colinsurprenant/tweitgeist) - realtime computation of the top trending hashtags on Twitter. [Live Demo](http://tweitgeist.needium.com).
+
 ## Author
 Colin Surprenant, [@colinsurprenant][twitter], [http://github.com/colinsurprenant][github], colin.surprenant@gmail.com, colin.surprenant@needium.com
 
+## Contributors
+Theo Hultberg, https://github.com/iconara
+
 ## License
 Apache License, Version 2.0. See the LICENSE.md file.
 

data/examples/simple/redis_word_count_topology.rb

@@ -1,6 +1,5 @@
-require 'rubygems'      # required for remote cluster exec where TopolyLauncher + require rubygems is not called
-require 'red_storm'     # must be required before bundler for environment setup and after rubygems
-require 'bundler/setup'
+require 'rubygems'
+require 'red_storm'
 
 require 'redis'
 require 'thread'
@@ -15,7 +14,7 @@ module RedStorm
     class RedisWordSpout < RedStorm::SimpleSpout
       output_fields :word
 
-      on_send {@q.pop if @q.size > 0}
+      on_send {@q.pop.to_s if @q.size > 0}
 
       on_init do
         @q = Queue.new
@@ -52,6 +51,7 @@ module RedStorm
         when :local
           max_task_parallelism 3
         when :cluster
+          max_task_parallelism 5
           num_workers 20
           max_spout_pending(1000);
         end

data/examples/simple/ruby_version_topology.rb

@@ -6,7 +6,7 @@ module RedStorm
   module Examples
     class VersionSpout < RedStorm::SimpleSpout
       output_fields :dummy
-      on_init {log.info("****** JRuby version #{RUBY_VERSION}")}
+      on_init {log.info("***************** RUBY_VERSION=#{RUBY_VERSION}")}
       on_send {}
     end
 
@@ -16,14 +16,14 @@ module RedStorm
       configure do |env|
         debug true
 
-        # set the JRuby version property for this topology. this will only affect remote cluster execution
+        # force the JRuby version property for this topology. this will only affect remote cluster execution
         # for local execution use the --1.8|--1.9 switch when launching
-        set "topology.worker.childopts", "-Djruby.compat.version=RUBY1_9"
+        # set "topology.worker.childopts", "-Djruby.compat.version=RUBY1_9"
       end
 
       on_submit do |env|
         if env == :local
-          sleep(1)
+          sleep(5)
           cluster.shutdown
         end
       end