kb-redstorm 0.6.4

data/CHANGELOG.md

@@ -0,0 +1,74 @@
+# 0.1.0, 11-07-2011
+- initial release
+
+# 0.1.1, 11-10-2011
+- issue #1 cannot find redstorm gem when using rbenv
+
+# 0.2.0, 11-16-2011
+- issue #2 redstorm examples fails when directory examples already exists
+- new *simple* DSL
+- examples using simple DSL
+- redstorm command usage syntax change
+- more doc in README
+
+# 0.2.1, 11-23-2011
+- gems support in production cluster
+
+# 0.3.0, 12-08-2011
+- Storm 0.6.0
+
+# 0.4.0, 02-08-2012
+- Storm 0.6.2
+- JRuby 1.6.6
+
+# 0.5.0, 05-28-2012
+- issue #16, Bundler support for topology gems
+- issue #19, support for multiple dirs in jar
+- issue #20, update to Storm 0.7.1
+- issue #21, proper support for 1.8/1.9 Ruby compatibility mode
+- issue #22, fixed Config class name clash
+- JRuby 1.6.7
+- DSL Support for per bolt/spout configuration and spout activate/deactivate in Storm 0.7.x
+- consistent workflow using the redstorm command in local dev or gem mode
+
+# 0.5.1, 06-05-2012
+- better handling of enviroments and paths
+- redstorm bundle command to install topology gems
+- issue #26, fixed examples/native for 0.5.1 compatibility
+
+# 0.6.0, 06-27-2012
+- issue #29, updated to Storm 0.7.3
+- issue #30, add redstorm cluster command for remote topology submission
+- issue #31, added support for localOrShuffleGrouping
+- issue #33, avoid forking or shelling out on redstorm commands
+- issue #35, automatically set JRuby 1.8/1.9 mode in remote cluster context
+- JRuby 1.6.7.2
+- better handling of JRuby 1.8/1.9 mode
+- topology gems are now specified using a Bundler group in the project Gemfile
+- refactored environment/paths handling for local vs cluster context
+
+# 0.6.1, 07-07-2012
+- issue #38, added support for spout reliable emit
+- gem path is always in target/gems for both local and cluster context
+- temp fix for slf4j dependencies conflict (issue #36)
+- Storm 0.7.4
+
+# 0.6.2, 07-10-2012
+- issue #39, spout on_receive block will not be evaluated if :emit => false
+- issue #40, bolt fail method missing
+- integration tests support
+
+# 0.6.3, 10-09-2012
+- issue #28, allow specification of output_fields in topology DSL
+- issue #41, add support for ShellBolt and ShellSpout
+- issue #49, redstorm bundle not picking up default group in Gemfile
+- support constructor parameters for Java spout/bolt in topology DSL
+
+# 0.6.4, 10-19-2012
+- Storm 0.8.1 and JRuby 1.6.8
+- improved and more flexible jar dependencies handling
+- issue #36, fix slf4j-api-1.6.3 and slf4j-log4j12-1.5.8 version conflict
+- issue #37, fix dependencies xml files naming
+- issue #47, Log4j Logger class conflict
+- issue #48, add support for external Jars
+- issue #50, update RedStorm to target Storm 0.8

data/LICENSE.md

@@ -0,0 +1,13 @@
+# License
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,375 @@
+# RedStorm v0.6.4 - JRuby on Storm
+
+[![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
+
+RedStorm provides a Ruby DSL using JRuby integration for the [Storm](https://github.com/nathanmarz/storm/) distributed realtime computation system.
+
+## Documentation
+
+Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary. 
+
+### Released gems
+
+- [RedStorm Gem v0.4.x Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.4.x-Documentation)
+- [RedStorm Gem v0.5.0 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.0-Documentation)
+- [RedStorm Gem v0.5.1 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.1-Documentation)
+- [RedStorm Gem v0.6.3 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.6.3-Documentation)
+
+## Dependencies
+
+Tested on **OSX 10.8.2** and **Ubuntu Linux 12.04** using **Storm 0.8.1** and **JRuby 1.6.8** and **OpenJDK 7**
+
+## Notes about 1.8/1.9 JRuby compatibility
+
+Up until the upcoming JRuby 1.7, JRuby runs in 1.8 Ruby compatibility mode by default. Unless you have a specific need to run topologies in 1.8 mode, you should use 1.9 mode, which will become the default in JRuby. 
+
+There are two ways to have JRuby 1.6.x run in 1.9 mode by default:
+- by setting the JRUBY_OPTS env variable
+
+  ``` sh
+  $ export JRUBY_OPTS=--1.9
+  ```
+- by installing JRuby using RVM with 1.9 mode by default
+
+  ``` sh
+  $ rvm install jruby --1.9
+  ```
+
+Otherwise, to manually choose the JRuby compatibility mode, this JRuby syntax can be used 
+
+``` sh
+$ jruby --1.9 -S redstorm ...
+```
+
+By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
+
+``` sh
+$ redstorm local|cluster [--1.8|--1.9] ...
+```
+
+## Installation
+
+- RubyGems
+
+  ``` sh
+  $ gem install redstorm
+  ```
+
+- Bundler
+
+  ``` ruby
+  source :rubygems
+  gem "redstorm", "~> 0.6.4"
+  ```
+
+## Usage overview
+
+- create a project directory.
+- install the [RedStorm gem](http://rubygems.org/gems/redstorm).
+- create a subdirectory for your topology code.
+- perform the initial setup as described below to build and install dependencies.
+- run your topology in local mode and/or on a remote cluster as described below.
+
+### Initial setup
+
+``` sh
+$ redstorm install
+```
+
+or if your default JRuby mode is 1.8 but you want to use 1.9 for your topology development, use
+
+``` sh
+$ jruby --1.9 -S redstorm install
+```
+
+This will basically install default Java jar dependencies in `target/dependency`, generate & compile the Java bindings in `target/classes`.
+
+### Create a topology
+
+Create a subdirectory for your topology code and create your topology class **using this naming convention**: *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
+
+### Gems in your topology
+
+RedStorm requires [Bundler](http://gembundler.com/) **if gems are needed** in your topology. Basically supply a `Gemfile` in the root of your project directory with the gems required in your topology. If you are using Bundler for other gems **you should** group the topology gems in a Bunder group of your choice.
+
+1. have Bundler install the gems locally
+
+  ``` sh
+  $ bundle install
+  ```
+
+  or if your default JRuby mode is 1.8 but you want to use 1.9 for your topology development, use
+
+  ``` sh
+  $ jruby --1.9 -S bundle install
+  ```
+
+2. copy the topology gems into the `target/gems` directory
+
+  ``` sh
+  $ redstorm bundle [BUNDLER_GROUP]
+  ```
+
+Basically, the `redstorm bundle` command copy the gems specified in the Gemfile (in a specific group if specified) into the `target/gems` directory. In order for the topology to run in a Storm cluster, the fully *installed* gems must be packaged and self-contained into a single jar file. **Note** you should **NOT** `require 'bundler/setup'` in the topology. 
+
+This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the jar file. This **could lead to problems** if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have **C or FFI** extensions.
+
+### Custom Jar dependencies in your topology
+
+By defaut, RedStorm installs Storm and JRuby jars dependencies. If you require custom dependencies, these can be specified by creating the `Dependencies` file in the root of your project. Note that this file overwrites the defaults dependencies so you must also include the Storm and JRuby dependencies. Here's an example of a `Dependencies` file which included the jars required to run the `KafkaTopology` in the examples.
+
+``` ruby
+{
+  :storm_artifacts => [
+    "storm:storm:0.8.1, transitive=true",
+  ],
+  :topology_artifacts => [
+    "org.jruby:jruby-complete:1.6.8, transitive=false",
+    "org.scala-lang:scala-library:2.8.0, transitive=false",
+    "storm:kafka:0.7.0-incubating, transitive=false",
+    "storm:storm-kafka:0.8.0-wip4, transitive=false",
+  ],
+}
+```
+
+Basically the dependendencies are speified as Maven artifacts. There are two sections, the `:storm_artifacts =>` contains the dependencies for running storm in local mode and the `:topology_artifacts =>` are the dependencies specific for your topology. The format is self explainatory and the attribute `transitive=[true|false]` controls the recursive dependencies resolution (using `true`).
+
+The jars repositories can be configured by adding the `ivy/setting.xml` file in the root of your project. For information on the Ivy settings format, see the [Ivy Settings Documentation](http://ant.apache.org/ivy/history/2.2.0/settings.html). I will try my best to eliminate all XML :) but for now I haven't figured how to get rid of this one. For an example Ivy settings file, RedStorm is using the following settings by default:
+
+``` xml
+<ivysettings>
+  <settings defaultResolver="repositories"/>
+  <resolvers>
+    <chain name="repositories">
+      <ibiblio name="ibiblio" m2compatible="true"/>
+      <ibiblio name="maven2" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
+      <ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/> 
+      <ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/> 
+    </chain>
+  </resolvers>
+</ivysettings>
+```
+
+### Run in local mode
+
+``` sh
+$ redstorm local [--1.8|--1.9]  <path/to/topology_class_file_name.rb>
+```
+
+By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
+
+**See examples below** to run examples in local mode or on a production cluster.
+
+### Run on production cluster
+
+1. download and unpack the [Storm 0.8.1 distribution](https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip) locally and **add** the Storm `bin/` directory to your `$PATH`.
+
+2. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
+
+  ``` sh
+  $ redstorm jar <sources_directory1> <sources_directory2> ...
+  ```
+
+3. submit the cluster topology jar file to the cluster
+
+  ``` sh
+  $ redstorm cluster [--1.8|--1.9]  <path/to/topology_class_file_name.rb>
+  ```
+
+  By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
+
+The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [setting up a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). You can also [manually submit your topology](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
+
+## Examples
+
+Install the [example files](https://github.com/colinsurprenant/redstorm/tree/master/examples) in your project. The `examples/` dir will be created in your project root dir.
+
+``` sh
+$ redstorm examples
+```
+
+All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) are located in `examples/simple`. Examples using the standard Java interface are in `examples/native`.
+
+### Local mode
+
+#### Example topologies without gems 
+
+``` sh
+$ redstorm local examples/simple/exclamation_topology.rb
+$ redstorm local examples/simple/exclamation_topology2.rb
+$ redstorm local examples/simple/word_count_topology.rb
+```
+
+#### Example topologies with gems 
+
+For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
+
+1. create a `Gemfile`
+
+  ``` ruby
+  source :rubygems
+
+  group :word_count do
+    gem "redis"
+  end
+  ```
+
+2. install the topology gems
+
+  ``` sh
+  $ bundle install
+  $ redstorm bundle word_count
+  ```
+
+3. run the topology in local mode
+
+  ``` sh
+  $ redstorm local examples/simple/redis_word_count_topology.rb
+  ```
+
+Using `redis-cli` push words into the `test` list and watch Storm pick them up
+
+### Remote cluster
+
+All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) can run in both local or on a remote cluster. The only **native** example compatible with a remote cluster is `examples/native/cluster_word_count_topology.rb`.
+
+
+#### Topologies without gems 
+
+1. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
+
+  ``` sh
+  $ redstorm jar examples
+  ```
+
+2. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+
+  ``` sh
+  $ redstorm cluster examples/simple/exclamation_topology.rb
+  $ redstorm cluster examples/simple/exclamation_topology2.rb
+  $ redstorm cluster examples/simple/word_count_topology.rb
+  ```
+
+
+#### Topologies with gems 
+
+For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
+
+1. create a `Gemfile`
+
+  ``` ruby
+  source :rubygems
+
+  group :word_count do
+      gem "redis"
+  end
+  ```
+
+2. install the topology gems
+
+  ``` sh
+  $ bundle install
+  $ redstorm bundle word_count
+  ```
+
+3. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
+
+  ``` sh
+  $ redstorm jar examples
+  ```
+
+4. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+
+  ``` sh
+  $ redstorm cluster examples/simple/redis_word_count_topology.rb
+  ```
+
+Using `redis-cli` push words into the `test` list and watch Storm pick them up
+
+The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [setting up a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). You can also [manually submit your topology](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
+
+## Ruby DSL
+
+[Ruby DSL Documentation](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation)
+
+## Multilang ShellSpout & ShellBolt support
+
+Please refer to [Using non JVM languages with Storm](https://github.com/nathanmarz/storm/wiki/Using-non-JVM-languages-with-Storm) for the complete information on Multilang & shelling in Storm.
+
+In RedStorm *ShellSpout* and *ShellBolt* are supported using the following construct in the topology definition:
+
+``` ruby
+bolt JRubyShellBolt, ["python", "splitsentence.py"] do
+  output_fields "word"
+  source SimpleSpout, :shuffle
+end
+```
+
+- `JRubyShellBolt` must be used for a *ShellBolt* and the array argument `["python", "splitsentence.py"]` are the arguments to the class constructor and are the *commands* to the *ShellBolt*.
+
+- The directory containing the topology class **must** contain a `resources/` directory with all the shell files.
+
+See the [shell topology example](https://github.com/colinsurprenant/redstorm/tree/master/examples/shell)
+
+## RedStorm Development
+
+It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.
+
+### Requirements
+
+- JRuby 1.6.8
+
+### Workflow
+
+- fork project and create branch
+
+- install RedStorm required gems
+
+  ```sh
+  $ bundle install
+  ```
+
+- install dependencies in `target/dependencies`
+
+  ```sh
+  $ bin/redstorm deps
+  ```
+
+- generate and build Java source into `target/classes`
+
+  ```sh
+  $ bin/redstorm build
+  ```
+
+  **if you modify any of the RedStorm Ruby code or Java binding code**, you need to run this to refresh code and rebuild the bindings
+
+- follow the normal usage patterns to run the topology in local or remote cluster.
+
+### How to Contribute
+
+Fork the project, create a branch and submit a pull request.
+
+Some ways you can contribute:
+
+- by reporting bugs using the issue tracker
+- by suggesting new features using the issue tracker
+- by writing or editing documentation
+- by writing specs
+- by writing code
+- by refactoring code
+- ...
+
+## Projects using RedStorm
+
+If you want to list your RedStorm project here, contact me.
+
+- [Tweigeist](https://github.com/colinsurprenant/tweitgeist) - realtime computation of the top trending hashtags on Twitter. See [Live Demo](http://tweitgeist.colinsurprenant.com/).
+
+## Author
+***Colin Surprenant***, [@colinsurprenant](http://twitter.com/colinsurprenant/), [http://github.com/colinsurprenant/](http://github.com/colinsurprenant/), colin.surprenant@gmail.com, [http://colinsurprenant.com/](http://colinsurprenant.com/)
+
+## Contributors
+Theo Hultberg, https://github.com/iconara
+
+## License
+Apache License, Version 2.0. See the LICENSE.md file.