kb-redstorm 0.6.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/CHANGELOG.md +74 -0
- data/LICENSE.md +13 -0
- data/README.md +375 -0
- data/Rakefile +11 -0
- data/bin/redstorm +15 -0
- data/examples/native/Gemfile +2 -0
- data/examples/native/cluster_word_count_topology.rb +25 -0
- data/examples/native/exclamation_bolt.rb +21 -0
- data/examples/native/local_exclamation_topology.rb +31 -0
- data/examples/native/local_exclamation_topology2.rb +48 -0
- data/examples/native/local_redis_word_count_topology.rb +69 -0
- data/examples/native/local_word_count_topology.rb +27 -0
- data/examples/native/random_sentence_spout.rb +30 -0
- data/examples/native/split_sentence_bolt.rb +20 -0
- data/examples/native/word_count_bolt.rb +26 -0
- data/examples/shell/resources/splitsentence.py +9 -0
- data/examples/shell/resources/storm.py +206 -0
- data/examples/shell/shell_topology.rb +41 -0
- data/examples/simple/exclamation_bolt.rb +10 -0
- data/examples/simple/exclamation_topology.rb +45 -0
- data/examples/simple/exclamation_topology2.rb +45 -0
- data/examples/simple/kafka_topology.rb +55 -0
- data/examples/simple/random_sentence_spout.rb +21 -0
- data/examples/simple/redis_word_count_topology.rb +61 -0
- data/examples/simple/ruby_version_topology.rb +32 -0
- data/examples/simple/split_sentence_bolt.rb +33 -0
- data/examples/simple/word_count_bolt.rb +19 -0
- data/examples/simple/word_count_topology.rb +38 -0
- data/ivy/settings.xml +11 -0
- data/lib/red_storm.rb +9 -0
- data/lib/red_storm/application.rb +85 -0
- data/lib/red_storm/configuration.rb +16 -0
- data/lib/red_storm/configurator.rb +26 -0
- data/lib/red_storm/environment.rb +41 -0
- data/lib/red_storm/loggable.rb +15 -0
- data/lib/red_storm/proxy/batch_spout.rb +71 -0
- data/lib/red_storm/proxy/bolt.rb +63 -0
- data/lib/red_storm/proxy/proxy_function.rb +48 -0
- data/lib/red_storm/proxy/spout.rb +87 -0
- data/lib/red_storm/simple_bolt.rb +135 -0
- data/lib/red_storm/simple_drpc_topology.rb +87 -0
- data/lib/red_storm/simple_spout.rb +184 -0
- data/lib/red_storm/simple_topology.rb +209 -0
- data/lib/red_storm/topology_launcher.rb +54 -0
- data/lib/red_storm/version.rb +3 -0
- data/lib/tasks/red_storm.rake +272 -0
- data/src/main/redstorm/storm/jruby/JRubyBatchSpout.java +89 -0
- data/src/main/redstorm/storm/jruby/JRubyBolt.java +88 -0
- data/src/main/redstorm/storm/jruby/JRubyProxyFunction.java +59 -0
- data/src/main/redstorm/storm/jruby/JRubyShellBolt.java +26 -0
- data/src/main/redstorm/storm/jruby/JRubyShellSpout.java +26 -0
- data/src/main/redstorm/storm/jruby/JRubySpout.java +107 -0
- metadata +134 -0
data/CHANGELOG.md
ADDED
@@ -0,0 +1,74 @@
|
|
1
|
+
# 0.1.0, 11-07-2011
|
2
|
+
- initial release
|
3
|
+
|
4
|
+
# 0.1.1, 11-10-2011
|
5
|
+
- issue #1 cannot find redstorm gem when using rbenv
|
6
|
+
|
7
|
+
# 0.2.0, 11-16-2011
|
8
|
+
- issue #2 redstorm examples fails when directory examples already exists
|
9
|
+
- new *simple* DSL
|
10
|
+
- examples using simple DSL
|
11
|
+
- redstorm command usage syntax change
|
12
|
+
- more doc in README
|
13
|
+
|
14
|
+
# 0.2.1, 11-23-2011
|
15
|
+
- gems support in production cluster
|
16
|
+
|
17
|
+
# 0.3.0, 12-08-2011
|
18
|
+
- Storm 0.6.0
|
19
|
+
|
20
|
+
# 0.4.0, 02-08-2012
|
21
|
+
- Storm 0.6.2
|
22
|
+
- JRuby 1.6.6
|
23
|
+
|
24
|
+
# 0.5.0, 05-28-2012
|
25
|
+
- issue #16, Bundler support for topology gems
|
26
|
+
- issue #19, support for multiple dirs in jar
|
27
|
+
- issue #20, update to Storm 0.7.1
|
28
|
+
- issue #21, proper support for 1.8/1.9 Ruby compatibility mode
|
29
|
+
- issue #22, fixed Config class name clash
|
30
|
+
- JRuby 1.6.7
|
31
|
+
- DSL Support for per bolt/spout configuration and spout activate/deactivate in Storm 0.7.x
|
32
|
+
- consistent workflow using the redstorm command in local dev or gem mode
|
33
|
+
|
34
|
+
# 0.5.1, 06-05-2012
|
35
|
+
- better handling of enviroments and paths
|
36
|
+
- redstorm bundle command to install topology gems
|
37
|
+
- issue #26, fixed examples/native for 0.5.1 compatibility
|
38
|
+
|
39
|
+
# 0.6.0, 06-27-2012
|
40
|
+
- issue #29, updated to Storm 0.7.3
|
41
|
+
- issue #30, add redstorm cluster command for remote topology submission
|
42
|
+
- issue #31, added support for localOrShuffleGrouping
|
43
|
+
- issue #33, avoid forking or shelling out on redstorm commands
|
44
|
+
- issue #35, automatically set JRuby 1.8/1.9 mode in remote cluster context
|
45
|
+
- JRuby 1.6.7.2
|
46
|
+
- better handling of JRuby 1.8/1.9 mode
|
47
|
+
- topology gems are now specified using a Bundler group in the project Gemfile
|
48
|
+
- refactored environment/paths handling for local vs cluster context
|
49
|
+
|
50
|
+
# 0.6.1, 07-07-2012
|
51
|
+
- issue #38, added support for spout reliable emit
|
52
|
+
- gem path is always in target/gems for both local and cluster context
|
53
|
+
- temp fix for slf4j dependencies conflict (issue #36)
|
54
|
+
- Storm 0.7.4
|
55
|
+
|
56
|
+
# 0.6.2, 07-10-2012
|
57
|
+
- issue #39, spout on_receive block will not be evaluated if :emit => false
|
58
|
+
- issue #40, bolt fail method missing
|
59
|
+
- integration tests support
|
60
|
+
|
61
|
+
# 0.6.3, 10-09-2012
|
62
|
+
- issue #28, allow specification of output_fields in topology DSL
|
63
|
+
- issue #41, add support for ShellBolt and ShellSpout
|
64
|
+
- issue #49, redstorm bundle not picking up default group in Gemfile
|
65
|
+
- support constructor parameters for Java spout/bolt in topology DSL
|
66
|
+
|
67
|
+
# 0.6.4, 10-19-2012
|
68
|
+
- Storm 0.8.1 and JRuby 1.6.8
|
69
|
+
- improved and more flexible jar dependencies handling
|
70
|
+
- issue #36, fix slf4j-api-1.6.3 and slf4j-log4j12-1.5.8 version conflict
|
71
|
+
- issue #37, fix dependencies xml files naming
|
72
|
+
- issue #47, Log4j Logger class conflict
|
73
|
+
- issue #48, add support for external Jars
|
74
|
+
- issue #50, update RedStorm to target Storm 0.8
|
data/LICENSE.md
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
# License
|
2
|
+
|
3
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
4
|
+
you may not use this file except in compliance with the License.
|
5
|
+
You may obtain a copy of the License at
|
6
|
+
|
7
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
8
|
+
|
9
|
+
Unless required by applicable law or agreed to in writing, software
|
10
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
11
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12
|
+
See the License for the specific language governing permissions and
|
13
|
+
limitations under the License.
|
data/README.md
ADDED
@@ -0,0 +1,375 @@
|
|
1
|
+
# RedStorm v0.6.4 - JRuby on Storm
|
2
|
+
|
3
|
+
[](http://travis-ci.org/colinsurprenant/redstorm)
|
4
|
+
|
5
|
+
RedStorm provides a Ruby DSL using JRuby integration for the [Storm](https://github.com/nathanmarz/storm/) distributed realtime computation system.
|
6
|
+
|
7
|
+
## Documentation
|
8
|
+
|
9
|
+
Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary.
|
10
|
+
|
11
|
+
### Released gems
|
12
|
+
|
13
|
+
- [RedStorm Gem v0.4.x Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.4.x-Documentation)
|
14
|
+
- [RedStorm Gem v0.5.0 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.0-Documentation)
|
15
|
+
- [RedStorm Gem v0.5.1 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.5.1-Documentation)
|
16
|
+
- [RedStorm Gem v0.6.3 Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.6.3-Documentation)
|
17
|
+
|
18
|
+
## Dependencies
|
19
|
+
|
20
|
+
Tested on **OSX 10.8.2** and **Ubuntu Linux 12.04** using **Storm 0.8.1** and **JRuby 1.6.8** and **OpenJDK 7**
|
21
|
+
|
22
|
+
## Notes about 1.8/1.9 JRuby compatibility
|
23
|
+
|
24
|
+
Up until the upcoming JRuby 1.7, JRuby runs in 1.8 Ruby compatibility mode by default. Unless you have a specific need to run topologies in 1.8 mode, you should use 1.9 mode, which will become the default in JRuby.
|
25
|
+
|
26
|
+
There are two ways to have JRuby 1.6.x run in 1.9 mode by default:
|
27
|
+
- by setting the JRUBY_OPTS env variable
|
28
|
+
|
29
|
+
``` sh
|
30
|
+
$ export JRUBY_OPTS=--1.9
|
31
|
+
```
|
32
|
+
- by installing JRuby using RVM with 1.9 mode by default
|
33
|
+
|
34
|
+
``` sh
|
35
|
+
$ rvm install jruby --1.9
|
36
|
+
```
|
37
|
+
|
38
|
+
Otherwise, to manually choose the JRuby compatibility mode, this JRuby syntax can be used
|
39
|
+
|
40
|
+
``` sh
|
41
|
+
$ jruby --1.9 -S redstorm ...
|
42
|
+
```
|
43
|
+
|
44
|
+
By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
|
45
|
+
|
46
|
+
``` sh
|
47
|
+
$ redstorm local|cluster [--1.8|--1.9] ...
|
48
|
+
```
|
49
|
+
|
50
|
+
## Installation
|
51
|
+
|
52
|
+
- RubyGems
|
53
|
+
|
54
|
+
``` sh
|
55
|
+
$ gem install redstorm
|
56
|
+
```
|
57
|
+
|
58
|
+
- Bundler
|
59
|
+
|
60
|
+
``` ruby
|
61
|
+
source :rubygems
|
62
|
+
gem "redstorm", "~> 0.6.4"
|
63
|
+
```
|
64
|
+
|
65
|
+
## Usage overview
|
66
|
+
|
67
|
+
- create a project directory.
|
68
|
+
- install the [RedStorm gem](http://rubygems.org/gems/redstorm).
|
69
|
+
- create a subdirectory for your topology code.
|
70
|
+
- perform the initial setup as described below to build and install dependencies.
|
71
|
+
- run your topology in local mode and/or on a remote cluster as described below.
|
72
|
+
|
73
|
+
### Initial setup
|
74
|
+
|
75
|
+
``` sh
|
76
|
+
$ redstorm install
|
77
|
+
```
|
78
|
+
|
79
|
+
or if your default JRuby mode is 1.8 but you want to use 1.9 for your topology development, use
|
80
|
+
|
81
|
+
``` sh
|
82
|
+
$ jruby --1.9 -S redstorm install
|
83
|
+
```
|
84
|
+
|
85
|
+
This will basically install default Java jar dependencies in `target/dependency`, generate & compile the Java bindings in `target/classes`.
|
86
|
+
|
87
|
+
### Create a topology
|
88
|
+
|
89
|
+
Create a subdirectory for your topology code and create your topology class **using this naming convention**: *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
|
90
|
+
|
91
|
+
### Gems in your topology
|
92
|
+
|
93
|
+
RedStorm requires [Bundler](http://gembundler.com/) **if gems are needed** in your topology. Basically supply a `Gemfile` in the root of your project directory with the gems required in your topology. If you are using Bundler for other gems **you should** group the topology gems in a Bunder group of your choice.
|
94
|
+
|
95
|
+
1. have Bundler install the gems locally
|
96
|
+
|
97
|
+
``` sh
|
98
|
+
$ bundle install
|
99
|
+
```
|
100
|
+
|
101
|
+
or if your default JRuby mode is 1.8 but you want to use 1.9 for your topology development, use
|
102
|
+
|
103
|
+
``` sh
|
104
|
+
$ jruby --1.9 -S bundle install
|
105
|
+
```
|
106
|
+
|
107
|
+
2. copy the topology gems into the `target/gems` directory
|
108
|
+
|
109
|
+
``` sh
|
110
|
+
$ redstorm bundle [BUNDLER_GROUP]
|
111
|
+
```
|
112
|
+
|
113
|
+
Basically, the `redstorm bundle` command copy the gems specified in the Gemfile (in a specific group if specified) into the `target/gems` directory. In order for the topology to run in a Storm cluster, the fully *installed* gems must be packaged and self-contained into a single jar file. **Note** you should **NOT** `require 'bundler/setup'` in the topology.
|
114
|
+
|
115
|
+
This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the jar file. This **could lead to problems** if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have **C or FFI** extensions.
|
116
|
+
|
117
|
+
### Custom Jar dependencies in your topology
|
118
|
+
|
119
|
+
By defaut, RedStorm installs Storm and JRuby jars dependencies. If you require custom dependencies, these can be specified by creating the `Dependencies` file in the root of your project. Note that this file overwrites the defaults dependencies so you must also include the Storm and JRuby dependencies. Here's an example of a `Dependencies` file which included the jars required to run the `KafkaTopology` in the examples.
|
120
|
+
|
121
|
+
``` ruby
|
122
|
+
{
|
123
|
+
:storm_artifacts => [
|
124
|
+
"storm:storm:0.8.1, transitive=true",
|
125
|
+
],
|
126
|
+
:topology_artifacts => [
|
127
|
+
"org.jruby:jruby-complete:1.6.8, transitive=false",
|
128
|
+
"org.scala-lang:scala-library:2.8.0, transitive=false",
|
129
|
+
"storm:kafka:0.7.0-incubating, transitive=false",
|
130
|
+
"storm:storm-kafka:0.8.0-wip4, transitive=false",
|
131
|
+
],
|
132
|
+
}
|
133
|
+
```
|
134
|
+
|
135
|
+
Basically the dependendencies are speified as Maven artifacts. There are two sections, the `:storm_artifacts =>` contains the dependencies for running storm in local mode and the `:topology_artifacts =>` are the dependencies specific for your topology. The format is self explainatory and the attribute `transitive=[true|false]` controls the recursive dependencies resolution (using `true`).
|
136
|
+
|
137
|
+
The jars repositories can be configured by adding the `ivy/setting.xml` file in the root of your project. For information on the Ivy settings format, see the [Ivy Settings Documentation](http://ant.apache.org/ivy/history/2.2.0/settings.html). I will try my best to eliminate all XML :) but for now I haven't figured how to get rid of this one. For an example Ivy settings file, RedStorm is using the following settings by default:
|
138
|
+
|
139
|
+
``` xml
|
140
|
+
<ivysettings>
|
141
|
+
<settings defaultResolver="repositories"/>
|
142
|
+
<resolvers>
|
143
|
+
<chain name="repositories">
|
144
|
+
<ibiblio name="ibiblio" m2compatible="true"/>
|
145
|
+
<ibiblio name="maven2" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/>
|
146
|
+
<ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/>
|
147
|
+
<ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/>
|
148
|
+
</chain>
|
149
|
+
</resolvers>
|
150
|
+
</ivysettings>
|
151
|
+
```
|
152
|
+
|
153
|
+
### Run in local mode
|
154
|
+
|
155
|
+
``` sh
|
156
|
+
$ redstorm local [--1.8|--1.9] <path/to/topology_class_file_name.rb>
|
157
|
+
```
|
158
|
+
|
159
|
+
By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
|
160
|
+
|
161
|
+
**See examples below** to run examples in local mode or on a production cluster.
|
162
|
+
|
163
|
+
### Run on production cluster
|
164
|
+
|
165
|
+
1. download and unpack the [Storm 0.8.1 distribution](https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip) locally and **add** the Storm `bin/` directory to your `$PATH`.
|
166
|
+
|
167
|
+
2. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
|
168
|
+
|
169
|
+
``` sh
|
170
|
+
$ redstorm jar <sources_directory1> <sources_directory2> ...
|
171
|
+
```
|
172
|
+
|
173
|
+
3. submit the cluster topology jar file to the cluster
|
174
|
+
|
175
|
+
``` sh
|
176
|
+
$ redstorm cluster [--1.8|--1.9] <path/to/topology_class_file_name.rb>
|
177
|
+
```
|
178
|
+
|
179
|
+
By defaut, a topology will be executed in the **same mode** as the interpreter running the `$ redstorm` command. You can force RedStorm to choose a specific JRuby compatibility mode using the [--1.8|--1.9] parameter for the topology execution in local or remote cluster.
|
180
|
+
|
181
|
+
The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [setting up a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). You can also [manually submit your topology](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
|
182
|
+
|
183
|
+
## Examples
|
184
|
+
|
185
|
+
Install the [example files](https://github.com/colinsurprenant/redstorm/tree/master/examples) in your project. The `examples/` dir will be created in your project root dir.
|
186
|
+
|
187
|
+
``` sh
|
188
|
+
$ redstorm examples
|
189
|
+
```
|
190
|
+
|
191
|
+
All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) are located in `examples/simple`. Examples using the standard Java interface are in `examples/native`.
|
192
|
+
|
193
|
+
### Local mode
|
194
|
+
|
195
|
+
#### Example topologies without gems
|
196
|
+
|
197
|
+
``` sh
|
198
|
+
$ redstorm local examples/simple/exclamation_topology.rb
|
199
|
+
$ redstorm local examples/simple/exclamation_topology2.rb
|
200
|
+
$ redstorm local examples/simple/word_count_topology.rb
|
201
|
+
```
|
202
|
+
|
203
|
+
#### Example topologies with gems
|
204
|
+
|
205
|
+
For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
|
206
|
+
|
207
|
+
1. create a `Gemfile`
|
208
|
+
|
209
|
+
``` ruby
|
210
|
+
source :rubygems
|
211
|
+
|
212
|
+
group :word_count do
|
213
|
+
gem "redis"
|
214
|
+
end
|
215
|
+
```
|
216
|
+
|
217
|
+
2. install the topology gems
|
218
|
+
|
219
|
+
``` sh
|
220
|
+
$ bundle install
|
221
|
+
$ redstorm bundle word_count
|
222
|
+
```
|
223
|
+
|
224
|
+
3. run the topology in local mode
|
225
|
+
|
226
|
+
``` sh
|
227
|
+
$ redstorm local examples/simple/redis_word_count_topology.rb
|
228
|
+
```
|
229
|
+
|
230
|
+
Using `redis-cli` push words into the `test` list and watch Storm pick them up
|
231
|
+
|
232
|
+
### Remote cluster
|
233
|
+
|
234
|
+
All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) can run in both local or on a remote cluster. The only **native** example compatible with a remote cluster is `examples/native/cluster_word_count_topology.rb`.
|
235
|
+
|
236
|
+
|
237
|
+
#### Topologies without gems
|
238
|
+
|
239
|
+
1. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
|
240
|
+
|
241
|
+
``` sh
|
242
|
+
$ redstorm jar examples
|
243
|
+
```
|
244
|
+
|
245
|
+
2. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
|
246
|
+
|
247
|
+
``` sh
|
248
|
+
$ redstorm cluster examples/simple/exclamation_topology.rb
|
249
|
+
$ redstorm cluster examples/simple/exclamation_topology2.rb
|
250
|
+
$ redstorm cluster examples/simple/word_count_topology.rb
|
251
|
+
```
|
252
|
+
|
253
|
+
|
254
|
+
#### Topologies with gems
|
255
|
+
|
256
|
+
For `examples/simple/redis_word_count_topology.rb` the `redis` gem is required and you need a [Redis](http://redis.io/) server running on `localhost:6379`
|
257
|
+
|
258
|
+
1. create a `Gemfile`
|
259
|
+
|
260
|
+
``` ruby
|
261
|
+
source :rubygems
|
262
|
+
|
263
|
+
group :word_count do
|
264
|
+
gem "redis"
|
265
|
+
end
|
266
|
+
```
|
267
|
+
|
268
|
+
2. install the topology gems
|
269
|
+
|
270
|
+
``` sh
|
271
|
+
$ bundle install
|
272
|
+
$ redstorm bundle word_count
|
273
|
+
```
|
274
|
+
|
275
|
+
3. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
|
276
|
+
|
277
|
+
``` sh
|
278
|
+
$ redstorm jar examples
|
279
|
+
```
|
280
|
+
|
281
|
+
4. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
|
282
|
+
|
283
|
+
``` sh
|
284
|
+
$ redstorm cluster examples/simple/redis_word_count_topology.rb
|
285
|
+
```
|
286
|
+
|
287
|
+
Using `redis-cli` push words into the `test` list and watch Storm pick them up
|
288
|
+
|
289
|
+
The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [setting up a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). You can also [manually submit your topology](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
|
290
|
+
|
291
|
+
## Ruby DSL
|
292
|
+
|
293
|
+
[Ruby DSL Documentation](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation)
|
294
|
+
|
295
|
+
## Multilang ShellSpout & ShellBolt support
|
296
|
+
|
297
|
+
Please refer to [Using non JVM languages with Storm](https://github.com/nathanmarz/storm/wiki/Using-non-JVM-languages-with-Storm) for the complete information on Multilang & shelling in Storm.
|
298
|
+
|
299
|
+
In RedStorm *ShellSpout* and *ShellBolt* are supported using the following construct in the topology definition:
|
300
|
+
|
301
|
+
``` ruby
|
302
|
+
bolt JRubyShellBolt, ["python", "splitsentence.py"] do
|
303
|
+
output_fields "word"
|
304
|
+
source SimpleSpout, :shuffle
|
305
|
+
end
|
306
|
+
```
|
307
|
+
|
308
|
+
- `JRubyShellBolt` must be used for a *ShellBolt* and the array argument `["python", "splitsentence.py"]` are the arguments to the class constructor and are the *commands* to the *ShellBolt*.
|
309
|
+
|
310
|
+
- The directory containing the topology class **must** contain a `resources/` directory with all the shell files.
|
311
|
+
|
312
|
+
See the [shell topology example](https://github.com/colinsurprenant/redstorm/tree/master/examples/shell)
|
313
|
+
|
314
|
+
## RedStorm Development
|
315
|
+
|
316
|
+
It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.
|
317
|
+
|
318
|
+
### Requirements
|
319
|
+
|
320
|
+
- JRuby 1.6.8
|
321
|
+
|
322
|
+
### Workflow
|
323
|
+
|
324
|
+
- fork project and create branch
|
325
|
+
|
326
|
+
- install RedStorm required gems
|
327
|
+
|
328
|
+
```sh
|
329
|
+
$ bundle install
|
330
|
+
```
|
331
|
+
|
332
|
+
- install dependencies in `target/dependencies`
|
333
|
+
|
334
|
+
```sh
|
335
|
+
$ bin/redstorm deps
|
336
|
+
```
|
337
|
+
|
338
|
+
- generate and build Java source into `target/classes`
|
339
|
+
|
340
|
+
```sh
|
341
|
+
$ bin/redstorm build
|
342
|
+
```
|
343
|
+
|
344
|
+
**if you modify any of the RedStorm Ruby code or Java binding code**, you need to run this to refresh code and rebuild the bindings
|
345
|
+
|
346
|
+
- follow the normal usage patterns to run the topology in local or remote cluster.
|
347
|
+
|
348
|
+
### How to Contribute
|
349
|
+
|
350
|
+
Fork the project, create a branch and submit a pull request.
|
351
|
+
|
352
|
+
Some ways you can contribute:
|
353
|
+
|
354
|
+
- by reporting bugs using the issue tracker
|
355
|
+
- by suggesting new features using the issue tracker
|
356
|
+
- by writing or editing documentation
|
357
|
+
- by writing specs
|
358
|
+
- by writing code
|
359
|
+
- by refactoring code
|
360
|
+
- ...
|
361
|
+
|
362
|
+
## Projects using RedStorm
|
363
|
+
|
364
|
+
If you want to list your RedStorm project here, contact me.
|
365
|
+
|
366
|
+
- [Tweigeist](https://github.com/colinsurprenant/tweitgeist) - realtime computation of the top trending hashtags on Twitter. See [Live Demo](http://tweitgeist.colinsurprenant.com/).
|
367
|
+
|
368
|
+
## Author
|
369
|
+
***Colin Surprenant***, [@colinsurprenant](http://twitter.com/colinsurprenant/), [http://github.com/colinsurprenant/](http://github.com/colinsurprenant/), colin.surprenant@gmail.com, [http://colinsurprenant.com/](http://colinsurprenant.com/)
|
370
|
+
|
371
|
+
## Contributors
|
372
|
+
Theo Hultberg, https://github.com/iconara
|
373
|
+
|
374
|
+
## License
|
375
|
+
Apache License, Version 2.0. See the LICENSE.md file.
|