redstorm 0.4.0 → 0.5.0
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG.md +11 -1
- data/README.md +121 -390
- data/Rakefile +7 -11
- data/TODO.md +0 -3
- data/examples/simple/Gemfile +2 -0
- data/examples/simple/exclamation_topology.rb +10 -4
- data/examples/simple/exclamation_topology2.rb +1 -0
- data/examples/simple/random_sentence_spout.rb +0 -1
- data/examples/simple/redis_word_count_topology.rb +5 -3
- data/examples/simple/ruby_version_topology.rb +32 -0
- data/examples/simple/word_count_topology.rb +1 -0
- data/lib/red_storm.rb +39 -1
- data/lib/red_storm/application.rb +19 -10
- data/lib/red_storm/configurator.rb +25 -0
- data/lib/red_storm/loggable.rb +13 -0
- data/lib/red_storm/proxy/bolt.rb +9 -0
- data/lib/red_storm/proxy/spout.rb +21 -6
- data/lib/red_storm/simple_bolt.rb +26 -3
- data/lib/red_storm/simple_spout.rb +50 -17
- data/lib/red_storm/simple_topology.rb +27 -24
- data/lib/red_storm/topology_launcher.rb +6 -2
- data/lib/red_storm/version.rb +1 -1
- data/lib/tasks/red_storm.rake +59 -29
- data/pom.xml +6 -5
- data/src/main/redstorm/storm/jruby/JRubyBolt.java +10 -0
- data/src/main/redstorm/storm/jruby/JRubySpout.java +19 -9
- metadata +12 -8
data/CHANGELOG.md
CHANGED
@@ -19,4 +19,14 @@
|
|
19
19
|
|
20
20
|
# 0.4.0, 02-08-2012
|
21
21
|
- Storm 0.6.2
|
22
|
-
- JRuby 1.6.6
|
22
|
+
- JRuby 1.6.6
|
23
|
+
|
24
|
+
# 0.5.0, 05-28-2012
|
25
|
+
- issue #16, Bundler support for topology gems
|
26
|
+
- issue #19, support for multiple dirs in jar
|
27
|
+
- issue #20, update to Storm 0.7.1
|
28
|
+
- issue #21, proper support for 1.8/1.9 Ruby compatibility mode
|
29
|
+
- issue #22, fixed Config class name clash
|
30
|
+
- JRuby 1.6.7
|
31
|
+
- DSL Support for per bolt/spout configuration and spout activate/deactivate in Storm 0.7.x
|
32
|
+
- consistent workflow using the redstorm command in local dev or gem mode
|
data/README.md
CHANGED
@@ -1,16 +1,47 @@
|
|
1
|
-
# RedStorm v0.
|
1
|
+
# RedStorm v0.5.0 - JRuby on Storm
|
2
2
|
|
3
|
-
|
3
|
+
[![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
|
4
|
+
|
5
|
+
RedStorm provides a Ruby DSL using JRuby integration for the [Storm][storm] distributed realtime computation system.
|
6
|
+
|
7
|
+
## Documentation
|
8
|
+
|
9
|
+
Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept in the wiki when necessary. This README reflects the current/master developement state.
|
10
|
+
|
11
|
+
### v0.4.x gem
|
12
|
+
|
13
|
+
- [RedStorm Gem v0.4.x Documentation](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.4.x-Documentation)
|
4
14
|
|
5
15
|
## Dependencies
|
6
16
|
|
7
|
-
|
17
|
+
Tested on OSX 10.6.8 and Linux 10.04 using Storm 0.6.2 and JRuby 1.6.7
|
18
|
+
|
19
|
+
## Notes about 1.8/1.9 JRuby compatibility
|
20
|
+
|
21
|
+
Up until the upcoming JRuby 1.7, JRuby runs in 1.8 Ruby compatibility mode by default. Unless you have a specific need to run topologies in 1.8 mode, you should use 1.9 mode, which will become the default in JRuby. Things are a bit tricky with Storm/RedStorm. There are 3 contexts where the Ruby compatibility mode has to be controlled.
|
22
|
+
|
23
|
+
- when installing the topology required gems. the installation path embeds the Ruby version
|
24
|
+
- when running in local mode or for the submission phase in remote/cluster mode
|
25
|
+
- when Storm runs the topology in remote/cluster mode
|
26
|
+
|
27
|
+
For each of these contexts, 1.9 mode has to be explicitly specified to avoid any problems. All commands/examples below will use the 1.9 compatibility mode. If you want to avoid the explicit --1.9 mode option, using [RVM][rvm] you can compile your JRuby to run in 1.9 mode by default. If you run your topology in remote/cluster mode, you will still need to include some bits of 1.9 options and configuration since in this case JRuby and your topology is run independently by Storm.
|
8
28
|
|
9
29
|
## Installation
|
30
|
+
|
31
|
+
### Released v0.4.0 gem
|
10
32
|
``` sh
|
11
33
|
$ gem install redstorm
|
12
34
|
```
|
13
35
|
|
36
|
+
### From github master
|
37
|
+
|
38
|
+
- clone/fork project
|
39
|
+
|
40
|
+
``` sh
|
41
|
+
$ gem build redstorm.gemspec
|
42
|
+
$ gem install redstorm-x.y.z.gem
|
43
|
+
```
|
44
|
+
|
14
45
|
## Usage overview
|
15
46
|
|
16
47
|
- create a new empty project directory.
|
@@ -21,34 +52,34 @@ $ gem install redstorm
|
|
21
52
|
|
22
53
|
### Initial setup
|
23
54
|
|
24
|
-
- install RedStom dependencies
|
55
|
+
- install RedStom dependencies. From your project root directory execute:
|
25
56
|
|
26
|
-
``` sh
|
27
|
-
$ redstorm install
|
28
|
-
```
|
57
|
+
``` sh
|
58
|
+
$ redstorm --1.9 install
|
59
|
+
```
|
29
60
|
|
30
|
-
The `install` command will install all Java jars dependencies using [ruby-maven][ruby-maven] in `target/dependency
|
61
|
+
The `install` command will install all Java jars dependencies using [ruby-maven][ruby-maven] in `target/dependency`, generate & compile the Java bindings in `target/classes` and install gems in `target/gems`.
|
31
62
|
|
32
|
-
***DON'T PANIC*** it's Maven. The first time you run `$ redstorm install` Maven will take a few minutes resolving dependencies and in the end will download and install the dependency jar files.
|
63
|
+
***DON'T PANIC*** it's Maven. The first time you run `$ redstorm --1.9 install` Maven will take a few minutes resolving dependencies and in the end will download and install the dependency jar files.
|
33
64
|
|
34
|
-
- create a topology class. The *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
|
65
|
+
- create a topology class in your sources subdirectory. The *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
|
35
66
|
|
36
|
-
### Gems
|
67
|
+
### Gems in your topology
|
37
68
|
|
38
|
-
|
69
|
+
RedStorm now support [Bundler](http://gembundler.com/) for using gems in your topology. Basically supply a `Gemfile` in the root of your project directory and execute this command to install the gems into the `target/gems` directory. **Note that if you change the Gemfile you must rerun this command**.
|
39
70
|
|
40
|
-
|
71
|
+
``` sh
|
72
|
+
$ redstorm --1.9 gems [--gemfile=GEMFILE]
|
73
|
+
```
|
41
74
|
|
42
|
-
|
75
|
+
All `bundle install` command options can be passed as options to `redstorm --1.9 gem` like `--gemfile=GEMFILE` to specify a Gemfile in an alternate path.
|
43
76
|
|
44
|
-
|
45
|
-
gem install <the gem> --install-dir target/gems/ --no-ri --no-rdoc
|
46
|
-
```
|
77
|
+
Basically, the `redstorm --1.9 gems` command installs the *Bundler* and *Rake* gems and all the gems specified in the Gemfile into the `target/gems` directory. The idea is that in order for the topology to run in a Storm cluster, everything, including the fully *installed* gems, must be packaged and self-contained into a single JAR file. This has an important consequence: the gems will not be *installed* on the cluster target machines, they are already *installed* in the JAR file. This could possibly lead to problems if the machine used to *install* the gems is of a different architecture than the cluster target machines **and** some of these gems have *native* C/FFI extensions.
|
47
78
|
|
48
79
|
### Run in local mode
|
49
80
|
|
50
81
|
``` sh
|
51
|
-
$ redstorm local <path/to/topology_class_file_name.rb>
|
82
|
+
$ redstorm --1.9 local <path/to/topology_class_file_name.rb>
|
52
83
|
```
|
53
84
|
|
54
85
|
**See examples below** to run examples in local mode or on a production cluster.
|
@@ -57,15 +88,17 @@ $ redstorm local <path/to/topology_class_file_name.rb>
|
|
57
88
|
|
58
89
|
- generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies from the `target/` directory:
|
59
90
|
|
60
|
-
``` sh
|
61
|
-
$ redstorm jar <
|
62
|
-
```
|
91
|
+
``` sh
|
92
|
+
$ redstorm --1.9 jar <sources_directory1> <sources_directory2> ...
|
93
|
+
```
|
63
94
|
|
64
95
|
- submit the cluster topology jar file to the cluster. Assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
|
65
96
|
|
66
|
-
``` sh
|
67
|
-
storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher cluster <path/to/topology_class_file_name.rb>
|
68
|
-
```
|
97
|
+
``` sh
|
98
|
+
storm jar ./target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster <path/to/topology_class_file_name.rb>
|
99
|
+
```
|
100
|
+
|
101
|
+
Note the **-Djruby.compat.version=RUBY1_9** parameter.
|
69
102
|
|
70
103
|
Basically you must follow the [Storm instructions](https://github.com/nathanmarz/storm/wiki) to [setup a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster) and [submit your topology to the cluster](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
|
71
104
|
|
@@ -77,391 +110,119 @@ Install the [example files](https://github.com/colinsurprenant/redstorm/tree/mas
|
|
77
110
|
$ redstorm examples
|
78
111
|
```
|
79
112
|
|
80
|
-
All examples using the
|
113
|
+
All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) are located in `examples/simple`. Examples using the standard Java interface are in `examples/native`.
|
81
114
|
|
82
115
|
### Local mode
|
83
116
|
|
84
117
|
``` sh
|
85
|
-
$ redstorm local examples/simple/exclamation_topology.rb
|
86
|
-
$ redstorm local examples/simple/exclamation_topology2.rb
|
87
|
-
$ redstorm local examples/simple/word_count_topology.rb
|
118
|
+
$ redstorm --1.9 local examples/simple/exclamation_topology.rb
|
119
|
+
$ redstorm --1.9 local examples/simple/exclamation_topology2.rb
|
120
|
+
$ redstorm --1.9 local examples/simple/word_count_topology.rb
|
121
|
+
```
|
122
|
+
|
123
|
+
To run `examples/simple/redis_word_count_topology.rb` you need a [Redis][redis] server running on `localhost:6379`
|
124
|
+
|
125
|
+
``` sh
|
126
|
+
$ redstorm --1.9 gems --gemfile examples/simple/Gemfile
|
88
127
|
```
|
89
128
|
|
90
|
-
|
129
|
+
Run the topology in local mode
|
91
130
|
|
92
131
|
``` sh
|
93
|
-
$ redstorm local examples/simple/redis_word_count_topology.rb
|
132
|
+
$ redstorm --1.9 local examples/simple/redis_word_count_topology.rb
|
94
133
|
```
|
95
134
|
|
96
135
|
Using `redis-cli`, push words into the `test` list and watch Storm pick them up
|
97
136
|
|
98
137
|
### Production cluster
|
99
138
|
|
100
|
-
All examples using the
|
139
|
+
All examples using the [simple DSL](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation) can also run on a productions cluster. The only **native** example compatible with a production cluster is the [ClusterWordCountTopology](https://github.com/colinsurprenant/redstorm/tree/master/examples/native/cluster_word_count_topology.rb)
|
101
140
|
|
102
141
|
- genererate the `target/cluster-topology.jar` and include the `examples/` directory.
|
103
142
|
|
104
|
-
``` sh
|
105
|
-
$ redstorm jar examples
|
106
|
-
```
|
143
|
+
``` sh
|
144
|
+
$ redstorm --1.9 jar examples
|
145
|
+
```
|
107
146
|
|
108
147
|
- submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
|
109
148
|
|
110
|
-
``` sh
|
111
|
-
$ storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher cluster examples/simple/word_count_topology.rb
|
112
|
-
```
|
113
|
-
|
114
|
-
- to run `examples/simple/redis_word_count_topology.rb` you need a [Redis][redis] server running on `localhost:6379` and the Redis gem in `target/gems` using:
|
115
|
-
|
116
|
-
```sh
|
117
|
-
gem install redis --install-dir target/gems/ --no-ri --no-rdoc
|
118
|
-
```
|
149
|
+
``` sh
|
150
|
+
$ storm jar ./target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster examples/simple/word_count_topology.rb
|
151
|
+
```
|
119
152
|
|
120
|
-
|
153
|
+
Note the **-Djruby.compat.version=RUBY1_9** parameter.
|
121
154
|
|
122
|
-
|
123
|
-
$ redstorm jar examples
|
124
|
-
$ storm jar ./target/cluster-topology.jar redstorm.TopologyLauncher cluster examples/simple/redis_word_count_topology.rb
|
125
|
-
```
|
155
|
+
- to run `examples/simple/redis_word_count_topology.rb` you need a [Redis][redis] server running on `localhost:6379`
|
126
156
|
|
127
|
-
|
157
|
+
``` sh
|
158
|
+
$ redstorm --1.9 gems --gemfile examples/simple/Gemfile
|
159
|
+
$ redstorm --1.9 jar examples
|
160
|
+
$ storm jar ./target/cluster-topology.jar -Djruby.compat.version=RUBY1_9 redstorm.TopologyLauncher cluster examples/simple/redis_word_count_topology.rb
|
161
|
+
```
|
128
162
|
|
163
|
+
- using `redis-cli`, push words into the `test` list and watch Storm pick them up
|
129
164
|
|
130
165
|
Basically you must follow the [Storm instructions](https://github.com/nathanmarz/storm/wiki) to [setup a production cluster](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster) and [submit your topology to the cluster](https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster).
|
131
166
|
|
132
|
-
## DSL
|
133
|
-
|
134
|
-
Your project can be created in a single file containing all spouts, bolts and topology classes or each classes can be in its own file, your choice. There are [many examples](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple) for the *simple* DSL.
|
135
|
-
|
136
|
-
The DSL uses a **callback metaphor** to attach code to the topology/spout/bolt execution contexts using `on_*` DSL constructs (ex.: on_submit, on_send, ...). When using `on_*` you can attach you code in 3 different ways:
|
137
|
-
|
138
|
-
- using a code block
|
139
|
-
|
140
|
-
```ruby
|
141
|
-
on_receive (:ack => true, :anchor => true) {|tuple| do_something_with(tuple)}
|
142
|
-
|
143
|
-
on_receive :ack => true, :anchor => true do |tuple|
|
144
|
-
do_something_with(tuple)
|
145
|
-
end
|
146
|
-
```
|
147
|
-
|
148
|
-
- defining the corresponding method
|
149
|
-
|
150
|
-
```ruby
|
151
|
-
on_receive :ack => true, :anchor => true
|
152
|
-
def on_receive(tuple)
|
153
|
-
do_something_with(tuple)
|
154
|
-
end
|
155
|
-
```
|
156
|
-
|
157
|
-
- defining an arbitrary method
|
158
|
-
|
159
|
-
```ruby
|
160
|
-
on_receive :my_method, :ack => true, :anchor => true
|
161
|
-
def my_method(tuple)
|
162
|
-
do_something_with(tuple)
|
163
|
-
end
|
164
|
-
```
|
165
|
-
|
166
|
-
The [example SplitSentenceBolt](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/split_sentence_bolt.rb) shows the 3 different coding style.
|
167
|
-
|
168
|
-
### Topology DSL
|
169
|
-
|
170
|
-
Normally Storm topology components are assigned and referenced using numeric ids. In the SimpleTopology DSL **ids are optional**. By default the DSL will use the component class name as an implicit symbolic id and bolt source ids can use these implicit ids. The DSL will automatically resolve and assign numeric ids upon topology submission. If two components are of the same class, creating a conflict, then the id can be explicitly defined using either a numeric value, a symbol or a string. Numeric values will be used as-is at topology submission while symbols and strings will be resolved and assigned a numeric id.
|
171
|
-
|
172
|
-
```ruby
|
173
|
-
require 'red_storm'
|
174
|
-
|
175
|
-
class MyTopology < RedStorm::SimpleTopology
|
176
|
-
|
177
|
-
spout spout_class, options
|
178
|
-
|
179
|
-
bolt bolt_class, options do
|
180
|
-
source source_id, grouping
|
181
|
-
...
|
182
|
-
end
|
183
|
-
|
184
|
-
configure topology_name do |env|
|
185
|
-
config_attribute value
|
186
|
-
...
|
187
|
-
end
|
188
|
-
|
189
|
-
on_submit do |env|
|
190
|
-
...
|
191
|
-
end
|
192
|
-
end
|
193
|
-
```
|
194
|
-
|
195
|
-
#### spout statement
|
196
|
-
|
197
|
-
```ruby
|
198
|
-
spout spout_class, options
|
199
|
-
```
|
200
|
-
|
201
|
-
- `spout_class` — spout Ruby class
|
202
|
-
- `options`
|
203
|
-
- `:id` — spout explicit id (**default** is spout class name)
|
204
|
-
- `:parallelism` — spout parallelism (**default** is 1)
|
205
|
-
|
206
|
-
#### bolt statement
|
207
|
-
|
208
|
-
```ruby
|
209
|
-
bolt bolt_class, options do
|
210
|
-
source source_id, grouping
|
211
|
-
...
|
212
|
-
end
|
213
|
-
```
|
214
|
-
|
215
|
-
- `bolt_class` — bolt Ruby class
|
216
|
-
- `options`
|
217
|
-
- `:id` — bolt explicit id (**default** is bolt class name)
|
218
|
-
- `:parallelism` — bolt parallelism (**default** is 1)
|
219
|
-
- `source_id` — source id reference. can be the source class name if unique or the explicit id if defined
|
220
|
-
- `grouping`
|
221
|
-
- `:fields => ["field", ...]` — fieldsGrouping using fields on the source_id
|
222
|
-
- `:shuffle` — shuffleGrouping on the source_id
|
223
|
-
- `:global` — globalGrouping on the source_id
|
224
|
-
- `:none` — noneGrouping on the source_id
|
225
|
-
- `:all` — allGrouping on the source_id
|
226
|
-
- `:direct` — directGrouping on the source_id
|
227
|
-
|
228
|
-
#### configure statement
|
229
|
-
|
230
|
-
```ruby
|
231
|
-
configure topology_name do |env|
|
232
|
-
configuration_field value
|
233
|
-
...
|
234
|
-
end
|
235
|
-
```
|
236
|
-
|
237
|
-
The `configure` statement is **required**.
|
238
|
-
|
239
|
-
- `topology_name` — alternate topology name (**default** is topology class name)
|
240
|
-
- `env` — is set to `:local` or `:cluster` for you to set enviroment specific configurations
|
241
|
-
- `config_attribute` — the Storm Config attribute name. See Storm for complete list. The attribute name correspond to the Java setter method, without the "set" prefix and the suffix converted from CamelCase to underscore. Ex.: `setMaxTaskParallelism` is `:max_task_parallelism`.
|
242
|
-
- `:debug`
|
243
|
-
- `:max_task_parallelism`
|
244
|
-
- `:num_workers`
|
245
|
-
- `:max_spout_pending`
|
246
|
-
- ...
|
247
|
-
|
248
|
-
#### on_submit statement
|
249
|
-
|
250
|
-
```ruby
|
251
|
-
on_submit do |env|
|
252
|
-
...
|
253
|
-
end
|
254
|
-
```
|
255
|
-
|
256
|
-
The `on_submit` statement is **optional**. Use it to execute code after the topology submission.
|
257
|
-
|
258
|
-
- `env` — is set to `:local` or `:cluster`
|
259
|
-
|
260
|
-
For example, you can use `on_submit` to shutdown the LocalCluster after some time. The LocalCluster instance is available usign the `cluster` method.
|
261
|
-
|
262
|
-
```ruby
|
263
|
-
on_submit do |env|
|
264
|
-
if env == :local
|
265
|
-
sleep(5)
|
266
|
-
cluster.shutdown
|
267
|
-
end
|
268
|
-
end
|
269
|
-
```
|
270
|
-
|
271
|
-
#### Examples
|
167
|
+
## Ruby DSL
|
272
168
|
|
273
|
-
|
274
|
-
- [ExclamationTopology2](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/exclamation_topology2.rb)
|
275
|
-
- [WordCountTopology](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/word_count_topology.rb)
|
276
|
-
- [RedisWordCountTopology](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/redis_word_count_topology.rb)
|
169
|
+
[Ruby DSL Documentation](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation)
|
277
170
|
|
278
|
-
|
171
|
+
## RedStorm Development
|
279
172
|
|
280
|
-
|
281
|
-
require 'red_storm'
|
173
|
+
It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.
|
282
174
|
|
283
|
-
|
284
|
-
set spout_attribute => value
|
285
|
-
...
|
286
|
-
|
287
|
-
output_fields :field, ...
|
288
|
-
|
289
|
-
on_send options do
|
290
|
-
...
|
291
|
-
end
|
292
|
-
|
293
|
-
on_init do
|
294
|
-
...
|
295
|
-
end
|
296
|
-
|
297
|
-
on_close do
|
298
|
-
...
|
299
|
-
end
|
300
|
-
|
301
|
-
on_ack do |msg_id|
|
302
|
-
...
|
303
|
-
end
|
304
|
-
|
305
|
-
on_fail do |msg_id|
|
306
|
-
...
|
307
|
-
end
|
308
|
-
end
|
309
|
-
```
|
310
|
-
|
311
|
-
#### set statement
|
312
|
-
|
313
|
-
```ruby
|
314
|
-
set spout_attribute => value
|
315
|
-
```
|
316
|
-
|
317
|
-
The `set` statement is **optional**. Use it to set spout specific attributes.
|
318
|
-
|
319
|
-
- `spout_attributes`
|
320
|
-
- `:is_distributed` — set to `true` for a distributed spout (**default** is `false`)
|
321
|
-
|
322
|
-
#### output_fields statement
|
323
|
-
|
324
|
-
```ruby
|
325
|
-
output_fields :field, ...
|
326
|
-
```
|
327
|
-
|
328
|
-
Define the output fields for this spout.
|
329
|
-
|
330
|
-
- `:field` — the field name, can be symbol or string.
|
331
|
-
|
332
|
-
#### on_send statement
|
333
|
-
|
334
|
-
```ruby
|
335
|
-
on_send options do
|
336
|
-
...
|
337
|
-
end
|
338
|
-
```
|
339
|
-
|
340
|
-
`on_send` relates to the Java spout `nextTuple` method and is called periodically by storm to allow the spout to output a tuple. When using auto-emit (default), the block return value will be auto emited. A single value return will be emited as a single-field tuple. An array of values `[a, b]` will be emited as a multiple-fields tuple. Normally a spout [should only output a single tuple per on_send invocation](https://groups.google.com/forum/#!topic/storm-user/SGwih7vPiDE/discussion).
|
341
|
-
|
342
|
-
- `:options`
|
343
|
-
- `:emit` — set to `false` to disable auto-emit (**default** is `true`)
|
344
|
-
|
345
|
-
#### on_init statement
|
346
|
-
|
347
|
-
```ruby
|
348
|
-
on_init do
|
349
|
-
...
|
350
|
-
end
|
351
|
-
```
|
352
|
-
|
353
|
-
`on_init` relates to the Java spout `open` method. When `on_init` is called, the `config`, `context` and `collector` are set to return the Java spout config `Map`, `TopologyContext` and `SpoutOutputCollector`.
|
354
|
-
|
355
|
-
#### on_close statement
|
356
|
-
|
357
|
-
```ruby
|
358
|
-
on_close do
|
359
|
-
...
|
360
|
-
end
|
361
|
-
```
|
362
|
-
|
363
|
-
`on_close` relates to the Java spout `close` method.
|
364
|
-
|
365
|
-
#### on_ack statement
|
366
|
-
|
367
|
-
```ruby
|
368
|
-
on_ack do |msg_id|
|
369
|
-
...
|
370
|
-
end
|
371
|
-
```
|
372
|
-
|
373
|
-
`on_ack` relates to the Java spout `ack` method.
|
374
|
-
|
375
|
-
#### on_fail statement
|
376
|
-
|
377
|
-
```ruby
|
378
|
-
on_fail do |msg_id|
|
379
|
-
...
|
380
|
-
end
|
381
|
-
```
|
382
|
-
|
383
|
-
`on_fail` relates to the Java spout `fail` method.
|
384
|
-
|
385
|
-
#### Examples
|
386
|
-
|
387
|
-
- [RandomSentenceSpout](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/random_sentence_spout.rb)
|
388
|
-
- [RedisWordSpout](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/redis_word_count_topology.rb)
|
389
|
-
|
390
|
-
### Bolt DSL
|
391
|
-
|
392
|
-
```ruby
|
393
|
-
require 'red_storm'
|
394
|
-
|
395
|
-
class MyBolt < RedStorm::SimpleBolt
|
396
|
-
output_fields :field, ...
|
397
|
-
|
398
|
-
on_receive options do
|
399
|
-
...
|
400
|
-
end
|
175
|
+
### Requirements
|
401
176
|
|
402
|
-
|
403
|
-
...
|
404
|
-
end
|
177
|
+
- JRuby 1.6.7
|
405
178
|
|
406
|
-
|
407
|
-
...
|
408
|
-
end
|
409
|
-
end
|
410
|
-
```
|
179
|
+
### Workflow
|
411
180
|
|
412
|
-
|
181
|
+
- fork project and create branch
|
182
|
+
- install required gems
|
413
183
|
|
414
|
-
```
|
415
|
-
|
416
|
-
|
417
|
-
end
|
418
|
-
```
|
184
|
+
```sh
|
185
|
+
$ jruby --1.9 -S bundle install
|
186
|
+
```
|
419
187
|
|
420
|
-
|
188
|
+
- install dependencies in `target/dependencies`
|
421
189
|
|
422
|
-
|
190
|
+
```sh
|
191
|
+
$ bin/redstorm --1.9 deps
|
192
|
+
```
|
423
193
|
|
424
|
-
-
|
425
|
-
- `:emit` — set to `false` to disable auto-emit (**default** is `true`)
|
426
|
-
- `:ack` — set to `true` to enable auto-ack (**default** is `false`)
|
427
|
-
- `:anchor` — set to `true` to enable auto-anchor (**default** is `false`)
|
194
|
+
- generate and build Java source into `target/classes`
|
428
195
|
|
429
|
-
|
196
|
+
```sh
|
197
|
+
$ bin/redstorm --1.9 build
|
198
|
+
```
|
430
199
|
|
431
|
-
|
432
|
-
on_init do
|
433
|
-
...
|
434
|
-
end
|
435
|
-
```
|
200
|
+
if you modify any of the Java binding code, you need to run this to rebuild the bindings
|
436
201
|
|
437
|
-
|
202
|
+
- run topology in **local** Storm mode
|
438
203
|
|
439
|
-
|
204
|
+
```sh
|
205
|
+
$ bin/redstorm --1.9 local path/to/topology_class.rb
|
206
|
+
```
|
440
207
|
|
441
|
-
|
442
|
-
on_close do
|
443
|
-
...
|
444
|
-
end
|
445
|
-
```
|
208
|
+
If you only make changes to your topology code, this is the only step you need to repeat to try your updated code.
|
446
209
|
|
447
|
-
|
210
|
+
- generate remote cluster topology jar into `target/cluster-topology.jar`, including the `mydir/` directory.
|
448
211
|
|
449
|
-
|
212
|
+
```sh
|
213
|
+
$ bin/redstorm --1.9 jar mydir otherdir1 otherdir2 ...
|
214
|
+
```
|
450
215
|
|
451
|
-
-
|
452
|
-
- [SplitSentenceBolt](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/split_sentence_bolt.rb)
|
453
|
-
- [WordCountBolt](https://github.com/colinsurprenant/redstorm/tree/master/examples/simple/word_count_bolt.rb)
|
216
|
+
- **if you add/change** Gemfile for your topology, install gems in `target/gems`. Alternate gemfile path can be specified using --gemfile=GEMFILE
|
454
217
|
|
455
|
-
|
218
|
+
```sh
|
219
|
+
$ bin/redstorm --1.9 gems [--gemfile=GEMFILE]
|
220
|
+
```
|
456
221
|
|
457
|
-
|
222
|
+
**do not forget** to rerurn `bin/redstorm --1.9 jar ...` to pick up these gems, before submitting your topology on a remote cluster.
|
458
223
|
|
459
|
-
- JRuby 1.6.6
|
460
|
-
- rake gem ~> 0.9.2.2
|
461
|
-
- ruby-maven gem ~> 3.0.3.0.28.5
|
462
|
-
- rspec gem ~> 2.8.0
|
463
224
|
|
464
|
-
### Contribute
|
225
|
+
### How to Contribute
|
465
226
|
|
466
227
|
Fork the project, create a branch and submit a pull request.
|
467
228
|
|
@@ -475,42 +236,12 @@ Some ways you can contribute:
|
|
475
236
|
- by refactoring code
|
476
237
|
- ...
|
477
238
|
|
478
|
-
### Workflow
|
479
|
-
|
480
|
-
- fork project
|
481
|
-
- create branch
|
482
|
-
- install dependencies in `target/dependencies`
|
483
|
-
|
484
|
-
```sh
|
485
|
-
$ rake deps
|
486
|
-
```
|
487
|
-
|
488
|
-
- generate and build Java source into `target/classes`
|
489
|
-
|
490
|
-
```sh
|
491
|
-
$ rake build
|
492
|
-
```
|
493
|
-
|
494
|
-
- run topology in local dev cluster
|
495
|
-
|
496
|
-
```sh
|
497
|
-
$ bin/redstorm local path/to/topology_class.rb
|
498
|
-
```
|
499
|
-
|
500
|
-
- generate remote cluster topology jar into `target/cluster-topology.jar`, including the `examples/` directory.
|
501
|
-
|
502
|
-
```sh
|
503
|
-
$ rake jar['examples']
|
504
|
-
```
|
505
|
-
|
506
239
|
## Author
|
507
|
-
Colin Surprenant, [@colinsurprenant][twitter], [
|
240
|
+
Colin Surprenant, [@colinsurprenant][twitter], [http://github.com/colinsurprenant][github], colin.surprenant@gmail.com, colin.surprenant@needium.com
|
508
241
|
|
509
242
|
## License
|
510
243
|
Apache License, Version 2.0. See the LICENSE.md file.
|
511
244
|
|
512
|
-
[needium]: colin.surprenant@needium.com
|
513
|
-
[gmail]: colin.surprenant@gmail.com
|
514
245
|
[twitter]: http://twitter.com/colinsurprenant
|
515
246
|
[github]: http://github.com/colinsurprenant
|
516
247
|
[rvm]: http://beginrescueend.com/
|