redstorm 0.6.2 → 0.6.3
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG.md +7 -1
- data/README.md +23 -4
- data/examples/shell/resources/splitsentence.py +9 -0
- data/examples/shell/resources/storm.py +206 -0
- data/examples/shell/shell_topology.rb +41 -0
- data/lib/red_storm/application.rb +8 -1
- data/lib/red_storm/simple_topology.rb +41 -9
- data/lib/red_storm/version.rb +1 -1
- data/lib/tasks/red_storm.rake +17 -3
- data/src/main/redstorm/storm/jruby/JRubyBolt.java +12 -4
- data/src/main/redstorm/storm/jruby/JRubyShellBolt.java +26 -0
- data/src/main/redstorm/storm/jruby/JRubyShellSpout.java +26 -0
- data/src/main/redstorm/storm/jruby/JRubySpout.java +11 -4
- metadata +7 -2
data/CHANGELOG.md
CHANGED
@@ -56,4 +56,10 @@
|
|
56
56
|
# 0.6.2, 07-10-2012
|
57
57
|
- issue #39, spout on_receive block will not be evaluated if :emit => false
|
58
58
|
- issue #40, bolt fail method missing
|
59
|
-
- integration tests support
|
59
|
+
- integration tests support
|
60
|
+
|
61
|
+
# 0.6.3, 10-09-2012
|
62
|
+
- issue #28, allow specification of output_fields in topology DSL
|
63
|
+
- issue #41, add support for ShellBolt and ShellSpout
|
64
|
+
- issue #49, redstorm bundle not picking up default group in Gemfile
|
65
|
+
- support constructor parameters for Java spout/bolt in topology DSL
|
data/README.md
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# RedStorm v0.6.
|
1
|
+
# RedStorm v0.6.3 - JRuby on Storm
|
2
2
|
|
3
3
|
[![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
|
4
4
|
|
@@ -58,7 +58,7 @@ $ redstorm local|cluster [--1.8|--1.9] ...
|
|
58
58
|
|
59
59
|
``` ruby
|
60
60
|
source :rubygems
|
61
|
-
gem "redstorm", "~> 0.6.
|
61
|
+
gem "redstorm", "~> 0.6.3"
|
62
62
|
```
|
63
63
|
|
64
64
|
## Usage overview
|
@@ -255,6 +255,25 @@ The [Storm wiki](https://github.com/nathanmarz/storm/wiki) has instructions on [
|
|
255
255
|
|
256
256
|
[Ruby DSL Documentation](https://github.com/colinsurprenant/redstorm/wiki/Ruby-DSL-Documentation)
|
257
257
|
|
258
|
+
## Multilang ShellSpout & ShellBolt support
|
259
|
+
|
260
|
+
Please refer to [Using non JVM languages with Storm](https://github.com/nathanmarz/storm/wiki/Using-non-JVM-languages-with-Storm) for the complete information on Multilang & shelling in Storm.
|
261
|
+
|
262
|
+
In RedStorm *ShellSpout* and *ShellBolt* are supported using the following construct in the topology definition:
|
263
|
+
|
264
|
+
``` ruby
|
265
|
+
bolt JRubyShellBolt, ["python", "splitsentence.py"] do
|
266
|
+
output_fields "word"
|
267
|
+
source SimpleSpout, :shuffle
|
268
|
+
end
|
269
|
+
```
|
270
|
+
|
271
|
+
- `JRubyShellBolt` must be used for a *ShellBolt* and the array argument `["python", "splitsentence.py"]` are the arguments to the class constructor and are the *commands* to the *ShellBolt*.
|
272
|
+
|
273
|
+
- The directory containing the topology class **must** contain a `resources/` directory with all the shell files.
|
274
|
+
|
275
|
+
See the [shell topology example](https://github.com/colinsurprenant/redstorm/tree/master/examples/shell)
|
276
|
+
|
258
277
|
## RedStorm Development
|
259
278
|
|
260
279
|
It is possible to fork the RedStorm project and run local and remote/cluster topologies directly from the project sources without installing the gem. This is a useful setup when contributing to the project.
|
@@ -307,10 +326,10 @@ Some ways you can contribute:
|
|
307
326
|
|
308
327
|
If you want to list your RedStorm project here, contact me.
|
309
328
|
|
310
|
-
- [Tweigeist](https://github.com/colinsurprenant/tweitgeist) - realtime computation of the top trending hashtags on Twitter.
|
329
|
+
- [Tweigeist](https://github.com/colinsurprenant/tweitgeist) - realtime computation of the top trending hashtags on Twitter. Live Demo in search of a new home.
|
311
330
|
|
312
331
|
## Author
|
313
|
-
Colin Surprenant, [@colinsurprenant][twitter], [http://github.com/colinsurprenant][github], colin.surprenant@gmail.com
|
332
|
+
Colin Surprenant, [@colinsurprenant][twitter], [http://github.com/colinsurprenant][github], colin.surprenant@gmail.com
|
314
333
|
|
315
334
|
## Contributors
|
316
335
|
Theo Hultberg, https://github.com/iconara
|
@@ -0,0 +1,206 @@
|
|
1
|
+
import sys
|
2
|
+
import os
|
3
|
+
import traceback
|
4
|
+
from collections import deque
|
5
|
+
|
6
|
+
try:
|
7
|
+
import simplejson as json
|
8
|
+
except ImportError:
|
9
|
+
import json
|
10
|
+
|
11
|
+
json_encode = lambda x: json.dumps(x)
|
12
|
+
json_decode = lambda x: json.loads(x)
|
13
|
+
|
14
|
+
#reads lines and reconstructs newlines appropriately
|
15
|
+
def readMsg():
|
16
|
+
msg = ""
|
17
|
+
while True:
|
18
|
+
line = sys.stdin.readline()[0:-1]
|
19
|
+
if line == "end":
|
20
|
+
break
|
21
|
+
msg = msg + line + "\n"
|
22
|
+
return json_decode(msg[0:-1])
|
23
|
+
|
24
|
+
MODE = None
|
25
|
+
ANCHOR_TUPLE = None
|
26
|
+
|
27
|
+
#queue up commands we read while trying to read taskids
|
28
|
+
pending_commands = deque()
|
29
|
+
|
30
|
+
def readTaskIds():
|
31
|
+
if pending_taskids:
|
32
|
+
return pending_taskids.popleft()
|
33
|
+
else:
|
34
|
+
msg = readMsg()
|
35
|
+
while type(msg) is not list:
|
36
|
+
pending_commands.append(msg)
|
37
|
+
msg = readMsg()
|
38
|
+
return msg
|
39
|
+
|
40
|
+
#queue up taskids we read while trying to read commands/tuples
|
41
|
+
pending_taskids = deque()
|
42
|
+
|
43
|
+
def readCommand():
|
44
|
+
if pending_commands:
|
45
|
+
return pending_commands.popleft()
|
46
|
+
else:
|
47
|
+
msg = readMsg()
|
48
|
+
while type(msg) is list:
|
49
|
+
pending_taskids.append(msg)
|
50
|
+
msg = readMsg()
|
51
|
+
return msg
|
52
|
+
|
53
|
+
def readTuple():
|
54
|
+
cmd = readCommand()
|
55
|
+
return Tuple(cmd["id"], cmd["comp"], cmd["stream"], cmd["task"], cmd["tuple"])
|
56
|
+
|
57
|
+
def sendMsgToParent(msg):
|
58
|
+
print json_encode(msg)
|
59
|
+
print "end"
|
60
|
+
sys.stdout.flush()
|
61
|
+
|
62
|
+
def sync():
|
63
|
+
sendMsgToParent({'command':'sync'})
|
64
|
+
|
65
|
+
def sendpid(heartbeatdir):
|
66
|
+
pid = os.getpid()
|
67
|
+
sendMsgToParent({'pid':pid})
|
68
|
+
open(heartbeatdir + "/" + str(pid), "w").close()
|
69
|
+
|
70
|
+
def emit(*args, **kwargs):
|
71
|
+
__emit(*args, **kwargs)
|
72
|
+
return readTaskIds()
|
73
|
+
|
74
|
+
def emitDirect(task, *args, **kwargs):
|
75
|
+
kwargs[directTask] = task
|
76
|
+
__emit(*args, **kwargs)
|
77
|
+
|
78
|
+
def __emit(*args, **kwargs):
|
79
|
+
global MODE
|
80
|
+
if MODE == Bolt:
|
81
|
+
emitBolt(*args, **kwargs)
|
82
|
+
elif MODE == Spout:
|
83
|
+
emitSpout(*args, **kwargs)
|
84
|
+
|
85
|
+
def emitBolt(tup, stream=None, anchors = [], directTask=None):
|
86
|
+
global ANCHOR_TUPLE
|
87
|
+
if ANCHOR_TUPLE is not None:
|
88
|
+
anchors = [ANCHOR_TUPLE]
|
89
|
+
m = {"command": "emit"}
|
90
|
+
if stream is not None:
|
91
|
+
m["stream"] = stream
|
92
|
+
m["anchors"] = map(lambda a: a.id, anchors)
|
93
|
+
if directTask is not None:
|
94
|
+
m["task"] = directTask
|
95
|
+
m["tuple"] = tup
|
96
|
+
sendMsgToParent(m)
|
97
|
+
|
98
|
+
def emitSpout(tup, stream=None, id=None, directTask=None):
|
99
|
+
m = {"command": "emit"}
|
100
|
+
if id is not None:
|
101
|
+
m["id"] = id
|
102
|
+
if stream is not None:
|
103
|
+
m["stream"] = stream
|
104
|
+
if directTask is not None:
|
105
|
+
m["task"] = directTask
|
106
|
+
m["tuple"] = tup
|
107
|
+
sendMsgToParent(m)
|
108
|
+
|
109
|
+
def ack(tup):
|
110
|
+
sendMsgToParent({"command": "ack", "id": tup.id})
|
111
|
+
|
112
|
+
def fail(tup):
|
113
|
+
sendMsgToParent({"command": "fail", "id": tup.id})
|
114
|
+
|
115
|
+
def log(msg):
|
116
|
+
sendMsgToParent({"command": "log", "msg": msg})
|
117
|
+
|
118
|
+
def initComponent():
|
119
|
+
setupInfo = readMsg()
|
120
|
+
sendpid(setupInfo['pidDir'])
|
121
|
+
return [setupInfo['conf'], setupInfo['context']]
|
122
|
+
|
123
|
+
class Tuple:
|
124
|
+
def __init__(self, id, component, stream, task, values):
|
125
|
+
self.id = id
|
126
|
+
self.component = component
|
127
|
+
self.stream = stream
|
128
|
+
self.task = task
|
129
|
+
self.values = values
|
130
|
+
|
131
|
+
def __repr__(self):
|
132
|
+
return '<%s%s>' % (
|
133
|
+
self.__class__.__name__,
|
134
|
+
''.join(' %s=%r' % (k, self.__dict__[k]) for k in sorted(self.__dict__.keys())))
|
135
|
+
|
136
|
+
class Bolt:
|
137
|
+
def initialize(self, stormconf, context):
|
138
|
+
pass
|
139
|
+
|
140
|
+
def process(self, tuple):
|
141
|
+
pass
|
142
|
+
|
143
|
+
def run(self):
|
144
|
+
global MODE
|
145
|
+
MODE = Bolt
|
146
|
+
conf, context = initComponent()
|
147
|
+
self.initialize(conf, context)
|
148
|
+
try:
|
149
|
+
while True:
|
150
|
+
tup = readTuple()
|
151
|
+
self.process(tup)
|
152
|
+
except Exception, e:
|
153
|
+
log(traceback.format_exc(e))
|
154
|
+
|
155
|
+
class BasicBolt:
|
156
|
+
def initialize(self, stormconf, context):
|
157
|
+
pass
|
158
|
+
|
159
|
+
def process(self, tuple):
|
160
|
+
pass
|
161
|
+
|
162
|
+
def run(self):
|
163
|
+
global MODE
|
164
|
+
MODE = Bolt
|
165
|
+
global ANCHOR_TUPLE
|
166
|
+
conf, context = initComponent()
|
167
|
+
self.initialize(conf, context)
|
168
|
+
try:
|
169
|
+
while True:
|
170
|
+
tup = readTuple()
|
171
|
+
ANCHOR_TUPLE = tup
|
172
|
+
self.process(tup)
|
173
|
+
ack(tup)
|
174
|
+
except Exception, e:
|
175
|
+
log(traceback.format_exc(e))
|
176
|
+
|
177
|
+
class Spout:
|
178
|
+
def initialize(self, conf, context):
|
179
|
+
pass
|
180
|
+
|
181
|
+
def ack(self, id):
|
182
|
+
pass
|
183
|
+
|
184
|
+
def fail(self, id):
|
185
|
+
pass
|
186
|
+
|
187
|
+
def nextTuple(self):
|
188
|
+
pass
|
189
|
+
|
190
|
+
def run(self):
|
191
|
+
global MODE
|
192
|
+
MODE = Spout
|
193
|
+
conf, context = initComponent()
|
194
|
+
self.initialize(conf, context)
|
195
|
+
try:
|
196
|
+
while True:
|
197
|
+
msg = readCommand()
|
198
|
+
if msg["command"] == "next":
|
199
|
+
self.nextTuple()
|
200
|
+
if msg["command"] == "ack":
|
201
|
+
self.ack(msg["id"])
|
202
|
+
if msg["command"] == "fail":
|
203
|
+
self.fail(msg["id"])
|
204
|
+
sync()
|
205
|
+
except Exception, e:
|
206
|
+
log(traceback.format_exc(e))
|
@@ -0,0 +1,41 @@
|
|
1
|
+
require 'red_storm'
|
2
|
+
require 'thread'
|
3
|
+
|
4
|
+
java_import 'redstorm.storm.jruby.JRubyShellBolt'
|
5
|
+
|
6
|
+
class SimpleSpout < RedStorm::SimpleSpout
|
7
|
+
on_init do
|
8
|
+
@q = Queue.new
|
9
|
+
@q << "the quick red fox"
|
10
|
+
end
|
11
|
+
|
12
|
+
on_send do
|
13
|
+
# avoid putting the thread to sleep endlessly on @q.pop which will prevent local cluster.shutdown
|
14
|
+
sleep(1)
|
15
|
+
@q.pop unless @q.empty?
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
class ShellTopology < RedStorm::SimpleTopology
|
20
|
+
spout SimpleSpout do
|
21
|
+
output_fields :string
|
22
|
+
end
|
23
|
+
|
24
|
+
bolt JRubyShellBolt, ["python", "splitsentence.py"] do
|
25
|
+
output_fields "word"
|
26
|
+
source SimpleSpout, :shuffle
|
27
|
+
end
|
28
|
+
|
29
|
+
configure do |env|
|
30
|
+
debug true
|
31
|
+
end
|
32
|
+
|
33
|
+
on_submit do |env|
|
34
|
+
case env
|
35
|
+
when :local
|
36
|
+
sleep(10)
|
37
|
+
cluster.shutdown
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
@@ -25,7 +25,8 @@ module RedStorm
|
|
25
25
|
TASKS_FILE = "#{RedStorm::REDSTORM_HOME}/lib/tasks/red_storm.rake"
|
26
26
|
|
27
27
|
def self.local_storm_command(class_file, ruby_mode = nil)
|
28
|
-
|
28
|
+
src_dir = File.expand_path(File.dirname(class_file))
|
29
|
+
"java -Djruby.compat.version=#{RedStorm.jruby_mode_token(ruby_mode)} -cp \"#{TARGET_CLASSES_DIR}:#{TARGET_DEPENDENCY_DIR}/*:#{src_dir}/\" redstorm.TopologyLauncher local #{class_file}"
|
29
30
|
end
|
30
31
|
|
31
32
|
def self.cluster_storm_command(class_file, ruby_mode = nil)
|
@@ -67,6 +68,12 @@ module RedStorm
|
|
67
68
|
end
|
68
69
|
usage
|
69
70
|
end
|
71
|
+
|
72
|
+
def self.subshell(command)
|
73
|
+
out = IO.popen([command, {:err => [:child, :out]}]) {|io| io.read}
|
74
|
+
[!!$?.success?, out]
|
75
|
+
end
|
76
|
+
|
70
77
|
end
|
71
78
|
|
72
79
|
end
|
@@ -2,6 +2,7 @@ require 'java'
|
|
2
2
|
require 'red_storm/configuration'
|
3
3
|
require 'red_storm/configurator'
|
4
4
|
|
5
|
+
|
5
6
|
module RedStorm
|
6
7
|
|
7
8
|
class TopologyDefinitionError < StandardError; end
|
@@ -13,14 +14,20 @@ module RedStorm
|
|
13
14
|
DEFAULT_BOLT_PARALLELISM = 1
|
14
15
|
|
15
16
|
class ComponentDefinition < Configurator
|
16
|
-
attr_reader :clazz, :parallelism
|
17
|
+
attr_reader :clazz, :constructor_args, :parallelism
|
17
18
|
attr_accessor :id # ids are forced to string
|
18
19
|
|
19
|
-
def initialize(component_class, id, parallelism)
|
20
|
+
def initialize(component_class, constructor_args, id, parallelism)
|
20
21
|
super()
|
21
22
|
@clazz = component_class
|
23
|
+
@constructor_args = constructor_args
|
22
24
|
@id = id.to_s
|
23
25
|
@parallelism = parallelism
|
26
|
+
@output_fields = []
|
27
|
+
end
|
28
|
+
|
29
|
+
def output_fields(*args)
|
30
|
+
args.empty? ? @output_fields : @output_fields = args.map(&:to_s)
|
24
31
|
end
|
25
32
|
|
26
33
|
def is_java?
|
@@ -29,13 +36,22 @@ module RedStorm
|
|
29
36
|
end
|
30
37
|
|
31
38
|
class SpoutDefinition < ComponentDefinition
|
39
|
+
|
40
|
+
# WARNING non-dry see BoltDefinition#new_instance
|
32
41
|
def new_instance(base_class_path)
|
33
|
-
|
42
|
+
if @clazz.name == "Java::RedstormStormJruby::JRubyShellSpout"
|
43
|
+
@clazz.new(constructor_args, @output_fields)
|
44
|
+
elsif is_java?
|
45
|
+
@clazz.new(*constructor_args)
|
46
|
+
else
|
47
|
+
JRubySpout.new(base_class_path, @clazz.name, @output_fields)
|
48
|
+
end
|
49
|
+
# is_java? ? @clazz.new : JRubySpout.new(base_class_path, @clazz.name)
|
34
50
|
end
|
35
51
|
end
|
36
52
|
|
37
53
|
class BoltDefinition < ComponentDefinition
|
38
|
-
attr_accessor :sources
|
54
|
+
attr_accessor :sources, :command
|
39
55
|
|
40
56
|
def initialize(*args)
|
41
57
|
super
|
@@ -72,7 +88,15 @@ module RedStorm
|
|
72
88
|
end
|
73
89
|
|
74
90
|
def new_instance(base_class_path)
|
75
|
-
|
91
|
+
# WARNING non-dry see BoltDefinition#new_instance
|
92
|
+
if @clazz.name == "Java::RedstormStormJruby::JRubyShellBolt"
|
93
|
+
@clazz.new(constructor_args, @output_fields)
|
94
|
+
elsif is_java?
|
95
|
+
@clazz.new(*constructor_args)
|
96
|
+
else
|
97
|
+
JRubyBolt.new(base_class_path, @clazz.name, @output_fields)
|
98
|
+
end
|
99
|
+
# is_java? ? @clazz.new : @clazz.is_a?(SimpleBolt) ? JRubyBolt.new(base_class_path, @clazz.name) : @clazz.new
|
76
100
|
end
|
77
101
|
end
|
78
102
|
|
@@ -80,16 +104,24 @@ module RedStorm
|
|
80
104
|
@log ||= org.apache.log4j.Logger.getLogger(self.name)
|
81
105
|
end
|
82
106
|
|
83
|
-
def self.spout(spout_class, options = {}, &spout_block)
|
107
|
+
# def self.spout(spout_class, contructor_args = [], options = {}, &spout_block)
|
108
|
+
def self.spout(spout_class, *args, &spout_block)
|
109
|
+
options = args.last.is_a?(Hash) ? args.pop : {}
|
110
|
+
contructor_args = !args.empty? ? args.pop : []
|
84
111
|
spout_options = {:id => self.underscore(spout_class), :parallelism => DEFAULT_SPOUT_PARALLELISM}.merge(options)
|
85
|
-
|
112
|
+
|
113
|
+
spout = SpoutDefinition.new(spout_class, contructor_args, spout_options[:id], spout_options[:parallelism])
|
86
114
|
spout.instance_exec(&spout_block) if block_given?
|
87
115
|
self.components << spout
|
88
116
|
end
|
89
117
|
|
90
|
-
def self.bolt(bolt_class, options = {}, &bolt_block)
|
118
|
+
# def self.bolt(bolt_class, contructor_args = [], options = {}, &bolt_block)
|
119
|
+
def self.bolt(bolt_class, *args, &bolt_block)
|
120
|
+
options = args.last.is_a?(Hash) ? args.pop : {}
|
121
|
+
contructor_args = !args.empty? ? args.pop : []
|
91
122
|
bolt_options = {:id => self.underscore(bolt_class), :parallelism => DEFAULT_BOLT_PARALLELISM}.merge(options)
|
92
|
-
|
123
|
+
|
124
|
+
bolt = BoltDefinition.new(bolt_class, contructor_args, bolt_options[:id], bolt_options[:parallelism])
|
93
125
|
raise(TopologyDefinitionError, "#{bolt.clazz.name}, #{bolt.id}, bolt definition body required") unless block_given?
|
94
126
|
bolt.instance_exec(&bolt_block)
|
95
127
|
self.components << bolt
|
data/lib/red_storm/version.rb
CHANGED
data/lib/tasks/red_storm.rake
CHANGED
@@ -88,8 +88,22 @@ task :jar, [:include_dir] => [:unpack, :clean_jar] do |t, args|
|
|
88
88
|
exclude :name => "tasks/**"
|
89
89
|
end
|
90
90
|
if args[:include_dir]
|
91
|
+
dirs = args[:include_dir].split(":")
|
92
|
+
|
93
|
+
# first add any resources/ dir in the tree in the jar root - requirement for ShellBolt multilang resources
|
94
|
+
dirs.each do |dir|
|
95
|
+
resources_dirs = Dir.glob("#{dir}/**/resources")
|
96
|
+
resources_dirs.each do |resources_dir|
|
97
|
+
resources_parent = resources_dir.gsub("/resources", "")
|
98
|
+
fileset :dir => resources_parent do
|
99
|
+
include :name => "resources/**/*"
|
100
|
+
end
|
101
|
+
end
|
102
|
+
end
|
103
|
+
|
104
|
+
# include complete source dir tree (note we don't care about potential duplicated resources dir)
|
91
105
|
fileset :dir => CWD do
|
92
|
-
|
106
|
+
dirs.each{|dir| include :name => "#{dir}/**/*"}
|
93
107
|
end
|
94
108
|
end
|
95
109
|
manifest do
|
@@ -157,8 +171,8 @@ end
|
|
157
171
|
|
158
172
|
task :bundle, [:groups] => :setup do |t, args|
|
159
173
|
require 'bundler'
|
160
|
-
|
161
|
-
groups =
|
174
|
+
defaulted_args = {:groups => 'default'}.merge(args.to_hash.delete_if{|k, v| v.to_s.empty?})
|
175
|
+
groups = defaulted_args[:groups].split(':').map(&:to_sym)
|
162
176
|
Bundler.definition.specs_for(groups).each do |spec|
|
163
177
|
unless spec.full_name =~ /^bundler-\d+/
|
164
178
|
destination_path = "#{TARGET_GEM_DIR}/#{spec.full_name}"
|
@@ -5,6 +5,7 @@ import backtype.storm.task.TopologyContext;
|
|
5
5
|
import backtype.storm.topology.IRichBolt;
|
6
6
|
import backtype.storm.topology.OutputFieldsDeclarer;
|
7
7
|
import backtype.storm.tuple.Tuple;
|
8
|
+
import backtype.storm.tuple.Fields;
|
8
9
|
import java.util.Map;
|
9
10
|
|
10
11
|
/**
|
@@ -20,16 +21,19 @@ import java.util.Map;
|
|
20
21
|
public class JRubyBolt implements IRichBolt {
|
21
22
|
IRichBolt _proxyBolt;
|
22
23
|
String _realBoltClassName;
|
23
|
-
String _baseClassPath;
|
24
|
+
String _baseClassPath;
|
25
|
+
String[] _fields;
|
26
|
+
|
24
27
|
/**
|
25
28
|
* create a new JRubyBolt
|
26
29
|
*
|
27
30
|
* @param baseClassPath the topology/project base JRuby class file path
|
28
31
|
* @param realBoltClassName the fully qualified JRuby bolt implementation class name
|
29
32
|
*/
|
30
|
-
public JRubyBolt(String baseClassPath, String realBoltClassName) {
|
33
|
+
public JRubyBolt(String baseClassPath, String realBoltClassName, String[] fields) {
|
31
34
|
_baseClassPath = baseClassPath;
|
32
35
|
_realBoltClassName = realBoltClassName;
|
36
|
+
_fields = fields;
|
33
37
|
}
|
34
38
|
|
35
39
|
@Override
|
@@ -54,8 +58,12 @@ public class JRubyBolt implements IRichBolt {
|
|
54
58
|
// declareOutputFields is executed in the topology creation time, before serialisation.
|
55
59
|
// do not set the _proxyBolt instance variable here to avoid JRuby serialization
|
56
60
|
// issues. Just create tmp bolt instance to call declareOutputFields.
|
57
|
-
|
58
|
-
|
61
|
+
if (_fields.length > 0) {
|
62
|
+
declarer.declare(new Fields(_fields));
|
63
|
+
} else {
|
64
|
+
IRichBolt bolt = newProxyBolt(_baseClassPath, _realBoltClassName);
|
65
|
+
bolt.declareOutputFields(declarer);
|
66
|
+
}
|
59
67
|
}
|
60
68
|
|
61
69
|
@Override
|
@@ -0,0 +1,26 @@
|
|
1
|
+
package redstorm.storm.jruby;
|
2
|
+
|
3
|
+
import backtype.storm.task.ShellBolt;
|
4
|
+
import backtype.storm.topology.IRichBolt;
|
5
|
+
import backtype.storm.topology.OutputFieldsDeclarer;
|
6
|
+
import backtype.storm.tuple.Fields;
|
7
|
+
import java.util.Map;
|
8
|
+
|
9
|
+
public class JRubyShellBolt extends ShellBolt implements IRichBolt {
|
10
|
+
private String[] _fields;
|
11
|
+
|
12
|
+
public JRubyShellBolt(String[] command, String[] fields) {
|
13
|
+
super(command);
|
14
|
+
_fields = fields;
|
15
|
+
}
|
16
|
+
|
17
|
+
@Override
|
18
|
+
public void declareOutputFields(OutputFieldsDeclarer declarer) {
|
19
|
+
declarer.declare(new Fields(_fields));
|
20
|
+
}
|
21
|
+
|
22
|
+
@Override
|
23
|
+
public Map<String, Object> getComponentConfiguration() {
|
24
|
+
return null;
|
25
|
+
}
|
26
|
+
}
|
@@ -0,0 +1,26 @@
|
|
1
|
+
package backtype.storm.clojure;
|
2
|
+
|
3
|
+
import backtype.storm.spout.ShellSpout;
|
4
|
+
import backtype.storm.topology.IRichSpout;
|
5
|
+
import backtype.storm.topology.OutputFieldsDeclarer;
|
6
|
+
import backtype.storm.tuple.Fields;
|
7
|
+
import java.util.Map;
|
8
|
+
|
9
|
+
public class JRubyShellSpout extends ShellSpout implements IRichSpout {
|
10
|
+
private String[] _fields;
|
11
|
+
|
12
|
+
public JRubyShellSpout(String[] command, String[] fields) {
|
13
|
+
super(command);
|
14
|
+
_fields = fields;
|
15
|
+
}
|
16
|
+
|
17
|
+
@Override
|
18
|
+
public void declareOutputFields(OutputFieldsDeclarer declarer) {
|
19
|
+
declarer.declare(new Fields(_fields));
|
20
|
+
}
|
21
|
+
|
22
|
+
@Override
|
23
|
+
public Map<String, Object> getComponentConfiguration() {
|
24
|
+
return null;
|
25
|
+
}
|
26
|
+
}
|
@@ -5,6 +5,7 @@ import backtype.storm.task.TopologyContext;
|
|
5
5
|
import backtype.storm.topology.IRichSpout;
|
6
6
|
import backtype.storm.topology.OutputFieldsDeclarer;
|
7
7
|
import backtype.storm.tuple.Tuple;
|
8
|
+
import backtype.storm.tuple.Fields;
|
8
9
|
import java.util.Map;
|
9
10
|
|
10
11
|
/**
|
@@ -21,16 +22,18 @@ public class JRubySpout implements IRichSpout {
|
|
21
22
|
IRichSpout _proxySpout;
|
22
23
|
String _realSpoutClassName;
|
23
24
|
String _baseClassPath;
|
24
|
-
|
25
|
+
String[] _fields;
|
26
|
+
|
25
27
|
/**
|
26
28
|
* create a new JRubySpout
|
27
29
|
*
|
28
30
|
* @param baseClassPath the topology/project base JRuby class file path
|
29
31
|
* @param realSpoutClassName the fully qualified JRuby spout implementation class name
|
30
32
|
*/
|
31
|
-
public JRubySpout(String baseClassPath, String realSpoutClassName) {
|
33
|
+
public JRubySpout(String baseClassPath, String realSpoutClassName, String[] fields) {
|
32
34
|
_baseClassPath = baseClassPath;
|
33
35
|
_realSpoutClassName = realSpoutClassName;
|
36
|
+
_fields = fields;
|
34
37
|
}
|
35
38
|
|
36
39
|
@Override
|
@@ -75,8 +78,12 @@ public class JRubySpout implements IRichSpout {
|
|
75
78
|
// declareOutputFields is executed in the topology creation time before serialisation.
|
76
79
|
// do not set the _proxySpout instance variable here to avoid JRuby serialization
|
77
80
|
// issues. Just create tmp spout instance to call declareOutputFields.
|
78
|
-
|
79
|
-
|
81
|
+
if (_fields.length > 0) {
|
82
|
+
declarer.declare(new Fields(_fields));
|
83
|
+
} else {
|
84
|
+
IRichSpout spout = newProxySpout(_baseClassPath, _realSpoutClassName);
|
85
|
+
spout.declareOutputFields(declarer);
|
86
|
+
}
|
80
87
|
}
|
81
88
|
|
82
89
|
@Override
|
metadata
CHANGED
@@ -2,14 +2,14 @@
|
|
2
2
|
name: redstorm
|
3
3
|
version: !ruby/object:Gem::Version
|
4
4
|
prerelease:
|
5
|
-
version: 0.6.
|
5
|
+
version: 0.6.3
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
8
8
|
- Colin Surprenant
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-
|
12
|
+
date: 2012-10-10 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: rspec
|
@@ -91,6 +91,9 @@ files:
|
|
91
91
|
- examples/native/random_sentence_spout.rb
|
92
92
|
- examples/native/split_sentence_bolt.rb
|
93
93
|
- examples/native/word_count_bolt.rb
|
94
|
+
- examples/shell/shell_topology.rb
|
95
|
+
- examples/shell/resources/splitsentence.py
|
96
|
+
- examples/shell/resources/storm.py
|
94
97
|
- examples/simple/exclamation_bolt.rb
|
95
98
|
- examples/simple/exclamation_topology.rb
|
96
99
|
- examples/simple/exclamation_topology2.rb
|
@@ -102,6 +105,8 @@ files:
|
|
102
105
|
- examples/simple/word_count_bolt.rb
|
103
106
|
- examples/simple/word_count_topology.rb
|
104
107
|
- src/main/redstorm/storm/jruby/JRubyBolt.java
|
108
|
+
- src/main/redstorm/storm/jruby/JRubyShellBolt.java
|
109
|
+
- src/main/redstorm/storm/jruby/JRubyShellSpout.java
|
105
110
|
- src/main/redstorm/storm/jruby/JRubySpout.java
|
106
111
|
- bin/redstorm
|
107
112
|
- Rakefile
|