redstorm 0.6.6.beta2 → 0.6.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/CHANGELOG.md +5 -2
- data/README.md +18 -9
- data/bin/redstorm +6 -1
- data/examples/dsl/kafka_topology.rb +21 -13
- data/examples/dsl/redis_word_count_topology.rb +4 -4
- data/ivy/settings.xml +3 -3
- data/lib/red_storm/application.rb +0 -27
- data/lib/red_storm/environment.rb +40 -13
- data/lib/red_storm/version.rb +1 -1
- data/lib/tasks/red_storm.rake +56 -54
- metadata +5 -5
data/CHANGELOG.md
CHANGED
@@ -91,7 +91,10 @@
|
|
91
91
|
- improved DSL performance by refactoring closures invocation, see [redstorm-benchmark](https://github.com/colinsurprenant/redstorm-benchmark/)
|
92
92
|
- fixed FFI support
|
93
93
|
- exposed topology builder for specs
|
94
|
-
- [issue #11](https://github.com/colinsurprenant/redstorm/issues/11), [#17](https://github.com/colinsurprenant/redstorm/issues/17) - example specs using the Storm testing API see [redstorm-starter](https://github.com/colinsurprenant/
|
94
|
+
- [issue #11](https://github.com/colinsurprenant/redstorm/issues/11), [#17](https://github.com/colinsurprenant/redstorm/issues/17) - example specs using the Storm testing API see [redstorm-starter](https://github.com/colinsurprenant/redsto rm-starter/)
|
95
95
|
|
96
96
|
# 0.6.6.beta2, 07-20-2013
|
97
|
-
- [issue #76](https://github.com/colinsurprenant/redstorm/issues/76) - avoid shelling out to storm jar command for cluster submission
|
97
|
+
- [issue #76](https://github.com/colinsurprenant/redstorm/issues/76) - avoid shelling out to storm jar command for cluster submission
|
98
|
+
|
99
|
+
# 0.6.6, 07-25-2013
|
100
|
+
- updated example Kafka topology for new dependencies for Storm KafkaSpout
|
data/README.md
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# RedStorm
|
1
|
+
# RedStorm - JRuby on Storm
|
2
2
|
|
3
3
|
[](http://travis-ci.org/colinsurprenant/redstorm)
|
4
4
|
|
@@ -13,11 +13,12 @@ Check also these related projects:
|
|
13
13
|
|
14
14
|
## Documentation
|
15
15
|
|
16
|
+
<!--
|
16
17
|
---
|
17
|
-
This is the documentation for the **current
|
18
|
+
This is the documentation for the **current 0.6.6-beta2 version of RedStorm** - the **[latest released Gem is v0.6.5](https://github.com/colinsurprenant/redstorm/wiki/RedStorm-Gem-v0.6.5-Documentation)**
|
18
19
|
|
19
20
|
---
|
20
|
-
|
21
|
+
-->
|
21
22
|
Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept [in the wiki](https://github.com/colinsurprenant/redstorm/wiki) when necessary.
|
22
23
|
|
23
24
|
## Dependencies
|
@@ -29,14 +30,14 @@ Tested on **OSX 10.8.3** and **Ubuntu Linux 12.10** using **Storm 0.9.0-wip16**
|
|
29
30
|
- RubyGems
|
30
31
|
|
31
32
|
``` sh
|
32
|
-
$ gem install redstorm
|
33
|
+
$ gem install redstorm
|
33
34
|
```
|
34
35
|
|
35
36
|
- Bundler
|
36
37
|
|
37
38
|
``` ruby
|
38
39
|
source "https://rubygems.org"
|
39
|
-
gem "redstorm", "~> 0.6.6
|
40
|
+
gem "redstorm", "~> 0.6.6"
|
40
41
|
```
|
41
42
|
|
42
43
|
## Usage
|
@@ -140,7 +141,7 @@ You can supply custom `storm` and `topology` dependencies by creating `ivy/storm
|
|
140
141
|
|
141
142
|
``` xml
|
142
143
|
<?xml version="1.0"?>
|
143
|
-
<ivy-module version="2.0">
|
144
|
+
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
|
144
145
|
<info organisation="redstorm" module="storm-deps"/>
|
145
146
|
<dependencies>
|
146
147
|
<dependency org="storm" name="storm" rev="0.9.0-wip16" conf="default" transitive="true" />
|
@@ -153,10 +154,18 @@ You can supply custom `storm` and `topology` dependencies by creating `ivy/storm
|
|
153
154
|
|
154
155
|
``` xml
|
155
156
|
<?xml version="1.0"?>
|
156
|
-
<ivy-module version="2.0">
|
157
|
+
<ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
|
157
158
|
<info organisation="redstorm" module="topology-deps"/>
|
158
159
|
<dependencies>
|
159
160
|
<dependency org="org.jruby" name="jruby-core" rev="1.7.4" conf="default" transitive="true"/>
|
161
|
+
|
162
|
+
<!-- explicitely specify jffi to also fetch the native jar. make sure to update jffi version matching jruby-core version -->
|
163
|
+
<!-- this is the only way I found using Ivy to fetch the native jar -->
|
164
|
+
<dependency org="com.github.jnr" name="jffi" rev="1.2.5" conf="default" transitive="true">
|
165
|
+
<artifact name="jffi" type="jar" />
|
166
|
+
<artifact name="jffi" type="jar" m:classifier="native"/>
|
167
|
+
</dependency>
|
168
|
+
|
160
169
|
</dependencies>
|
161
170
|
</ivy-module>
|
162
171
|
```
|
@@ -172,9 +181,9 @@ The jars repositories can be configured by adding the `ivy/settings.xml` file in
|
|
172
181
|
<resolvers>
|
173
182
|
<chain name="repositories">
|
174
183
|
<ibiblio name="ibiblio" m2compatible="true"/>
|
175
|
-
<ibiblio name="maven2" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/>
|
176
184
|
<ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/>
|
177
185
|
<ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/>
|
186
|
+
<ibiblio name="conjars" root="http://conjars.org/repo/" m2compatible="true"/>
|
178
187
|
</chain>
|
179
188
|
</resolvers>
|
180
189
|
</ivysettings>
|
@@ -189,7 +198,7 @@ $ redstorm local <sources_directory_path/topology_class_file_name.rb>
|
|
189
198
|
note that the topology can also be launched with the following command:
|
190
199
|
|
191
200
|
``` sh
|
192
|
-
$ java -
|
201
|
+
$ java -cp "target/classes:target/dependency/storm/default/*:target/dependency/topology/default/*:<sources_directory_path>" redstorm.TopologyLauncher local <sources_directory_path/topology_class_file_name.rb>
|
193
202
|
```
|
194
203
|
|
195
204
|
**See examples below** to run examples in local mode or on a production cluster.
|
data/bin/redstorm
CHANGED
@@ -1,6 +1,11 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
|
-
|
3
|
+
unless RUBY_PLATFORM =~ /java/
|
4
|
+
puts("error: RedStorm requires JRuby. Check http://jruby.org/ or https://github.com/sstephenson/rbenv or https://rvm.io/")
|
5
|
+
exit(1)
|
6
|
+
end
|
7
|
+
|
8
|
+
require 'rubygems'
|
4
9
|
require 'rake'
|
5
10
|
|
6
11
|
begin
|
@@ -1,7 +1,6 @@
|
|
1
|
-
java_import 'storm.kafka.KafkaConfig'
|
2
1
|
java_import 'storm.kafka.SpoutConfig'
|
3
|
-
java_import 'storm.kafka.StringScheme'
|
4
2
|
java_import 'storm.kafka.KafkaSpout'
|
3
|
+
java_import 'storm.kafka.KafkaConfig'
|
5
4
|
|
6
5
|
require 'red_storm'
|
7
6
|
|
@@ -12,28 +11,37 @@ require 'red_storm'
|
|
12
11
|
# dependencies in the "ivy/topology_dependencies.xml" file in the root of your RedStorm project:
|
13
12
|
#
|
14
13
|
# <?xml version="1.0"?>
|
15
|
-
# <ivy-module version="2.0">
|
14
|
+
# <ivy-module version="2.0" xmlns:m="http://ant.apache.org/ivy/maven">
|
16
15
|
# <info organisation="redstorm" module="topology-deps"/>
|
17
16
|
# <dependencies>
|
18
17
|
# <dependency org="org.jruby" name="jruby-core" rev="1.7.4" conf="default" transitive="true"/>
|
19
|
-
|
20
|
-
# <dependency org="
|
21
|
-
# <dependency org="
|
18
|
+
|
19
|
+
# <dependency org="org.scala-lang" name="scala-library" rev="2.9.2" conf="default" transitive="false"/>
|
20
|
+
# <dependency org="com.twitter" name="kafka_2.9.2" rev="0.7.0" conf="default" transitive="false"/>
|
21
|
+
# <dependency org="storm" name="storm-kafka" rev="0.9.0-wip16a-scala292" conf="default" transitive="true"/>
|
22
|
+
|
23
|
+
# <!-- explicitely specify jffi to also fetch the native jar. make sure to update jffi version matching jruby-core version -->
|
24
|
+
# <!-- this is the only way I found using Ivy to fetch the native jar -->
|
25
|
+
# <dependency org="com.github.jnr" name="jffi" rev="1.2.5" conf="default" transitive="true">
|
26
|
+
# <artifact name="jffi" type="jar" />
|
27
|
+
# <artifact name="jffi" type="jar" m:classifier="native"/>
|
28
|
+
# </dependency>
|
29
|
+
|
22
30
|
# </dependencies>
|
23
31
|
# </ivy-module>
|
24
32
|
|
33
|
+
class SplitStringBolt < RedStorm::DSL::Bolt
|
34
|
+
on_receive {|tuple| tuple[0].split.map{|w| [w]}}
|
35
|
+
end
|
36
|
+
|
25
37
|
class KafkaTopology < RedStorm::DSL::Topology
|
38
|
+
|
26
39
|
spout_config = SpoutConfig.new(
|
27
40
|
KafkaConfig::ZkHosts.new("localhost:2181", "/brokers"),
|
28
41
|
"words", # topic to read from
|
29
|
-
"/
|
30
|
-
"
|
42
|
+
"/kafkaspout", # Zookeeper root path to store the consumer offsets
|
43
|
+
"someid" # Zookeeper consumer id to store the consumer offsets
|
31
44
|
)
|
32
|
-
spout_config.scheme = StringScheme.new
|
33
|
-
|
34
|
-
class SplitStringBolt < RedStorm::DSL::Bolt
|
35
|
-
on_receive {|tuple| tuple[0].split.map{|w| [w]}}
|
36
|
-
end
|
37
45
|
|
38
46
|
spout KafkaSpout, [spout_config]
|
39
47
|
|
@@ -38,7 +38,7 @@ module RedStorm
|
|
38
38
|
class RedisWordCountTopology < DSL::Topology
|
39
39
|
spout RedisWordSpout
|
40
40
|
|
41
|
-
bolt WordCountBolt, :parallelism =>
|
41
|
+
bolt WordCountBolt, :parallelism => 2 do
|
42
42
|
source RedisWordSpout, :fields => ["word"]
|
43
43
|
end
|
44
44
|
|
@@ -46,10 +46,10 @@ module RedStorm
|
|
46
46
|
debug true
|
47
47
|
case env
|
48
48
|
when :local
|
49
|
-
max_task_parallelism
|
49
|
+
max_task_parallelism 2
|
50
50
|
when :cluster
|
51
|
-
max_task_parallelism
|
52
|
-
num_workers
|
51
|
+
max_task_parallelism 2
|
52
|
+
num_workers 2
|
53
53
|
max_spout_pending(1000)
|
54
54
|
end
|
55
55
|
end
|
data/ivy/settings.xml
CHANGED
@@ -4,9 +4,9 @@
|
|
4
4
|
<resolvers>
|
5
5
|
<chain name="repositories">
|
6
6
|
<ibiblio name="ibiblio" m2compatible="true"/>
|
7
|
-
<ibiblio name="
|
8
|
-
<ibiblio name="
|
9
|
-
<ibiblio name="
|
7
|
+
<ibiblio name="sonatype" root="http://repo.maven.apache.org/maven2/" m2compatible="true"/>
|
8
|
+
<ibiblio name="clojars" root="http://clojars.org/repo/" m2compatible="true"/>
|
9
|
+
<ibiblio name="conjars" root="http://conjars.org/repo/" m2compatible="true"/>
|
10
10
|
</chain>
|
11
11
|
</resolvers>
|
12
12
|
</ivysettings>
|
@@ -1,33 +1,6 @@
|
|
1
1
|
require 'red_storm/version'
|
2
2
|
require 'red_storm/environment'
|
3
3
|
|
4
|
-
CWD = Dir.pwd
|
5
|
-
TARGET_DIR = "#{CWD}/target"
|
6
|
-
TARGET_LIB_DIR = "#{TARGET_DIR}/lib"
|
7
|
-
TARGET_SRC_DIR = "#{TARGET_DIR}/src"
|
8
|
-
TARGET_GEM_DIR = "#{TARGET_DIR}/gems/gems"
|
9
|
-
TARGET_SPECS_DIR = "#{TARGET_DIR}/gems/specifications"
|
10
|
-
TARGET_CLASSES_DIR = "#{TARGET_DIR}/classes"
|
11
|
-
TARGET_DEPENDENCY_DIR = "#{TARGET_DIR}/dependency"
|
12
|
-
TARGET_DEPENDENCY_UNPACKED_DIR = "#{TARGET_DIR}/dependency-unpacked"
|
13
|
-
TARGET_CLUSTER_JAR = "#{TARGET_DIR}/cluster-topology.jar"
|
14
|
-
|
15
|
-
REDSTORM_JAVA_SRC_DIR = "#{RedStorm::REDSTORM_HOME}/src/main"
|
16
|
-
REDSTORM_LIB_DIR = "#{RedStorm::REDSTORM_HOME}/lib"
|
17
|
-
|
18
|
-
SRC_EXAMPLES = "#{RedStorm::REDSTORM_HOME}/examples"
|
19
|
-
DST_EXAMPLES = "#{CWD}/examples"
|
20
|
-
|
21
|
-
SRC_IVY_DIR = "#{RedStorm::REDSTORM_HOME}/ivy"
|
22
|
-
DST_IVY_DIR = "#{CWD}/ivy"
|
23
|
-
DEFAULT_IVY_SETTINGS = "#{SRC_IVY_DIR}/settings.xml"
|
24
|
-
CUSTOM_IVY_SETTINGS = "#{DST_IVY_DIR}/settings.xml"
|
25
|
-
DEFAULT_IVY_STORM_DEPENDENCIES = "#{SRC_IVY_DIR}/storm_dependencies.xml"
|
26
|
-
CUSTOM_IVY_STORM_DEPENDENCIES = "#{DST_IVY_DIR}/storm_dependencies.xml"
|
27
|
-
DEFAULT_IVY_TOPOLOGY_DEPENDENCIES = "#{SRC_IVY_DIR}/topology_dependencies.xml"
|
28
|
-
CUSTOM_IVY_TOPOLOGY_DEPENDENCIES = "#{DST_IVY_DIR}/topology_dependencies.xml"
|
29
|
-
|
30
|
-
DEFAULT_STORM_CONF_FILE = File.expand_path("~/.storm/storm.yaml")
|
31
4
|
|
32
5
|
module RedStorm
|
33
6
|
|
@@ -2,30 +2,57 @@ require 'java'
|
|
2
2
|
java_import 'java.lang.System'
|
3
3
|
|
4
4
|
module RedStorm
|
5
|
+
CWD = Dir.pwd
|
5
6
|
|
6
|
-
|
7
|
-
|
7
|
+
launch_path = File.expand_path(File.dirname(__FILE__))
|
8
|
+
jar_context = !!(launch_path =~ /\.jar!\/red_storm$/)
|
8
9
|
|
9
|
-
if
|
10
|
-
BASE_PATH = File.expand_path(
|
10
|
+
if jar_context
|
11
|
+
BASE_PATH = File.expand_path(launch_path + '/..')
|
11
12
|
REDSTORM_HOME = BASE_PATH
|
12
|
-
|
13
|
+
GEM_PATH = "#{REDSTORM_HOME}/gems/"
|
13
14
|
else
|
14
|
-
BASE_PATH =
|
15
|
-
REDSTORM_HOME = File.expand_path(
|
16
|
-
|
15
|
+
BASE_PATH = CWD
|
16
|
+
REDSTORM_HOME = File.expand_path(launch_path + '/../..')
|
17
|
+
GEM_PATH = "#{BASE_PATH}/target/gems/"
|
17
18
|
end
|
18
19
|
|
19
20
|
unless defined?(SPECS_CONTEXT)
|
20
|
-
GEM_PATH = "#{TARGET_PATH}/gems/"
|
21
21
|
ENV["GEM_PATH"] = GEM_PATH
|
22
22
|
ENV["GEM_HOME"] = GEM_PATH
|
23
23
|
end
|
24
24
|
|
25
|
+
TARGET_DIR = "#{CWD}/target"
|
26
|
+
TARGET_LIB_DIR = "#{TARGET_DIR}/lib"
|
27
|
+
TARGET_SRC_DIR = "#{TARGET_DIR}/src"
|
28
|
+
TARGET_GEM_DIR = "#{TARGET_DIR}/gems/gems"
|
29
|
+
TARGET_SPECS_DIR = "#{TARGET_DIR}/gems/specifications"
|
30
|
+
TARGET_CLASSES_DIR = "#{TARGET_DIR}/classes"
|
31
|
+
TARGET_DEPENDENCY_DIR = "#{TARGET_DIR}/dependency"
|
32
|
+
TARGET_DEPENDENCY_UNPACKED_DIR = "#{TARGET_DIR}/dependency-unpacked"
|
33
|
+
TARGET_CLUSTER_JAR = "#{TARGET_DIR}/cluster-topology.jar"
|
34
|
+
|
35
|
+
REDSTORM_JAVA_SRC_DIR = "#{REDSTORM_HOME}/src/main"
|
36
|
+
REDSTORM_LIB_DIR = "#{REDSTORM_HOME}/lib"
|
37
|
+
|
38
|
+
SRC_EXAMPLES = "#{REDSTORM_HOME}/examples"
|
39
|
+
DST_EXAMPLES = "#{CWD}/examples"
|
40
|
+
|
41
|
+
SRC_IVY_DIR = "#{REDSTORM_HOME}/ivy"
|
42
|
+
DST_IVY_DIR = "#{CWD}/ivy"
|
43
|
+
DEFAULT_IVY_SETTINGS = "#{SRC_IVY_DIR}/settings.xml"
|
44
|
+
CUSTOM_IVY_SETTINGS = "#{DST_IVY_DIR}/settings.xml"
|
45
|
+
DEFAULT_IVY_STORM_DEPENDENCIES = "#{SRC_IVY_DIR}/storm_dependencies.xml"
|
46
|
+
CUSTOM_IVY_STORM_DEPENDENCIES = "#{DST_IVY_DIR}/storm_dependencies.xml"
|
47
|
+
DEFAULT_IVY_TOPOLOGY_DEPENDENCIES = "#{SRC_IVY_DIR}/topology_dependencies.xml"
|
48
|
+
CUSTOM_IVY_TOPOLOGY_DEPENDENCIES = "#{DST_IVY_DIR}/topology_dependencies.xml"
|
49
|
+
|
50
|
+
DEFAULT_STORM_CONF_FILE = File.expand_path("~/.storm/storm.yaml")
|
51
|
+
|
25
52
|
def current_ruby_mode
|
26
|
-
|
27
|
-
raise("unknown Ruby version #{$1}") unless
|
28
|
-
|
53
|
+
version = RUBY_VERSION[/(\d+\.\d+)(\.\d+)*/, 1]
|
54
|
+
raise("unknown Ruby version #{$1}") unless ["1.8", "1.9"].include?(version)
|
55
|
+
version
|
29
56
|
end
|
30
57
|
|
31
58
|
def jruby_mode_token(ruby_version = nil)
|
@@ -34,7 +61,7 @@ module RedStorm
|
|
34
61
|
end
|
35
62
|
|
36
63
|
def java_runtime_version
|
37
|
-
System.properties["java.runtime.version"].to_s
|
64
|
+
System.properties["java.runtime.version"].to_s[/^(\d+\.\d+).[^\s]+$/, 1] || raise("unknown java runtime version #{System.properties["java.runtime.version"].to_s}")
|
38
65
|
end
|
39
66
|
|
40
67
|
module_function :current_ruby_mode, :jruby_mode_token, :java_runtime_version
|
data/lib/red_storm/version.rb
CHANGED
data/lib/tasks/red_storm.rake
CHANGED
@@ -1,8 +1,8 @@
|
|
1
1
|
begin
|
2
2
|
require 'ant'
|
3
3
|
rescue
|
4
|
-
puts("
|
5
|
-
puts("\nerror
|
4
|
+
puts("error: unable to load Ant, make sure Ant is installed, in your PATH and $ANT_HOME is defined properly")
|
5
|
+
puts("\nerror details:\n#{$!}")
|
6
6
|
exit(1)
|
7
7
|
end
|
8
8
|
|
@@ -10,7 +10,9 @@ require 'jruby/jrubyc'
|
|
10
10
|
require 'red_storm/environment'
|
11
11
|
require 'red_storm/application'
|
12
12
|
|
13
|
-
|
13
|
+
module RedStorm
|
14
|
+
INSTALL_IVY_VERSION = "2.3.0"
|
15
|
+
end
|
14
16
|
|
15
17
|
task :launch, :env, :storm_conf, :ruby_mode, :class_file do |t, args|
|
16
18
|
# use ruby mode parameter or default to current interpreter version
|
@@ -20,8 +22,8 @@ task :launch, :env, :storm_conf, :ruby_mode, :class_file do |t, args|
|
|
20
22
|
when "local"
|
21
23
|
RedStorm::Application.local_storm_command(args[:class_file], args[:ruby_mode])
|
22
24
|
when "cluster"
|
23
|
-
unless File.exist?(TARGET_CLUSTER_JAR)
|
24
|
-
puts("error: cluster jar file #{TARGET_CLUSTER_JAR} not found. Generate it using $redstorm jar DIR1 [DIR2, ...]")
|
25
|
+
unless File.exist?(RedStorm::TARGET_CLUSTER_JAR)
|
26
|
+
puts("error: cluster jar file #{RedStorm::TARGET_CLUSTER_JAR} not found. Generate it using $redstorm jar DIR1 [DIR2, ...]")
|
25
27
|
exit(1)
|
26
28
|
end
|
27
29
|
unless File.exist?(args[:storm_conf])
|
@@ -38,50 +40,50 @@ task :launch, :env, :storm_conf, :ruby_mode, :class_file do |t, args|
|
|
38
40
|
end
|
39
41
|
|
40
42
|
task :clean do
|
41
|
-
ant.delete 'dir' => TARGET_DIR
|
43
|
+
ant.delete 'dir' => RedStorm::TARGET_DIR
|
42
44
|
end
|
43
45
|
|
44
46
|
task :clean_jar do
|
45
|
-
ant.delete 'file' => TARGET_CLUSTER_JAR
|
47
|
+
ant.delete 'file' => RedStorm::TARGET_CLUSTER_JAR
|
46
48
|
end
|
47
49
|
|
48
50
|
task :setup do
|
49
51
|
puts("\n--> Setting up target directories")
|
50
|
-
ant.mkdir 'dir' => TARGET_DIR
|
51
|
-
ant.mkdir 'dir' => TARGET_CLASSES_DIR
|
52
|
-
ant.mkdir 'dir' => TARGET_DEPENDENCY_DIR
|
53
|
-
ant.mkdir 'dir' => TARGET_SRC_DIR
|
54
|
-
ant.mkdir 'dir' => TARGET_GEM_DIR
|
55
|
-
ant.mkdir 'dir' => TARGET_SPECS_DIR
|
52
|
+
ant.mkdir 'dir' => RedStorm::TARGET_DIR
|
53
|
+
ant.mkdir 'dir' => RedStorm::TARGET_CLASSES_DIR
|
54
|
+
ant.mkdir 'dir' => RedStorm::TARGET_DEPENDENCY_DIR
|
55
|
+
ant.mkdir 'dir' => RedStorm::TARGET_SRC_DIR
|
56
|
+
ant.mkdir 'dir' => RedStorm::TARGET_GEM_DIR
|
57
|
+
ant.mkdir 'dir' => RedStorm::TARGET_SPECS_DIR
|
56
58
|
ant.path 'id' => 'classpath' do
|
57
|
-
fileset 'dir' => TARGET_DEPENDENCY_DIR
|
58
|
-
fileset 'dir' => TARGET_CLASSES_DIR
|
59
|
+
fileset 'dir' => RedStorm::TARGET_DEPENDENCY_DIR
|
60
|
+
fileset 'dir' => RedStorm::TARGET_CLASSES_DIR
|
59
61
|
end
|
60
62
|
end
|
61
63
|
|
62
64
|
desc "install dependencies and compile proxy classes"
|
63
65
|
task :install, [:jvm_version] => [:deps, :build] do |t, args|
|
64
|
-
puts("\nRedStorm install completed. All dependencies installed in #{TARGET_DIR}")
|
66
|
+
puts("\nRedStorm install completed. All dependencies installed in #{RedStorm::TARGET_DIR}")
|
65
67
|
end
|
66
68
|
|
67
69
|
desc "locally install examples"
|
68
70
|
task :examples do
|
69
|
-
if File.identical?(SRC_EXAMPLES, DST_EXAMPLES)
|
71
|
+
if File.identical?(RedStorm::SRC_EXAMPLES, RedStorm::DST_EXAMPLES)
|
70
72
|
STDERR.puts("error: cannot copy examples into itself")
|
71
73
|
exit(1)
|
72
74
|
end
|
73
|
-
if File.exist?(DST_EXAMPLES)
|
74
|
-
STDERR.puts("error: directory #{DST_EXAMPLES} already exists")
|
75
|
+
if File.exist?(RedStorm::DST_EXAMPLES)
|
76
|
+
STDERR.puts("error: directory #{RedStorm::DST_EXAMPLES} already exists")
|
75
77
|
exit(1)
|
76
78
|
end
|
77
79
|
|
78
|
-
puts("\n--> Installing examples into #{DST_EXAMPLES}")
|
79
|
-
FileUtils.mkdir(DST_EXAMPLES)
|
80
|
-
FileUtils.cp_r(Dir["#{SRC_EXAMPLES}/*"], DST_EXAMPLES)
|
80
|
+
puts("\n--> Installing examples into #{RedStorm::DST_EXAMPLES}")
|
81
|
+
FileUtils.mkdir(RedStorm::DST_EXAMPLES)
|
82
|
+
FileUtils.cp_r(Dir["#{RedStorm::SRC_EXAMPLES}/*"], RedStorm::DST_EXAMPLES)
|
81
83
|
end
|
82
84
|
|
83
85
|
task :copy_red_storm do
|
84
|
-
FileUtils.cp_r(REDSTORM_LIB_DIR, TARGET_DIR)
|
86
|
+
FileUtils.cp_r(RedStorm::REDSTORM_LIB_DIR, RedStorm::TARGET_DIR)
|
85
87
|
end
|
86
88
|
|
87
89
|
desc "compile JRuby and Java proxy classes"
|
@@ -89,22 +91,22 @@ task :build, [:jvm_version] => [:setup, :copy_red_storm] do |t, args|
|
|
89
91
|
jvm_version = args[:jvm_version].to_s =~ /--(1.\d)/ ? $1 : RedStorm.java_runtime_version
|
90
92
|
|
91
93
|
# compile the JRuby proxy classes to Java
|
92
|
-
build_jruby("#{REDSTORM_LIB_DIR}/red_storm/proxy")
|
94
|
+
build_jruby("#{RedStorm::REDSTORM_LIB_DIR}/red_storm/proxy")
|
93
95
|
|
94
96
|
# compile the generated Java proxy classes
|
95
|
-
build_java_dir("#{TARGET_SRC_DIR}", jvm_version)
|
97
|
+
build_java_dir("#{RedStorm::TARGET_SRC_DIR}", jvm_version)
|
96
98
|
|
97
99
|
# generate the JRuby topology launcher
|
98
|
-
build_jruby("#{REDSTORM_LIB_DIR}/red_storm/topology_launcher.rb")
|
100
|
+
build_jruby("#{RedStorm::REDSTORM_LIB_DIR}/red_storm/topology_launcher.rb")
|
99
101
|
|
100
102
|
# compile the JRuby proxy classes
|
101
|
-
build_java_dir("#{REDSTORM_JAVA_SRC_DIR}", jvm_version)
|
103
|
+
build_java_dir("#{RedStorm::REDSTORM_JAVA_SRC_DIR}", jvm_version)
|
102
104
|
|
103
105
|
# compile the JRuby proxy classes
|
104
|
-
build_java_dir("#{TARGET_SRC_DIR}", jvm_version)
|
106
|
+
build_java_dir("#{RedStorm::TARGET_SRC_DIR}", jvm_version)
|
105
107
|
end
|
106
108
|
|
107
|
-
desc "package topology gems into #{TARGET_GEM_DIR}"
|
109
|
+
desc "package topology gems into #{RedStorm::TARGET_GEM_DIR}"
|
108
110
|
task :bundle, [:groups] => :setup do |t, args|
|
109
111
|
require 'bundler'
|
110
112
|
defaulted_args = {:groups => 'default'}.merge(args.to_hash.delete_if{|k, v| v.to_s.empty?})
|
@@ -113,9 +115,9 @@ task :bundle, [:groups] => :setup do |t, args|
|
|
113
115
|
next if spec.name == 'bundler'
|
114
116
|
|
115
117
|
# try to avoid infinite recursion
|
116
|
-
next if TARGET_GEM_DIR.start_with?(spec.full_gem_path)
|
118
|
+
next if RedStorm::TARGET_GEM_DIR.start_with?(spec.full_gem_path)
|
117
119
|
|
118
|
-
destination_path = "#{TARGET_GEM_DIR}/#{spec.full_name}"
|
120
|
+
destination_path = "#{RedStorm::TARGET_GEM_DIR}/#{spec.full_name}"
|
119
121
|
next if File.directory?(destination_path)
|
120
122
|
|
121
123
|
puts("installing gem #{spec.full_name} into #{destination_path}")
|
@@ -124,7 +126,7 @@ task :bundle, [:groups] => :setup do |t, args|
|
|
124
126
|
# copy the evaluated gemspec into the specifications/ dir (we
|
125
127
|
# may not have enough info to reconstruct once we delete the
|
126
128
|
# .git directory)
|
127
|
-
File.open(File.join(TARGET_SPECS_DIR, File.basename(spec.loaded_from)), 'w'){|f| f.write(spec.to_ruby)}
|
129
|
+
File.open(File.join(RedStorm::TARGET_SPECS_DIR, File.basename(spec.loaded_from)), 'w'){|f| f.write(spec.to_ruby)}
|
128
130
|
# strip the .git directory from git dependencies, it can be huge
|
129
131
|
FileUtils.rm_rf("#{destination_path}/.git")
|
130
132
|
end
|
@@ -132,17 +134,17 @@ end
|
|
132
134
|
|
133
135
|
namespace :ivy do
|
134
136
|
task :download do
|
135
|
-
mkdir_p DST_IVY_DIR
|
137
|
+
mkdir_p RedStorm::DST_IVY_DIR
|
136
138
|
ant.get({
|
137
|
-
'src' => "http://repo1.maven.org/maven2/org/apache/ivy/ivy/#{INSTALL_IVY_VERSION}/ivy-#{INSTALL_IVY_VERSION}.jar",
|
138
|
-
'dest' => "#{DST_IVY_DIR}/ivy-#{INSTALL_IVY_VERSION}.jar",
|
139
|
+
'src' => "http://repo1.maven.org/maven2/org/apache/ivy/ivy/#{RedStorm::INSTALL_IVY_VERSION}/ivy-#{RedStorm::INSTALL_IVY_VERSION}.jar",
|
140
|
+
'dest' => "#{RedStorm::DST_IVY_DIR}/ivy-#{RedStorm::INSTALL_IVY_VERSION}.jar",
|
139
141
|
'usetimestamp' => true,
|
140
142
|
})
|
141
143
|
end
|
142
144
|
|
143
145
|
task :install => :download do
|
144
146
|
ant.path 'id' => 'ivy.lib.path' do
|
145
|
-
fileset 'dir' => DST_IVY_DIR, 'includes' => '*.jar'
|
147
|
+
fileset 'dir' => RedStorm::DST_IVY_DIR, 'includes' => '*.jar'
|
146
148
|
end
|
147
149
|
|
148
150
|
ant.taskdef({
|
@@ -154,44 +156,44 @@ namespace :ivy do
|
|
154
156
|
end
|
155
157
|
|
156
158
|
task :ivy_config do
|
157
|
-
ant.configure 'file' => File.exists?(CUSTOM_IVY_SETTINGS) ? CUSTOM_IVY_SETTINGS : DEFAULT_IVY_SETTINGS
|
159
|
+
ant.configure 'file' => File.exists?(RedStorm::CUSTOM_IVY_SETTINGS) ? RedStorm::CUSTOM_IVY_SETTINGS : RedStorm::DEFAULT_IVY_SETTINGS
|
158
160
|
end
|
159
161
|
|
160
162
|
task :storm_deps => ["ivy:install", :ivy_config] do
|
161
163
|
puts("\n--> Installing Storm dependencies")
|
162
164
|
|
163
|
-
ant.resolve 'file' => File.exists?(CUSTOM_IVY_STORM_DEPENDENCIES) ? CUSTOM_IVY_STORM_DEPENDENCIES : DEFAULT_IVY_STORM_DEPENDENCIES
|
164
|
-
ant.retrieve 'pattern' => "#{TARGET_DEPENDENCY_DIR}/storm/[conf]/[artifact](-[classifier])-[revision].[ext]", 'sync' => "true"
|
165
|
+
ant.resolve 'file' => File.exists?(RedStorm::CUSTOM_IVY_STORM_DEPENDENCIES) ? RedStorm::CUSTOM_IVY_STORM_DEPENDENCIES : RedStorm::DEFAULT_IVY_STORM_DEPENDENCIES
|
166
|
+
ant.retrieve 'pattern' => "#{RedStorm::TARGET_DEPENDENCY_DIR}/storm/[conf]/[artifact](-[classifier])-[revision].[ext]", 'sync' => "true"
|
165
167
|
end
|
166
168
|
|
167
169
|
task :topology_deps => ["ivy:install", :ivy_config] do
|
168
170
|
puts("\n--> Installing topology dependencies")
|
169
171
|
|
170
|
-
ant.resolve 'file' => File.exists?(CUSTOM_IVY_TOPOLOGY_DEPENDENCIES) ? CUSTOM_IVY_TOPOLOGY_DEPENDENCIES : DEFAULT_IVY_TOPOLOGY_DEPENDENCIES
|
171
|
-
ant.retrieve 'pattern' => "#{TARGET_DEPENDENCY_DIR}/topology/[conf]/[artifact](-[classifier])-[revision].[ext]", 'sync' => "true"
|
172
|
+
ant.resolve 'file' => File.exists?(RedStorm::CUSTOM_IVY_TOPOLOGY_DEPENDENCIES) ? RedStorm::CUSTOM_IVY_TOPOLOGY_DEPENDENCIES : RedStorm::DEFAULT_IVY_TOPOLOGY_DEPENDENCIES
|
173
|
+
ant.retrieve 'pattern' => "#{RedStorm::TARGET_DEPENDENCY_DIR}/topology/[conf]/[artifact](-[classifier])-[revision].[ext]", 'sync' => "true"
|
172
174
|
end
|
173
175
|
|
174
|
-
desc "install storm and topology dependencies in #{TARGET_DEPENDENCY_DIR}"
|
176
|
+
desc "install storm and topology dependencies in #{RedStorm::TARGET_DEPENDENCY_DIR}"
|
175
177
|
task :deps => ["ivy:install", :ivy_config, :storm_deps, :topology_deps] do
|
176
178
|
end
|
177
179
|
|
178
|
-
desc "generate #{TARGET_CLUSTER_JAR}"
|
180
|
+
desc "generate #{RedStorm::TARGET_CLUSTER_JAR}"
|
179
181
|
task :jar, [:include_dir] => [:clean_jar] do |t, args|
|
180
|
-
puts("\n--> Generating JAR file #{TARGET_CLUSTER_JAR}")
|
182
|
+
puts("\n--> Generating JAR file #{RedStorm::TARGET_CLUSTER_JAR}")
|
181
183
|
|
182
|
-
ant.jar 'destfile' => TARGET_CLUSTER_JAR do
|
184
|
+
ant.jar 'destfile' => RedStorm::TARGET_CLUSTER_JAR do
|
183
185
|
# rejar all topology jars
|
184
186
|
Dir["target/dependency/topology/default/*.jar"].each do |jar|
|
185
187
|
puts("Extracting #{jar}")
|
186
188
|
zipfileset 'src' => jar, 'includes' => "**/*"
|
187
189
|
end
|
188
|
-
fileset 'dir' => TARGET_DIR do
|
190
|
+
fileset 'dir' => RedStorm::TARGET_DIR do
|
189
191
|
include 'name' => "gems/**"
|
190
192
|
end
|
191
|
-
fileset 'dir' => TARGET_CLASSES_DIR
|
193
|
+
fileset 'dir' => RedStorm::TARGET_CLASSES_DIR
|
192
194
|
# red_storm.rb and red_storm/* must be in root of jar so that "require 'red_storm'"
|
193
195
|
# in bolts/spouts works in jar context
|
194
|
-
fileset 'dir' => TARGET_LIB_DIR do
|
196
|
+
fileset 'dir' => RedStorm::TARGET_LIB_DIR do
|
195
197
|
exclude 'name' => "tasks/**"
|
196
198
|
end
|
197
199
|
if args[:include_dir]
|
@@ -209,7 +211,7 @@ task :jar, [:include_dir] => [:clean_jar] do |t, args|
|
|
209
211
|
end
|
210
212
|
|
211
213
|
# include complete source dir tree (note we don't care about potential duplicated resources dir)
|
212
|
-
fileset 'dir' => CWD do
|
214
|
+
fileset 'dir' => RedStorm::CWD do
|
213
215
|
dirs.each{|dir| include 'name' => "#{dir}/**/*"}
|
214
216
|
end
|
215
217
|
end
|
@@ -217,14 +219,14 @@ task :jar, [:include_dir] => [:clean_jar] do |t, args|
|
|
217
219
|
attribute 'name' => "Main-Class", 'value' => "redstorm.TopologyLauncher"
|
218
220
|
end
|
219
221
|
end
|
220
|
-
puts("\nRedStorm generated JAR file #{TARGET_CLUSTER_JAR}")
|
222
|
+
puts("\nRedStorm generated JAR file #{RedStorm::TARGET_CLUSTER_JAR}")
|
221
223
|
end
|
222
224
|
|
223
225
|
def build_java_dir(source_folder, jvm_version)
|
224
226
|
puts("\n--> Compiling Java for JVM #{jvm_version}")
|
225
227
|
ant.javac(
|
226
228
|
'srcdir' => source_folder,
|
227
|
-
'destdir' => TARGET_CLASSES_DIR,
|
229
|
+
'destdir' => RedStorm::TARGET_CLASSES_DIR,
|
228
230
|
'classpathref' => 'classpath',
|
229
231
|
'source' => jvm_version,
|
230
232
|
'target' => jvm_version,
|
@@ -242,13 +244,13 @@ def build_jruby(source_path)
|
|
242
244
|
puts("\n--> Compiling JRuby")
|
243
245
|
Dir.chdir(RedStorm::REDSTORM_HOME) do
|
244
246
|
argv = []
|
245
|
-
argv << '-t' << TARGET_SRC_DIR
|
247
|
+
argv << '-t' << RedStorm::TARGET_SRC_DIR
|
246
248
|
argv << '--verbose'
|
247
249
|
argv << '--java'
|
248
|
-
Dir["#{TARGET_DEPENDENCY_DIR}/storm/default/*.jar"].each do |jar|
|
250
|
+
Dir["#{RedStorm::TARGET_DEPENDENCY_DIR}/storm/default/*.jar"].each do |jar|
|
249
251
|
argv << '-c' << %("#{jar}")
|
250
252
|
end
|
251
|
-
argv << '-c' << %("#{TARGET_CLASSES_DIR}")
|
253
|
+
argv << '-c' << %("#{RedStorm::TARGET_CLASSES_DIR}")
|
252
254
|
argv << source_path
|
253
255
|
status = JRuby::Compiler::compile_argv(argv)
|
254
256
|
end
|
metadata
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: redstorm
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.6.6
|
5
|
-
prerelease:
|
4
|
+
version: 0.6.6
|
5
|
+
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
8
8
|
- Colin Surprenant
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2013-07-
|
12
|
+
date: 2013-07-25 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: rspec
|
@@ -131,9 +131,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
131
131
|
none: false
|
132
132
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
133
133
|
requirements:
|
134
|
-
- - '
|
134
|
+
- - '>='
|
135
135
|
- !ruby/object:Gem::Version
|
136
|
-
version:
|
136
|
+
version: '0'
|
137
137
|
none: false
|
138
138
|
requirements: []
|
139
139
|
rubyforge_project: redstorm
|