redstorm 0.6.6.beta1 → 0.6.6.beta2

data/CHANGELOG.md

@@ -82,7 +82,7 @@
 - issue #73, external jars dependencies Ivy configurations completely externalized and configurable
 - lots of other [bug fixes and improvements](https://github.com/colinsurprenant/redstorm/issues?milestone=9&page=1&state=closed)
 
-# 0.6.6, WIP
+# 0.6.6.beta1, 07-02-2013
 - Storm 0.9.0-wip16 and JRuby 1.7.4
 - [issue #79](https://github.com/colinsurprenant/redstorm/issues/79) - fix example Kafka topology dependency doc
 - [issue #82](https://github.com/colinsurprenant/redstorm/issues/82) - configurable javac compilation source/target
@@ -92,3 +92,6 @@
 - fixed FFI support
 - exposed topology builder for specs
 - [issue #11](https://github.com/colinsurprenant/redstorm/issues/11), [#17](https://github.com/colinsurprenant/redstorm/issues/17) -  example specs using the Storm testing API see [redstorm-starter](https://github.com/colinsurprenant/redstorm-starter/)
+
+# 0.6.6.beta2, 07-20-2013
+- [issue #76](https://github.com/colinsurprenant/redstorm/issues/76) - avoid shelling out to storm jar command for cluster submission

data/README.md

@@ -1,4 +1,4 @@
-# RedStorm v0.6.6.beta1 - JRuby on Storm
+# RedStorm v0.6.6.beta2 - JRuby on Storm
 
 [![build status](https://secure.travis-ci.org/colinsurprenant/redstorm.png)](http://travis-ci.org/colinsurprenant/redstorm)
 
@@ -13,6 +13,11 @@ Check also these related projects:
 
 ## Documentation
 
+---
+This is the documentation for the **current WIP version of RedStorm** - the **[latest released Gem is v0.6.5](wiki/RedStorm-Gem-v0.6.5-Documentation)**
+
+---
+
 Chances are new versions of RedStorm will introduce changes that will break compatibility or change the developement workflow. To prevent out-of-sync documentation, per version specific documentation are kept [in the wiki](https://github.com/colinsurprenant/redstorm/wiki) when necessary.
 
 ## Dependencies
@@ -24,14 +29,14 @@ Tested on **OSX 10.8.3** and **Ubuntu Linux 12.10** using **Storm 0.9.0-wip16**
 - RubyGems
 
   ``` sh
-  $ gem install redstorm
+  $ gem install redstorm --pre
   ```
 
 - Bundler
 
   ``` ruby
   source "https://rubygems.org"
-  gem "redstorm", "~> 0.6.6.beta1"
+  gem "redstorm", "~> 0.6.6.beta2"
   ```
 
 ## Usage
@@ -60,6 +65,12 @@ $ redstorm install
 
 This will install default Java jar dependencies in `target/dependency`, generate & compile the Java bindings in `target/classes`.
 
+By default, the Java compilation will use the current JVM version as source/target compilation compatibility. If you want to force a specific source/target version, use the `--[JVM VERSION]` option. For example, to force compiling for 1.6 use:
+
+``` sh
+$ redstorm install --1.6
+```
+
 ### Create a topology
 
 Create a subdirectory for your topology code and create your topology class **using this naming convention**: *underscore* topology_class_file_name.rb **MUST** correspond to its *CamelCase* class name.
@@ -76,7 +87,7 @@ end
 
 class HelloWorldBolt < RedStorm::DSL::Bolt
   on_receive :emit => false do |tuple|
-    log.info(tuple.getString(0))
+    log.info(tuple[0])
   end
 end
 
@@ -185,30 +196,34 @@ $ java -Djruby.compat.version=RUBY1_9 -cp "target/classes:target/dependency/stor
 
 ### Run on production cluster
 
-The Storm distribution is currently required for the cluster topology submission.
-
-1. download and unpack the [Storm 0.9.0-wip16 distribution](https://dl.dropbox.com/u/133901206/storm-0.9.0-wip16.zip) locally
+Locally installing the Storm distribution is **not required**. Note that RedStorm does not provide the Storm command line utilities and you will need to install the Storm distribution to use the Storm command line utilities.
 
-2. add the Storm `bin/` directory to your `$PATH`
-
-3. create `~/.storm/storm.yaml` and add the following
+1. create the Storm config file `~/.storm/storm.yaml` and add the following
 
   ```yaml
   nimbus.host: "host_name_or_ip"
   ```
 
-4. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
+  you can also use an alternate path and use the `--config <other/path/to/config.yaml>`
+
+2. generate `target/cluster-topology.jar`. This jar file will include your sources directory plus the required dependencies
 
   ``` sh
   $ redstorm jar <sources_directory1> <sources_directory2> ...
   ```
 
-5. submit the cluster topology jar file to the cluster
+3. submit the cluster topology jar file to the cluster
 
   ``` sh
   $ redstorm cluster <sources_directory/topology_class_file_name.rb>
   ```
 
+  or if you have an alternate Storm config path:
+
+  ``` sh
+  $ redstorm cluster --config <some/other/path/to/config.yaml> <sources_directory/topology_class_file_name.rb>
+  ```
+
   note that the cluster topology jar can also be submitted using the storm command with:
 
   ``` sh
@@ -273,13 +288,13 @@ All examples using the [DSL](https://github.com/colinsurprenant/redstorm/wiki/Ru
 
 #### Topologies without gems
 
-1. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
+1. genererate the `target/cluster-topology.jar` and include the `examples/` directory
 
   ``` sh
   $ redstorm jar examples
   ```
 
-2. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+2. submit the cluster topology jar file to the cluster
 
   ``` sh
   $ redstorm cluster examples/dsl/exclamation_topology.rb
@@ -309,13 +324,13 @@ For `examples/dsl/redis_word_count_topology.rb` the `redis` gem is required and
   $ redstorm bundle word_count
   ```
 
-3. genererate the `target/cluster-topology.jar` and include the `examples/` directory.
+3. genererate the `target/cluster-topology.jar` and include the `examples/` directory
 
   ``` sh
   $ redstorm jar examples
   ```
 
-4. submit the cluster topology jar file to the cluster, assuming you have the Storm distribution installed and the Storm `bin/` directory in your path:
+4. submit the cluster topology jar file to the cluster
 
   ``` sh
   $ redstorm cluster examples/dsl/redis_word_count_topology.rb
@@ -404,6 +419,12 @@ It is possible to fork the RedStorm project and run local and remote/cluster top
   $ bundle exec redstorm build
   ```
 
+  By default, the Java compilation will use the current JVM version as source/target compilation compatibility. If you want to force a specific source/target version, use the `--[JVM VERSION]` option. For example, to force compiling for 1.6 use:
+
+  ``` sh
+  $ bundle exec redstorm build --1.6
+  ```
+
   **if you modify any of the RedStorm Ruby code or Java binding code**, you need to run this to refresh code and rebuild the bindings
 
 - follow the normal usage patterns to run the topology in local or remote cluster.

data/lib/red_storm/application.rb

@@ -27,6 +27,8 @@ CUSTOM_IVY_STORM_DEPENDENCIES = "#{DST_IVY_DIR}/storm_dependencies.xml"
 DEFAULT_IVY_TOPOLOGY_DEPENDENCIES = "#{SRC_IVY_DIR}/topology_dependencies.xml"
 CUSTOM_IVY_TOPOLOGY_DEPENDENCIES = "#{DST_IVY_DIR}/topology_dependencies.xml"
 
+DEFAULT_STORM_CONF_FILE = File.expand_path("~/.storm/storm.yaml")
+
 module RedStorm
 
   class Application
@@ -34,11 +36,11 @@ module RedStorm
 
     def self.local_storm_command(class_file, ruby_mode = nil)
       src_dir = File.expand_path(File.dirname(class_file))
-      "java -Djruby.compat.version=#{RedStorm.jruby_mode_token(ruby_mode)} -cp \"#{TARGET_CLASSES_DIR}:#{TARGET_DEPENDENCY_DIR}/storm/default/*:#{TARGET_DEPENDENCY_DIR}/topology/default/*:#{src_dir}/\" redstorm.TopologyLauncher local #{class_file}"
+      "java -server -Djruby.compat.version=#{RedStorm.jruby_mode_token(ruby_mode)} -cp \"#{TARGET_CLASSES_DIR}:#{TARGET_DEPENDENCY_DIR}/storm/default/*:#{TARGET_DEPENDENCY_DIR}/topology/default/*:#{src_dir}/\" redstorm.TopologyLauncher local #{class_file}"
     end
 
-    def self.cluster_storm_command(class_file, ruby_mode = nil)
-      "storm jar #{TARGET_CLUSTER_JAR} -Djruby.compat.version=#{RedStorm.jruby_mode_token(ruby_mode)} redstorm.TopologyLauncher cluster #{class_file}"
+    def self.cluster_storm_command(storm_conf, class_file, ruby_mode = nil)
+      "java -client -Dstorm.conf.file=#{File.basename(storm_conf)} -Dstorm.jar=#{TARGET_CLUSTER_JAR} -Djruby.compat.version=#{RedStorm.jruby_mode_token(ruby_mode)} -cp #{TARGET_DEPENDENCY_DIR}/storm/default/*:#{TARGET_CLUSTER_JAR}:#{File.dirname(storm_conf)} redstorm.TopologyLauncher cluster #{class_file}"
     end
 
     def self.usage
@@ -50,10 +52,12 @@ module RedStorm
       puts("       redstorm bundle [BUNDLER_GROUP]")
       puts("       redstorm jar DIR1, [DIR2, ...]")
       puts("       redstorm local [--1.8|--1.9] TOPOLOGY_CLASS_PATH")
-      puts("       redstorm cluster [--1.8|--1.9] TOPOLOGY_CLASS_PATH")
+      puts("       redstorm cluster [--1.8|--1.9] [--config STORM_CONFIG_PATH] TOPOLOGY_CLASS_PATH")
       exit(1)
     end
 
+    # TODO: refactor args parsing... becoming a mess.
+
     def self.run(args)
       if args.size > 0
         if args[0] == "version"
@@ -66,10 +70,11 @@ module RedStorm
         elsif args.size >= 2 && ["local", "cluster"].include?(args[0])
           env = args.delete_at(0)
           version = args.delete("--1.8") || args.delete("--1.9")
+          storm_conf = args.delete("--config") ? File.expand_path(args.delete_at(0)) : DEFAULT_STORM_CONF_FILE
           if args.size == 1
             file = args[0]
             load(TASKS_FILE)
-            Rake::Task['launch'].invoke(env, version, file)
+            Rake::Task['launch'].invoke(env, storm_conf, version, file)
             exit
           end
         end

data/lib/red_storm/version.rb

@@ -1,3 +1,3 @@
 module RedStorm
-  VERSION = '0.6.6.beta1'
+  VERSION = '0.6.6.beta2'
 end

data/lib/tasks/red_storm.rake

@@ -12,7 +12,7 @@ require 'red_storm/application'
 
 INSTALL_IVY_VERSION = "2.3.0"
 
-task :launch, :env, :ruby_mode, :class_file do |t, args|
+task :launch, :env, :storm_conf, :ruby_mode, :class_file do |t, args|
   # use ruby mode parameter or default to current interpreter version
   version_token = RedStorm.jruby_mode_token(args[:ruby_mode])
 
@@ -24,7 +24,11 @@ task :launch, :env, :ruby_mode, :class_file do |t, args|
       puts("error: cluster jar file #{TARGET_CLUSTER_JAR} not found. Generate it using $redstorm jar DIR1 [DIR2, ...]")
       exit(1)
     end
-    RedStorm::Application.cluster_storm_command(args[:class_file], args[:ruby_mode])
+    unless File.exist?(args[:storm_conf])
+      puts("error: Storm config file #{args[:storm_conf]} not found. Create it or supply alternate path using $redstorm cluster --config STORM_CONFIG_PATH ...")
+      exit(1)
+    end
+    RedStorm::Application.cluster_storm_command(args[:storm_conf], args[:class_file], args[:ruby_mode])
   end
 
   puts("launching #{command}")

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: redstorm
 version: !ruby/object:Gem::Version
-  version: 0.6.6.beta1
+  version: 0.6.6.beta2
   prerelease: 6
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2013-07-02 00:00:00.000000000 Z
+date: 2013-07-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec