wukong-storm 0.0.2 → 0.1.0

data/.gitignore

@@ -1,2 +1,59 @@
-pkg/
+## OS
+.DS_Store
+Icon
+nohup.out
+.bak
+
+*.pem
+
+## EDITORS
+\#*
+.\#*
+\#*\#
+*~
+*.swp
+REVISION
+TAGS*
+tmtags
+*_flymake.*
+*_flymake
+*.tmproj
+.project
+.settings
+
+## COMPILED
+a.out
+*.o
+*.pyc
+*.so
+
+## OTHER SCM
+.bzr
+.hg
+.svn
+
+## PROJECT::GENERAL
+
+log/*
+tmp/*
+pkg/*
+
+coverage
+rdoc
+doc
+pkg
+.rake_test_cache
+.bundle
+.yardoc
+
+.vendor
+
+## PROJECT::SPECIFIC
+
+old/*
+docpages
+away
+
+.rbx
 Gemfile.lock
+Backup*of*.numbers

data/.yardopts

@@ -0,0 +1,5 @@
+--readme   README.md
+--markup   markdown
+-
+LICENSE.md
+README.md

data/Gemfile

@@ -5,4 +5,6 @@ gemspec
 group :development do
   gem 'rake',  '~> 0.9'
   gem 'rspec', '~> 2'
+  gem 'yard'
+  gem 'redcarpet'
 end

data/LICENSE.md

@@ -0,0 +1,95 @@
+# License for Wukong-Storm
+
+The wukong code is __Copyright (c) 2011, 2012 Infochimps, Inc__
+
+This code is licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an **AS IS** BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+
+__________________________________________________________________________
+
+# Apache License
+
+
+                              Apache License
+                        Version 2.0, January 2004
+                     http://www.apache.org/licenses/
+
+_TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION_
+
+## 1. Definitions.
+
+* **License** shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+
+* **Licensor** shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+
+* **Legal Entity** shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+* **You** (or **Your**) shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+* **Source** form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+* **Object** form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+* **Work** shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+
+* **Derivative Works** shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+* **Contribution** shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+* **Contributor** shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+## 2. Grant of Copyright License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+## 3. Grant of Patent License.
+
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+## 4. Redistribution.
+
+You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+  - (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
+  - (b) You must cause any modified files to carry prominent notices stating that You changed the files; and
+  - (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+  - (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+## 5. Submission of Contributions.
+
+Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+## 6. Trademarks.
+
+This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+## 7. Disclaimer of Warranty.
+
+Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+## 8. Limitation of Liability.
+
+In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+## 9. Accepting Warranty or Additional Liability.
+
+While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+_END OF TERMS AND CONDITIONS_
+
+## APPENDIX: How to apply the Apache License to your work.
+
+To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets `[]` replaced with your own identifying information. (Don't include the brackets!)  The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
+
+> Copyright [yyyy] [name of copyright owner]
+>
+> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+>
+>     http://www.apache.org/licenses/LICENSE-2.0
+>
+> Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

data/README.md

@@ -12,13 +12,13 @@ a storm or trident topology.
 
 wu-storm operates over STDIN and STDOUT and has a one-to-one message guarantee.
 For example, when using an identity processor, wu-storm, given an event 'foo', will return
-'foo|'. The '|' character is the specified End-Of-File delimiter. 
+'foo\n|\n'. The '|' character is the specified End-Of-File delimiter.
 
-If there is ever a suppressed error in pricessing, or a skipped record for any reason,
-wu-storm will still respond with a '|', signifying an empty return event.
+If there is ever a suppressed error in processing, or a skipped record for any reason,
+wu-storm will still respond with a '|\n', signifying an empty return event.
 
 If there are multiple messages that have resulted from a single event, wu-storm will return
-them newline separated, followed by the delimite, e.g. 'foo\nbar\nbaz|'.
+them newline separated, followed by the delimite, e.g. 'foo\nbar\nbaz\n|\n'.
 
 
 Params:

data/bin/wu-storm

@@ -1,52 +1,4 @@
 #!/usr/bin/env ruby
-require 'wukong-storm'
-require 'configliere'
-
-Settings.use(:commandline)
-Settings.define :run,       description: 'Name of the processor or dataflow to use. Defaults to basename of the given path', flag: 'r'
-Settings.define :delimiter, description: 'The EOF specifier when returning events', default: '|',                            flag: 't'
-
-def Settings.usage() "usage: #{File.basename($0)} PROCESSOR|FLOW [...--param=value...]" ; end
-  
-Settings.description = <<'EOF'
-wu-storm is a commandline tool for running Wukong processors and flows in
-a storm or trident topology.
-
-wu-storm operates over STDIN and STDOUT and has a one-to-one message guarantee.
-For example, when using an identity processor, wu-storm, given an event 'foo', will return
-'foo|'. The '|' character is the specified End-Of-File delimiter. 
-
-If there is ever a suppressed error in pricessing, or a skipped record for any reason,
-wu-storm will still respond with a '|', signifying an empty return event.
 
-If there are multiple messages that have resulted from a single event, wu-storm will return
-them newline separated, followed by the delimite, e.g. 'foo\nbar\nbaz|'.
-EOF
-
-require 'wukong/boot'
-Wukong.boot!(Settings)
-
-runnable = Settings.rest.first
-
-case
-when runnable.nil?
-  Settings.dump_help
-  exit(1)  
-when Wukong.registry.registered?(runnable.to_sym)
-  processor = runnable
-when File.exist?(runnable)
-  load runnable
-  processor = Settings.run || File.basename(runnable, '.rb')
-else
-  Settings.dump_help
-  exit(1)
-end
-
-begin
-  EM.run do
-    Wu::StormRunner.start(processor.to_sym, Settings)
-  end
-rescue Wu::Error => e
-  $stderr.puts e.message
-  exit(1)
-end
+require 'wukong-storm'
+Wukong::Storm::StormRunner.run

data/lib/wukong-storm.rb

@@ -1,3 +1,36 @@
 require 'wukong'
+
+module Wukong
+
+  # Connects Wukong to Storm.
+  module Storm
+    
+    include Plugin
+
+    # Configure the given settings object for use with Wukong::Storm.
+    #
+    # @param [Configliere::Param] settings the settings to configure
+    # @param [String] program the name of the currently executing program
+    def self.configure settings, program
+      return unless program == 'wu-storm'
+      settings.define :zookeepers_servers, description: 'storm.zookeeper.servers'
+      settings.define :zookeepers_port,    description: 'storm.zookeeper.port'
+      settings.define :local_dir,          description: 'storm.local.dir'
+      settings.define :scheduler,          description: 'storm.scheduler'
+      settings.define :cluster_mode,       description: 'storm.cluster.mode'
+      settings.define :local_hostname,     description: 'storm.local.hostname'
+      settings.define :run,                description: 'Name of the processor or dataflow to use. Defaults to basename of the given path', flag: 'r'
+      settings.define :delimiter,          description: 'Emitted as a single record to mark the end of the batch ', default: '---', flag: 't'
+    end
+
+    # Boots the Wukong::Storm plugin.
+    #
+    # @param [Configliere::Param] settings the settings to boot from
+    # @param [String] root the root directory to boot in
+    def self.boot settings, root
+    end
+    
+  end
+end
+
 require 'wukong-storm/runner'
-#  require 'wukong-storm/configuration'

data/lib/wukong-storm/driver.rb

@@ -0,0 +1,58 @@
+module Wukong
+  module Storm
+
+    # A driver to connect events passed in over STDIN to STDOUT.
+    # Differs from the vanilla Wukong::Local::LocalDriver in some
+    # Storm-specific ways.
+    class StormDriver < EM::P::LineAndTextProtocol
+      include DriverMethods
+
+      attr_accessor :dataflow, :settings
+
+      def self.start(label, settings = {})
+        EM.attach($stdin, self, label, settings)
+      end
+
+      def initialize(label, settings)
+        super
+        @settings = settings
+        @dataflow = construct_dataflow(label, settings)
+        @messages = []
+      end
+
+      def post_init
+        setup_dataflow      
+      end
+
+      def receive_line line
+        driver.send_through_dataflow(line)
+        send_messages
+      rescue => e
+        raise Wukong::Error.new(e)
+        EM.stop
+      end
+
+      def send_messages
+        # message newline message newline message delimiter
+        # message newline message newline message newline delimiter newline
+        @messages.each do |message|
+          $stdout.write(message)
+          $stdout.write("\n")
+        end
+        $stdout.write(settings.delimiter)
+        $stdout.write("\n")
+        $stdout.flush
+        @messages.clear
+      end
+
+      def unbind
+        EM.stop
+      end
+
+      def setup()                             ; end
+      def process(record) @messages << record ; end
+      def stop()                              ; end
+
+    end
+  end
+end

data/lib/wukong-storm/runner.rb

@@ -1,45 +1,40 @@
+require_relative('driver')
+
 module Wukong
-  class StormRunner < EM::P::LineAndTextProtocol
-    include DriverMethods
+  module Storm
 
-    attr_accessor :dataflow, :settings
+    # Implements the runner for wu-storm.
+    class StormRunner < Wukong::Local::LocalRunner
 
-    def self.start(label, settings = {})
-      EM.attach($stdin, self, label, settings)
-    end
+      include Wukong::Logging
 
-    def initialize(label, settings)
-      super
-      @settings = settings
-      @dataflow = construct_dataflow(label, settings)
-      @messages = []
-    end
-
-    def post_init
-      setup_dataflow      
-    end
+      usage "PROCESSOR|FLOW"
 
-    def receive_line line
-      driver.send_through_dataflow(line)
-      send_messages
-    rescue => e
-      $stderr.puts e.message
-      EM.stop
-    end
+      description <<-EOF.gsub(/^ {8}/,'')
+        wu-storm is a commandline tool for running Wukong processors and flows
+        in a storm or trident topology.
 
-    def send_messages
-      $stdout.write(@messages.join("\n") + settings.delimiter)
-      $stdout.flush
-      @messages.clear
-    end
+        wu-storm operates over STDIN and STDOUT and has a one-to-one message
+        guarantee.  For example, when using an identity processor, wu-storm,
+        given an event 'foo', will return 'foo|'. The '|' character is the
+        specified End-Of-File delimiter.
 
-    def unbind
-      EM.stop
-    end
+        If there is ever a suppressed error in pricessing, or a skipped record
+        for any reason, wu-storm will still respond with a '|', signifying an
+        empty return event.
 
-    def setup()                             ; end
-    def process(record) @messages << record ; end
-    def stop()                              ; end
+        If there are multiple messages that have resulted from a single event,
+        wu-storm will return them newline separated, followed by the
+        delimiter, e.g. 'foo\nbar\nbaz|'.
+      EOF
 
+      # :nodoc:
+      def driver
+        StormDriver
+      end
+      
+    end
   end
 end
+
+  

data/lib/wukong-storm/{configuration.rb → storm_settings.java}

@@ -1,326 +1,321 @@
-module Wukong
-  module Storm
-
-    Configuration = Configliere::Param.new unless defined? Configuration
-
-    Configuration.define :zookeepers_servers, description: 'storm.zookeeper.servers'
-    Configuration.define :zookeepers_port,    description: 'storm.zookeeper.port'
-    Configuration.define :local_dir,          description: 'storm.local.dir'
-    Configuration.define :scheduler,          description: 'storm.scheduler'
-    Configuration.define :cluster_mode,       description: 'storm.cluster.mode'
-    Configuration.define :local_hostname,     description: 'storm.local.hostname'
-
-    /**
-     * Whether or not to use ZeroMQ for messaging in local mode. If this is set 
-     * to false, then Storm will use a pure-Java messaging system. The purpose 
-     * of this flag is to make it easy to run Storm in local mode by eliminating 
-     * the need for native dependencies, which can be difficult to install.
-     *
-     * Defaults to false.
-     */
-    public static String STORM_LOCAL_MODE_ZMQ = "storm.local.mode.zmq";
-
-    /**
-     * The root location at which Storm stores data in ZooKeeper.
-     */
-    public static String STORM_ZOOKEEPER_ROOT = "storm.zookeeper.root";
-
-    /**
-     * The session timeout for clients to ZooKeeper.
-     */
-    public static String STORM_ZOOKEEPER_SESSION_TIMEOUT = "storm.zookeeper.session.timeout";
-
-    /**
-     * The connection timeout for clients to ZooKeeper.
-     */
-    public static String STORM_ZOOKEEPER_CONNECTION_TIMEOUT = "storm.zookeeper.connection.timeout";
-    
-    
-    /**
-     * The number of times to retry a Zookeeper operation.
-     */
-    public static String STORM_ZOOKEEPER_RETRY_TIMES="storm.zookeeper.retry.times";
-    
-    /**
-     * The interval between retries of a Zookeeper operation.
-     */
-    public static String STORM_ZOOKEEPER_RETRY_INTERVAL="storm.zookeeper.retry.interval";
-
-    /**
-     * The Zookeeper authentication scheme to use, e.g. "digest". Defaults to no authentication.
-     */
-    public static String STORM_ZOOKEEPER_AUTH_SCHEME="storm.zookeeper.auth.scheme";
-    
-    /**
-     * A string representing the payload for Zookeeper authentication. It gets serialized using UTF-8 encoding during authentication.
-     */
-    public static String STORM_ZOOKEEPER_AUTH_PAYLOAD="storm.zookeeper.auth.payload";
-    
-    /**
-     * The id assigned to a running topology. The id is the storm name with a unique nonce appended.
-     */
-    public static String STORM_ID = "storm.id";
-    
-    /**
-     * The host that the master server is running on.
-     */
-    public static String NIMBUS_HOST = "nimbus.host";
-
-    /**
-     * Which port the Thrift interface of Nimbus should run on. Clients should
-     * connect to this port to upload jars and submit topologies.
-     */
-    public static String NIMBUS_THRIFT_PORT = "nimbus.thrift.port";
-
-
-    /**
-     * This parameter is used by the storm-deploy project to configure the
-     * jvm options for the nimbus daemon.
-     */
-    public static String NIMBUS_CHILDOPTS = "nimbus.childopts";
-
-
-    /**
-     * How long without heartbeating a task can go before nimbus will consider the
-     * task dead and reassign it to another location.
-     */
-    public static String NIMBUS_TASK_TIMEOUT_SECS = "nimbus.task.timeout.secs";
-
-
-    /**
-     * How often nimbus should wake up to check heartbeats and do reassignments. Note
-     * that if a machine ever goes down Nimbus will immediately wake up and take action.
-     * This parameter is for checking for failures when there's no explicit event like that
-     * occuring.
-     */
-    public static String NIMBUS_MONITOR_FREQ_SECS = "nimbus.monitor.freq.secs";
-
-    /**
-     * How often nimbus should wake the cleanup thread to clean the inbox.
-     * @see NIMBUS_INBOX_JAR_EXPIRATION_SECS
-     */
-    public static String NIMBUS_CLEANUP_INBOX_FREQ_SECS = "nimbus.cleanup.inbox.freq.secs";
-
-    /**
-     * The length of time a jar file lives in the inbox before being deleted by the cleanup thread.
-     *
-     * Probably keep this value greater than or equal to NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS.
-     * Note that the time it takes to delete an inbox jar file is going to be somewhat more than
-     * NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS (depending on how often NIMBUS_CLEANUP_FREQ_SECS
-     * is set to).
-     * @see NIMBUS_CLEANUP_FREQ_SECS
-     */
-    public static String NIMBUS_INBOX_JAR_EXPIRATION_SECS = "nimbus.inbox.jar.expiration.secs";
-
-    /**
-     * How long before a supervisor can go without heartbeating before nimbus considers it dead
-     * and stops assigning new work to it.
-     */
-    public static String NIMBUS_SUPERVISOR_TIMEOUT_SECS = "nimbus.supervisor.timeout.secs";
-
-    /**
-     * A special timeout used when a task is initially launched. During launch, this is the timeout
-     * used until the first heartbeat, overriding nimbus.task.timeout.secs.
-     *
-     * <p>A separate timeout exists for launch because there can be quite a bit of overhead
-     * to launching new JVM's and configuring them.</p>
-     */
-    public static String NIMBUS_TASK_LAUNCH_SECS = "nimbus.task.launch.secs";
-
-    /**
-     * Whether or not nimbus should reassign tasks if it detects that a task goes down. 
-     * Defaults to true, and it's not recommended to change this value.
-     */
-    public static String NIMBUS_REASSIGN = "nimbus.reassign";
-
-    /**
-     * During upload/download with the master, how long an upload or download connection is idle
-     * before nimbus considers it dead and drops the connection.
-     */
-    public static String NIMBUS_FILE_COPY_EXPIRATION_SECS = "nimbus.file.copy.expiration.secs";
-
-    /**
-     * A custom class that implements ITopologyValidator that is run whenever a
-     * topology is submitted. Can be used to provide business-specific logic for
-     * whether topologies are allowed to run or not.
-     */
-    public static String NIMBUS_TOPOLOGY_VALIDATOR = "nimbus.topology.validator";
-    
-    
-    /**
-     * Storm UI binds to this port.
-     */
-    public static String UI_PORT = "ui.port";
-
-    /**
-     * Childopts for Storm UI Java process.
-     */
-    public static String UI_CHILDOPTS = "ui.childopts";
-    
-    
-    /**
-     * List of DRPC servers so that the DRPCSpout knows who to talk to.
-     */
-    public static String DRPC_SERVERS = "drpc.servers";
-
-    /**
-     * This port is used by Storm DRPC for receiving DPRC requests from clients.
-     */
-    public static String DRPC_PORT = "drpc.port";
-    
-    /**
-     * This port on Storm DRPC is used by DRPC topologies to receive function invocations and send results back. 
-     */
-    public static String DRPC_INVOCATIONS_PORT = "drpc.invocations.port";  
-    
-    /**
-     * The timeout on DRPC requests within the DRPC server. Defaults to 10 minutes. Note that requests can also
-     * timeout based on the socket timeout on the DRPC client, and separately based on the topology message
-     * timeout for the topology implementing the DRPC function.
-     */
-    public static String DRPC_REQUEST_TIMEOUT_SECS  = "drpc.request.timeout.secs";  
-    
-    /**
-     * the metadata configed on the supervisor
-     */    
-    public static String SUPERVISOR_SCHEDULER_META = "supervisor.scheduler.meta";
-    /**
-     * A list of ports that can run workers on this supervisor. Each worker uses one port, and
-     * the supervisor will only run one worker per port. Use this configuration to tune
-     * how many workers run on each machine.
-     */
-    public static String SUPERVISOR_SLOTS_PORTS = "supervisor.slots.ports";
-
-
-
-    /**
-     * This parameter is used by the storm-deploy project to configure the
-     * jvm options for the supervisor daemon.
-     */
-    public static String SUPERVISOR_CHILDOPTS = "supervisor.childopts";
-
-
-    /**
-     * How long a worker can go without heartbeating before the supervisor tries to
-     * restart the worker process.
-     */
-    public static String SUPERVISOR_WORKER_TIMEOUT_SECS = "supervisor.worker.timeout.secs";
-
-
-    /**
-     * How long a worker can go without heartbeating during the initial launch before
-     * the supervisor tries to restart the worker process. This value override
-     * supervisor.worker.timeout.secs during launch because there is additional
-     * overhead to starting and configuring the JVM on launch.
-     */
-    public static String SUPERVISOR_WORKER_START_TIMEOUT_SECS = "supervisor.worker.start.timeout.secs";
-
-
-    /**
-     * Whether or not the supervisor should launch workers assigned to it. Defaults
-     * to true -- and you should probably never change this value. This configuration
-     * is used in the Storm unit tests.
-     */
-    public static String SUPERVISOR_ENABLE = "supervisor.enable";
-
-
-    /**
-     * how often the supervisor sends a heartbeat to the master.
-     */
-    public static String SUPERVISOR_HEARTBEAT_FREQUENCY_SECS = "supervisor.heartbeat.frequency.secs";
-
-
-    /**
-     * How often the supervisor checks the worker heartbeats to see if any of them
-     * need to be restarted.
-     */
-    public static String SUPERVISOR_MONITOR_FREQUENCY_SECS = "supervisor.monitor.frequency.secs";
-    
-    /**
-     * The jvm opts provided to workers launched by this supervisor. All "%ID%" substrings are replaced
-     * with an identifier for this worker.
-     */
-    public static String WORKER_CHILDOPTS = "worker.childopts";
-
-
-    /**
-     * How often this worker should heartbeat to the supervisor.
-     */
-    public static String WORKER_HEARTBEAT_FREQUENCY_SECS = "worker.heartbeat.frequency.secs";
-
-    /**
-     * How often a task should heartbeat its status to the master.
-     */
-    public static String TASK_HEARTBEAT_FREQUENCY_SECS = "task.heartbeat.frequency.secs";
-
-
-    /**
-     * How often a task should sync its connections with other tasks (if a task is
-     * reassigned, the other tasks sending messages to it need to refresh their connections).
-     * In general though, when a reassignment happens other tasks will be notified
-     * almost immediately. This configuration is here just in case that notification doesn't
-     * come through.
-     */
-    public static String TASK_REFRESH_POLL_SECS = "task.refresh.poll.secs";
-
-    
-    
-    /**
-     * True if Storm should timeout messages or not. Defaults to true. This is meant to be used
-     * in unit tests to prevent tuples from being accidentally timed out during the test.
-     */
-    public static String TOPOLOGY_ENABLE_MESSAGE_TIMEOUTS = "topology.enable.message.timeouts";
-    
-    /**
-     * When set to true, Storm will log every message that's emitted.
-     */
-    public static String TOPOLOGY_DEBUG = "topology.debug";
-
-
-    /**
-     * Whether or not the master should optimize topologies by running multiple
-     * tasks in a single thread where appropriate.
-     */
-    public static String TOPOLOGY_OPTIMIZE = "topology.optimize";
-
-    /**
-     * How many processes should be spawned around the cluster to execute this
-     * topology. Each process will execute some number of tasks as threads within
-     * them. This parameter should be used in conjunction with the parallelism hints
-     * on each component in the topology to tune the performance of a topology.
-     */
-    public static String TOPOLOGY_WORKERS = "topology.workers";
-
-    /**
-     * How many instances to create for a spout/bolt. A task runs on a thread with zero or more
-     * other tasks for the same spout/bolt. The number of tasks for a spout/bolt is always
-     * the same throughout the lifetime of a topology, but the number of executors (threads) for 
-     * a spout/bolt can change over time. This allows a topology to scale to more or less resources 
-     * without redeploying the topology or violating the constraints of Storm (such as a fields grouping
-     * guaranteeing that the same value goes to the same task).
-     */
-    public static String TOPOLOGY_TASKS = "topology.tasks";
-
-    /**
-     * How many executors to spawn for ackers.
-     *
-     * <p>If this is set to 0, then Storm will immediately ack tuples as soon
-     * as they come off the spout, effectively disabling reliability.</p>
-     */
-    public static String TOPOLOGY_ACKER_EXECUTORS = "topology.acker.executors";
-
-
-    /**
-     * The maximum amount of time given to the topology to fully process a message
-     * emitted by a spout. If the message is not acked within this time frame, Storm
-     * will fail the message on the spout. Some spouts implementations will then replay
-     * the message at a later time.
-     */
-    public static String TOPOLOGY_MESSAGE_TIMEOUT_SECS = "topology.message.timeout.secs";
-
-    /**
-     * A list of serialization registrations for Kryo ( http://code.google.com/p/kryo/ ),
-     * the underlying serialization framework for Storm. A serialization can either
+/**
+   This file contains a bunch Storm settings ripped from the Storm
+   Java code.  Some of these might need wrapped at the Ruby layer by
+   this plugin for added configurability.  Others may not.
+*/
+
+
+/**
+ * Whether or not to use ZeroMQ for messaging in local mode. If this is set 
+ * to false, then Storm will use a pure-Java messaging system. The purpose 
+ * of this flag is to make it easy to run Storm in local mode by eliminating 
+ * the need for native dependencies, which can be difficult to install.
+ *
+ * Defaults to false.
+ */
+public static String STORM_LOCAL_MODE_ZMQ = "storm.local.mode.zmq";
+
+/**
+ * The root location at which Storm stores data in ZooKeeper.
+ */
+public static String STORM_ZOOKEEPER_ROOT = "storm.zookeeper.root";
+
+/**
+ * The session timeout for clients to ZooKeeper.
+ */
+public static String STORM_ZOOKEEPER_SESSION_TIMEOUT = "storm.zookeeper.session.timeout";
+
+/**
+ * The connection timeout for clients to ZooKeeper.
+ */
+public static String STORM_ZOOKEEPER_CONNECTION_TIMEOUT = "storm.zookeeper.connection.timeout";
+    
+    
+/**
+ * The number of times to retry a Zookeeper operation.
+ */
+public static String STORM_ZOOKEEPER_RETRY_TIMES="storm.zookeeper.retry.times";
+    
+/**
+ * The interval between retries of a Zookeeper operation.
+ */
+public static String STORM_ZOOKEEPER_RETRY_INTERVAL="storm.zookeeper.retry.interval";
+
+/**
+ * The Zookeeper authentication scheme to use, e.g. "digest". Defaults to no authentication.
+ */
+public static String STORM_ZOOKEEPER_AUTH_SCHEME="storm.zookeeper.auth.scheme";
+    
+/**
+ * A string representing the payload for Zookeeper authentication. It gets serialized using UTF-8 encoding during authentication.
+ */
+public static String STORM_ZOOKEEPER_AUTH_PAYLOAD="storm.zookeeper.auth.payload";
+    
+/**
+ * The id assigned to a running topology. The id is the storm name with a unique nonce appended.
+ */
+public static String STORM_ID = "storm.id";
+    
+/**
+ * The host that the master server is running on.
+ */
+public static String NIMBUS_HOST = "nimbus.host";
+
+/**
+ * Which port the Thrift interface of Nimbus should run on. Clients should
+ * connect to this port to upload jars and submit topologies.
+ */
+public static String NIMBUS_THRIFT_PORT = "nimbus.thrift.port";
+
+
+/**
+ * This parameter is used by the storm-deploy project to configure the
+ * jvm options for the nimbus daemon.
+ */
+public static String NIMBUS_CHILDOPTS = "nimbus.childopts";
+
+
+/**
+ * How long without heartbeating a task can go before nimbus will consider the
+ * task dead and reassign it to another location.
+ */
+public static String NIMBUS_TASK_TIMEOUT_SECS = "nimbus.task.timeout.secs";
+
+
+/**
+ * How often nimbus should wake up to check heartbeats and do reassignments. Note
+ * that if a machine ever goes down Nimbus will immediately wake up and take action.
+ * This parameter is for checking for failures when there's no explicit event like that
+ * occuring.
+ */
+public static String NIMBUS_MONITOR_FREQ_SECS = "nimbus.monitor.freq.secs";
+
+/**
+ * How often nimbus should wake the cleanup thread to clean the inbox.
+ * @see NIMBUS_INBOX_JAR_EXPIRATION_SECS
+ */
+public static String NIMBUS_CLEANUP_INBOX_FREQ_SECS = "nimbus.cleanup.inbox.freq.secs";
+
+/**
+ * The length of time a jar file lives in the inbox before being deleted by the cleanup thread.
+ *
+ * Probably keep this value greater than or equal to NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS.
+ * Note that the time it takes to delete an inbox jar file is going to be somewhat more than
+ * NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS (depending on how often NIMBUS_CLEANUP_FREQ_SECS
+ * is set to).
+ * @see NIMBUS_CLEANUP_FREQ_SECS
+ */
+public static String NIMBUS_INBOX_JAR_EXPIRATION_SECS = "nimbus.inbox.jar.expiration.secs";
+
+/**
+ * How long before a supervisor can go without heartbeating before nimbus considers it dead
+ * and stops assigning new work to it.
+ */
+public static String NIMBUS_SUPERVISOR_TIMEOUT_SECS = "nimbus.supervisor.timeout.secs";
+
+/**
+ * A special timeout used when a task is initially launched. During launch, this is the timeout
+ * used until the first heartbeat, overriding nimbus.task.timeout.secs.
+ *
+ * <p>A separate timeout exists for launch because there can be quite a bit of overhead
+ * to launching new JVM's and configuring them.</p>
+ */
+public static String NIMBUS_TASK_LAUNCH_SECS = "nimbus.task.launch.secs";
+
+/**
+ * Whether or not nimbus should reassign tasks if it detects that a task goes down. 
+ * Defaults to true, and it's not recommended to change this value.
+ */
+public static String NIMBUS_REASSIGN = "nimbus.reassign";
+
+/**
+ * During upload/download with the master, how long an upload or download connection is idle
+ * before nimbus considers it dead and drops the connection.
+ */
+public static String NIMBUS_FILE_COPY_EXPIRATION_SECS = "nimbus.file.copy.expiration.secs";
+
+/**
+ * A custom class that implements ITopologyValidator that is run whenever a
+ * topology is submitted. Can be used to provide business-specific logic for
+ * whether topologies are allowed to run or not.
+ */
+public static String NIMBUS_TOPOLOGY_VALIDATOR = "nimbus.topology.validator";
+    
+    
+/**
+ * Storm UI binds to this port.
+ */
+public static String UI_PORT = "ui.port";
+
+/**
+ * Childopts for Storm UI Java process.
+ */
+public static String UI_CHILDOPTS = "ui.childopts";
+    
+    
+/**
+ * List of DRPC servers so that the DRPCSpout knows who to talk to.
+ */
+public static String DRPC_SERVERS = "drpc.servers";
+
+/**
+ * This port is used by Storm DRPC for receiving DPRC requests from clients.
+ */
+public static String DRPC_PORT = "drpc.port";
+    
+/**
+ * This port on Storm DRPC is used by DRPC topologies to receive function invocations and send results back. 
+ */
+public static String DRPC_INVOCATIONS_PORT = "drpc.invocations.port";  
+    
+/**
+ * The timeout on DRPC requests within the DRPC server. Defaults to 10 minutes. Note that requests can also
+ * timeout based on the socket timeout on the DRPC client, and separately based on the topology message
+ * timeout for the topology implementing the DRPC function.
+ */
+public static String DRPC_REQUEST_TIMEOUT_SECS  = "drpc.request.timeout.secs";  
+    
+/**
+ * the metadata configed on the supervisor
+ */    
+public static String SUPERVISOR_SCHEDULER_META = "supervisor.scheduler.meta";
+/**
+ * A list of ports that can run workers on this supervisor. Each worker uses one port, and
+ * the supervisor will only run one worker per port. Use this configuration to tune
+ * how many workers run on each machine.
+ */
+public static String SUPERVISOR_SLOTS_PORTS = "supervisor.slots.ports";
+
+
+
+/**
+ * This parameter is used by the storm-deploy project to configure the
+ * jvm options for the supervisor daemon.
+ */
+public static String SUPERVISOR_CHILDOPTS = "supervisor.childopts";
+
+
+/**
+ * How long a worker can go without heartbeating before the supervisor tries to
+ * restart the worker process.
+ */
+public static String SUPERVISOR_WORKER_TIMEOUT_SECS = "supervisor.worker.timeout.secs";
+
+
+/**
+ * How long a worker can go without heartbeating during the initial launch before
+ * the supervisor tries to restart the worker process. This value override
+ * supervisor.worker.timeout.secs during launch because there is additional
+ * overhead to starting and configuring the JVM on launch.
+ */
+public static String SUPERVISOR_WORKER_START_TIMEOUT_SECS = "supervisor.worker.start.timeout.secs";
+
+
+/**
+ * Whether or not the supervisor should launch workers assigned to it. Defaults
+ * to true -- and you should probably never change this value. This configuration
+ * is used in the Storm unit tests.
+ */
+public static String SUPERVISOR_ENABLE = "supervisor.enable";
+
+
+/**
+ * how often the supervisor sends a heartbeat to the master.
+ */
+public static String SUPERVISOR_HEARTBEAT_FREQUENCY_SECS = "supervisor.heartbeat.frequency.secs";
+
+
+/**
+ * How often the supervisor checks the worker heartbeats to see if any of them
+ * need to be restarted.
+ */
+public static String SUPERVISOR_MONITOR_FREQUENCY_SECS = "supervisor.monitor.frequency.secs";
+    
+/**
+ * The jvm opts provided to workers launched by this supervisor. All "%ID%" substrings are replaced
+ * with an identifier for this worker.
+ */
+public static String WORKER_CHILDOPTS = "worker.childopts";
+
+
+/**
+ * How often this worker should heartbeat to the supervisor.
+ */
+public static String WORKER_HEARTBEAT_FREQUENCY_SECS = "worker.heartbeat.frequency.secs";
+
+/**
+ * How often a task should heartbeat its status to the master.
+ */
+public static String TASK_HEARTBEAT_FREQUENCY_SECS = "task.heartbeat.frequency.secs";
+
+
+/**
+ * How often a task should sync its connections with other tasks (if a task is
+ * reassigned, the other tasks sending messages to it need to refresh their connections).
+ * In general though, when a reassignment happens other tasks will be notified
+ * almost immediately. This configuration is here just in case that notification doesn't
+ * come through.
+ */
+public static String TASK_REFRESH_POLL_SECS = "task.refresh.poll.secs";
+
+    
+    
+/**
+ * True if Storm should timeout messages or not. Defaults to true. This is meant to be used
+ * in unit tests to prevent tuples from being accidentally timed out during the test.
+ */
+public static String TOPOLOGY_ENABLE_MESSAGE_TIMEOUTS = "topology.enable.message.timeouts";
+    
+/**
+ * When set to true, Storm will log every message that's emitted.
+ */
+public static String TOPOLOGY_DEBUG = "topology.debug";
+
+
+/**
+ * Whether or not the master should optimize topologies by running multiple
+ * tasks in a single thread where appropriate.
+ */
+public static String TOPOLOGY_OPTIMIZE = "topology.optimize";
+
+/**
+ * How many processes should be spawned around the cluster to execute this
+ * topology. Each process will execute some number of tasks as threads within
+ * them. This parameter should be used in conjunction with the parallelism hints
+ * on each component in the topology to tune the performance of a topology.
+ */
+public static String TOPOLOGY_WORKERS = "topology.workers";
+
+/**
+ * How many instances to create for a spout/bolt. A task runs on a thread with zero or more
+ * other tasks for the same spout/bolt. The number of tasks for a spout/bolt is always
+ * the same throughout the lifetime of a topology, but the number of executors (threads) for 
+ * a spout/bolt can change over time. This allows a topology to scale to more or less resources 
+ * without redeploying the topology or violating the constraints of Storm (such as a fields grouping
+ * guaranteeing that the same value goes to the same task).
+ */
+public static String TOPOLOGY_TASKS = "topology.tasks";
+
+/**
+ * How many executors to spawn for ackers.
+ *
+ * <p>If this is set to 0, then Storm will immediately ack tuples as soon
+ * as they come off the spout, effectively disabling reliability.</p>
+ */
+public static String TOPOLOGY_ACKER_EXECUTORS = "topology.acker.executors";
+
+
+/**
+ * The maximum amount of time given to the topology to fully process a message
+ * emitted by a spout. If the message is not acked within this time frame, Storm
+ * will fail the message on the spout. Some spouts implementations will then replay
+ * the message at a later time.
+ */
+public static String TOPOLOGY_MESSAGE_TIMEOUT_SECS = "topology.message.timeout.secs";
+
+/**
+ * A list of serialization registrations for Kryo ( http://code.google.com/p/kryo/ ),
+ * the underlying serialization framework for Storm. A serialization can either
      * be the name of a class (in which case Kryo will automatically create a serializer for the class
      * that saves all the object's fields), or an implementation of com.esotericsoftware.kryo.Serializer.
      *
@@ -670,7 +665,3 @@ module Wukong
         return ret;
     }
 }
-
-
-  end
-end