wukong-storm 0.1.1 → 0.2.0

data/.gitignore

@@ -57,3 +57,4 @@ away
 .rbx
 Gemfile.lock
 Backup*of*.numbers
+target/*

data/.rspec

@@ -1,3 +1,2 @@
---format documentation
+--format progress
 --color
---drb

data/Gemfile

@@ -1,4 +1,4 @@
-source :rubygems
+source 'https://rubygems.org'
 
 gemspec
 

data/README.md

@@ -1,31 +1,187 @@
-# Wukong Storm
+# Wukong-Storm
 
-## Usage
+The Hadoop plugin for Wukong lets you run <a
+href="http://github.com/infochimps-labs/wukong/tree/3.0.0">Wukong</a>
+processors and dataflows as <a
+href="https://github.com/nathanmarz/storm">Storm</a> topologies reading data in and out from <a href="http://kafka.apache.org/">Kafka</a>.
 
-The Wukong Storm plugin is very basic at the moment. It functions entirely over STDIN and STDOUT. Taken from the `wu-storm` executable:
+Before you use Wukong-Storm to develop, test, and write your Hadoop
+jobs, you might want to read about <a
+href="http://github.com/infochimps-labs/wukong/tree/3.0.0">Wukong</a>,
+write some <a
+href="http://github.com/infochimps-labs/wukong/tree/3.0.0#writing-simple-processors">simple
+processors</a>, and read about some of Storm's <a
+href="https://github.com/nathanmarz/storm/wiki/Concepts">core
+concepts</a>.
 
+You might also want to check out some other projects which enrich the
+Wukong and Hadoop experience:
+
+* <a href="http://github.com/infochimps-labs/wukong-hadoop">wukong-hadoop</a>: Run Wukong processors and dataflows as mappers and/or reducers within the Hadoop framework.  Model jobs locally before you run them.
+* <a href="http://github.com/infochimps-labs/wukong-load">wukong-load</a>: Load the output data from your local Wukong jobs and flows into a variety of different data stores.
+* <a href="http://github.com/infochimps-labs/wukong-deploy">wukong-deploy</a>: Orchestrate Wukong and other wu-tools together to support an application running on the Infochimps Platform.
+
+<a name="installation"></a>
+## Installation & Setup
+
+Wukong-Storm can be installed as a RubyGem:
+
+```
+$ sudo gem install wukong-storm
 ```
-usage: wu-storm PROCESSOR|FLOW [...--param=value...]
 
-wu-storm is a commandline tool for running Wukong processors and flows in
-a storm or trident topology.
+If you actually want to run your dataflows as functioning Storm
+topologies reading/writing to/from Kafka, you'll of course need access
+to Storm and Kafka installations.  <a
+href="http://github.com/infochimps-labs/ironfan">Ironfan</a> is a
+great tool for building and managing Storm clusters and other
+distributed infrastructure quickly and easily.
+
+To run Storm jobs through Wukong-Storm, you'll need to move your your
+Wukong code to each worker of the Storm cluster, install Wukong-Storm
+on each, and log in and launch your job fron one of them.  Ironfan
+again helps with configuring this.
 
-wu-storm operates over STDIN and STDOUT and has a one-to-one message guarantee.
-For example, when using an identity processor, wu-storm, given an event 'foo', will return
-'foo\n|\n'. The '|' character is the specified End-Of-File delimiter.
+<a name="anatomy"></a>
+## Anatomy of a running topology
 
-If there is ever a suppressed error in processing, or a skipped record for any reason,
-wu-storm will still respond with a '|\n', signifying an empty return event.
+Storm defines the concept of a **topology**.  A topology contains
+spouts and bolts.  A **spout** is a source of data.  A **bolt**
+processes data.  Bolts can be connected to each other and to spouts in
+arbitrary ways.
 
-If there are multiple messages that have resulted from a single event, wu-storm will return
-them newline separated, followed by the delimite, e.g. 'foo\nbar\nbaz\n|\n'.
+Tooplogies submitted to Storm's Nimbus but run within a Storm
+supervisor.  Each supervisor can dedicate a certain number of
+**workers** to a topology. Within each worker, **parallelism**
+controls the number of threads the worker assigns to the topology.
 
+Wukong-Storm runs each Wukong dataflow as a single bolt within a
+single topology.  Data is passed to this bolt over STDIN and collected
+over STDOUT, similar to the way <a
+href="http://hadoop.apache.org/docs/r0.15.2/streaming.html">Hadoop
+streaming </a> operates.
 
-Params:
-   -t, --delimiter=String       The EOF specifier when returning events [Default: |]
-   -r, --run=String             Name of the processor or dataflow to use. Defaults to basename of the given path
+This topology is hooked up to a
+`storm.kafka.trident.OpaqueTridentKafkaSpout` (part of
+[storm-contrib](https://github.com/nathanmarz/storm-contrib)) which
+reads from a single input topic within Kafka.
+
+Output records are written to a default Kafka topic but this can be
+overridden on a per-record basis.
+
+<a name="protocol"></a>
+### Communication protocol
+
+A Wukong dataflow launched within Storm runs as a single bolt (see
+[`com.infochimps.wukong.storm.SubprocessFunction`](https://github.com/infochimps-labs/wukong-storm/blob/master/src/main/java/com/infochimps/wukong/storm/SubprocessFunction.java)).
+This bolt works by launching an arbitrary command-line and sending it
+records over STDIN and reading its output over STDOUT.  The
+`SubprocessFunction` class expects whatever command it launched to
+obey a protocol under which the output after **each** input consists
+of each output record followed by a newline, with the full batch of
+output records followed by a batch terminator (default: `---`) then
+another newline.
+
+Wukong-Storm comes with a command `wu-bolt` which works very similarly
+to `wu-local` but implements this protocol.  Here's an example of
+using `wu-bolt` directly with a processor:
+
+```
+$ echo 2 | wu-bolt prime_factorizer.rb
+2
+---
+$ echo 12 | wu-bolt prime_factorizer.rb
+2
+2
+3
+---
+$ echo 19 | wu-bolt prime_factorizer.rb
+---
 ```
 
-## TODO
+Notice that in the last example, the presence of the batch delimiter
+after each input record make it easy to tell the difference between
+"no output records" and "no output records yet" which, over
+STDIN/STDOUT, is rather hard to tell otherwise.
+
+## Running a dataflow
+
+### A simple processor
+
+Assuming you have correctly installed Wukong-Storm, Storm, Kafka,
+Zookeeper, &c., and you have defined a simple dataflow (or in this
+case, just a single processor) like this:
+
+```ruby
+# in upcaser.rb
+Wukong.processor(:upcaser) do
+  def process line
+    yield line.upcase
+  end
+end
+```
+
+Then you can launch it directly into Storm:
+
+```
+$ wu-storm upcaser.rb --input=some_input_topic --output=some_output_topic
+```
+
+If a topology named `upcaser` already exists, you'll get an error.
+Add the `--rm` flag to first kill the running topology before
+launching the new one:
+
+```
+$ wu-storm upcaser.rb --input=some_input_topic --output=some_output_topic --rm
+```
+
+The default amount of time to wait for the topology to die is 300
+seconds (5 minutes), just like the `storm kill` command (which is used
+under the hood).  When debugging a topology in development, it's
+helpful to add `--wait=1` to immediately kill the topology.
+
+See exactly what happened behind the scenes by adding the `--dry_run`
+flag which will print commands and not execute them:
+
+```
+$ wu-storm upcaser.rb --input=some_input_topic --output=some_output_topic --rm --dry_run
+```
+
+### A more complicated example
+
+Say you have a dataflow:
+
+```ruby
+# in my_flow.rb
+Wukong.dataflow(:my_flow) do
+  my_parser | does_something | then_something_else | to_json
+end
+```
+
+You can launch it using a different topology name as well as target
+arbitrary locations for your Zookeeper, Kafka, and Storm servers:
+
+```
+$ wu-storm my_flow.rb --name=my_flow_attempt_3 --zookeeper_hosts=10.121.121.121,10.122.122.122 --kafka_hosts=10.123.123.123 --nimbus_host=10.124.124.124 --input=some_input_topic --output=some_output_topic
+```
+
+### Running non-Wukong or non-Ruby code
+
+You can also use Wukong-Storm as a harness to run non-Wukong or
+non-Ruby code.  As long as you can specificy a command-line to run
+which supports the [communication protocol](#protocol), then you can
+run it with `wu-storm`:
+
+```
+$ wu-storm --bolt_command='my_cmd --some-option=value -af -q 3' --input=some_input_topic --output=some_output_topic
+```
+
+### Scaling options
+
+Storm provides several options for scaling up or down a topology.
+Wukong-Storm makes them accessible at launch time via the following
+options:
 
-The configuration file has __all__ of the options for storm listed. Slowly translating into real Configliere options.
+* `--workers` specify the number of workers (a.k.a. "executors" or "slots") for the topology.  Defaults to 1.
+* `--input_parallelism` specify the number of threads within the spout reading from Kafka within each worker.  Defaults to 1.
+* `--parallelism` specify the number of threads within the bolt running Wukong code within each worker.  Defaults to 1.

data/bin/wu-bolt

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+
+require 'wukong-storm'
+Wukong::Storm::StormBoltRunner.run

data/lib/wukong-storm.rb

@@ -12,15 +12,54 @@ module Wukong
     # @param [Configliere::Param] settings the settings to configure
     # @param [String] program the name of the currently executing program
     def self.configure settings, program
-      return unless program == 'wu-storm'
-      settings.define :zookeepers_servers, description: 'storm.zookeeper.servers'
-      settings.define :zookeepers_port,    description: 'storm.zookeeper.port'
-      settings.define :local_dir,          description: 'storm.local.dir'
-      settings.define :scheduler,          description: 'storm.scheduler'
-      settings.define :cluster_mode,       description: 'storm.cluster.mode'
-      settings.define :local_hostname,     description: 'storm.local.hostname'
-      settings.define :run,                description: 'Name of the processor or dataflow to use. Defaults to basename of the given path', flag: 'r'
-      settings.define :delimiter,          description: 'Emitted as a single record to mark the end of the batch ', default: '---', flag: 't'
+      case program
+      when 'wu-bolt'
+        settings.define :run,                description: 'Name of the processor or dataflow to use. Defaults to basename of the given path', flag: 'r'
+        settings.define :delimiter,          description: 'Emitted as a single record to mark the end of the batch ', default: 'X', flag: 't'
+      when 'wu-storm'
+        settings.define :name,               wukong_storm: true, description: "Name for the launched topology"
+        settings.define :command_prefix,     wukong_storm: true, description: "Prefix to insert before all Wukong commands"
+        settings.define :bolt_command,       wukong_storm: true, description: "Command-line to run within the spawned Storm bolt"
+        settings.define :dry_run,            wukong_storm: true, description: "Echo commands that will be run, but don't run them", type: :boolean, default: false
+        settings.define :wait,               wukong_storm: true, description: "How many seconds to wait when killing a topology",  type: Integer, default: 300
+        settings.define :rm,                 wukong_storm: true, description: "Will kill any running topology of the same name before launching", type: :boolean, default: false
+        settings.define :delimiter,          wukong_storm: true, description: "Batch delimiter to use with wu-bolt"
+        settings.define :parallelism,        wukong_storm: true, description: "Parallelism hint for wu-bolt", default: 1
+
+        settings.define :input,              wukong_storm: true, description: "Input URI for the topology.  The scheme of the URI determines the type of spout."
+        settings.define :input_parallelism,  wukong_storm: true, description: "Parallelism (number of simultaneous threads) reading input.  Only used by some spouts.", default: 1
+        settings.define :offset,             wukong_storm: true, description: "Offset to use when starting to read from input.  Interpreted in a spout-dependent way."
+
+        settings.define :from_beginning,     wukong_storm: true, description: "Start reading from the beginning of the input.", type: :boolean, default: false
+        settings.define :from_end,           wukong_storm: true, description: "Start reading from the end of the input.", type: :boolean, default: false
+        settings.define :resume,             wukong_storm: true, description: "Start reading from where the topology left off.  This is the default behavior.", type: :boolean, default: true
+        
+        settings.define :kafka_partitions,   wukong_storm: true, description: "Number of Kafka partitions on the input topic", default: 1
+        settings.define :kafka_batch,        wukong_storm: true, description: "Batch size when reading from input topic (bytes)", default: 1_048_576
+
+        settings.define :aws_key,            wukong_storm: true, description: "AWS access key. (Required for S3 input)"
+        settings.define :aws_secret,         wukong_storm: true, description: "AWS secret key. (Required for S3 input)"
+        settings.define :aws_region,         wukong_storm: true, description: "AWS region, one of: us-east-1, us-west-[1,2], eu-west-1, ap-southeast-[1,2], ap-northeast-1, sa-east-1.  (Required for S3 input)", default: 'us-east-1'
+        
+        settings.define :output,             wukong_storm: true, description: "Output URI for the topology.  The schee of the URI determines the type of state used."
+        
+        settings.define :debug,              wukong_storm: true, storm: true, description: 'topology.debug'
+        settings.define :optimize,           wukong_storm: true, storm: true, description: 'topology.optimize'
+        settings.define :timeout,            wukong_storm: true, storm: true, description: 'topology.message.timeout.secs'
+        settings.define :workers,            wukong_storm: true, storm: true, description: 'topology.workers'
+        settings.define :worker_opts,        wukong_storm: true, storm: true, description: 'topology.worker.childopts'
+        settings.define :ackers,             wukong_storm: true, storm: true, description: 'topology.acker.executors'
+        settings.define :sample_rate,        wukong_storm: true, storm: true, description: 'topology.stats.sample.rate'
+        
+        settings.define :nimbus_host,        wukong_storm: true, storm: true, description: 'nimbus.host',                default: 'localhost'
+        settings.define :nimbus_port,        wukong_storm: true, storm: true, description: 'nimbus.thrift.port',         default: 6627
+        settings.define :kafka_hosts,        wukong_storm: true, description: "Comma-separated list of Kafka hosts",     default: 'localhost'
+        settings.define :zookeeper_hosts,    wukong_storm: true, description: "Comma-separated list of Zookeeper hosts", default: 'localhost'
+
+        settings.define :storm_home,         wukong_storm: true, description: "Path to Storm installation", env_var: "STORM_HOME", default: "/usr/lib/storm"
+        settings.define :storm_runner,       wukong_storm: true, description: "Path to Storm executable.  Use this for non-standard Storm installations"
+        
+      end
     end
 
     # Boots the Wukong::Storm plugin.
@@ -33,4 +72,5 @@ module Wukong
   end
 end
 
-require 'wukong-storm/runner'
+require 'wukong-storm/storm_runner'
+require 'wukong-storm/bolt_runner'

data/lib/wukong-storm/bolt_driver.rb

@@ -0,0 +1,81 @@
+module Wukong
+  module Storm
+
+    # Modifies the behavior of Wukong::Local::StdioDriver by appending
+    # a batch delimiter after each set of output records, including
+    # when there are 0 output records or if an error occurs.
+    class BoltDriver < Local::StdioDriver
+      
+      include Logging
+
+      #
+      # == Startup == 
+      #
+      
+      # Override the behavior of StdioDriver by initializing an empty
+      # array of output records.
+      def initialize(label, settings)
+        super(label, settings)
+        @output = []
+      end
+
+      # Do *not* sync $stdout as in the StdioDriver.
+      def setup()
+      end
+
+      #
+      # == Reading Input == 
+      #
+      
+      # Called by EventMachine framework after successfully reading a
+      # line from $stdin.
+      #
+      # Relies on StdioDriver, but calls #write_output afterwards to
+      # ensure that a delimiter is also sent.
+      #
+      # @param [String] line
+      def receive_line line
+        super(line)
+        write_output
+      end
+
+      #
+      # == Handling Output == 
+      #
+
+      # Don't write the record to $stdout, but store it in an array of
+      # output records instead.
+      #
+      # @param [Object] record
+      # 
+      # @see #write_output
+      def process(record)
+        @output << record
+      end
+      
+      # Writes all output records out in a single batch write with a
+      # batch delimiter appended to the end.
+      #
+      # All output records are newline delimited within the batch.
+      #
+      # The batch itself includes a newline character after the final
+      # batch delimiter.
+      #
+      # $stdout is flushed after the write and accumulated outputs are
+      # cleared.
+      #
+      # @see #process
+      def write_output
+        @output.each do |record|
+          $stdout.write(record)
+          $stdout.write("\n")
+        end
+        $stdout.write(settings.delimiter)
+        $stdout.write("\n")
+        $stdout.flush
+        @output.clear
+      end
+
+    end
+  end
+end

data/lib/wukong-storm/bolt_runner.rb

@@ -0,0 +1,44 @@
+require_relative('bolt_driver')
+
+module Wukong
+  module Storm
+
+    # Implements the runner for wu-bolt.
+    class StormBoltRunner < Wukong::Local::LocalRunner
+
+      include Logging
+
+      usage "PROCESSOR|FLOW"
+
+      description <<-EOF.gsub(/^ {8}/,'')
+        wu-bolt is a commandline tool for running Wukong dataflows as
+        bolts within a Storm topology.
+
+        wu-bolt behaves like wu-local except it adds a batch
+        terminator after the output generated from each input record.
+        This allows Storm to differentiate "no output" from "no output
+        yet", important for back-propagating acks.
+
+        For example
+
+          $ echo "adds a terminator" | wu-bolt tokenizer.rb
+          adds
+          a
+          terminator
+          ---
+          $ echo "" | wu-bolt tokenizer.rb
+          ---
+
+        If there is ever a suppressed error in pricessing, or a
+        skipped record for any reason, wu-bolt will still output the
+        batch terminator.
+      EOF
+
+      # :nodoc:
+      def driver
+        BoltDriver
+      end
+      
+    end
+  end
+end

data/lib/wukong-storm/storm_invocation.rb

@@ -0,0 +1,386 @@
+require 'shellwords'
+
+module Wukong
+  module Storm
+
+    # This module defines several methods that generate command lines
+    # that interact with Storm using the `storm` program.
+    module StormInvocation
+
+      #
+      # == Topology Structure & Properties
+      #
+
+      # Return the name of the Storm topology from the given settings
+      # and/or commandline args.
+      #
+      # @return [String] the name of the Storm topology
+      def topology_name
+        settings[:name] || dataflow
+      end
+
+      # Name of the Wukong dataflow to be launched.
+      #
+      # Obtained from either the first non-option argument passed to
+      # `wu-storm` or the `--run` option.
+      #
+      # @return [String]
+      def dataflow_name
+        args.first || settings[:run]
+      end
+
+      # The input URI for the topology.  Will determine the Trident
+      # spout that will be used.
+      #
+      # @return [URI]
+      def input_uri
+        @input_uri ||= URI.parse(settings[:input])
+      end
+
+      # Does this topology read from Kafka?
+      #
+      # @return [true, false]
+      def kafka_input?
+        ! blob_input?
+      end
+      
+      # Does this topology read from a filesystem?
+      #
+      # @return [true, false]
+      def blob_input?
+        s3_input? || file_input?
+      end
+      
+      # Does this topology read from Amazon's S3?
+      #
+      # @return [true, false]
+      def s3_input?
+        input_uri.scheme == 's3'
+      end
+
+      # Does this topology read from a local filesystem?
+      #
+      # @return [true, false]
+      def file_input?
+        input_uri.scheme == 'file'
+      end
+
+      # The input URI for the topology.  Will determine the Trident
+      # state that will be used.
+      #
+      # @return [URI]
+      def output_uri
+        @output_uri ||= URI.parse(settings[:output])
+      end
+      
+      # Does this topology write to Kafka?
+      #
+      # @return [true, false]
+      def kafka_output?
+        true                    # only option right now
+      end
+
+      #
+      # == Interaction w/Storm ==
+      #
+      
+      # Generates a commandline that can be used to launch a new Storm
+      # topology based on the given dataflow, input and output topics,
+      # and settings.
+      #
+      # @return [String]
+      def storm_launch_commandline
+        [
+         storm_runner,
+         "jar #{wukong_topology_submitter_jar}",
+         fully_qualified_class_name,
+         native_storm_options,
+         storm_topology_options,
+        ].flatten.compact.join("\ \t\\\n ")
+      end
+
+      # Generates a commandline that can be used to kill a running
+      # Storm topology based on the given topology name.
+      #
+      # @return [String]
+      def storm_kill_commandline
+        "#{storm_runner} kill #{topology_name} #{storm_kill_options} > /dev/null 2>&1"
+      end
+
+      # Generates the commandline that will be used to launch wu-bolt
+      # within each bolt of the Storm topology.
+      #
+      # @return [String]
+      def wu_bolt_commandline
+        return settings[:bolt_command] if settings[:bolt_command]
+        [settings[:command_prefix], 'wu-bolt', dataflow_name, non_wukong_storm_params_string].compact.map(&:to_s).reject(&:empty?).join(' ')
+      end
+
+      # Return the path to the `storm` program.
+      #
+      # Will pay attention to `--storm_runner` and `--storm_home`
+      # options.
+      #
+      # @return [String]
+      def storm_runner
+        explicit_runner = settings[:storm_runner]
+        home_runner     = File.join(settings[:storm_home], 'bin/storm')
+        default_runner  = 'storm'
+        case
+        when explicit_runner then explicit_runner
+        when File.exist?(home_runner) then home_runner
+        else default_runner
+        end
+      end
+
+      # Path to the Java jar file containing the submitter class.
+      #
+      # @return [String]
+      #
+      # @see #fully_qualified_class_name
+      def wukong_topology_submitter_jar
+        File.expand_path("wukong-storm.jar", File.dirname(__FILE__))
+      end
+
+      # The default Java Submitter class.
+      #
+      # @see #fully_qualified_class_name
+      TOPOLOGY_SUBMITTER_CLASS = "com.infochimps.wukong.storm.TopologySubmitter"
+
+      # Returns the fully qualified name of the Java submitter class.
+      #
+      # @see TOPOLOGY_SUBMITTER_CLASS
+      def fully_qualified_class_name
+        TOPOLOGY_SUBMITTER_CLASS
+      end
+      
+      # Return Java `-D` options constructed from mapping the passed
+      # in "friendly" options (`--timeout`) to native, Storm options
+      # (`topology.message.timeout.secs`).
+      #
+      # @return [Array<String>] an array of each `-D` option
+      def native_storm_options
+        settings.params_with(:storm).map do |option, value|
+          defn = settings.definition_of(option, :description)
+          [defn, settings[option.to_sym]]
+        end.map { |option, value| java_option(option, value) }
+      end
+
+      # Return Java `-D` options for Wukong-specific options.
+      #
+      # @return [Array<String>]
+      def storm_topology_options
+        (services_options + topology_options + spout_options + dataflow_options + state_options).reject do |pair|
+          key, value = pair
+          value.nil? || value.to_s.strip.empty?
+        end.map { |pair|  java_option(*pair) }.sort
+      end
+
+      # Return Java `-D` option key-value pairs related to services
+      # used by the topology.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def services_options
+        [
+         ["wukong.kafka.hosts",       settings[:kafka_hosts]],
+         ["wukong.zookeeper.hosts",   settings[:zookeeper_hosts]],
+        ]
+      end
+
+      # Return Java `-D` option key-value pairs related to the overall
+      # topology.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def topology_options
+        [
+         ["wukong.topology",          topology_name],
+        ]
+      end
+
+      # Return Java `-D` option key-value pairs related to the
+      # topology's spout.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def spout_options
+        case
+        when blob_input?
+          blob_spout_options + (s3_input? ? s3_spout_options : file_spout_options)
+        else
+          kafka_spout_options
+        end
+      end
+
+      # Return Java `-D` option key-value pairs related to the
+      # topology's spout if it is reading from a generic filesystem.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def blob_spout_options
+        [
+         ["wukong.input.type", "blob"],
+        ].tap do |so|
+          so << ["wukong.input.blob.marker",       settings[:offset]] if settings[:offset]
+          so << case
+          when settings[:from_beginning]
+            ["wukong.input.blob.start",        "EARLIEST"]
+          when settings[:from_end]
+            ["wukong.input.blob.start",        "LATEST"]
+          when settings[:offset]
+            ["wukong.input.blob.start",        "EXPLICIT"]
+          else
+            ["wukong.input.blob.start",        "RESUME"]
+          end
+        end
+      end
+
+      # Return Java `-D` option key-value pairs related to the
+      # topology's spout if it is reading from S3.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def s3_spout_options
+        [
+         ["wukong.input.blob.type",         "s3"],
+         ["wukong.input.blob.path",         input_uri.path.gsub(%r{^/},'')],
+         ["wukong.input.blob.s3_bucket",    input_uri.host],
+         ["wukong.input.blob.aws_key",      settings[:aws_key]],
+         ["wukong.input.blob.aws_secret",   settings[:aws_secret]],
+         ["wukong.input.blob.s3_endpoint",  s3_endpoint]
+        ]
+      end
+
+      # The AWS endpoint used to communicate with AWS for S3 access.
+      #
+      # Determined by the AWS region the S3 bucket was declared to be
+      # in.
+      #
+      # @see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
+      def s3_endpoint
+        case settings[:aws_region]
+        when 'us-east-1'       then 's3.amazonaws.com'
+        when 'us-west-1'       then 's3-us-west-1.amazonaws.com'
+        when 'us-west-2'       then 's3-us-west-2.amazonaws.com'
+        when /EU/, 'eu-west-1' then 's3-eu-west-1.amazonaws.com'
+        when 'ap-southeast-1'  then 's3-ap-southeast-1.amazonaws.com'
+        when 'ap-southeast-2'  then 's3-ap-southeast-2.amazonaws.com'
+        when 'ap-northeast-1'  then 's3-ap-northeast-1.amazonaws.com'
+        when 'sa-east-1'       then 's3-sa-east-1.amazonaws.com'
+        end
+      end
+
+      # Return Java `-D` option key-value pairs related to the
+      # topology's spout if it is reading from a local file.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def file_spout_options
+        [
+         ["wukong.input.blob.type", "file"],
+         ["wukong.input.blob.path", input_uri.path],
+        ]
+      end
+
+      # Return Java `-D` option key-value pairs related to the
+      # topology's spout if it is reading from Kafka.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def kafka_spout_options
+        [
+         ["wukong.input.type",              'kafka'],
+         ["wukong.input.kafka.topic",       settings[:input]],
+         ["wukong.input.kafka.partitions",  settings[:kafka_partitions]],
+         ["wukong.input.kafka.batch",       settings[:kafka_batch]],
+         
+         ["wukong.input.parallelism",       settings[:input_parallelism]],
+         case
+         when settings[:from_beginning]
+           ["wukong.input.kafka.offset",      "-2"]
+         when settings[:from_end]
+           ["wukong.input.kafka.offset",      "-1"]
+         when settings[:offset]
+           ["wukong.input.kafka.offset",      settings[:offset]]
+         else
+           # Do *not* set anything and the spout will attempt to
+           # resume and, finding no prior offset, will start from the
+           # end, as though we'd passed "-1"
+         end
+        ]
+      end
+
+      # Return Java `-D` option key-value pairs related to the Wukong
+      # dataflow run by the topology.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def dataflow_options
+        [
+         ["wukong.directory",         Dir.pwd],
+         ["wukong.dataflow",          dataflow_name],
+         ["wukong.command",           wu_bolt_commandline],
+         ["wukong.parallelism",       settings[:parallelism]],
+        ].tap do |opts|
+          opts << ["wukong.environment", settings[:environment]] if settings[:environment]
+        end
+      end
+
+      # Return Java `-D` option key-value pairs related to the final
+      # state used by the topology.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def state_options
+        case
+        when kafka_output?
+          kafka_state_options
+        end
+      end
+
+      # Return Java `-D` option key-value pairs related to the final
+      # state used by the topology when it is writing to Kafka.
+      #
+      # @return [Array<Array>] an Array of key-value pairs
+      def kafka_state_options
+        [
+         ["wukong.output.kafka.topic", settings[:output]],
+        ]
+      end
+
+      protected
+
+      # Return a String of options used when attempting to kill a
+      # running Storm topology.
+      #
+      # @return [String]
+      def storm_kill_options
+        "-w #{settings[:wait]}"
+      end
+
+      # Format the given `option` and `value` into a Java option
+      # (`-D`).
+      #
+      # @param [Object] option
+      # @param [Object] value
+      # @return [String]
+      def java_option option, value
+        return unless value
+        return if value.to_s.strip.empty?
+        "-D#{option}=#{Shellwords.escape(value.to_s)}"
+      end
+
+      # Parameters that should be passed onto subprocesses.
+      #
+      # @return [Configliere::Param]
+      def params_to_pass
+        settings
+      end
+
+      # Return a String stripped of any `wu-storm`-specific params but
+      # still including any other params.
+      #
+      # @return [String]
+      def non_wukong_storm_params_string
+        params_to_pass.reject do |param, val|
+          (params_to_pass.definition_of(param, :wukong_storm) || params_to_pass.definition_of(param, :wukong))
+        end.map do |param, val|
+          "--#{param}=#{Shellwords.escape(val.to_s)}"
+        end.join(" ")
+      end
+
+    end
+  end
+end