neptune 0.1.1 → 0.1.2

data/doc/README.html

@@ -66,6 +66,8 @@
         
           <li><a href="./CommonFunctions.html">CommonFunctions</a></li>
         
+          <li><a href="./Kernel.html">Kernel</a></li>
+        
           <li><a href="./Object.html">Object</a></li>
         
         </ul>
@@ -119,7 +121,9 @@ in the test folder, with the standard naming convention</p>
 <p>cases for each type of job that Neptune offers. Before running ts_neptune,
 you should export the environment variable APPSCALE_HEAD_NODE, which should
 be set to the IP address of the AppScale machine that runs the Shadow
-daemon (a.k.a. the Master AppController).</p>
+daemon (a.k.a. the Master AppController). Running generate_coverage.sh in
+the top-level directory will run rcov and generate the coverage reports
+automatically via unit tests.</p>
 
 <p>Developed by Chris Bunch as part of the AppScale project. See <a
 href="LICENSE.html">LICENSE</a> for the specifics of the New BSD License by
@@ -138,15 +142,32 @@ as adding capabilities for other types of computation. We would also like
 to refactor Neptune to use symbols instead of instance variables for
 running jobs: this will likely appear in a future release as well.</p>
 
-<p>An academic paper describing Neptune is in the works - check in for a link
-to that as it becomes available.</p>
+<p>Our academic paper on Neptune won best paper at ACM ScienceCloud 2011!
+&lt;a href=‘<a
+href="http://www.neptune-lang.org/2011/6/Neptune-Picks-up-Best-Paper-at-ScienceCloud-2011">www.neptune-lang.org/2011/6/Neptune-Picks-up-Best-Paper-at-ScienceCloud-2011</a>’&gt;
+Here’s a link&lt;/a&gt; to the abstract of the paper and the PDF.</p>
 
 <p>Version History:</p>
 
-<p>April 2, 2001 - 0.0.7 released, adding automatic test suite and many bug
+<p>November 10, 2011 - 0.1.2 released, adding unit tests and refactoring all
+around.</p>
+
+<p>June 6, 2011 - 0.1.1 released, adding support for code written in Go and R</p>
+
+<p>June 4, 2011 - 0.1.0 released, adding verbose / quiet options for users
+wishing to suppress stdout from Neptune jobs.</p>
+
+<p>May 28, 2011 - 0.0.9 released, adding generic SSA support for users wanting
+to use StochKit and other SSA codes.</p>
+
+<p>April 8, 2011 - 0.0.8 released, fixing MapReduce support for both regular
+Hadoop and Hadoop Streaming. Also increased code coverage to cover a number
+of failure scenarios.</p>
+
+<p>April 2, 2011 - 0.0.7 released, adding automatic test suite and many bug
 fixes for all scenarios. rcov can also be used to generate test coverage
 information: current coverage stats can be found in coverage directory.
-mapreduce broken at the moment - will fix in next release</p>
+MapReduce broken at the moment - will fix in next release</p>
 
 <p>March 28, 2011 - 0.0.6 released, adding support for input jobs, so users
 can place data in the datastore without having to run any computation</p>

data/doc/bin/neptune.html

@@ -24,7 +24,7 @@
   <div id="metadata">
     <dl>
       <dt class="modified-date">Last Modified</dt>
-      <dd class="modified-date">Fri Feb 04 19:49:21 -0800 2011</dd>
+      <dd class="modified-date">Sun May 15 22:52:56 -0700 2011</dd>
 
       
       <dt class="requires">Requires</dt>

data/doc/created.rid

@@ -1,7 +1,7 @@
-Sat, 02 Apr 2011 09:03:26 -0700
-./lib/common_functions.rb	Wed, 30 Mar 2011 21:31:10 -0700
+Thu, 10 Nov 2011 12:27:04 -0800
+./lib/common_functions.rb	Wed, 26 Oct 2011 10:37:34 -0700
 ./LICENSE	Thu, 27 Jan 2011 13:24:30 -0800
-./lib/app_controller_client.rb	Sun, 27 Mar 2011 23:37:41 -0700
-./lib/neptune.rb	Thu, 31 Mar 2011 10:11:31 -0700
-./README	Sat, 02 Apr 2011 09:03:23 -0700
-./bin/neptune	Fri, 04 Feb 2011 19:49:21 -0800
+./lib/app_controller_client.rb	Tue, 25 Oct 2011 23:47:47 -0700
+./lib/neptune.rb	Wed, 09 Nov 2011 10:54:28 -0800
+./README	Thu, 10 Nov 2011 11:48:11 -0800
+./bin/neptune	Sun, 15 May 2011 22:52:56 -0700

data/doc/index.html

@@ -44,6 +44,8 @@
     
     <li class="module"><a href="CommonFunctions.html">CommonFunctions</a></li>
     
+    <li class="module"><a href="Kernel.html">Kernel</a></li>
+    
     <li class="class"><a href="Object.html">Object</a></li>
     
   </ul>
@@ -63,12 +65,20 @@
     
       <li><a href="AppControllerClient.html#method-i-compile_code">#compile_code &mdash; AppControllerClient</a></li>
     
+      <li><a href="Object.html#method-i-compile_code">#compile_code &mdash; Object</a></li>
+    
       <li><a href="Object.html#method-i-do_preprocessing">#do_preprocessing &mdash; Object</a></li>
     
       <li><a href="AppControllerClient.html#method-i-get_acl">#get_acl &mdash; AppControllerClient</a></li>
     
+      <li><a href="Object.html#method-i-get_input">#get_input &mdash; Object</a></li>
+    
+      <li><a href="Object.html#method-i-get_job_data">#get_job_data &mdash; Object</a></li>
+    
       <li><a href="AppControllerClient.html#method-i-get_output">#get_output &mdash; AppControllerClient</a></li>
     
+      <li><a href="Object.html#method-i-get_std_out_and_err">#get_std_out_and_err &mdash; Object</a></li>
+    
       <li><a href="AppControllerClient.html#method-i-make_call">#make_call &mdash; AppControllerClient</a></li>
     
       <li><a href="Object.html#method-i-neptune">#neptune &mdash; Object</a></li>
@@ -77,16 +87,24 @@
     
       <li><a href="Object.html#method-i-preprocess_erlang">#preprocess_erlang &mdash; Object</a></li>
     
-      <li><a href="Object.html#method-i-preprocess_mapreduce">#preprocess_mapreduce &mdash; Object</a></li>
-    
       <li><a href="Object.html#method-i-preprocess_mpi">#preprocess_mpi &mdash; Object</a></li>
     
+      <li><a href="Object.html#method-i-preprocess_ssa">#preprocess_ssa &mdash; Object</a></li>
+    
       <li><a href="AppControllerClient.html#method-i-put_input">#put_input &mdash; AppControllerClient</a></li>
     
+      <li><a href="Object.html#method-i-run_job">#run_job &mdash; Object</a></li>
+    
       <li><a href="AppControllerClient.html#method-i-set_acl">#set_acl &mdash; AppControllerClient</a></li>
     
+      <li><a href="Kernel.html#method-i-shell">#shell &mdash; Kernel</a></li>
+    
       <li><a href="AppControllerClient.html#method-i-start_neptune_job">#start_neptune_job &mdash; AppControllerClient</a></li>
     
+      <li><a href="Object.html#method-i-validate_storage_params">#validate_storage_params &mdash; Object</a></li>
+    
+      <li><a href="Object.html#method-i-wait_for_compilation_to_finish">#wait_for_compilation_to_finish &mdash; Object</a></li>
+    
   </ul>
 
   <div id="validator-badges">

data/doc/lib/app_controller_client_rb.html

@@ -24,7 +24,7 @@
   <div id="metadata">
     <dl>
       <dt class="modified-date">Last Modified</dt>
-      <dd class="modified-date">Sun Mar 27 23:37:41 -0700 2011</dd>
+      <dd class="modified-date">Tue Oct 25 23:47:47 -0700 2011</dd>
 
       
       <dt class="requires">Requires</dt>
@@ -50,7 +50,7 @@
     <div class="description">
       <h2>Description</h2>
       
-<p>Programmer: Chris Bunch</p>
+<p>Programmer: Chris Bunch (cgb@cs.ucsb.edu)</p>
 
     </div>
     

data/doc/lib/common_functions_rb.html

@@ -24,7 +24,7 @@
   <div id="metadata">
     <dl>
       <dt class="modified-date">Last Modified</dt>
-      <dd class="modified-date">Wed Mar 30 21:31:10 -0700 2011</dd>
+      <dd class="modified-date">Wed Oct 26 10:37:34 -0700 2011</dd>
 
       
       <dt class="requires">Requires</dt>
@@ -58,7 +58,7 @@
     <div class="description">
       <h2>Description</h2>
       
-<p>Programmer: Chris Bunch</p>
+<p>Programmer: Chris Bunch (cgb@cs.ucsb.edu)</p>
 
     </div>
     

data/doc/lib/neptune_rb.html

@@ -24,7 +24,7 @@
   <div id="metadata">
     <dl>
       <dt class="modified-date">Last Modified</dt>
-      <dd class="modified-date">Thu Mar 31 10:11:31 -0700 2011</dd>
+      <dd class="modified-date">Wed Nov 09 10:54:28 -0800 2011</dd>
 
       
       <dt class="requires">Requires</dt>
@@ -48,6 +48,8 @@
     <div class="description">
       <h2>Description</h2>
       
+<p>Programmer: Chris Bunch (cgb@cs.ucsb.edu)</p>
+
     </div>
     
   </div>

data/lib/app_controller_client.rb

@@ -1,5 +1,5 @@
 #!/usr/bin/ruby -w
-# Programmer: Chris Bunch
+# Programmer: Chris Bunch (cgb@cs.ucsb.edu)
 
 require 'openssl'
 require 'soap/rpc/driver'
@@ -16,7 +16,7 @@ NO_TIMEOUT = -1
 # platform (here, AppScale). This client is similar to that used in the AppScale
 # Tools, but with non-Neptune SOAP calls removed.
 class AppControllerClient
-  attr_reader :conn, :ip, :secret
+  attr_accessor :conn, :ip, :secret
   
   # A constructor that requires both the IP address of the machine to communicate
   # with as well as the secret (string) needed to perform communication.

data/lib/common_functions.rb

@@ -1,5 +1,5 @@
 #!/usr/bin/ruby -w
-# Programmer: Chris Bunch
+# Programmer: Chris Bunch (cgb@cs.ucsb.edu)
 
 require 'digest/sha1'
 require 'fileutils'
@@ -9,6 +9,12 @@ require 'socket'
 require 'timeout'
 require 'yaml'
 
+module Kernel
+  def shell(command)
+    return `#{command}`
+  end
+end
+
 # A helper module that aggregates functions that are not part of Neptune's
 # core functionality. Specifically, this module contains methods to scp
 # files to other machines and the ability to read YAML files, which are
@@ -21,11 +27,17 @@ module CommonFunctions
   # placed, and the name of the key to use. The keyname is typically
   # specified by the Neptune job given, but defaults to ''appscale''
   # if not provided.
-  def self.scp_to_shadow(local_file_loc, remote_file_loc, keyname, is_dir=false)
-    shadow_ip = CommonFunctions.get_from_yaml(keyname, :shadow)
-    ssh_key = File.expand_path("~/.appscale/#{keyname}.key")
+  def self.scp_to_shadow(local_file_loc,
+                         remote_file_loc,
+                         keyname,
+                         is_dir=false,
+                         file=File,
+                         get_from_yaml=CommonFunctions.method(:get_from_yaml),
+                         scp_file=CommonFunctions.method(:scp_file))
 
-    self.scp_file(local_file_loc, remote_file_loc, shadow_ip, ssh_key, is_dir)
+    shadow_ip = get_from_yaml.call(keyname, :shadow, file)
+    ssh_key = file.expand_path("~/.appscale/#{keyname}.key")
+    scp_file.call(local_file_loc, remote_file_loc, shadow_ip, ssh_key, is_dir)
   end 
  
   # Performs the actual remote copying of files: given the IP address
@@ -34,33 +46,39 @@ module CommonFunctions
   # if the network is down, if a bad keyname is provided, or if the 
   # wrong IP is given. If the user specifies that the file to copy is
   # actually a directory, we append the -r flag to scp as well.
-  def self.scp_file(local_file_loc, remote_file_loc, target_ip, public_key_loc, is_dir=false)
+  def self.scp_file(local_file_loc, remote_file_loc, target_ip, public_key_loc,
+    is_dir=false, file=File, fileutils=FileUtils, kernel=Kernel)
     cmd = ""
-    local_file_loc = File.expand_path(local_file_loc)
+    local_file_loc = file.expand_path(local_file_loc)
 
     ssh_args = "-o StrictHostkeyChecking=no 2>&1"
     ssh_args << " -r " if is_dir
 
-    public_key_loc = File.expand_path(public_key_loc)
+    public_key_loc = file.expand_path(public_key_loc)
     cmd = "scp -i #{public_key_loc} #{ssh_args} #{local_file_loc} root@#{target_ip}:#{remote_file_loc}"
     cmd << "; echo $? >> ~/.appscale/retval"
 
-    retval_loc = File.expand_path("~/.appscale/retval")
-    FileUtils.rm_f(retval_loc)
+    retval_loc = file.expand_path("~/.appscale/retval")
+    fileutils.rm_f(retval_loc)
 
     begin
-      Timeout::timeout(-1) { `#{cmd}` }
+      Timeout::timeout(-1) { kernel.shell("#{cmd}") }
     rescue Timeout::Error
-      abort("Remotely copying over files failed. Is the destination machine on and reachable from this computer? We tried the following command:\n\n#{cmd}")
+      abort("Remotely copying over files failed. Is the destination machine" +
+        " on and reachable from this computer? We tried the following" +
+        " command:\n\n#{cmd}")
     end
 
     loop {
-      break if File.exists?(retval_loc)
+      break if file.exists?(retval_loc)
       sleep(5)
     }
 
-    retval = (File.open(retval_loc) { |f| f.read }).chomp
-    abort("\n\n[#{cmd}] returned #{retval} instead of 0 as expected. Is your environment set up properly?") if retval != "0"
+    retval = (file.open(retval_loc) { |f| f.read }).chomp
+    if retval != "0"
+      abort("\n\n[#{cmd}] returned #{retval} instead of 0 as expected. Is " +
+        "your environment set up properly?")
+    end
     return cmd
   end
 
@@ -70,16 +88,20 @@ module CommonFunctions
   # method aborts if the value doesn't exist or the YAML file is malformed.
   # If the required flag is set to false, it returns nil in either scenario
   # instead.
-  def self.get_from_yaml(keyname, tag, required=true)
-    location_file = File.expand_path("~/.appscale/locations-#{keyname}.yaml")
+  def self.get_from_yaml(keyname, tag, required=true, file=File, yaml=YAML)
+    location_file = file.expand_path("~/.appscale/locations-#{keyname}.yaml")
   
-    abort("An AppScale instance is not currently running with the provided keyname, \"#{keyname}\".") unless File.exists?(location_file)  
+    if !file.exists?(location_file)
+      abort("An AppScale instance is not currently running with the provided" +
+        " keyname, \"#{keyname}\".")
+    end
     
     begin
-      tree = YAML.load_file(location_file)
+      tree = yaml.load_file(location_file)
     rescue ArgumentError
       if required
-        abort("The yaml file you provided was malformed. Please correct any errors in it and try again.")
+        abort("The yaml file you provided was malformed. Please correct any" +
+          " errors in it and try again.")
       else
         return nil
       end
@@ -87,15 +109,19 @@ module CommonFunctions
     
     value = tree[tag]
     
-    bad_yaml_format_msg = "The file #{location_file} is in the wrong format and doesn't contain a #{tag} tag. Please make sure the file is in the correct format and try again"
-    abort(bad_yaml_format_msg) if value.nil? and required
+    if value.nil? and required
+      abort("The file #{location_file} is in the wrong format and doesn't" +
+        " contain a #{tag} tag. Please make sure the file is in the correct" +
+        " format and try again.")
+    end
+
     return value
   end
 
   # Returns the secret key needed for communication with AppScale's
   # Shadow node. This method is a nice frontend to the get_from_yaml
   # function, as the secret is stored in a YAML file.
-  def self.get_secret_key(keyname, required=true)
-    return CommonFunctions.get_from_yaml(keyname, :secret)
+  def self.get_secret_key(keyname, required=true, file=File, yaml=YAML)
+    return CommonFunctions.get_from_yaml(keyname, :secret, required, file, yaml)
   end
 end

data/lib/neptune.rb

@@ -1,4 +1,5 @@
 #!/usr/bin/ruby
+# Programmer: Chris Bunch (cgb@cs.ucsb.edu)
 
 require 'app_controller_client'
 require 'common_functions'
@@ -39,6 +40,8 @@ NEED_PREPROCESSING = ["compile", "erlang", "mpi", "ssa"]
 # support. In the future, it is likely that the only exposed / monkey-patched
 # method should be job, while the others could probably be folded into either
 # a Neptune-specific class or into CommonFunctions.
+# TODO(cbunch): This doesn't look like it does anything - run the integration
+# test and confirm one way or the other.
 class Object
 end
 
@@ -48,7 +51,9 @@ end
 # on the type of the job that the user has asked to run.
 def do_preprocessing(job_data)
   job_type = job_data["@type"]
-  return unless NEED_PREPROCESSING.include?(job_type)
+  if !NEED_PREPROCESSING.include?(job_type)
+    return
+  end
 
   preprocess = "preprocess_#{job_type}".to_sym
   send(preprocess, job_data)
@@ -57,10 +62,9 @@ end
 # This preprocessing method copies over the user's code to the
 # Shadow node so that it can be compiled there. A future version
 # of this method may also copy over libraries as well.
-def preprocess_compile(job_data)
-  verbose = job_data["@verbose"]
+def preprocess_compile(job_data, shell=Kernel.method(:`))
   code = File.expand_path(job_data["@code"])
-  unless File.exists?(code)
+  if !File.exists?(code)
     abort("The source file #{code} does not exist.")
   end
 
@@ -71,94 +75,88 @@ def preprocess_compile(job_data)
 
   ssh_args = "-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no root@#{shadow_ip}"
   remove_dir = "ssh #{ssh_args} 'rm -rf #{dest}' 2>&1"
-  puts remove_dir if verbose
-  `#{remove_dir}`
+  puts remove_dir
+  shell.call(remove_dir)
 
   CommonFunctions.scp_to_shadow(code, dest, keyname, is_dir=true)
 
   job_data["@code"] = dest
 end
 
-def preprocess_erlang(job_data)
-  source_code = File.expand_path(job_data["@code"])
-  unless File.exists?(source_code)
-    file_not_found = "The specified code, #{job_data['@code']}," +
-      " didn't exist. Please specify one that exists and try again"
-    abort(file_not_found)
+def preprocess_erlang(job_data, file=File, common_functions=CommonFunctions)
+  if !job_data["@code"]
+    abort("When running Erlang jobs, :code must be specified.")
+  end
+
+  source_code = file.expand_path(job_data["@code"])
+  if !file.exists?(source_code)
+    abort("The specified code, #{job_data['@code']}," +
+      " didn't exist. Please specify one that exists and try again")
   end
   dest_code = "/tmp/"
 
   keyname = job_data["@keyname"]
-  CommonFunctions.scp_to_shadow(source_code, dest_code, keyname)
+  common_functions.scp_to_shadow(source_code, dest_code, keyname)
 end
 
-# This preprocessing method copies over the user's MPI
-# code to the master node in AppScale - this node will
-# then copy it to whoever will run the MPI job.
+# This preprocessing method verifies that the user specified the number of nodes
+# to use. If they also specified the number of processes to use, we also verify
+# that this value is at least as many as the number of nodes (that is, nodes
+# can't be underprovisioned in MPI).
 def preprocess_mpi(job_data)
-  verbose = job_data["@verbose"]
+  if !job_data["@nodes_to_use"]
+    abort("When running MPI jobs, :nodes_to_use must be specified.")
+  end
+
+  if !job_data["@procs_to_use"]
+    abort("When running MPI jobs, :procs_to_use must be specified.")
+  end
+
   if job_data["@procs_to_use"]
     p = job_data["@procs_to_use"]
     n = job_data["@nodes_to_use"]
     if p < n
-      not_enough_procs = "When specifying both :procs_to_use and :nodes_to_use" +
+      abort("When specifying both :procs_to_use and :nodes_to_use" +
         ", :procs_to_use must be at least as large as :nodes_to_use. Please " +
         "change this and try again. You specified :procs_to_use = #{p} and" +
-        ":nodes_to_use = #{n}."
-      abort(not_enough_procs)
+        ":nodes_to_use = #{n}.")
     end
   end
 
-  # TODO: verify that source_code is in repo
+  return job_data
 end
 
+# This preprocessing method verifies that the user specified the number of
+# trajectories to run, via either :trajectories or :simulations. Both should
+# not be specified - only one or the other, and regardless of which they
+# specify, convert it to be :trajectories.
 def preprocess_ssa(job_data)
+  if job_data["@simulations"] and job_data["@trajectories"]
+    abort("Both :simulations and :trajectories cannot be specified - use one" +
+      " or the other.")
+  end
+
   if job_data["@simulations"]
     job_data["@trajectories"] = job_data["@simulations"]
+    job_data.delete("@simulations")
   end
 
-  unless job_data["@trajectories"]
+  if !job_data["@trajectories"]
     abort(":trajectories needs to be specified when running ssa jobs")
   end
-end
-
-# TODO: actually use me!
-#def validate_args(list)
-#  list.each do |item|
-#    val = instance_variable_get("@#{item}".to_sym)
-#    abort("FATAL: #{item} was not defined") if val.nil?
-#  end
-#end
 
-# This method is the heart of Neptune - here, we take
-# blocks of code that the user has written and convert them
-# into HPC job requests. At a high level, the user can
-# request to run a job, retrieve a job's output, or
-# modify the access policy (ACL) for the output of a
-# job. By default, job data is private, but a Neptune
-# job can be used to set it to public later (and
-# vice-versa).
-def neptune(params)
-  verbose = params[:verbose]
-
-  puts "Received a request to run a job." if verbose
-  puts params[:type] if verbose
-
-  keyname = params[:keyname] || "appscale"
-
-  shadow_ip = CommonFunctions.get_from_yaml(keyname, :shadow)
-  secret = CommonFunctions.get_secret_key(keyname)
-  controller = AppControllerClient.new(shadow_ip, secret)
-  ssh_key = File.expand_path("~/.appscale/#{keyname}.key")
+  return job_data
+end
 
+def get_job_data(params)
   job_data = {}
   params.each { |k, v|
     key = "@#{k}"
     job_data[key] = v
   }
 
-  job_data["@job"] = nil
-  job_data["@keyname"] = keyname || "appscale"
+  job_data.delete("@job")
+  job_data["@keyname"] = params[:keyname] || "appscale"
 
   job_data["@type"] = job_data["@type"].to_s
   type = job_data["@type"]
@@ -182,132 +180,174 @@ def neptune(params)
     end
   end
 
-  if job_data["@storage"]
-    storage = job_data["@storage"]
-    unless ALLOWED_STORAGE_TYPES.include?(storage)
-      msg = "Supported storage types are #{ALLOWED_STORAGE_TYPES.join(', ')}" +
-        " - we do not support #{storage}."
-      abort(msg)
-    end
-
-    # Our implementation for storing / retrieving via Google Storage
-    # and Walrus uses
-    # the same library as we do for S3 - so just tell it that it's S3
-    if storage == "gstorage" or storage == "walrus"
-      storage = "s3"
-      job_data["@storage"] = "s3"
-    end
+  return job_data
+end
 
-    if storage == "s3"
-      ["EC2_ACCESS_KEY", "EC2_SECRET_KEY", "S3_URL"].each { |item|
-        unless job_data["@#{item}"]
-          if ENV[item]
-            puts "Using #{item} from environment" if verbose
-            job_data["@#{item}"] = ENV[item]
-          else
-            msg = "When storing data to S3, #{item} must be specified or be in " + 
-              "your environment. Please do so and try again."
-            abort(msg)
-          end
-        end
-      }
-    end
-  else
+def validate_storage_params(job_data)
+  if !job_data["@storage"]
     job_data["@storage"] = "appdb"
   end
 
-  #if job_data["@can_run_on"].class == Range
-  #  job_data["@can_run_on"] = job_data["@can_run_on"].to_a
-  #elsif job_data["@can_run_on"].class == Fixnum
-  #  job_data["@can_run_on"] = [job_data["@can_run_on"]]
-  #end
+  storage = job_data["@storage"]
+  if !ALLOWED_STORAGE_TYPES.include?(storage)
+    abort("Supported storage types are #{ALLOWED_STORAGE_TYPES.join(', ')}" +
+      " - we do not support #{storage}.")
+  end
 
-  puts "job data = #{job_data.inspect}" if verbose
+  # Our implementation for storing / retrieving via Google Storage
+  # and Walrus uses
+  # the same library as we do for S3 - so just tell it that it's S3
+  if storage == "gstorage" or storage == "walrus"
+    storage = "s3"
+    job_data["@storage"] = "s3"
+  end
 
-  do_preprocessing(job_data) 
+  if storage == "s3"
+    ["EC2_ACCESS_KEY", "EC2_SECRET_KEY", "S3_URL"].each { |item|
+      if job_data["@#{item}"]
+        puts "Using specified #{item}"
+      else
+        if ENV[item]
+          puts "Using #{item} from environment"
+          job_data["@#{item}"] = ENV[item]
+        else
+          abort("When storing data to S3, #{item} must be specified or be in " + 
+            "your environment. Please do so and try again.")
+        end
+      end
+    }
+  end
 
-  ssh_args = "-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no "
+  return job_data
+end
 
-  # TODO - right now the job is assumed to succeed in many cases
-  # need to investigate the various failure scenarios
-  result = { :result => :success }
+# This method takes a file on the local user's computer and stores it remotely
+# via AppScale. It returns a hash map indicating whether or not the job
+# succeeded and if it failed, the reason for it.
+def get_input(job_data, ssh_args, shadow_ip, controller, file=File,
+  shell=Kernel.method(:`))
+  result = {:result => :success}
 
-  if type == "input"
-    # copy file to remote
-    # set location
-    local_file = File.expand_path(job_data["@local"])
-    if !File.exists?(local_file)
-      msg = "the file you specified to copy, #{local_file}, doesn't exist." + 
+  if !job_data["@local"]
+    abort("You failed to specify a file to copy over via the :local flag.")
+  end
+
+  local_file = file.expand_path(job_data["@local"])
+  if !file.exists?(local_file)
+    reason = "the file you specified to copy, #{local_file}, doesn't exist." + 
         " Please specify a file that exists and try again."
-      abort(msg)
+    return {:result => :failure, :reason => reason}  
+  end
+
+  remote = "/tmp/neptune-input-#{rand(100000)}"
+  scp_cmd = "scp -r #{ssh_args} #{local_file} root@#{shadow_ip}:#{remote}"
+  puts scp_cmd
+  shell.call(scp_cmd)
+
+  job_data["@local"] = remote
+  puts "job data = #{job_data.inspect}"
+  response = controller.put_input(job_data)
+  if response
+    return {:result => :success}
+  else
+    # TODO - expand this to include the reason why it failed
+    return {:result => :failure}
+  end
+end
+
+# This method waits for AppScale to finish compiling the user's code, indicated
+# by AppScale copying the finished code to a pre-determined location.
+def wait_for_compilation_to_finish(ssh_args, shadow_ip, compiled_location,
+  shell=Kernel.method(:`))
+  loop {
+    ssh_command = "ssh #{ssh_args} root@#{shadow_ip} 'ls #{compiled_location}' 2>&1"
+    puts ssh_command
+    ssh_result = shell.call(ssh_command)
+    puts "result was [#{ssh_result}]"
+    if ssh_result =~ /No such file or directory/
+      puts "Still waiting for code to be compiled..."
+    else
+      puts "compilation complete! Copying compiled code to #{copy_to}"
+      return
     end
+    sleep(5)
+  }
+end
 
-    remote = "/tmp/neptune-input-#{rand(100000)}"
-    scp_cmd = "scp -r #{ssh_args} #{local_file} root@#{shadow_ip}:#{remote}"
-    puts scp_cmd if verbose
-    `#{scp_cmd}`
+# This method sends out a request to compile code, waits for it to finish, and
+# gets the standard out and error returned from the compilation. This method
+# returns a hash containing the standard out, error, and a result that indicates
+# whether or not the compilation was successful.
+def compile_code(job_data, ssh_args, shadow_ip, shell=Kernel.method(:`))
+  compiled_location = controller.compile_code(job_data)
 
-    job_data["@local"] = remote
-    puts "job data = #{job_data.inspect}" if verbose
-    result[:input] = controller.put_input(job_data)
-  elsif type == "output"
-    result[:output] = controller.get_output(job_data)
-  elsif type == "get-acl"
-    job_data["@type"] = "acl"
-    result[:acl] = controller.get_acl(job_data)
-  elsif type == "set-acl"
-    job_data["@type"] = "acl"
-    result[:acl] = controller.set_acl(job_data)
-  elsif type == "compile"
-    compiled_location = controller.compile_code(job_data)
-
-    copy_to = job_data["@copy_to"]
-
-    loop {
-      ssh_command = "ssh #{ssh_args} root@#{shadow_ip} 'ls #{compiled_location}' 2>&1"
-      puts ssh_command if verbose
-      result = `#{ssh_command}`
-      puts "result was [#{result}]" if verbose
-      if result =~ /No such file or directory/
-        puts "Still waiting for code to be compiled..." if verbose
-      else
-        puts "compilation complete! Copying compiled code to #{copy_to}" if verbose
-        break
-      end
-      sleep(5)
-    }
+  copy_to = job_data["@copy_to"]
 
-    rm_local = "rm -rf #{copy_to}"
-    puts rm_local if verbose
-    `#{rm_local}`
+  wait_for_compilation_to_finish(ssh_args, shadow_ip, compiled_location)
 
-    scp_command = "scp -r #{ssh_args} root@#{shadow_ip}:#{compiled_location} #{copy_to} 2>&1"
-    puts scp_command if verbose
-    `#{scp_command}`
+  FileUtils.rm_rf(copy_to)
 
-    code = job_data["@code"]
-    dirs = code.split(/\//)
-    remote_dir = "/tmp/" + dirs[-1] 
+  scp_command = "scp -r #{ssh_args} root@#{shadow_ip}:#{compiled_location} #{copy_to} 2>&1"
+  puts scp_command
+  shell.call(scp_command)
 
-    ssh_command = "ssh #{ssh_args} root@#{shadow_ip} 'rm -rf #{remote_dir}' 2>&1"
-    puts ssh_command if verbose
-    `#{ssh_command}`
+  code = job_data["@code"]
+  dirs = code.split(/\//)
+  remote_dir = "/tmp/" + dirs[-1] 
 
-    ssh_command = "ssh #{ssh_args} root@#{shadow_ip} 'rm -rf #{compiled_location}' 2>&1"
-    puts ssh_command if verbose
-    `#{ssh_command}`
+  [remote_dir, compiled_location].each { |remote_files|
+    ssh_command = "ssh #{ssh_args} root@#{shadow_ip} 'rm -rf #{remote_files}' 2>&1"
+    puts ssh_command
+    shell.call(ssh_command)
+  }
 
-    out = File.open("#{copy_to}/compile_out") { |f| f.read.chomp! }
-    err = File.open("#{copy_to}/compile_err") { |f| f.read.chomp! }
-    result = {}
-    result[:out] = out
-    result[:err] = err
+  return get_std_out_and_err(copy_to)
+end
 
-    if result[:err]
-      result[:result] = :failure
-    else
-      result[:result] = :success
-    end    
+# This method returns a hash containing the standard out and standard error
+# from a completed job, as well as a result field that indicates whether or
+# not the job completed successfully (success = no errors).
+def get_std_out_and_err(location)
+  result = {}
+
+  out = File.open("#{location}/compile_out") { |f| f.read.chomp! }
+  result[:out] = out
+
+  err = File.open("#{location}/compile_err") { |f| f.read.chomp! }
+  result[:err] = err
+
+  if result[:err]
+    result[:result] = :failure
+  else
+    result[:result] = :success
+  end    
+
+  return result
+end
+
+# This method actually runs the Neptune job, given information about the job
+# as well as information about the node to send the request to.
+def run_job(job_data, ssh_args, shadow_ip, secret,
+  controller=AppControllerClient, file=File)
+  controller = controller.new(shadow_ip, secret)
+
+  # TODO - right now the job is assumed to succeed in many cases
+  # need to investigate the various failure scenarios
+  result = { :result => :success }
+
+  case job_data["@type"]
+  when "input"
+    result = get_input(job_data, ssh_args, shadow_ip, controller, file)
+  when "output"
+    result[:output] = controller.get_output(job_data)
+  when "get-acl"
+    job_data["@type"] = "acl"
+    result[:acl] = controller.get_acl(job_data)
+  when "set-acl"
+    job_data["@type"] = "acl"
+    result[:acl] = controller.set_acl(job_data)
+  when "compile"
+    result = compile_code(job_data, ssh_args, shadow_ip)
   else
     msg = controller.start_neptune_job(job_data)
     result[:msg] = msg
@@ -317,3 +357,28 @@ def neptune(params)
   return result
 end
 
+# This method is the heart of Neptune - here, we take
+# blocks of code that the user has written and convert them
+# into HPC job requests. At a high level, the user can
+# request to run a job, retrieve a job's output, or
+# modify the access policy (ACL) for the output of a
+# job. By default, job data is private, but a Neptune
+# job can be used to set it to public later (and
+# vice-versa).
+def neptune(params)
+  puts "Received a request to run a job."
+  puts params[:type]
+
+  job_data = get_job_data(params)
+  validate_storage_params(job_data)
+  puts "job data = #{job_data.inspect}"
+  do_preprocessing(job_data) 
+  keyname = job_data["@keyname"]
+
+  shadow_ip = CommonFunctions.get_from_yaml(keyname, :shadow)
+  secret = CommonFunctions.get_secret_key(keyname)
+  ssh_key = File.expand_path("~/.appscale/#{keyname}.key")
+  ssh_args = "-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no "
+
+  return run_job(job_data, ssh_args, shadow_ip, secret)
+end