neptune 0.2.0 → 0.2.1

data/README

@@ -67,6 +67,12 @@ http://www.neptune-lang.org/2011/6/Neptune-Picks-up-Best-Paper-at-ScienceCloud-2
 
 Version History:
 
+February 12, 2012 - 0.2.1 released, adding support for the babel() method on
+non-Babel job types (e.g., MPI, MapReduce), and removed the is_remote
+parameter for babel() calls (since it can always be inferred from :storage).
+Added more unit tests accordingly for babel() calls, and completed documentation
+per rdoc standards.
+
 February 11, 2012 - 0.2.0 released, adding support for Babel jobs (code that
 runs over multiple clouds, using their queues automatically) and the babel()
 method, to automatically upload code, run it, and retrieve the output.

data/bin/neptune

@@ -27,4 +27,3 @@ else
   params = Hash[*ARGV]
   puts neptune(params).inspect
 end
-

data/doc/AppControllerClient.html

@@ -172,7 +172,7 @@ AppScale Tools, but with non-Neptune SOAP calls removed.</p>
 
         <div class="method-description">
         
-        
+        <p>The SOAP client that we use to communicate with the AppController.</p>
         
         </div>
       </div>
@@ -189,7 +189,7 @@ AppScale Tools, but with non-Neptune SOAP calls removed.</p>
 
         <div class="method-description">
         
-        
+        <p>The IP address of the AppController that we will be connecting to.</p>
         
         </div>
       </div>
@@ -206,7 +206,9 @@ AppScale Tools, but with non-Neptune SOAP calls removed.</p>
 
         <div class="method-description">
         
-        
+        <p>The secret string that is used to authenticate this client with
+AppControllers. It is initially generated by appscale-run-instances and can
+be found on the machine that ran that tool, or on any AppScale machine.</p>
         
         </div>
       </div>
@@ -245,7 +247,7 @@ for us.</p>
           <div class="method-source-code"
             id="new-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 26</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 41</span>
 def initialize(ip, secret)
   <span class="ruby-ivar">@ip</span> = ip
   <span class="ruby-ivar">@secret</span> = secret
@@ -289,14 +291,15 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>Instructs the AppController to fetch the code specified and compile it. The
+result should then be placed in a location specified in the job data.</p>
           
 
           
           <div class="method-source-code"
             id="compile_code-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 154</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 178</span>
 def compile_code(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -328,14 +331,16 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>Asks the AppController to see if the given file exists in the remote
+datastore. If extra credentials are needed for this operation, they are
+searched for within the job data.</p>
           
 
           
           <div class="method-source-code"
             id="does_file_exist-3F-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 171</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 203</span>
 def does_file_exist?(file, job_data)
   result = false
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) {
@@ -376,7 +381,7 @@ exceptions mirror that of start_neptune_job.</p>
           <div class="method-source-code"
             id="get_acl-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 131</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 151</span>
 def get_acl(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -425,7 +430,7 @@ well as the input format used for job_data.</p>
           <div class="method-source-code"
             id="get_output-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 117</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 136</span>
 def get_output(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -457,14 +462,16 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>Asks the AppController for a list of all the Babel engines (each of which
+is a queue to store jobs and something that executes tasks) that are
+supported for the given credentials.</p>
           
 
           
           <div class="method-source-code"
             id="get_supported_babel_engines-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 163</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 191</span>
 def get_supported_babel_engines(job_data)
   result = []
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) {
@@ -513,7 +520,7 @@ The result of the block is returned to the caller.</p>
           <div class="method-source-code"
             id="make_call-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 54</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 70</span>
 def make_call(time, retry_on_except)
   begin
     <span class="ruby-constant">Timeout</span>::timeout(time) {
@@ -569,7 +576,7 @@ parameter.</p>
           <div class="method-source-code"
             id="put_input-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 98</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 116</span>
 def put_input(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) {
@@ -613,7 +620,7 @@ that of start_neptune_job.</p>
           <div class="method-source-code"
             id="set_acl-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 145</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 166</span>
 def set_acl(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -660,7 +667,7 @@ AppController.</p>
           <div class="method-source-code"
             id="start_neptune_job-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 84</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 101</span>
 def start_neptune_job(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 

data/doc/AppControllerException.html

@@ -118,7 +118,8 @@
 
     <div id="description">
       
-<p>Programmer: Chris Bunch</p>
+<p>A special class of exceptions that are thrown whenever the AppController
+experiences an unexpected result.</p>
 
     </div>
 

data/doc/BabelHelper.html

@@ -181,7 +181,7 @@ just re-converts it - how can we remove it?</p>
           <div class="method-source-code"
             id="convert_from_neptune_params-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 198</span>
+<span class="ruby-comment"># File lib/babel.rb, line 211</span>
 def self.convert_from_neptune_params(params)
   job_data = {}
   params.each { |k, v|
@@ -221,7 +221,7 @@ converted them to the form @name, this function reverses that conversion.</p>
           <div class="method-source-code"
             id="convert_to_neptune_params-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 209</span>
+<span class="ruby-comment"># File lib/babel.rb, line 223</span>
 def self.convert_to_neptune_params(job_data)
   neptune_params = {}
 
@@ -264,11 +264,11 @@ with the given name.</p>
           <div class="method-source-code"
             id="ensure_output_does_not_exist-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 134</span>
+<span class="ruby-comment"># File lib/babel.rb, line 142</span>
 def self.ensure_output_does_not_exist(job_data)
   file = job_data[<span class="ruby-string">&quot;@output&quot;</span>]
   controller = self.get_appcontroller(job_data)
-  puts job_data.inspect
+  <span class="ruby-comment"># Kernel.puts job_data.inspect</span>
   <span class="ruby-constant">NeptuneHelper</span>.require_file_to_not_exist(file, job_data, controller)
 end</pre>
           </div>
@@ -304,9 +304,9 @@ code).</p>
           <div class="method-source-code"
             id="generate_output_location-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 81</span>
+<span class="ruby-comment"># File lib/babel.rb, line 86</span>
 def self.generate_output_location(job_data)
-  if job_data[<span class="ruby-string">&quot;@is_remote&quot;</span>]
+  if job_data[<span class="ruby-string">&quot;@storage&quot;</span>]
     <span class="ruby-comment"># We already know the bucket name - the same one that the user</span>
     <span class="ruby-comment"># has told us their code is located in.</span>
     prefix = job_data[<span class="ruby-string">&quot;@code&quot;</span>].scan(<span class="ruby-regexp">/\/(.*?)\//</span>)[0].to_s
@@ -347,7 +347,7 @@ the given job data.</p>
           <div class="method-source-code"
             id="get_appcontroller-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 142</span>
+<span class="ruby-comment"># File lib/babel.rb, line 151</span>
 def self.get_appcontroller(job_data)
   keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>] || <span class="ruby-string">&quot;appscale&quot;</span>
   shadow_ip = <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :shadow)
@@ -385,7 +385,7 @@ be used for Neptune jobs where the code is stored locally.</p>
           <div class="method-source-code"
             id="get_bucket_for_local_data-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 95</span>
+<span class="ruby-comment"># File lib/babel.rb, line 101</span>
 def self.get_bucket_for_local_data(job_data)
   bucket_name = job_data[<span class="ruby-string">&quot;@bucket_name&quot;</span>] || <span class="ruby-constant">ENV</span>[<span class="ruby-string">'BABEL_BUCKET_NAME'</span>]
 
@@ -432,7 +432,7 @@ used) in the remote datastore.</p>
           <div class="method-source-code"
             id="put_code-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 152</span>
+<span class="ruby-comment"># File lib/babel.rb, line 162</span>
 def self.put_code(job_data)
   code_dir = <span class="ruby-constant">File</span>.dirname(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
   code = <span class="ruby-constant">File</span>.basename(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
@@ -471,7 +471,7 @@ Neptune ‘input’ job to store the data remotely.</p>
           <div class="method-source-code"
             id="put_file-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 181</span>
+<span class="ruby-comment"># File lib/babel.rb, line 193</span>
 def self.put_file(local_path, job_data)
   input_data = self.convert_to_neptune_params(job_data)
   input_data[:type] = <span class="ruby-string">&quot;input&quot;</span>
@@ -517,7 +517,7 @@ remote location of these files.</p>
           <div class="method-source-code"
             id="put_inputs-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 164</span>
+<span class="ruby-comment"># File lib/babel.rb, line 175</span>
 def self.put_inputs(job_data)
   if job_data[<span class="ruby-string">&quot;@argv&quot;</span>].nil? or job_data[<span class="ruby-string">&quot;@argv&quot;</span>].empty?
     return job_data
@@ -563,10 +563,14 @@ queue) from the given parameters.</p>
           <div class="method-source-code"
             id="run_job-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 222</span>
+<span class="ruby-comment"># File lib/babel.rb, line 237</span>
 def self.run_job(job_data)
   run_data = self.convert_to_neptune_params(job_data)
-  run_data[:type] = <span class="ruby-string">&quot;babel&quot;</span>
+
+  <span class="ruby-comment"># Default to babel as the job type, if the user doesn't specify one.</span>
+  if run_data[:type].nil? or run_data[:type].empty?
+    run_data[:type] = <span class="ruby-string">&quot;babel&quot;</span>
+  end
 
   <span class="ruby-comment"># TODO(cgb): Once AppScale+Babel gets support for RabbitMQ, change this to</span>
   <span class="ruby-comment"># exec tasks over it, instead of locally.</span>
@@ -609,7 +613,7 @@ them for computation.</p>
           <div class="method-source-code"
             id="validate_inputs-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 113</span>
+<span class="ruby-comment"># File lib/babel.rb, line 120</span>
 def self.validate_inputs(job_data)
   controller = self.get_appcontroller(job_data)
 
@@ -659,7 +663,7 @@ output of the job.</p>
           <div class="method-source-code"
             id="wait_and_get_output-source">
 <pre>
-<span class="ruby-comment"># File lib/babel.rb, line 239</span>
+<span class="ruby-comment"># File lib/babel.rb, line 259</span>
 def self.wait_and_get_output(job_data)
   output_data = self.convert_to_neptune_params(job_data)
   output_data[:type] = <span class="ruby-string">&quot;output&quot;</span>

data/doc/BadConfigurationException.html

@@ -118,6 +118,9 @@
 
     <div id="description">
       
+<p>A class of exceptions that are thrown when the user tries to run a Neptune
+job but fails to give us the correct parameters to do so.</p>
+
     </div>
 
     <!-- Constants -->

data/doc/CommonFunctions.html

@@ -175,7 +175,7 @@ instead.</p>
           <div class="method-source-code"
             id="get_from_yaml-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 105</span>
+<span class="ruby-comment"># File lib/common_functions.rb, line 108</span>
 def self.get_from_yaml(keyname, tag, required=true)
   location_file = <span class="ruby-constant">File</span>.expand_path(&quot;~/.appscale/locations-#{keyname}.yaml&quot;)
 
@@ -236,14 +236,14 @@ user requests.</p>
           <div class="method-source-code"
             id="get_random_alphanumeric-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 29</span>
+<span class="ruby-comment"># File lib/common_functions.rb, line 33</span>
 def self.get_random_alphanumeric(length=10)
   random = <span class="ruby-string">&quot;&quot;</span>
   possible = <span class="ruby-string">&quot;0123456789abcdefghijklmnopqrstuvxwyzABCDEFGHIJKLMNOPQRSTUVWXYZ&quot;</span>
   possibleLength = possible.length
   
   length.times { |index|
-    random &lt;&lt; possible[rand(possibleLength)]
+    random &lt;&lt; possible[<span class="ruby-constant">Kernel</span>.rand(possibleLength)]
   }
    
   return random
@@ -281,7 +281,7 @@ function, as the secret is stored in a YAML file.</p>
           <div class="method-source-code"
             id="get_secret_key-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 138</span>
+<span class="ruby-comment"># File lib/common_functions.rb, line 142</span>
 def self.get_secret_key(keyname, required=true)
   return <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :secret, required)
 end</pre>
@@ -321,12 +321,11 @@ a directory, we append the -r flag to scp as well.</p>
           <div class="method-source-code"
             id="scp_file-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 63</span>
+<span class="ruby-comment"># File lib/common_functions.rb, line 66</span>
 def self.scp_file(local_file_loc, remote_file_loc, target_ip, public_key_loc,
   is_dir=false)
-  cmd = <span class="ruby-string">&quot;&quot;</span>
-  local_file_loc = <span class="ruby-constant">File</span>.expand_path(local_file_loc)
 
+  local_file_loc = <span class="ruby-constant">File</span>.expand_path(local_file_loc)
   ssh_args = <span class="ruby-string">&quot;-o StrictHostkeyChecking=no 2&gt;&amp;1&quot;</span>
   ssh_args &lt;&lt; <span class="ruby-string">&quot; -r &quot;</span> if is_dir
 
@@ -347,7 +346,7 @@ def self.scp_file(local_file_loc, remote_file_loc, target_ip, public_key_loc,
 
   loop {
     break if <span class="ruby-constant">File</span>.exists?(retval_loc)
-    sleep(5)
+    <span class="ruby-constant">Kernel</span>.sleep(5)
   }
 
   retval = (<span class="ruby-constant">File</span>.open(retval_loc) { |f| f.read }).chomp
@@ -390,15 +389,12 @@ by the Neptune job given, but defaults to ”appscale” if not provided.</p>
           <div class="method-source-code"
             id="scp_to_shadow-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 47</span>
-def self.scp_to_shadow(local_file_loc,
-                       remote_file_loc,
-                       keyname,
-                       is_dir=false)
-
+<span class="ruby-comment"># File lib/common_functions.rb, line 52</span>
+def self.scp_to_shadow(local_file_loc, remote_file_loc, keyname, is_dir=false)
   shadow_ip = <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :shadow)
   ssh_key = <span class="ruby-constant">File</span>.expand_path(&quot;~/.appscale/#{keyname}.key&quot;)
-  <span class="ruby-constant">CommonFunctions</span>.scp_file(local_file_loc, remote_file_loc, shadow_ip, ssh_key, is_dir)
+  <span class="ruby-constant">CommonFunctions</span>.scp_file(local_file_loc, remote_file_loc, shadow_ip,
+    ssh_key, is_dir)
 end</pre>
           </div>
           
@@ -432,7 +428,7 @@ method).</p>
           <div class="method-source-code"
             id="shell-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 23</span>
+<span class="ruby-comment"># File lib/common_functions.rb, line 26</span>
 def self.shell(cmd)
   return `#{cmd}`
 end</pre>

data/doc/FileNotFoundException.html

@@ -118,6 +118,9 @@
 
     <div id="description">
       
+<p>An exception that is thrown whenever the user specifies a file to use that
+does not exist.</p>
+
     </div>
 
     <!-- Constants -->

data/doc/NeptuneHelper.html

@@ -152,6 +152,10 @@
 
     <div id="description">
       
+<p><a href="NeptuneHelper.html">NeptuneHelper</a> provides methods that are
+used by neptune() and babel() to  validate parameters and run the user’s
+job.</p>
+
     </div>
 
     <!-- Constants -->
@@ -189,7 +193,7 @@ indicates whether or not the compilation was successful.</p>
           <div class="method-source-code"
             id="compile_code-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 369</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 424</span>
 def self.compile_code(job_data, ssh_args, shadow_ip)
   compiled_location = controller.compile_code(job_data)
   copy_to = job_data[<span class="ruby-string">&quot;@copy_to&quot;</span>]
@@ -198,7 +202,7 @@ def self.compile_code(job_data, ssh_args, shadow_ip)
   <span class="ruby-constant">FileUtils</span>.rm_rf(copy_to)
 
   scp_command = &quot;scp -r #{ssh_args} root@#{shadow_ip}:#{compiled_location} #{copy_to} 2&gt;&amp;1&quot;
-  <span class="ruby-constant">Kernel</span>.puts scp_command
+  <span class="ruby-comment"># Kernel.puts scp_command</span>
   <span class="ruby-constant">CommonFunctions</span>.shell(scp_command)
 
   code = job_data[<span class="ruby-string">&quot;@code&quot;</span>]
@@ -207,7 +211,7 @@ def self.compile_code(job_data, ssh_args, shadow_ip)
 
   [remote_dir, compiled_location].each { |remote_files|
     ssh_command = &quot;ssh #{ssh_args} root@#{shadow_ip} 'rm -rf #{remote_files}' 2&gt;&amp;1&quot;
-    <span class="ruby-constant">Kernel</span>.puts ssh_command
+    <span class="ruby-comment"># Kernel.puts ssh_command</span>
     <span class="ruby-constant">CommonFunctions</span>.shell(ssh_command)
   }
 
@@ -245,7 +249,7 @@ method to use based on the type of the job that the user has asked to run.</p>
           <div class="method-source-code"
             id="do_preprocessing-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 60</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 87</span>
 def self.do_preprocessing(job_data, controller)
   job_type = job_data[<span class="ruby-string">&quot;@type&quot;</span>]
   if !<span class="ruby-constant">NEED_PREPROCESSING</span>.include?(job_type)
@@ -289,7 +293,7 @@ job succeeded and if it failed, the reason for it.</p>
           <div class="method-source-code"
             id="get_input-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 319</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 372</span>
 def self.get_input(job_data, ssh_args, shadow_ip, controller)
   result = {:result =&gt; :success}
 
@@ -304,11 +308,11 @@ def self.get_input(job_data, ssh_args, shadow_ip, controller)
 
   remote = &quot;/tmp/neptune-input-#{rand(100000)}&quot;
   scp_cmd = &quot;scp -r #{ssh_args} #{local_file} root@#{shadow_ip}:#{remote}&quot;
-  <span class="ruby-constant">Kernel</span>.puts scp_cmd
+  <span class="ruby-comment"># Kernel.puts scp_cmd</span>
   <span class="ruby-constant">CommonFunctions</span>.shell(scp_cmd)
 
   job_data[<span class="ruby-string">&quot;@local&quot;</span>] = remote
-  <span class="ruby-constant">Kernel</span>.puts &quot;job data = #{job_data.inspect}&quot;
+  <span class="ruby-comment"># Kernel.puts &quot;job data = #{job_data.inspect}&quot;</span>
   response = controller.put_input(job_data)
   if response
     return {:result =&gt; :success}
@@ -340,14 +344,17 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>This method takes in a hash in the format that users write neptune/babel
+jobs in {:a =&gt; “b”} and converts it to the legacy format that
+Neptune used to use {“@a” =&gt; “b”}, and is understood by the
+AppController.</p>
           
 
           
           <div class="method-source-code"
             id="get_job_data-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 232</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 279</span>
 def self.get_job_data(params)
   job_data = {}
   params.each { |k, v|
@@ -427,7 +434,7 @@ not the job completed successfully (success = no errors).</p>
           <div class="method-source-code"
             id="get_std_out_and_err-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 396</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 452</span>
 def self.get_std_out_and_err(location)
   result = {}
 
@@ -478,7 +485,7 @@ engines can be found by contacting an AppScale node.</p>
           <div class="method-source-code"
             id="preprocess_babel-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 188</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 231</span>
 def self.preprocess_babel(job_data, controller)
   self.require_param(<span class="ruby-string">&quot;@code&quot;</span>, job_data)
   self.require_param(<span class="ruby-string">&quot;@engine&quot;</span>, job_data)
@@ -554,7 +561,7 @@ copy over libraries as well.</p>
           <div class="method-source-code"
             id="preprocess_compile-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 75</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 103</span>
 def self.preprocess_compile(job_data, controller)
   code = <span class="ruby-constant">File</span>.expand_path(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
   if !<span class="ruby-constant">File</span>.exists?(code)
@@ -568,7 +575,7 @@ def self.preprocess_compile(job_data, controller)
 
   ssh_args = &quot;-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no root@#{shadow_ip}&quot;
   remove_dir = &quot;ssh #{ssh_args} 'rm -rf #{dest}' 2&gt;&amp;1&quot;
-  <span class="ruby-constant">Kernel</span>.puts remove_dir
+  <span class="ruby-comment"># Kernel.puts remove_dir</span>
   <span class="ruby-constant">CommonFunctions</span>.shell(remove_dir)
   <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(code, dest, keyname, is_dir=true)
 
@@ -597,14 +604,15 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>This preprocessing method makes sure that the user’s Erlang code exists
+and copies it over to the AppScale Shadow node.</p>
           
 
           
           <div class="method-source-code"
             id="preprocess_erlang-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 95</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 126</span>
 def self.preprocess_erlang(job_data, controller)
   self.require_param(<span class="ruby-string">&quot;@code&quot;</span>, job_data)
 
@@ -651,7 +659,7 @@ also verify that this value is at least as many as the number of nodes
           <div class="method-source-code"
             id="preprocess_mpi-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 113</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 145</span>
 def self.preprocess_mpi(job_data, controller)
   self.require_param(<span class="ruby-string">&quot;@nodes_to_use&quot;</span>, job_data)
   self.require_param(<span class="ruby-string">&quot;@procs_to_use&quot;</span>, job_data)
@@ -712,7 +720,7 @@ specify, convert it to be :trajectories.</p>
           <div class="method-source-code"
             id="preprocess_ssa-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 145</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 178</span>
 def self.preprocess_ssa(job_data, controller)
   if job_data[<span class="ruby-string">&quot;@simulations&quot;</span>] and job_data[<span class="ruby-string">&quot;@trajectories&quot;</span>]
     raise <span class="ruby-constant">BadConfigurationException</span>.new(<span class="ruby-string">&quot;:simulations and :trajectories &quot;</span> +
@@ -750,14 +758,15 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>This helper method asks the AppController if the named file exists, and if
+it does not, throws an exception.</p>
           
 
           
           <div class="method-source-code"
             id="require_file_to_exist-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 166</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 205</span>
 def self.require_file_to_exist(file, job_data, controller)
   if controller.does_file_exist?(file, job_data)
     return
@@ -788,14 +797,16 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>This helper method performs the opposite function of <a
+href="NeptuneHelper.html#method-c-require_file_to_exist">require_file_to_exist</a>,
+raising an exception if the named file does exist.</p>
           
 
           
           <div class="method-source-code"
             id="require_file_to_not_exist-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 174</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 216</span>
 def self.require_file_to_not_exist(file, job_data, controller)
   begin
     self.require_file_to_exist(file, job_data, controller)
@@ -828,14 +839,15 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>This helper method aborts if the given parameter is not present in the job
+data provided.</p>
           
 
           
           <div class="method-source-code"
             id="require_param-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 160</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 196</span>
 def self.require_param(param, job_data)
   if !job_data[param]
     raise <span class="ruby-constant">BadConfigurationException</span>.new(&quot;#{param} must be specified&quot;)
@@ -872,7 +884,7 @@ as well as information about the node to send the request to.</p>
           <div class="method-source-code"
             id="run_job-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 442</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 502</span>
 def self.run_job(job_data, ssh_args, shadow_ip, secret)
   controller = <span class="ruby-constant">AppControllerClient</span>.new(shadow_ip, secret)
 
@@ -929,17 +941,18 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>This method uploads a Google App Engine application into AppScale, for use
+with Cicero jobs. It requires the AppScale tools to be installed.</p>
           
 
           
           <div class="method-source-code"
             id="upload_app_for_cicero-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 414</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 473</span>
 def self.upload_app_for_cicero(job_data)
   if !job_data[<span class="ruby-string">&quot;@app&quot;</span>]
-    <span class="ruby-constant">Kernel</span>.puts <span class="ruby-string">&quot;No app specified, not uploading...&quot;</span> 
+    <span class="ruby-comment"># Kernel.puts &quot;No app specified, not uploading...&quot; </span>
     return
   end
 
@@ -957,10 +970,10 @@ def self.upload_app_for_cicero(job_data)
     upload_app = <span class="ruby-string">&quot;appscale-upload-app&quot;</span>
   end
 
-  <span class="ruby-constant">Kernel</span>.puts &quot;Uploading AppEngine app at #{app_location}&quot;
+  <span class="ruby-comment"># Kernel.puts &quot;Uploading AppEngine app at #{app_location}&quot;</span>
   upload_command = &quot;#{upload_app} --file #{app_location} --test --keyname #{keyname}&quot;
-  <span class="ruby-constant">Kernel</span>.puts upload_command
-  <span class="ruby-constant">Kernel</span>.puts `#{upload_command}`
+  <span class="ruby-comment"># Kernel.puts upload_command</span>
+  <span class="ruby-comment"># Kernel.puts `#{upload_command}`</span>
 end</pre>
           </div>
           
@@ -985,14 +998,17 @@ end</pre>
 
         <div class="method-description">
           
-          
+          <p>This method looks through the given job data and makes sure that the
+correct parameters are present for the storage mechanism specified. It
+throws an exception if there are errors in the job data or if a needed
+parameter is missing.</p>
           
 
           
           <div class="method-source-code"
             id="validate_storage_params-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 280</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 332</span>
 def self.validate_storage_params(job_data)
   job_data[<span class="ruby-string">&quot;@storage&quot;</span>] ||= <span class="ruby-string">&quot;appdb&quot;</span>
 
@@ -1013,10 +1029,10 @@ def self.validate_storage_params(job_data)
   if storage == <span class="ruby-string">&quot;s3&quot;</span>
     [<span class="ruby-string">&quot;EC2_ACCESS_KEY&quot;</span>, <span class="ruby-string">&quot;EC2_SECRET_KEY&quot;</span>, <span class="ruby-string">&quot;S3_URL&quot;</span>].each { |item|
       if job_data[&quot;@#{item}&quot;]
-        <span class="ruby-constant">Kernel</span>.puts &quot;Using specified #{item}&quot;
+        <span class="ruby-comment"># Kernel.puts &quot;Using specified #{item}&quot;</span>
       else
         if <span class="ruby-constant">ENV</span>[item]
-          <span class="ruby-constant">Kernel</span>.puts &quot;Using #{item} from environment&quot;
+          <span class="ruby-comment"># Kernel.puts &quot;Using #{item} from environment&quot;</span>
           job_data[&quot;@#{item}&quot;] = <span class="ruby-constant">ENV</span>[item]
         else
           raise <span class="ruby-constant">BadConfigurationException</span>.new(&quot;When storing data to S3, #{item} must be specified or be in &quot; + 
@@ -1060,17 +1076,17 @@ location.</p>
           <div class="method-source-code"
             id="wait_for_compilation_to_finish-source">
 <pre>
-<span class="ruby-comment"># File lib/neptune.rb, line 349</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 403</span>
 def self.wait_for_compilation_to_finish(ssh_args, shadow_ip, compiled_location)
   loop {
     ssh_command = &quot;ssh #{ssh_args} root@#{shadow_ip} 'ls #{compiled_location}' 2&gt;&amp;1&quot;
-    <span class="ruby-constant">Kernel</span>.puts ssh_command
+    <span class="ruby-comment"># Kernel.puts ssh_command</span>
     ssh_result = <span class="ruby-constant">CommonFunctions</span>.shell(ssh_command)
-    <span class="ruby-constant">Kernel</span>.puts &quot;result was [#{ssh_result}]&quot;
+    <span class="ruby-comment"># Kernel.puts &quot;result was [#{ssh_result}]&quot;</span>
     if ssh_result =~ <span class="ruby-regexp">/No such file or directory/</span>
-      <span class="ruby-constant">Kernel</span>.puts <span class="ruby-string">&quot;Still waiting for code to be compiled...&quot;</span>
+      <span class="ruby-comment"># Kernel.puts &quot;Still waiting for code to be compiled...&quot;</span>
     else
-      <span class="ruby-constant">Kernel</span>.puts &quot;compilation complete! Copying compiled code to #{copy_to}&quot;
+      <span class="ruby-comment"># Kernel.puts &quot;compilation complete! Copying compiled code to #{copy_to}&quot;</span>
       return
     end
     sleep(5)