neptune 0.0.6 → 0.0.7

data/README

@@ -32,8 +32,13 @@ By default, Neptune jobs store their outputs in the underlying database
 that AppScale is running over. As of Neptune 0.0.5, job outputs can
 also be stored in Amazon S3, Eucalyptus Walrus, and Google Storage.
 
-Sample Neptune job scripts can be found in samples. Test scripts will
-be added to the 'test' folder soon.
+Sample Neptune job scripts can be found in samples. Test cases can
+be found in the test folder, with the standard naming convention
+- ts_neptune is the test suite runner, with tc_* containing test
+cases for each type of job that Neptune offers. Before running
+ts_neptune, you should export the environment variable APPSCALE_HEAD_NODE,
+which should be set to the IP address of the AppScale machine that runs
+the Shadow daemon (a.k.a. the Master AppController).
 
 Developed by Chris Bunch as part of the AppScale project.
 See LICENSE for the specifics of the New BSD License
@@ -41,8 +46,9 @@ by which Neptune is released.
 
 Check us out on the web:
 
-http://code.google.com/p/appscale
+http://neptune-lang.org
 
+http://code.google.com/p/appscale
 http://appscale.cs.ucsb.edu
 
 Contributors welcome! We would love to add support for other
@@ -57,6 +63,16 @@ in for a link to that as it becomes available.
 
 Version History:
 
+April 2, 2001 - 0.0.7 released, adding automatic test suite
+and many bug fixes for all scenarios. rcov can also be used
+to generate test coverage information: current coverage stats
+can be found in coverage directory. mapreduce broken at the
+moment - will fix in next release
+
+March 28, 2011 - 0.0.6 released, adding support for input jobs,
+so users can place data in the datastore without having to run
+any computation
+
 March 18, 2011 - 0.0.5 released, adding support for storage outside
 of AppScale to be used. Tested and working with Amazon S3 and Google
 Storage

data/doc/AppControllerClient.html

@@ -79,6 +79,8 @@
           
           <li><a href="#method-i-make_call">#make_call</a></li>
           
+          <li><a href="#method-i-put_input">#put_input</a></li>
+          
           <li><a href="#method-i-set_acl">#set_acl</a></li>
           
           <li><a href="#method-i-start_neptune_job">#start_neptune_job</a></li>
@@ -241,6 +243,7 @@ def initialize(ip, secret)
   
   <span class="ruby-ivar">@conn</span> = <span class="ruby-constant">SOAP</span>::<span class="ruby-constant">RPC</span>::<span class="ruby-constant">Driver</span>.new(&quot;https://#{@ip}:17443&quot;)
   <span class="ruby-ivar">@conn</span>.add_method(<span class="ruby-string">&quot;neptune_start_job&quot;</span>, <span class="ruby-string">&quot;job_data&quot;</span>, <span class="ruby-string">&quot;secret&quot;</span>)
+  <span class="ruby-ivar">@conn</span>.add_method(<span class="ruby-string">&quot;neptune_put_input&quot;</span>, <span class="ruby-string">&quot;job_data&quot;</span>, <span class="ruby-string">&quot;secret&quot;</span>)
   <span class="ruby-ivar">@conn</span>.add_method(<span class="ruby-string">&quot;neptune_get_output&quot;</span>, <span class="ruby-string">&quot;job_data&quot;</span>, <span class="ruby-string">&quot;secret&quot;</span>)
   <span class="ruby-ivar">@conn</span>.add_method(<span class="ruby-string">&quot;neptune_get_acl&quot;</span>, <span class="ruby-string">&quot;job_data&quot;</span>, <span class="ruby-string">&quot;secret&quot;</span>)
   <span class="ruby-ivar">@conn</span>.add_method(<span class="ruby-string">&quot;neptune_set_acl&quot;</span>, <span class="ruby-string">&quot;job_data&quot;</span>, <span class="ruby-string">&quot;secret&quot;</span>)
@@ -282,7 +285,7 @@ end</pre>
           <div class="method-source-code"
             id="compile_code-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 137</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 152</span>
 def compile_code(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -324,7 +327,7 @@ exceptions mirror that of start_neptune_job.</p>
           <div class="method-source-code"
             id="get_acl-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 114</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 129</span>
 def get_acl(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -361,7 +364,7 @@ Within AppScale, a special application runs, referred to as the Repository,
 which provides a key-value interface to Neptune job data. Data is stored as
 though it were on a file system, therefore output be of the usual form
 /folder/filename . Currently the contents of the file is returned as a
-string to the caller, but as this is inefficient for non-trivial output
+string to the caller, but as this may be inefficient for non-trivial output
 jobs, the next version of Neptune will add an additional call to directly
 copy the output to a file on the local filesystem. See <a
 href="AppControllerClient.html#method-i-start_neptune_job">start_neptune_job</a>
@@ -373,7 +376,7 @@ used for job_data.</p>
           <div class="method-source-code"
             id="get_output-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 100</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 115</span>
 def get_output(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -423,7 +426,7 @@ The result of the block is returned to the caller.</p>
           <div class="method-source-code"
             id="make_call-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 51</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 52</span>
 def make_call(time, retry_on_except)
   begin
     <span class="ruby-constant">Timeout</span>::timeout(time) {
@@ -452,6 +455,49 @@ end</pre>
         
 
         
+      </div>
+
+    
+      <div id="put_input-method" class="method-detail ">
+        <a name="method-i-put_input"></a>
+
+        
+        <div class="method-heading">
+          <span class="method-name">put_input</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        
+
+        <div class="method-description">
+          
+          <p>Stores a file stored on the user’s local file system in the underlying
+database. The user can specify to use either the underlying database that
+AppScale is using, or alternative storage mechanisms (as of writing, Google
+Storage, Amazon S3, and Eucalyptus Walrus are supported) via the storage
+parameter.</p>
+          
+
+          
+          <div class="method-source-code"
+            id="put_input-source">
+<pre>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 96</span>
+def put_input(job_data)
+  result = <span class="ruby-string">&quot;&quot;</span>
+  make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
+    result = conn.neptune_put_input(job_data, <span class="ruby-ivar">@secret</span>)
+  }  
+  abort(result) if result =~ <span class="ruby-regexp">/Error:/</span>
+  return result
+end</pre>
+          </div>
+          
+        </div>
+
+        
+
+        
       </div>
 
     
@@ -480,7 +526,7 @@ that of start_neptune_job.</p>
           <div class="method-source-code"
             id="set_acl-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 128</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 143</span>
 def set_acl(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 
@@ -526,7 +572,7 @@ is the result returned from the AppController.</p>
           <div class="method-source-code"
             id="start_neptune_job-source">
 <pre>
-<span class="ruby-comment"># File lib/app_controller_client.rb, line 81</span>
+<span class="ruby-comment"># File lib/app_controller_client.rb, line 82</span>
 def start_neptune_job(job_data)
   result = <span class="ruby-string">&quot;&quot;</span>
   make_call(<span class="ruby-constant">NO_TIMEOUT</span>, false) { 

data/doc/CommonFunctions.html

@@ -172,7 +172,7 @@ instead.</p>
           <div class="method-source-code"
             id="get_from_yaml-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 79</span>
+<span class="ruby-comment"># File lib/common_functions.rb, line 82</span>
 def self.get_from_yaml(keyname, tag, required=true)
   location_file = <span class="ruby-constant">File</span>.expand_path(&quot;~/.appscale/locations-#{keyname}.yaml&quot;)
 
@@ -227,7 +227,7 @@ function, as the secret is stored in a YAML file.</p>
           <div class="method-source-code"
             id="get_secret_key-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 104</span>
+<span class="ruby-comment"># File lib/common_functions.rb, line 107</span>
 def self.get_secret_key(keyname, required=true)
   return <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :secret)
 end</pre>
@@ -247,7 +247,7 @@ end</pre>
         
         <div class="method-heading">
           <span class="method-name">scp_file</span><span
-            class="method-args">(local_file_loc, remote_file_loc, target_ip, public_key_loc)</span>
+            class="method-args">(local_file_loc, remote_file_loc, target_ip, public_key_loc, is_dir=false)</span>
           <span class="method-click-advice">click to toggle source</span>
         </div>
         
@@ -259,7 +259,8 @@ information from <a
 href="CommonFunctions.html#method-c-scp_to_shadow">scp_to_shadow</a>,
 attempts to use scp to copy the file over. Aborts if the scp fails, which
 can occur if the network is down, if a bad keyname is provided, or if the 
-wrong IP is given.</p>
+wrong IP is given. If the user specifies that the file to copy is actually
+a directory, we append the -r flag to scp as well.</p>
           
 
           
@@ -267,19 +268,22 @@ wrong IP is given.</p>
             id="scp_file-source">
 <pre>
 <span class="ruby-comment"># File lib/common_functions.rb, line 37</span>
-def self.scp_file(local_file_loc, remote_file_loc, target_ip, public_key_loc)
+def self.scp_file(local_file_loc, remote_file_loc, target_ip, public_key_loc, is_dir=false)
   cmd = <span class="ruby-string">&quot;&quot;</span>
   local_file_loc = <span class="ruby-constant">File</span>.expand_path(local_file_loc)
-  
+
+  ssh_args = <span class="ruby-string">&quot;-o StrictHostkeyChecking=no 2&gt;&amp;1&quot;</span>
+  ssh_args &lt;&lt; <span class="ruby-string">&quot; -r &quot;</span> if is_dir
+
   if public_key_loc.class == <span class="ruby-constant">Array</span>
     public_key_loc.each { |key|
       key = <span class="ruby-constant">File</span>.expand_path(key)
     }
     
-    cmd = &quot;scp -i #{public_key_loc.join(' -i ')} -o StrictHostkeyChecking=no 2&gt;&amp;1 #{local_file_loc} root@#{target_ip}:#{remote_file_loc}&quot;
+    cmd = &quot;scp -i #{public_key_loc.join(' -i ')} #{ssh_args} #{local_file_loc} root@#{target_ip}:#{remote_file_loc}&quot;
   else
     public_key_loc = <span class="ruby-constant">File</span>.expand_path(public_key_loc)
-    cmd = &quot;scp -i #{public_key_loc} -o StrictHostkeyChecking=no 2&gt;&amp;1 #{local_file_loc} root@#{target_ip}:#{remote_file_loc}&quot;
+    cmd = &quot;scp -i #{public_key_loc} #{ssh_args} #{local_file_loc} root@#{target_ip}:#{remote_file_loc}&quot;
   end
 
   cmd &lt;&lt; <span class="ruby-string">&quot;; echo $? &gt;&gt; ~/.appscale/retval&quot;</span>
@@ -318,7 +322,7 @@ end</pre>
         
         <div class="method-heading">
           <span class="method-name">scp_to_shadow</span><span
-            class="method-args">(local_file_loc, remote_file_loc, keyname)</span>
+            class="method-args">(local_file_loc, remote_file_loc, keyname, is_dir=false)</span>
           <span class="method-click-advice">click to toggle source</span>
         </div>
         
@@ -335,12 +339,12 @@ by the Neptune job given, but defaults to ”appscale” if not provided.</p>
           <div class="method-source-code"
             id="scp_to_shadow-source">
 <pre>
-<span class="ruby-comment"># File lib/common_functions.rb, line 25</span>
-def self.scp_to_shadow(local_file_loc, remote_file_loc, keyname)
+<span class="ruby-comment"># File lib/common_functions.rb, line 24</span>
+def self.scp_to_shadow(local_file_loc, remote_file_loc, keyname, is_dir=false)
   shadow_ip = <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :shadow)
   ssh_key = <span class="ruby-constant">File</span>.expand_path(&quot;~/.appscale/#{keyname}.key&quot;)
 
-  self.scp_file(local_file_loc, remote_file_loc, shadow_ip, ssh_key)
+  self.scp_file(local_file_loc, remote_file_loc, shadow_ip, ssh_key, is_dir)
 end</pre>
           </div>
           

data/doc/Object.html

@@ -41,8 +41,8 @@
             <li><a href="./lib/app_controller_client_rb.html?TB_iframe=true&amp;height=550&amp;width=785"
               class="thickbox" title="lib/app_controller_client.rb">lib/app_controller_client.rb</a></li>
           
-            <li><a href="./lib/job_rb.html?TB_iframe=true&amp;height=550&amp;width=785"
-              class="thickbox" title="lib/job.rb">lib/job.rb</a></li>
+            <li><a href="./lib/neptune_rb.html?TB_iframe=true&amp;height=550&amp;width=785"
+              class="thickbox" title="lib/neptune.rb">lib/neptune.rb</a></li>
           
           </ul>
         </div>
@@ -74,10 +74,12 @@
           
           <li><a href="#method-i-do_preprocessing">#do_preprocessing</a></li>
           
-          <li><a href="#method-i-job">#job</a></li>
+          <li><a href="#method-i-neptune">#neptune</a></li>
           
           <li><a href="#method-i-preprocess_compile">#preprocess_compile</a></li>
           
+          <li><a href="#method-i-preprocess_erlang">#preprocess_erlang</a></li>
+          
           <li><a href="#method-i-preprocess_mapreduce">#preprocess_mapreduce</a></li>
           
           <li><a href="#method-i-preprocess_mpi">#preprocess_mpi</a></li>
@@ -162,6 +164,24 @@ timeout. The next version should replace this and properly timeout and not
 use long calls unless necessary.</p></dd>
         
       
+        <dt><a name="NO_NODES_NEEDED">NO_NODES_NEEDED</a></dt>
+        
+        <dd class="description"><p>A list of Neptune jobs that do not require nodes to be spawned up for
+computation</p></dd>
+        
+      
+        <dt><a name="NO_OUTPUT_NEEDED">NO_OUTPUT_NEEDED</a></dt>
+        
+        <dd class="description"><p>A list of Neptune jobs that do not require the output to be specified
+beforehand</p></dd>
+        
+      
+        <dt><a name="ALLOWED_STORAGE_TYPES">ALLOWED_STORAGE_TYPES</a></dt>
+        
+        <dd class="description"><p>A list of storage mechanisms that we can use to store and retrieve data to
+for Neptune jobs.</p></dd>
+        
+      
         <dt><a name="NEED_PREPROCESSING">NEED_PREPROCESSING</a></dt>
         
         <dd class="description"><p>A list of jobs that require some kind of work to be done before the actual
@@ -203,7 +223,7 @@ method to use based on the type of the job that the user has asked to run.</p>
           <div class="method-source-code"
             id="do_preprocessing-source">
 <pre>
-<span class="ruby-comment"># File lib/job.rb, line 34</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 49</span>
 def do_preprocessing(job_data)
   job_type = job_data[<span class="ruby-string">&quot;@type&quot;</span>]
   return unless <span class="ruby-constant">NEED_PREPROCESSING</span>.include?(job_type)
@@ -221,13 +241,13 @@ end</pre>
       </div>
 
     
-      <div id="job-method" class="method-detail ">
-        <a name="method-i-job"></a>
+      <div id="neptune-method" class="method-detail ">
+        <a name="method-i-neptune"></a>
 
         
         <div class="method-heading">
-          <span class="method-name">job</span><span
-            class="method-args">(name, &block)</span>
+          <span class="method-name">neptune</span><span
+            class="method-args">(params)</span>
           <span class="method-click-advice">click to toggle source</span>
         </div>
         
@@ -244,15 +264,14 @@ vice-versa).</p>
 
           
           <div class="method-source-code"
-            id="job-source">
+            id="neptune-source">
 <pre>
-<span class="ruby-comment"># File lib/job.rb, line 119</span>
-def job(name, &amp;block)
+<span class="ruby-comment"># File lib/neptune.rb, line 175</span>
+def neptune(params)
   puts <span class="ruby-string">&quot;Received a request to run a job.&quot;</span>
-  puts name
-  block.call()
+  puts params[:type]
 
-  keyname = <span class="ruby-ivar">@keyname</span> || <span class="ruby-string">&quot;appscale&quot;</span>
+  keyname = params[:keyname] || <span class="ruby-string">&quot;appscale&quot;</span>
 
   shadow_ip = <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :shadow)
   secret = <span class="ruby-constant">CommonFunctions</span>.get_secret_key(keyname)
@@ -260,20 +279,65 @@ def job(name, &amp;block)
   ssh_key = <span class="ruby-constant">File</span>.expand_path(&quot;~/.appscale/#{keyname}.key&quot;)
 
   job_data = {}
-  instance_variables.each { |var|
-    job_data[var] = instance_variable_get(var)
+  params.each { |k, v|
+    key = &quot;@#{k}&quot;
+    job_data[key] = v
   }
 
   job_data[<span class="ruby-string">&quot;@job&quot;</span>] = nil
-  job_data[<span class="ruby-string">&quot;@type&quot;</span>] = name
-  job_data[<span class="ruby-string">&quot;@keyname&quot;</span>] = keyname
+  job_data[<span class="ruby-string">&quot;@keyname&quot;</span>] = keyname || <span class="ruby-string">&quot;appscale&quot;</span>
+  type = job_data[<span class="ruby-string">&quot;@type&quot;</span>]
+
+  if type == <span class="ruby-string">&quot;upc&quot;</span> or type == <span class="ruby-string">&quot;x10&quot;</span>
+    job_data[<span class="ruby-string">&quot;@type&quot;</span>] = <span class="ruby-string">&quot;mpi&quot;</span>
+    type = <span class="ruby-string">&quot;mpi&quot;</span>
+  end
 
-  if (job_data[<span class="ruby-string">&quot;@output&quot;</span>].nil? or job_data[<span class="ruby-string">&quot;@output&quot;</span>] == <span class="ruby-string">&quot;&quot;</span>)
-    abort(<span class="ruby-string">&quot;Job output must be specified&quot;</span>)
+  if job_data[<span class="ruby-string">&quot;@nodes_to_use&quot;</span>].class == <span class="ruby-constant">Hash</span>
+    job_data[<span class="ruby-string">&quot;@nodes_to_use&quot;</span>] = job_data[<span class="ruby-string">&quot;@nodes_to_use&quot;</span>].to_a.flatten
   end
 
-  if job_data[<span class="ruby-string">&quot;@output&quot;</span>][0].chr != <span class="ruby-string">&quot;/&quot;</span>
-    abort(<span class="ruby-string">&quot;Job output must begin with a slash ('/')&quot;</span>)
+  if !<span class="ruby-constant">NO_OUTPUT_NEEDED</span>.include?(type)
+    if (job_data[<span class="ruby-string">&quot;@output&quot;</span>].nil? or job_data[<span class="ruby-string">&quot;@output&quot;</span>] == <span class="ruby-string">&quot;&quot;</span>)
+      abort(<span class="ruby-string">&quot;Job output must be specified&quot;</span>)
+    end
+
+    if job_data[<span class="ruby-string">&quot;@output&quot;</span>][0].chr != <span class="ruby-string">&quot;/&quot;</span>
+      abort(<span class="ruby-string">&quot;Job output must begin with a slash ('/')&quot;</span>)
+    end
+  end
+
+  if job_data[<span class="ruby-string">&quot;@storage&quot;</span>]
+    storage = job_data[<span class="ruby-string">&quot;@storage&quot;</span>]
+    unless <span class="ruby-constant">ALLOWED_STORAGE_TYPES</span>.include?(storage)
+      msg = &quot;Supported storage types are #{ALLOWED_STORAGE_TYPES.join(', ')}&quot; +
+        &quot; - we do not support #{storage}.&quot;
+      abort(msg)
+    end
+
+    <span class="ruby-comment"># Our implementation for storing / retrieving via Google Storage uses</span>
+    <span class="ruby-comment"># the same library as we do for S3 - so just tell it that it's S3</span>
+    if storage == <span class="ruby-string">&quot;gstorage&quot;</span>
+      storage = <span class="ruby-string">&quot;s3&quot;</span>
+      job_data[<span class="ruby-string">&quot;@storage&quot;</span>] = <span class="ruby-string">&quot;s3&quot;</span>
+    end
+
+    if storage == <span class="ruby-string">&quot;s3&quot;</span>
+      [<span class="ruby-string">&quot;EC2_ACCESS_KEY&quot;</span>, <span class="ruby-string">&quot;EC2_SECRET_KEY&quot;</span>, <span class="ruby-string">&quot;S3_URL&quot;</span>].each { |item|
+        unless job_data[&quot;@#{item}&quot;]
+          if <span class="ruby-constant">ENV</span>[item]
+            puts &quot;Using #{item} from environment&quot;
+            job_data[&quot;@#{item}&quot;] = <span class="ruby-constant">ENV</span>[item]
+          else
+            msg = &quot;When storing data to S3, #{item} must be specified or be in &quot; + 
+              <span class="ruby-string">&quot;your environment. Please do so and try again.&quot;</span>
+            abort(msg)
+          end
+        end
+      }
+    end
+  else
+    job_data[<span class="ruby-string">&quot;@storage&quot;</span>] = <span class="ruby-string">&quot;appdb&quot;</span>
   end
 
   <span class="ruby-comment">#if job_data[&quot;@can_run_on&quot;].class == Range</span>
@@ -286,8 +350,27 @@ def job(name, &amp;block)
 
   do_preprocessing(job_data) 
 
-  type = job_data[<span class="ruby-string">&quot;@type&quot;</span>]
-  if type == <span class="ruby-string">&quot;output&quot;</span>
+  ssh_args = &quot;-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no &quot;
+
+  if type == <span class="ruby-string">&quot;input&quot;</span>
+    <span class="ruby-comment"># copy file to remote</span>
+    <span class="ruby-comment"># set location</span>
+    local_file = <span class="ruby-constant">File</span>.expand_path(job_data[<span class="ruby-string">&quot;@local&quot;</span>])
+    if !<span class="ruby-constant">File</span>.exists?(local_file)
+      msg = &quot;the file you specified to copy, #{local_file}, doesn't exist.&quot; + 
+        <span class="ruby-string">&quot; Please specify a file that exists and try again.&quot;</span>
+      abort(msg)
+    end
+
+    remote = &quot;/tmp/neptune-input-#{rand(100000)}&quot;
+    scp_cmd = &quot;scp #{ssh_args} #{local_file} root@#{shadow_ip}:#{remote}&quot;
+    puts scp_cmd
+    `#{scp_cmd}`
+
+    job_data[<span class="ruby-string">&quot;@local&quot;</span>] = remote
+    puts &quot;job data = #{job_data.inspect}&quot;
+    return controller.put_input(job_data)
+  elsif type == <span class="ruby-string">&quot;output&quot;</span>
     return controller.get_output(job_data)
   elsif type == <span class="ruby-string">&quot;get-acl&quot;</span>
     job_data[<span class="ruby-string">&quot;@type&quot;</span>] = <span class="ruby-string">&quot;acl&quot;</span>
@@ -298,14 +381,13 @@ def job(name, &amp;block)
   elsif type == <span class="ruby-string">&quot;compile&quot;</span>
     compiled_location = controller.compile_code(job_data)
 
-    ssh_args = &quot;-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no 2&gt;&amp;1 root@#{shadow_ip}&quot;
     copy_to = job_data[<span class="ruby-string">&quot;@copy_to&quot;</span>]
 
     loop {
-      ssh_command = &quot;ssh #{ssh_args} 'ls #{compiled_location}'&quot;
-      puts ssh_command
+      ssh_command = &quot;ssh #{ssh_args} root@#{shadow_ip} 'ls #{compiled_location}' 2&gt;&amp;1&quot;
+      <span class="ruby-comment">#puts ssh_command</span>
       result = `#{ssh_command}`
-      puts &quot;result was [#{result}]&quot;
+      <span class="ruby-comment">#puts &quot;result was [#{result}]&quot;</span>
       if result =~ <span class="ruby-regexp">/No such file or directory/</span>
         puts <span class="ruby-string">&quot;Still waiting for code to be compiled...&quot;</span>
       else
@@ -315,15 +397,23 @@ def job(name, &amp;block)
       sleep(5)
     }
 
-    scp_command = &quot;scp #{ssh_args}:#{compiled_location} #{copy_to}&quot;
+    rm_local = &quot;rm -rf #{copy_to}&quot;
+    <span class="ruby-comment">#puts rm_local</span>
+    `#{rm_local}`
+
+    scp_command = &quot;scp -r #{ssh_args} root@#{shadow_ip}:#{compiled_location} #{copy_to} 2&gt;&amp;1&quot;
     puts scp_command
     `#{scp_command}`
+
+    out = <span class="ruby-constant">File</span>.open(&quot;#{copy_to}/compile_out&quot;) { |f| f.read.chomp! }
+    err = <span class="ruby-constant">File</span>.open(&quot;#{copy_to}/compile_err&quot;) { |f| f.read.chomp! }
+    return {:out =&gt; out, :err =&gt; err }
   else
     result = controller.start_neptune_job(job_data)
     if result =~ <span class="ruby-regexp">/job is now running\Z/</span>
-      return :success
+      return {:result =&gt; :success, :msg =&gt; result}
     else
-      return :failure
+      return {:result =&gt; :failure, :msg =&gt; result}
     end
   end
 end</pre>
@@ -359,7 +449,7 @@ copy over libraries as well.</p>
           <div class="method-source-code"
             id="preprocess_compile-source">
 <pre>
-<span class="ruby-comment"># File lib/job.rb, line 45</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 60</span>
 def preprocess_compile(job_data)
   code = <span class="ruby-constant">File</span>.expand_path(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
   unless <span class="ruby-constant">File</span>.exists?(code)
@@ -368,9 +458,15 @@ def preprocess_compile(job_data)
 
   suffix = code.split(<span class="ruby-string">'/'</span>)[-1]
   dest = &quot;/tmp/#{suffix}&quot;
-
   keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>]
-  <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(code, dest, keyname)
+  shadow_ip = <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :shadow)
+
+  ssh_args = &quot;-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no root@#{shadow_ip}&quot;
+  remove_dir = &quot;ssh #{ssh_args} 'rm -rf #{dest}' 2&gt;&amp;1&quot;
+  <span class="ruby-comment">#puts remove_dir</span>
+  `#{remove_dir}`
+
+  <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(code, dest, keyname, is_dir=true)
 
   job_data[<span class="ruby-string">&quot;@code&quot;</span>] = dest
 end</pre>
@@ -381,6 +477,49 @@ end</pre>
         
 
         
+      </div>
+
+    
+      <div id="preprocess_erlang-method" class="method-detail ">
+        <a name="method-i-preprocess_erlang"></a>
+
+        
+        <div class="method-heading">
+          <span class="method-name">preprocess_erlang</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        
+
+        <div class="method-description">
+          
+          
+          
+
+          
+          <div class="method-source-code"
+            id="preprocess_erlang-source">
+<pre>
+<span class="ruby-comment"># File lib/neptune.rb, line 81</span>
+def preprocess_erlang(job_data)
+  source_code = <span class="ruby-constant">File</span>.expand_path(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
+  unless <span class="ruby-constant">File</span>.exists?(source_code)
+    file_not_found = &quot;The specified code, #{job_data['@code']},&quot; +
+      <span class="ruby-string">&quot; didn't exist. Please specify one that exists and try again&quot;</span>
+    abort(file_not_found)
+  end
+  dest_code = <span class="ruby-string">&quot;/tmp/&quot;</span>
+
+  keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>]
+  <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(source_code, dest_code, keyname)
+end</pre>
+          </div>
+          
+        </div>
+
+        
+
+        
       </div>
 
     
@@ -409,11 +548,12 @@ will copy it into HDFS for us.</p>
           <div class="method-source-code"
             id="preprocess_mapreduce-source">
 <pre>
-<span class="ruby-comment"># File lib/job.rb, line 67</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 101</span>
 def preprocess_mapreduce(job_data)
-  items_to_copy = [<span class="ruby-string">&quot;@map&quot;</span>, <span class="ruby-string">&quot;@reduce&quot;</span>] if job_data[<span class="ruby-string">&quot;@map&quot;</span>] and job_data[<span class="ruby-string">&quot;@reduce&quot;</span>]
+  return
+  <span class="ruby-comment">#items_to_copy = [&quot;@map&quot;, &quot;@reduce&quot;] if job_data[&quot;@map&quot;] and job_data[&quot;@reduce&quot;]</span>
   items_to_copy = [<span class="ruby-string">&quot;@mapreducejar&quot;</span>] if job_data[<span class="ruby-string">&quot;@mapreducejar&quot;</span>]
-  items_to_copy &lt;&lt; <span class="ruby-string">&quot;@input&quot;</span> if job_data[<span class="ruby-string">&quot;@copy_input&quot;</span>]
+  <span class="ruby-comment">#items_to_copy &lt;&lt; &quot;@input&quot; if job_data[&quot;@copy_input&quot;]</span>
   items_to_copy.each { |item|
     source = <span class="ruby-constant">File</span>.expand_path(job_data[item])
     unless <span class="ruby-constant">File</span>.exists?(source)
@@ -461,18 +601,39 @@ job.</p>
           <div class="method-source-code"
             id="preprocess_mpi-source">
 <pre>
-<span class="ruby-comment"># File lib/job.rb, line 90</span>
+<span class="ruby-comment"># File lib/neptune.rb, line 125</span>
 def preprocess_mpi(job_data)
+  if job_data[<span class="ruby-string">&quot;@procs_to_use&quot;</span>]
+    p = job_data[<span class="ruby-string">&quot;@procs_to_use&quot;</span>]
+    n = job_data[<span class="ruby-string">&quot;@nodes_to_use&quot;</span>]
+    if p &lt; n
+      not_enough_procs = <span class="ruby-string">&quot;When specifying both :procs_to_use and :nodes_to_use&quot;</span> +
+        <span class="ruby-string">&quot;, :procs_to_use must be at least as large as :nodes_to_use. Please &quot;</span> +
+        &quot;change this and try again. You specified :procs_to_use = #{p} and&quot; +
+        &quot;:nodes_to_use = #{n}.&quot;
+      abort(not_enough_procs)
+    end
+  end
+
   source_code = <span class="ruby-constant">File</span>.expand_path(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
   unless <span class="ruby-constant">File</span>.exists?(source_code)
-    file_not_found = &quot;The specified code, #{job_data['@code']},&quot; +
+    file_not_found = &quot;The specified code, #{source_code},&quot; +
       <span class="ruby-string">&quot; didn't exist. Please specify one that exists and try again&quot;</span>
     abort(file_not_found)
   end
+
+  unless <span class="ruby-constant">File</span>.file?(source_code)
+    should_be_file = &quot;The specified code, #{source_code}, was not a file - &quot; +
+      <span class="ruby-string">&quot; it was a directory or symbolic link. Please specify a file and try again.&quot;</span>
+    abort(should_be_file)
+  end
+
   dest_code = <span class="ruby-string">&quot;/tmp/thempicode&quot;</span>
 
   keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>]
+  puts <span class="ruby-string">&quot;Copying over code...&quot;</span>
   <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(source_code, dest_code, keyname)
+  puts <span class="ruby-string">&quot;Done copying code!&quot;</span>
 end</pre>
           </div>