neptune 0.0.6 → 0.0.7

data/doc/README.html

@@ -84,25 +84,42 @@
 Platforms</p>
 
 <p>Neptune provides programmers with a simple interface by which they can
-deploy MPI, X10, and MapReduce jobs to without needing to know the
-particulars of the underlying cloud platform. You only need to give Neptune
-your code, tell it how many machines to run on and where to put the output:
-Neptune handles everything else. No more writing configuration files,
-having to start up NFS on all your machines, yada yada yada. Neptune works
-together with supported cloud platforms (currently AppScale is recommended)
-and can deploy over anything AppScale can - Xen or KVM virtual machines as
-well as Eucalyptus and Amazon EC2. There’s nothing virtualization
-specific in there, so in theory any machine installed with the AppScale
-software should work fine.</p>
+deploy MPI, X10, MapReduce, UPC, and Erlang jobs to without needing to know
+the particulars of the underlying cloud platform. You only need to give
+Neptune your code, tell it how many machines to run on and where to put the
+output: Neptune handles everything else. No more writing configuration
+files, having to start up NFS on all your machines, yada yada yada. Neptune
+works together with supported cloud platforms (currently AppScale is
+recommended) and can deploy over anything AppScale can - Xen or KVM virtual
+machines as well as Eucalyptus and Amazon EC2. There’s nothing
+virtualization specific in there, so in theory any machine installed with
+the AppScale software should work fine.</p>
 
 <p>Although Neptune is designed to automate deploying HPC jobs, it also can be
 used to deploy other types of software. For example, Neptune has support
 for user-specified scaling of the underlying cloud platform: users can
 write Neptune jobs that manually add load balancers, application servers,
-or database servers to a running AppScale deployment.</p>
-
-<p>Sample Neptune job scripts can be found in samples. Test scripts will be
-added to test in the next release.</p>
+or database servers to a running AppScale deployment. Additionally, as of
+version 0.0.2, remote compiling can also be performed: just give Neptune
+the path to the directory you want to compile and be sure to include a
+Makefile in it! Neptune will run ‘make’ on it (you can specify which
+target to make as well) and return to you a folder containing the standard
+out and standard error of the make command.</p>
+
+<p>By default, Neptune jobs store their outputs in the underlying database
+that AppScale is running over. As of Neptune 0.0.5, job outputs can also be
+stored in Amazon S3, Eucalyptus Walrus, and Google Storage.</p>
+
+<p>Sample Neptune job scripts can be found in samples. Test cases can be found
+in the test folder, with the standard naming convention</p>
+<ul><li>
+<p>ts_neptune is the test suite runner, with tc_* containing test</p>
+</li></ul>
+
+<p>cases for each type of job that Neptune offers. Before running ts_neptune,
+you should export the environment variable APPSCALE_HEAD_NODE, which should
+be set to the IP address of the AppScale machine that runs the Shadow
+daemon (a.k.a. the Master AppController).</p>
 
 <p>Developed by Chris Bunch as part of the AppScale project. See <a
 href="LICENSE.html">LICENSE</a> for the specifics of the New BSD License by
@@ -110,9 +127,10 @@ which Neptune is released.</p>
 
 <p>Check us out on the web:</p>
 
-<p><a href="http://code.google.com/p/appscale">code.google.com/p/appscale</a></p>
+<p><a href="http://neptune-lang.org">neptune-lang.org</a></p>
 
-<p><a href="http://appscale.cs.ucsb.edu">appscale.cs.ucsb.edu</a></p>
+<p><a href="http://code.google.com/p/appscale">code.google.com/p/appscale</a>
+<a href="http://appscale.cs.ucsb.edu">appscale.cs.ucsb.edu</a></p>
 
 <p>Contributors welcome! We would love to add support for other cloud
 platforms and test out Neptune more on non-virtualized deployments, as well
@@ -125,7 +143,27 @@ to that as it becomes available.</p>
 
 <p>Version History:</p>
 
-<p>January 27, 20111 - 0.0.1 released, with initial support for MPI, X10, and
+<p>April 2, 2001 - 0.0.7 released, adding automatic test suite and many bug
+fixes for all scenarios. rcov can also be used to generate test coverage
+information: current coverage stats can be found in coverage directory.
+mapreduce broken at the moment - will fix in next release</p>
+
+<p>March 28, 2011 - 0.0.6 released, adding support for input jobs, so users
+can place data in the datastore without having to run any computation</p>
+
+<p>March 18, 2011 - 0.0.5 released, adding support for storage outside of
+AppScale to be used. Tested and working with Amazon S3 and Google Storage</p>
+
+<p>February 10, 2011 - 0.0.4 released, adding UPC and Erlang support, and
+restructuring syntax to pass in hashes to method calls instead of passing
+in blocks</p>
+
+<p>February 4, 2011 - 0.0.3 released, allowing users to use Neptune properly
+as a gem within Ruby code</p>
+
+<p>February 4, 2011 - 0.0.2 released, adding support for remote compiling</p>
+
+<p>January 27, 2011 - 0.0.1 released, with initial support for MPI, X10, and
 MapReduce.</p>
 
 <h1>end</h1>

data/doc/bin/neptune.html

@@ -24,18 +24,14 @@
   <div id="metadata">
     <dl>
       <dt class="modified-date">Last Modified</dt>
-      <dd class="modified-date">Thu Jan 27 22:04:31 -0800 2011</dd>
+      <dd class="modified-date">Fri Feb 04 19:49:21 -0800 2011</dd>
 
       
       <dt class="requires">Requires</dt>
       <dd class="requires">
         <ul>
         
-          <li>app_controller_client</li>
-        
-          <li>common_functions</li>
-        
-          <li>job</li>
+          <li>neptune</li>
         
         </ul>
       </dd>

data/doc/created.rid

@@ -1,7 +1,7 @@
-Tue, 01 Feb 2011 12:02:35 -0800
-./lib/common_functions.rb	Thu, 27 Jan 2011 12:52:03 -0800
+Sat, 02 Apr 2011 09:03:26 -0700
+./lib/common_functions.rb	Wed, 30 Mar 2011 21:31:10 -0700
 ./LICENSE	Thu, 27 Jan 2011 13:24:30 -0800
-./lib/job.rb	Mon, 31 Jan 2011 19:32:41 -0800
-./lib/app_controller_client.rb	Mon, 31 Jan 2011 17:47:55 -0800
-./README	Mon, 31 Jan 2011 19:32:13 -0800
-./bin/neptune	Thu, 27 Jan 2011 22:04:31 -0800
+./lib/app_controller_client.rb	Sun, 27 Mar 2011 23:37:41 -0700
+./lib/neptune.rb	Thu, 31 Mar 2011 10:11:31 -0700
+./README	Sat, 02 Apr 2011 09:03:23 -0700
+./bin/neptune	Fri, 04 Feb 2011 19:49:21 -0800

data/doc/index.html

@@ -69,16 +69,20 @@
     
       <li><a href="AppControllerClient.html#method-i-get_output">#get_output &mdash; AppControllerClient</a></li>
     
-      <li><a href="Object.html#method-i-job">#job &mdash; Object</a></li>
-    
       <li><a href="AppControllerClient.html#method-i-make_call">#make_call &mdash; AppControllerClient</a></li>
     
+      <li><a href="Object.html#method-i-neptune">#neptune &mdash; Object</a></li>
+    
       <li><a href="Object.html#method-i-preprocess_compile">#preprocess_compile &mdash; Object</a></li>
     
+      <li><a href="Object.html#method-i-preprocess_erlang">#preprocess_erlang &mdash; Object</a></li>
+    
       <li><a href="Object.html#method-i-preprocess_mapreduce">#preprocess_mapreduce &mdash; Object</a></li>
     
       <li><a href="Object.html#method-i-preprocess_mpi">#preprocess_mpi &mdash; Object</a></li>
     
+      <li><a href="AppControllerClient.html#method-i-put_input">#put_input &mdash; AppControllerClient</a></li>
+    
       <li><a href="AppControllerClient.html#method-i-set_acl">#set_acl &mdash; AppControllerClient</a></li>
     
       <li><a href="AppControllerClient.html#method-i-start_neptune_job">#start_neptune_job &mdash; AppControllerClient</a></li>

data/doc/lib/app_controller_client_rb.html

@@ -24,7 +24,7 @@
   <div id="metadata">
     <dl>
       <dt class="modified-date">Last Modified</dt>
-      <dd class="modified-date">Mon Jan 31 17:47:55 -0800 2011</dd>
+      <dd class="modified-date">Sun Mar 27 23:37:41 -0700 2011</dd>
 
       
       <dt class="requires">Requires</dt>

data/doc/lib/common_functions_rb.html

@@ -24,7 +24,7 @@
   <div id="metadata">
     <dl>
       <dt class="modified-date">Last Modified</dt>
-      <dd class="modified-date">Thu Jan 27 12:52:03 -0800 2011</dd>
+      <dd class="modified-date">Wed Mar 30 21:31:10 -0700 2011</dd>
 
       
       <dt class="requires">Requires</dt>

data/doc/lib/neptune_rb.html

@@ -0,0 +1,56 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
+
+  <title>File: neptune.rb [RDoc Documentation]</title>
+
+  <link type="text/css" media="screen" href="../rdoc.css" rel="stylesheet" />
+
+  <script src="../js/jquery.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="../js/thickbox-compressed.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="../js/quicksearch.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="../js/darkfish.js" type="text/javascript"
+    charset="utf-8"></script>
+</head>
+
+<body class="file file-popup">
+  <div id="metadata">
+    <dl>
+      <dt class="modified-date">Last Modified</dt>
+      <dd class="modified-date">Thu Mar 31 10:11:31 -0700 2011</dd>
+
+      
+      <dt class="requires">Requires</dt>
+      <dd class="requires">
+        <ul>
+        
+          <li>app_controller_client</li>
+        
+          <li>common_functions</li>
+        
+        </ul>
+      </dd>
+      
+
+      
+    </dl>
+  </div>
+
+  <div id="documentation">
+    
+    <div class="description">
+      <h2>Description</h2>
+      
+    </div>
+    
+  </div>
+</body>
+</html>
+

data/lib/common_functions.rb

@@ -15,7 +15,6 @@ require 'yaml'
 # often needed to determine which machine should be used for computation
 # or to copy over code and input files.
 module CommonFunctions
-
   # Copies a file to the Shadow node (head node) within AppScale. 
   # The caller specifies
   # the local file location, the destination where the file should be

data/lib/neptune.rb

@@ -68,8 +68,8 @@ def preprocess_compile(job_data)
   keyname = job_data["@keyname"]
   shadow_ip = CommonFunctions.get_from_yaml(keyname, :shadow)
 
-  ssh_args = "-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no 2>&1 root@#{shadow_ip}"
-  remove_dir = "ssh #{ssh_args} 'rm -rf #{dest}'"
+  ssh_args = "-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no root@#{shadow_ip}"
+  remove_dir = "ssh #{ssh_args} 'rm -rf #{dest}' 2>&1"
   #puts remove_dir
   `#{remove_dir}`
 
@@ -99,9 +99,10 @@ end
 # specified to us to copy over an input file, we do that as well:
 # AppScale will copy it into HDFS for us.
 def preprocess_mapreduce(job_data)
-  items_to_copy = ["@map", "@reduce"] if job_data["@map"] and job_data["@reduce"]
+  return
+  #items_to_copy = ["@map", "@reduce"] if job_data["@map"] and job_data["@reduce"]
   items_to_copy = ["@mapreducejar"] if job_data["@mapreducejar"]
-  items_to_copy << "@input" if job_data["@copy_input"]
+  #items_to_copy << "@input" if job_data["@copy_input"]
   items_to_copy.each { |item|
     source = File.expand_path(job_data[item])
     unless File.exists?(source)
@@ -150,7 +151,9 @@ def preprocess_mpi(job_data)
   dest_code = "/tmp/thempicode"
 
   keyname = job_data["@keyname"]
+  puts "Copying over code..."
   CommonFunctions.scp_to_shadow(source_code, dest_code, keyname)
+  puts "Done copying code!"
 end
 
 # TODO: actually use me!
@@ -190,6 +193,11 @@ def neptune(params)
   job_data["@keyname"] = keyname || "appscale"
   type = job_data["@type"]
 
+  if type == "upc" or type == "x10"
+    job_data["@type"] = "mpi"
+    type = "mpi"
+  end
+
   if job_data["@nodes_to_use"].class == Hash
     job_data["@nodes_to_use"] = job_data["@nodes_to_use"].to_a.flatten
   end
@@ -232,16 +240,6 @@ def neptune(params)
           end
         end
       }
-
-      # the rightscale gems won't take the s3 url if it has http or https on 
-      # the front, so rip it off first - it also doesn't like a trailing slash
-      s3_url = job_data["@S3_URL"]
-      puts "s3 url is now #{s3_url}"
-      if s3_url =~ /\Ahttp[s]?:\/\/(.*)\/\Z/
-        s3_url = $1
-      end
-      puts "s3 url is now #{s3_url}"
-      job_data["@S3_URL"] = s3_url
     end
   else
     job_data["@storage"] = "appdb"
@@ -257,7 +255,7 @@ def neptune(params)
 
   do_preprocessing(job_data) 
 
-  ssh_args = "-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no 2>&1"
+  ssh_args = "-i ~/.appscale/#{keyname}.key -o StrictHostkeyChecking=no "
 
   if type == "input"
     # copy file to remote
@@ -275,6 +273,7 @@ def neptune(params)
     `#{scp_cmd}`
 
     job_data["@local"] = remote
+    puts "job data = #{job_data.inspect}"
     return controller.put_input(job_data)
   elsif type == "output"
     return controller.get_output(job_data)
@@ -290,7 +289,7 @@ def neptune(params)
     copy_to = job_data["@copy_to"]
 
     loop {
-      ssh_command = "ssh #{ssh_args} root@#{shadow_ip} 'ls #{compiled_location}'"
+      ssh_command = "ssh #{ssh_args} root@#{shadow_ip} 'ls #{compiled_location}' 2>&1"
       #puts ssh_command
       result = `#{ssh_command}`
       #puts "result was [#{result}]"
@@ -307,8 +306,8 @@ def neptune(params)
     #puts rm_local
     `#{rm_local}`
 
-    scp_command = "scp -r #{ssh_args}:#{compiled_location} #{copy_to}"
-    #puts scp_command
+    scp_command = "scp -r #{ssh_args} root@#{shadow_ip}:#{compiled_location} #{copy_to} 2>&1"
+    puts scp_command
     `#{scp_command}`
 
     out = File.open("#{copy_to}/compile_out") { |f| f.read.chomp! }

data/test/tc_c.rb

@@ -0,0 +1,57 @@
+
+$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
+require 'neptune'
+
+$:.unshift File.join(File.dirname(__FILE__), "..", "test")
+require 'test_helper'
+
+require 'test/unit'
+
+class TestC < Test::Unit::TestCase
+  # unlike the other language interfaces, we don't run c code yet
+  # just compile it - this may change in the future
+
+  def test_c_compile
+    ring_code = <<BAZ
+#include <stdio.h>
+
+int main() {
+  printf("hello world!");
+  return 0;
+}
+BAZ
+
+    contents = TestHelper.get_random_alphanumeric(1024)
+    folder = "hello-world-#{TestHelper.get_random_alphanumeric}"
+    source = "HelloWorld.c"
+
+    tmp_folder = "/tmp/#{folder}"
+    FileUtils.mkdir_p(tmp_folder)
+    compiled = "#{tmp_folder}-compiled"
+    compiled_code = "#{compiled}/HelloWorld"
+
+    local = "#{tmp_folder}/#{source}"
+    TestHelper.write_file(local, ring_code)
+
+    output = TestHelper.get_output_location(folder)
+
+    compile_c_code(tmp_folder, source, compiled)
+
+    FileUtils.rm_rf(tmp_folder)
+    FileUtils.rm_rf(compiled)
+  end
+
+  def compile_c_code(location, main_file, compiled)
+    std_out, std_err = TestHelper.compile_code(location, main_file, compiled)
+
+    make = "gcc -o HelloWorld HelloWorld.c -Wall"
+    msg = "The C Hello World code did not compile as expected. It should have " +
+      "compiled with the command [#{make}] instead of [#{std_out}]."
+    assert(std_out.include?(make), msg)
+
+    msg = "The C Hello World code did not compile successfully. It reported " +
+      "the following error: #{std_err}"
+    assert_nil(std_err, msg)
+  end
+end 
+

data/test/tc_dfsp.rb

@@ -0,0 +1,37 @@
+
+class TestDFSP < Test::Unit::TestCase
+  def test_dfsp
+    STORAGE_TYPES.each { |storage|
+      run_dfsp(storage)
+    }
+  end
+
+  def run_dfsp(storage)
+    expected_output = "sim_output"
+    contents = TestHelper.get_random_alphanumeric(1024)
+    folder = "dfsp-#{TestHelper.get_random_alphanumeric}"
+    output = TestHelper.get_output_location(folder, storage)
+
+    start_dfsp_code(output, storage)
+    get_dfsp_output(output, expected_output, storage)
+ end
+
+ def start_dfsp_code(output, storage)
+   params = { :simulations => 10 }
+   status = TestHelper.start_job("dfsp", nil, output, storage, params)
+
+   msg = "Your job was not started successfully. The failure message " +
+     "reported was #{status[:msg]}"
+   assert_equal(status[:result], :success, msg)
+  end
+
+  def get_dfsp_output(output, expected, storage)
+    result = TestHelper.get_job_output(output, storage)
+
+    msg = "The DFSP job you ran did not return the expected result. " +
+      "We expected to see [#{expected}] but instead saw [#{result}]"
+    success = result.include?(expected)
+    assert(success, msg)
+  end
+end 
+

data/test/tc_dwssa.rb

@@ -0,0 +1,38 @@
+
+class TestDWSSA < Test::Unit::TestCase
+  def test_dwssa
+    STORAGE_TYPES.each { |storage|
+      run_dwssa(storage)
+    }
+  end
+
+  def run_dwssa(storage)
+    expected_output = ""
+    contents = TestHelper.get_random_alphanumeric(1024)
+    folder = "dwssa-#{TestHelper.get_random_alphanumeric}"
+    output = TestHelper.get_output_location(folder, storage)
+
+    start_dwssa_code(output, storage)
+    get_dwssa_output(output, expected_output, storage)
+ end
+
+  def start_dwssa_code(output, storage)
+    params = { :simulations => 10 }
+    status = TestHelper.start_job("cewssa", nil, output, storage, params)
+
+    msg = "Your job was not started successfully. The failure message " +
+      "reported was #{status[:msg]}"
+    assert_equal(status[:result], :success, msg)
+  end
+
+  def get_dwssa_output(output, expected, storage)
+    result = TestHelper.get_job_output(output, storage)
+
+    msg = "The dwSSA job you ran did not return the expected result. " +
+      "We expected to see [#{expected}] but instead saw [#{result}]"
+    success = result.include?(expected)
+    assert(success, msg)
+    sleep(30) # wait for appscale to free up nodes
+  end
+end 
+