RubyGems - neptune - Versions diffs - 0.0.1 - Mend

neptune 0.0.1

Files changed (79) hide show

data/LICENSE +39 -0
data/README +50 -0
data/bin/neptune +16 -0
data/doc/AppControllerClient.html +468 -0
data/doc/CommonFunctions.html +368 -0
data/doc/LICENSE.html +131 -0
data/doc/Object.html +431 -0
data/doc/README.html +135 -0
data/doc/bin/neptune.html +58 -0
data/doc/created.rid +7 -0
data/doc/images/brick.png +0 -0
data/doc/images/brick_link.png +0 -0
data/doc/images/bug.png +0 -0
data/doc/images/bullet_black.png +0 -0
data/doc/images/bullet_toggle_minus.png +0 -0
data/doc/images/bullet_toggle_plus.png +0 -0
data/doc/images/date.png +0 -0
data/doc/images/find.png +0 -0
data/doc/images/loadingAnimation.gif +0 -0
data/doc/images/macFFBgHack.png +0 -0
data/doc/images/package.png +0 -0
data/doc/images/page_green.png +0 -0
data/doc/images/page_white_text.png +0 -0
data/doc/images/page_white_width.png +0 -0
data/doc/images/plugin.png +0 -0
data/doc/images/ruby.png +0 -0
data/doc/images/tag_green.png +0 -0
data/doc/images/wrench.png +0 -0
data/doc/images/wrench_orange.png +0 -0
data/doc/images/zoom.png +0 -0
data/doc/index.html +90 -0
data/doc/js/darkfish.js +116 -0
data/doc/js/jquery.js +32 -0
data/doc/js/quicksearch.js +114 -0
data/doc/js/thickbox-compressed.js +10 -0
data/doc/lib/app_controller_client_rb.html +60 -0
data/doc/lib/common_functions_rb.html +68 -0
data/doc/lib/job_rb.html +55 -0
data/doc/scripts/appscale/add_appserver_rb.html +52 -0
data/doc/scripts/appscale/add_database_rb.html +52 -0
data/doc/scripts/appscale/add_loadbalancer_rb.html +52 -0
data/doc/scripts/appscale/add_slave_rb.html +52 -0
data/doc/scripts/cewssa/cewssa-output.html +104 -0
data/doc/scripts/cewssa/get_cewssa_output_rb.html +52 -0
data/doc/scripts/cewssa/run_cewssa_rb.html +52 -0
data/doc/scripts/cewssa/test_dfsp_rb.html +52 -0
data/doc/scripts/dfsp/get_dfsp_output_rb.html +52 -0
data/doc/scripts/dfsp/make_output_public_rb.html +52 -0
data/doc/scripts/dfsp/run_dfsp_rb.html +52 -0
data/doc/scripts/dfsp/see_acl_rb.html +52 -0
data/doc/scripts/dfsp/test_dfsp_rb.html +52 -0
data/doc/scripts/mapreduce/get_mapreduce_output_rb.html +52 -0
data/doc/scripts/mapreduce/input-10.html +114 -0
data/doc/scripts/mapreduce/input-30.html +114 -0
data/doc/scripts/mapreduce/input-7.html +104 -0
data/doc/scripts/mapreduce/map_rb.html +56 -0
data/doc/scripts/mapreduce/reduce_rb.html +56 -0
data/doc/scripts/mapreduce/run_java_mr_rb.html +52 -0
data/doc/scripts/mapreduce/run_mapreduce_rb.html +52 -0
data/doc/scripts/mpi/Makefile.html +125 -0
data/doc/scripts/mpi/get_mpi_output_rb.html +52 -0
data/doc/scripts/mpi/hw2_c.html +52 -0
data/doc/scripts/mpi/hw2harness_c.html +52 -0
data/doc/scripts/mpi/hw2harness_h.html +52 -0
data/doc/scripts/mpi/run_mpi_cpi_rb.html +52 -0
data/doc/scripts/mpi/run_mpi_powermethod_rb.html +52 -0
data/doc/scripts/mpi/test_rb.html +52 -0
data/doc/scripts/x10/Makefile.html +106 -0
data/doc/scripts/x10/get_x10_output_rb.html +52 -0
data/doc/scripts/x10/run_x10_kmeans_rb.html +52 -0
data/doc/scripts/x10/run_x10_montypi_rb.html +52 -0
data/doc/scripts/x10/run_x10_nqueens_rb.html +52 -0
data/doc/scripts/x10/run_x10_powermethod_rb.html +52 -0
data/doc/scripts/x10/run_x10_qsort_rb.html +52 -0
data/doc/scripts/x10/test_rb.html +52 -0
data/lib/app_controller_client.rb +135 -0
data/lib/common_functions.rb +107 -0
data/lib/job.rb +161 -0
metadata +145 -0

data/doc/Object.html ADDED Viewed

@@ -0,0 +1,431 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
+  <title>Class: Object</title>
+  <link rel="stylesheet" href="./rdoc.css" type="text/css" media="screen" />
+  <script src="./js/jquery.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/thickbox-compressed.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/quicksearch.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/darkfish.js" type="text/javascript"
+    charset="utf-8"></script>
+</head>
+<body class="class">
+  <div id="metadata">
+    <div id="home-metadata">
+      <div id="home-section" class="section">
+        <h3 class="section-header">
+          <a href="./index.html">Home</a>
+          <a href="./index.html#classes">Classes</a>
+          <a href="./index.html#methods">Methods</a>
+        </h3>
+      </div>
+    </div>
+    <div id="file-metadata">
+      <div id="file-list-section" class="section">
+        <h3 class="section-header">In Files</h3>
+        <div class="section-body">
+          <ul>
+            <li><a href="./lib/app_controller_client_rb.html?TB_iframe=true&amp;height=550&amp;width=785"
+              class="thickbox" title="lib/app_controller_client.rb">lib/app_controller_client.rb</a></li>
+            <li><a href="./lib/job_rb.html?TB_iframe=true&amp;height=550&amp;width=785"
+              class="thickbox" title="lib/job.rb">lib/job.rb</a></li>
+          </ul>
+        </div>
+      </div>
+    </div>
+    <div id="class-metadata">
+      <!-- Parent Class -->
+      <div id="parent-class-section" class="section">
+        <h3 class="section-header">Parent</h3>
+        <p class="link"></p>
+      </div>
+      <!-- Namespace Contents -->
+      <!-- Method Quickref -->
+      <div id="method-list-section" class="section">
+        <h3 class="section-header">Methods</h3>
+        <ul class="link-list">
+          <li><a href="#method-i-do_preprocessing">#do_preprocessing</a></li>
+          <li><a href="#method-i-job">#job</a></li>
+          <li><a href="#method-i-preprocess_mapreduce">#preprocess_mapreduce</a></li>
+          <li><a href="#method-i-preprocess_mpi">#preprocess_mpi</a></li>
+        </ul>
+      </div>
+      <!-- Included Modules -->
+    </div>
+    <div id="project-metadata">
+      <div id="fileindex-section" class="section project-section">
+        <h3 class="section-header">Files</h3>
+        <ul>
+          <li class="file"><a href="./LICENSE.html">LICENSE</a></li>
+          <li class="file"><a href="./README.html">README</a></li>
+        </ul>
+      </div>
+      <div id="classindex-section" class="section project-section">
+        <h3 class="section-header">Class/Module Index
+          <span class="search-toggle"><img src="./images/find.png"
+            height="16" width="16" alt="[+]"
+            title="show/hide quicksearch" /></span></h3>
+        <form action="#" method="get" accept-charset="utf-8" class="initially-hidden">
+        <fieldset>
+          <legend>Quicksearch</legend>
+          <input type="text" name="quicksearch" value=""
+            class="quicksearch-field" />
+        </fieldset>
+        </form>
+        <ul class="link-list">
+          <li><a href="./AppControllerClient.html">AppControllerClient</a></li>
+          <li><a href="./CommonFunctions.html">CommonFunctions</a></li>
+          <li><a href="./Object.html">Object</a></li>
+        </ul>
+        <div id="no-class-search-results" style="display: none;">No matching classes.</div>
+      </div>
+    </div>
+  </div>
+  <div id="documentation">
+    <h1 class="class">Object</h1>
+    <div id="description">
+<p>A set of methods and constants that we’ve monkey-patched to enable
+Neptune support. In the future, it is likely that the only exposed /
+monkey-patched method should be job, while the others could probably be
+folded into either a Neptune-specific class or into <a
+href="CommonFunctions.html">CommonFunctions</a>.</p>
+    </div>
+    <!-- Constants -->
+    <div id="constants-list" class="section">
+      <h3 class="section-header">Constants</h3>
+      <dl>
+        <dt><a name="NO_TIMEOUT">NO_TIMEOUT</a></dt>
+        <dd class="description"><p>Sometimes SOAP calls take a long time if large amounts of data are being
+sent over the network: for this first version we don’t want these calls
+to endlessly timeout and retry, so as a hack, just don’t let them
+timeout. The next version should replace this and properly timeout and not
+use long calls unless necessary.</p></dd>
+        <dt><a name="NEED_PREPROCESSING">NEED_PREPROCESSING</a></dt>
+        <dd class="description"><p>A list of jobs that require some kind of work to be done before the actual
+computation can be performed.</p></dd>
+      </dl>
+    </div>
+    <!-- Attributes -->
+    <!-- Methods -->
+    <div id="public-instance-method-details" class="method-section section">
+      <h3 class="section-header">Public Instance Methods</h3>
+      <div id="do_preprocessing-method" class="method-detail ">
+        <a name="method-i-do_preprocessing"></a>
+        <div class="method-heading">
+          <span class="method-name">do_preprocessing</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>Certain types of jobs need steps to be taken before they can be started
+(e.g., copying input data or code over). This method dispatches the right
+method to use based on the type of the job that the user has asked to run.</p>
+          <div class="method-source-code"
+            id="do_preprocessing-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 34</span>
+def do_preprocessing(job_data)
+  job_type = job_data[<span class="ruby-string">&quot;@type&quot;</span>]
+  return unless <span class="ruby-constant">NEED_PREPROCESSING</span>.include?(job_type)
+  preprocess = &quot;preprocess_#{job_type}&quot;.to_sym
+  send(preprocess, job_data)
+end</pre>
+          </div>
+        </div>
+      </div>
+      <div id="job-method" class="method-detail ">
+        <a name="method-i-job"></a>
+        <div class="method-heading">
+          <span class="method-name">job</span><span
+            class="method-args">(name, &block)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>This method is the heart of Neptune - here, we take blocks of code that the
+user has written and convert them into HPC job requests. At a high level,
+the user can request to run a job, retrieve a job’s output, or modify the
+access policy (ACL) for the output of a job. By default, job data is
+private, but a Neptune job can be used to set it to public later (and
+vice-versa).</p>
+          <div class="method-source-code"
+            id="job-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 101</span>
+def job(name, &amp;block)
+  puts <span class="ruby-string">&quot;Received a request to run a job.&quot;</span>
+  puts name
+  block.call()
+  if <span class="ruby-ivar">@keyname</span>.nil?
+    keyname = <span class="ruby-string">&quot;appscale&quot;</span>
+  else
+    keyname = <span class="ruby-ivar">@keyname</span>
+  end
+  shadow_ip = <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :shadow)
+  secret = <span class="ruby-constant">CommonFunctions</span>.get_secret_key(keyname)
+  controller = <span class="ruby-constant">AppControllerClient</span>.new(shadow_ip, secret)
+  ssh_key = <span class="ruby-constant">File</span>.expand_path(&quot;~/.appscale/#{keyname}.key&quot;)
+  job_data = {}
+  instance_variables.each { |var|
+    job_data[var] = instance_variable_get(var)
+  }
+  job_data[<span class="ruby-string">&quot;@job&quot;</span>] = nil
+  job_data[<span class="ruby-string">&quot;@type&quot;</span>] = name
+  job_data[<span class="ruby-string">&quot;@keyname&quot;</span>] = keyname
+  if job_data[<span class="ruby-string">&quot;@output&quot;</span>].nil? or job_data[<span class="ruby-string">&quot;@output&quot;</span>] == <span class="ruby-string">&quot;&quot;</span>
+    abort(<span class="ruby-string">&quot;Job output must be specified&quot;</span>)
+  end
+  if job_data[<span class="ruby-string">&quot;@output&quot;</span>][0].chr != <span class="ruby-string">&quot;/&quot;</span>
+    abort(<span class="ruby-string">&quot;Job output must begin with a slash ('/')&quot;</span>)
+  end
+  <span class="ruby-comment">#if job_data[&quot;@can_run_on&quot;].class == Range</span>
+  <span class="ruby-comment">#  job_data[&quot;@can_run_on&quot;] = job_data[&quot;@can_run_on&quot;].to_a</span>
+  <span class="ruby-comment">#elsif job_data[&quot;@can_run_on&quot;].class == Fixnum</span>
+  <span class="ruby-comment">#  job_data[&quot;@can_run_on&quot;] = [job_data[&quot;@can_run_on&quot;]]</span>
+  <span class="ruby-comment">#end</span>
+  puts &quot;job data = #{job_data.inspect}&quot;
+  do_preprocessing(job_data)
+  if job_data[<span class="ruby-string">&quot;@type&quot;</span>] == <span class="ruby-string">&quot;output&quot;</span>
+    return controller.get_output(job_data)
+  elsif job_data[<span class="ruby-string">&quot;@type&quot;</span>] == <span class="ruby-string">&quot;get-acl&quot;</span>
+    job_data[<span class="ruby-string">&quot;@type&quot;</span>] = <span class="ruby-string">&quot;acl&quot;</span>
+    return controller.get_acl(job_data)
+  elsif job_data[<span class="ruby-string">&quot;@type&quot;</span>] == <span class="ruby-string">&quot;set-acl&quot;</span>
+    job_data[<span class="ruby-string">&quot;@type&quot;</span>] = <span class="ruby-string">&quot;acl&quot;</span>
+    return controller.set_acl(job_data)
+  else
+    result = controller.start_neptune_job(job_data)
+    if result =~ <span class="ruby-regexp">/job is now running\Z/</span>
+      return :success
+    else
+      return :failure
+    end
+  end
+end</pre>
+          </div>
+        </div>
+      </div>
+      <div id="preprocess_mapreduce-method" class="method-detail ">
+        <a name="method-i-preprocess_mapreduce"></a>
+        <div class="method-heading">
+          <span class="method-name">preprocess_mapreduce</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>This preprocessing method handles copying data for regular Hadoop MapReduce
+and Hadoop MapReduce Streaming. For the former case, we copy over just the
+JAR the user has given us, and in the latter case, we copy over the Map and
+Reduce files that have been specified. In either case, if the user has
+specified to us to copy over an input file, we do that as well: AppScale
+will copy it into HDFS for us.</p>
+          <div class="method-source-code"
+            id="preprocess_mapreduce-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 65</span>
+def preprocess_mapreduce(job_data)
+  items_to_copy = [<span class="ruby-string">&quot;@map&quot;</span>, <span class="ruby-string">&quot;@reduce&quot;</span>] if job_data[<span class="ruby-string">&quot;@map&quot;</span>] and job_data[<span class="ruby-string">&quot;@reduce&quot;</span>]
+  items_to_copy = [<span class="ruby-string">&quot;@mapreducejar&quot;</span>] if job_data[<span class="ruby-string">&quot;@mapreducejar&quot;</span>]
+  items_to_copy &lt;&lt; <span class="ruby-string">&quot;@input&quot;</span> if job_data[<span class="ruby-string">&quot;@copy_input&quot;</span>]
+  items_to_copy.each { |item|
+    source = <span class="ruby-constant">File</span>.expand_path(job_data[item])
+    unless <span class="ruby-constant">File</span>.exists?(source)
+      abort(&quot;The #{item} file #{source} does not exist.&quot;)
+    end
+    suffix = source.split(<span class="ruby-string">'/'</span>)[-1]
+    dest = &quot;/tmp/#{suffix}&quot;
+    keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>]
+    <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(source, dest, keyname)
+    job_data[item] = dest
+  }
+end</pre>
+          </div>
+        </div>
+      </div>
+      <div id="preprocess_mpi-method" class="method-detail ">
+        <a name="method-i-preprocess_mpi"></a>
+        <div class="method-heading">
+          <span class="method-name">preprocess_mpi</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>This preprocessing method copies over the user’s MPI code to the master
+node in AppScale - this node will then copy it to whoever will run the MPI
+job.</p>
+          <div class="method-source-code"
+            id="preprocess_mpi-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 45</span>
+def preprocess_mpi(job_data)
+  source_code = <span class="ruby-constant">File</span>.expand_path(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
+  unless <span class="ruby-constant">File</span>.exists?(source_code)
+    file_not_found = &quot;The specified code, #{job_data['@code']},&quot; +
+      <span class="ruby-string">&quot; didn't exist. Please specify one that exists and try again&quot;</span>
+    abort(file_not_found)
+  end
+  dest_code = <span class="ruby-string">&quot;/tmp/thempicode&quot;</span>
+  keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>]
+  <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(source_code, dest_code, keyname)
+end</pre>
+          </div>
+        </div>
+      </div>
+    </div>
+  </div>
+  <div id="validator-badges">
+    <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+    <p><small>Generated with the <a href="http://deveiate.org/projects/Darkfish-Rdoc/">Darkfish
+      Rdoc Generator</a> 2</small>.</p>
+  </div>
+</body>
+</html>

data/doc/README.html ADDED Viewed

@@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
+  <title>File: README [RDoc Documentation]</title>
+  <link type="text/css" media="screen" href="./rdoc.css" rel="stylesheet" />
+  <script src="./js/jquery.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/thickbox-compressed.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/quicksearch.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/darkfish.js" type="text/javascript"
+    charset="utf-8"></script>
+</head>
+<body class="file">
+  <div id="metadata">
+    <div id="home-metadata">
+      <div id="home-section" class="section">
+        <h3 class="section-header">
+          <a href="./index.html">Home</a>
+          <a href="./index.html#classes">Classes</a>
+          <a href="./index.html#methods">Methods</a>
+        </h3>
+      </div>
+    </div>
+    <div id="project-metadata">
+      <div id="fileindex-section" class="section project-section">
+        <h3 class="section-header">Files</h3>
+        <ul>
+          <li class="file"><a href="./LICENSE.html">LICENSE</a></li>
+          <li class="file"><a href="./README.html">README</a></li>
+        </ul>
+      </div>
+      <div id="classindex-section" class="section project-section">
+        <h3 class="section-header">Class Index
+          <span class="search-toggle"><img src="./images/find.png"
+            height="16" width="16" alt="[+]"
+            title="show/hide quicksearch" /></span></h3>
+        <form action="#" method="get" accept-charset="utf-8" class="initially-hidden">
+        <fieldset>
+          <legend>Quicksearch</legend>
+          <input type="text" name="quicksearch" value=""
+            class="quicksearch-field" />
+        </fieldset>
+        </form>
+        <ul class="link-list">
+          <li><a href="./AppControllerClient.html">AppControllerClient</a></li>
+          <li><a href="./CommonFunctions.html">CommonFunctions</a></li>
+          <li><a href="./Object.html">Object</a></li>
+        </ul>
+        <div id="no-class-search-results" style="display: none;">No matching classes.</div>
+      </div>
+    </div>
+  </div>
+  <div id="documentation">
+<h1>begin rdoc</h1>
+<p>Neptune: A Domain Specific Language for Deploying HPC  Software on Cloud
+Platforms</p>
+<p>Neptune provides programmers with a simple interface by which they can
+deploy MPI, X10, and MapReduce jobs to without needing to know the
+particulars of the underlying cloud platform. You only need to give Neptune
+your code, tell it how many machines to run on and where to put the output:
+Neptune handles everything else. No more writing configuration files,
+having to start up NFS on all your machines, yada yada yada. Neptune works
+together with supported cloud platforms (currently AppScale is recommended)
+and can deploy over anything AppScale can - Xen or KVM virtual machines as
+well as Eucalyptus and Amazon EC2. There’s nothing virtualization
+specific in there, so in theory any machine installed with the AppScale
+software should work fine.</p>
+<p>Although Neptune is designed to automate deploying HPC jobs, it also can be
+used to deploy other types of software. For example, Neptune has support
+for user-specified scaling of the underlying cloud platform: users can
+write Neptune jobs that manually add load balancers, application servers,
+or database servers to a running AppScale deployment.</p>
+<p>Sample Neptune job scripts can be found in samples. Test scripts will be
+added to test in the next release.</p>
+<p>Developed by Chris Bunch as part of the AppScale project. See <a
+href="LICENSE.html">LICENSE</a> for the specifics of the New BSD License by
+which Neptune is released.</p>
+<p>Check us out on the web:</p>
+<p><a href="http://code.google.com/p/appscale">code.google.com/p/appscale</a></p>
+<p><a href="http://appscale.cs.ucsb.edu">appscale.cs.ucsb.edu</a></p>
+<p>Contributors welcome! We would love to add support for other cloud
+platforms and test out Neptune more on non-virtualized deployments, as well
+as adding capabilities for other types of computation.</p>
+<p>An academic paper describing Neptune is in the works - check in for a link
+to that as it becomes available.</p>
+<h1>end</h1>
+  </div>
+  <div id="validator-badges">
+    <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+    <p><small>Generated with the <a href="http://deveiate.org/projects/Darkfish-Rdoc/">Darkfish
+      Rdoc Generator</a> 2</small>.</p>
+  </div>
+</body>
+</html>