RubyGems - neptune - Versions diffs - 0.0.1 - Mend

neptune 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (79) hide show

data/LICENSE +39 -0
data/README +50 -0
data/bin/neptune +16 -0
data/doc/AppControllerClient.html +468 -0
data/doc/CommonFunctions.html +368 -0
data/doc/LICENSE.html +131 -0
data/doc/Object.html +431 -0
data/doc/README.html +135 -0
data/doc/bin/neptune.html +58 -0
data/doc/created.rid +7 -0
data/doc/images/brick.png +0 -0
data/doc/images/brick_link.png +0 -0
data/doc/images/bug.png +0 -0
data/doc/images/bullet_black.png +0 -0
data/doc/images/bullet_toggle_minus.png +0 -0
data/doc/images/bullet_toggle_plus.png +0 -0
data/doc/images/date.png +0 -0
data/doc/images/find.png +0 -0
data/doc/images/loadingAnimation.gif +0 -0
data/doc/images/macFFBgHack.png +0 -0
data/doc/images/package.png +0 -0
data/doc/images/page_green.png +0 -0
data/doc/images/page_white_text.png +0 -0
data/doc/images/page_white_width.png +0 -0
data/doc/images/plugin.png +0 -0
data/doc/images/ruby.png +0 -0
data/doc/images/tag_green.png +0 -0
data/doc/images/wrench.png +0 -0
data/doc/images/wrench_orange.png +0 -0
data/doc/images/zoom.png +0 -0
data/doc/index.html +90 -0
data/doc/js/darkfish.js +116 -0
data/doc/js/jquery.js +32 -0
data/doc/js/quicksearch.js +114 -0
data/doc/js/thickbox-compressed.js +10 -0
data/doc/lib/app_controller_client_rb.html +60 -0
data/doc/lib/common_functions_rb.html +68 -0
data/doc/lib/job_rb.html +55 -0
data/doc/scripts/appscale/add_appserver_rb.html +52 -0
data/doc/scripts/appscale/add_database_rb.html +52 -0
data/doc/scripts/appscale/add_loadbalancer_rb.html +52 -0
data/doc/scripts/appscale/add_slave_rb.html +52 -0
data/doc/scripts/cewssa/cewssa-output.html +104 -0
data/doc/scripts/cewssa/get_cewssa_output_rb.html +52 -0
data/doc/scripts/cewssa/run_cewssa_rb.html +52 -0
data/doc/scripts/cewssa/test_dfsp_rb.html +52 -0
data/doc/scripts/dfsp/get_dfsp_output_rb.html +52 -0
data/doc/scripts/dfsp/make_output_public_rb.html +52 -0
data/doc/scripts/dfsp/run_dfsp_rb.html +52 -0
data/doc/scripts/dfsp/see_acl_rb.html +52 -0
data/doc/scripts/dfsp/test_dfsp_rb.html +52 -0
data/doc/scripts/mapreduce/get_mapreduce_output_rb.html +52 -0
data/doc/scripts/mapreduce/input-10.html +114 -0
data/doc/scripts/mapreduce/input-30.html +114 -0
data/doc/scripts/mapreduce/input-7.html +104 -0
data/doc/scripts/mapreduce/map_rb.html +56 -0
data/doc/scripts/mapreduce/reduce_rb.html +56 -0
data/doc/scripts/mapreduce/run_java_mr_rb.html +52 -0
data/doc/scripts/mapreduce/run_mapreduce_rb.html +52 -0
data/doc/scripts/mpi/Makefile.html +125 -0
data/doc/scripts/mpi/get_mpi_output_rb.html +52 -0
data/doc/scripts/mpi/hw2_c.html +52 -0
data/doc/scripts/mpi/hw2harness_c.html +52 -0
data/doc/scripts/mpi/hw2harness_h.html +52 -0
data/doc/scripts/mpi/run_mpi_cpi_rb.html +52 -0
data/doc/scripts/mpi/run_mpi_powermethod_rb.html +52 -0
data/doc/scripts/mpi/test_rb.html +52 -0
data/doc/scripts/x10/Makefile.html +106 -0
data/doc/scripts/x10/get_x10_output_rb.html +52 -0
data/doc/scripts/x10/run_x10_kmeans_rb.html +52 -0
data/doc/scripts/x10/run_x10_montypi_rb.html +52 -0
data/doc/scripts/x10/run_x10_nqueens_rb.html +52 -0
data/doc/scripts/x10/run_x10_powermethod_rb.html +52 -0
data/doc/scripts/x10/run_x10_qsort_rb.html +52 -0
data/doc/scripts/x10/test_rb.html +52 -0
data/lib/app_controller_client.rb +135 -0
data/lib/common_functions.rb +107 -0
data/lib/job.rb +161 -0
metadata +145 -0

data/doc/Object.html ADDED Viewed

@@ -0,0 +1,431 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
+  <title>Class: Object</title>
+  <link rel="stylesheet" href="./rdoc.css" type="text/css" media="screen" />
+  <script src="./js/jquery.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/thickbox-compressed.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/quicksearch.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/darkfish.js" type="text/javascript"
+    charset="utf-8"></script>
+</head>
+<body class="class">
+  <div id="metadata">
+    <div id="home-metadata">
+      <div id="home-section" class="section">
+        <h3 class="section-header">
+          <a href="./index.html">Home</a>
+          <a href="./index.html#classes">Classes</a>
+          <a href="./index.html#methods">Methods</a>
+        </h3>
+      </div>
+    </div>
+    <div id="file-metadata">
+      <div id="file-list-section" class="section">
+        <h3 class="section-header">In Files</h3>
+        <div class="section-body">
+          <ul>
+            <li><a href="./lib/app_controller_client_rb.html?TB_iframe=true&amp;height=550&amp;width=785"
+              class="thickbox" title="lib/app_controller_client.rb">lib/app_controller_client.rb</a></li>
+            <li><a href="./lib/job_rb.html?TB_iframe=true&amp;height=550&amp;width=785"
+              class="thickbox" title="lib/job.rb">lib/job.rb</a></li>
+          </ul>
+        </div>
+      </div>
+    </div>
+    <div id="class-metadata">
+      <!-- Parent Class -->
+      <div id="parent-class-section" class="section">
+        <h3 class="section-header">Parent</h3>
+        <p class="link"></p>
+      </div>
+      <!-- Namespace Contents -->
+      <!-- Method Quickref -->
+      <div id="method-list-section" class="section">
+        <h3 class="section-header">Methods</h3>
+        <ul class="link-list">
+          <li><a href="#method-i-do_preprocessing">#do_preprocessing</a></li>
+          <li><a href="#method-i-job">#job</a></li>
+          <li><a href="#method-i-preprocess_mapreduce">#preprocess_mapreduce</a></li>
+          <li><a href="#method-i-preprocess_mpi">#preprocess_mpi</a></li>
+        </ul>
+      </div>
+      <!-- Included Modules -->
+    </div>
+    <div id="project-metadata">
+      <div id="fileindex-section" class="section project-section">
+        <h3 class="section-header">Files</h3>
+        <ul>
+          <li class="file"><a href="./LICENSE.html">LICENSE</a></li>
+          <li class="file"><a href="./README.html">README</a></li>
+        </ul>
+      </div>
+      <div id="classindex-section" class="section project-section">
+        <h3 class="section-header">Class/Module Index
+          <span class="search-toggle"><img src="./images/find.png"
+            height="16" width="16" alt="[+]"
+            title="show/hide quicksearch" /></span></h3>
+        <form action="#" method="get" accept-charset="utf-8" class="initially-hidden">
+        <fieldset>
+          <legend>Quicksearch</legend>
+          <input type="text" name="quicksearch" value=""
+            class="quicksearch-field" />
+        </fieldset>
+        </form>
+        <ul class="link-list">
+          <li><a href="./AppControllerClient.html">AppControllerClient</a></li>
+          <li><a href="./CommonFunctions.html">CommonFunctions</a></li>
+          <li><a href="./Object.html">Object</a></li>
+        </ul>
+        <div id="no-class-search-results" style="display: none;">No matching classes.</div>
+      </div>
+    </div>
+  </div>
+  <div id="documentation">
+    <h1 class="class">Object</h1>
+    <div id="description">
+<p>A set of methods and constants that we’ve monkey-patched to enable
+Neptune support. In the future, it is likely that the only exposed /
+monkey-patched method should be job, while the others could probably be
+folded into either a Neptune-specific class or into <a
+href="CommonFunctions.html">CommonFunctions</a>.</p>
+    </div>
+    <!-- Constants -->
+    <div id="constants-list" class="section">
+      <h3 class="section-header">Constants</h3>
+      <dl>
+        <dt><a name="NO_TIMEOUT">NO_TIMEOUT</a></dt>
+        <dd class="description"><p>Sometimes SOAP calls take a long time if large amounts of data are being
+sent over the network: for this first version we don’t want these calls
+to endlessly timeout and retry, so as a hack, just don’t let them
+timeout. The next version should replace this and properly timeout and not
+use long calls unless necessary.</p></dd>
+        <dt><a name="NEED_PREPROCESSING">NEED_PREPROCESSING</a></dt>
+        <dd class="description"><p>A list of jobs that require some kind of work to be done before the actual
+computation can be performed.</p></dd>
+      </dl>
+    </div>
+    <!-- Attributes -->
+    <!-- Methods -->
+    <div id="public-instance-method-details" class="method-section section">
+      <h3 class="section-header">Public Instance Methods</h3>
+      <div id="do_preprocessing-method" class="method-detail ">
+        <a name="method-i-do_preprocessing"></a>
+        <div class="method-heading">
+          <span class="method-name">do_preprocessing</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>Certain types of jobs need steps to be taken before they can be started
+(e.g., copying input data or code over). This method dispatches the right
+method to use based on the type of the job that the user has asked to run.</p>
+          <div class="method-source-code"
+            id="do_preprocessing-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 34</span>
+def do_preprocessing(job_data)
+  job_type = job_data[<span class="ruby-string">&quot;@type&quot;</span>]
+  return unless <span class="ruby-constant">NEED_PREPROCESSING</span>.include?(job_type)
+  preprocess = &quot;preprocess_#{job_type}&quot;.to_sym
+  send(preprocess, job_data)
+end</pre>
+          </div>
+        </div>
+      </div>
+      <div id="job-method" class="method-detail ">
+        <a name="method-i-job"></a>
+        <div class="method-heading">
+          <span class="method-name">job</span><span
+            class="method-args">(name, &block)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>This method is the heart of Neptune - here, we take blocks of code that the
+user has written and convert them into HPC job requests. At a high level,
+the user can request to run a job, retrieve a job’s output, or modify the
+access policy (ACL) for the output of a job. By default, job data is
+private, but a Neptune job can be used to set it to public later (and
+vice-versa).</p>
+          <div class="method-source-code"
+            id="job-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 101</span>
+def job(name, &amp;block)
+  puts <span class="ruby-string">&quot;Received a request to run a job.&quot;</span>
+  puts name
+  block.call()
+  if <span class="ruby-ivar">@keyname</span>.nil?
+    keyname = <span class="ruby-string">&quot;appscale&quot;</span>
+  else
+    keyname = <span class="ruby-ivar">@keyname</span>
+  end
+  shadow_ip = <span class="ruby-constant">CommonFunctions</span>.get_from_yaml(keyname, :shadow)
+  secret = <span class="ruby-constant">CommonFunctions</span>.get_secret_key(keyname)
+  controller = <span class="ruby-constant">AppControllerClient</span>.new(shadow_ip, secret)
+  ssh_key = <span class="ruby-constant">File</span>.expand_path(&quot;~/.appscale/#{keyname}.key&quot;)
+  job_data = {}
+  instance_variables.each { |var|
+    job_data[var] = instance_variable_get(var)
+  }
+  job_data[<span class="ruby-string">&quot;@job&quot;</span>] = nil
+  job_data[<span class="ruby-string">&quot;@type&quot;</span>] = name
+  job_data[<span class="ruby-string">&quot;@keyname&quot;</span>] = keyname
+  if job_data[<span class="ruby-string">&quot;@output&quot;</span>].nil? or job_data[<span class="ruby-string">&quot;@output&quot;</span>] == <span class="ruby-string">&quot;&quot;</span>
+    abort(<span class="ruby-string">&quot;Job output must be specified&quot;</span>)
+  end
+  if job_data[<span class="ruby-string">&quot;@output&quot;</span>][0].chr != <span class="ruby-string">&quot;/&quot;</span>
+    abort(<span class="ruby-string">&quot;Job output must begin with a slash ('/')&quot;</span>)
+  end
+  <span class="ruby-comment">#if job_data[&quot;@can_run_on&quot;].class == Range</span>
+  <span class="ruby-comment">#  job_data[&quot;@can_run_on&quot;] = job_data[&quot;@can_run_on&quot;].to_a</span>
+  <span class="ruby-comment">#elsif job_data[&quot;@can_run_on&quot;].class == Fixnum</span>
+  <span class="ruby-comment">#  job_data[&quot;@can_run_on&quot;] = [job_data[&quot;@can_run_on&quot;]]</span>
+  <span class="ruby-comment">#end</span>
+  puts &quot;job data = #{job_data.inspect}&quot;
+  do_preprocessing(job_data)
+  if job_data[<span class="ruby-string">&quot;@type&quot;</span>] == <span class="ruby-string">&quot;output&quot;</span>
+    return controller.get_output(job_data)
+  elsif job_data[<span class="ruby-string">&quot;@type&quot;</span>] == <span class="ruby-string">&quot;get-acl&quot;</span>
+    job_data[<span class="ruby-string">&quot;@type&quot;</span>] = <span class="ruby-string">&quot;acl&quot;</span>
+    return controller.get_acl(job_data)
+  elsif job_data[<span class="ruby-string">&quot;@type&quot;</span>] == <span class="ruby-string">&quot;set-acl&quot;</span>
+    job_data[<span class="ruby-string">&quot;@type&quot;</span>] = <span class="ruby-string">&quot;acl&quot;</span>
+    return controller.set_acl(job_data)
+  else
+    result = controller.start_neptune_job(job_data)
+    if result =~ <span class="ruby-regexp">/job is now running\Z/</span>
+      return :success
+    else
+      return :failure
+    end
+  end
+end</pre>
+          </div>
+        </div>
+      </div>
+      <div id="preprocess_mapreduce-method" class="method-detail ">
+        <a name="method-i-preprocess_mapreduce"></a>
+        <div class="method-heading">
+          <span class="method-name">preprocess_mapreduce</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>This preprocessing method handles copying data for regular Hadoop MapReduce
+and Hadoop MapReduce Streaming. For the former case, we copy over just the
+JAR the user has given us, and in the latter case, we copy over the Map and
+Reduce files that have been specified. In either case, if the user has
+specified to us to copy over an input file, we do that as well: AppScale
+will copy it into HDFS for us.</p>
+          <div class="method-source-code"
+            id="preprocess_mapreduce-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 65</span>
+def preprocess_mapreduce(job_data)
+  items_to_copy = [<span class="ruby-string">&quot;@map&quot;</span>, <span class="ruby-string">&quot;@reduce&quot;</span>] if job_data[<span class="ruby-string">&quot;@map&quot;</span>] and job_data[<span class="ruby-string">&quot;@reduce&quot;</span>]
+  items_to_copy = [<span class="ruby-string">&quot;@mapreducejar&quot;</span>] if job_data[<span class="ruby-string">&quot;@mapreducejar&quot;</span>]
+  items_to_copy &lt;&lt; <span class="ruby-string">&quot;@input&quot;</span> if job_data[<span class="ruby-string">&quot;@copy_input&quot;</span>]
+  items_to_copy.each { |item|
+    source = <span class="ruby-constant">File</span>.expand_path(job_data[item])
+    unless <span class="ruby-constant">File</span>.exists?(source)
+      abort(&quot;The #{item} file #{source} does not exist.&quot;)
+    end
+    suffix = source.split(<span class="ruby-string">'/'</span>)[-1]
+    dest = &quot;/tmp/#{suffix}&quot;
+    keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>]
+    <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(source, dest, keyname)
+    job_data[item] = dest
+  }
+end</pre>
+          </div>
+        </div>
+      </div>
+      <div id="preprocess_mpi-method" class="method-detail ">
+        <a name="method-i-preprocess_mpi"></a>
+        <div class="method-heading">
+          <span class="method-name">preprocess_mpi</span><span
+            class="method-args">(job_data)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        <div class="method-description">
+          <p>This preprocessing method copies over the user’s MPI code to the master
+node in AppScale - this node will then copy it to whoever will run the MPI
+job.</p>
+          <div class="method-source-code"
+            id="preprocess_mpi-source">
+<pre>
+<span class="ruby-comment"># File lib/job.rb, line 45</span>
+def preprocess_mpi(job_data)
+  source_code = <span class="ruby-constant">File</span>.expand_path(job_data[<span class="ruby-string">&quot;@code&quot;</span>])
+  unless <span class="ruby-constant">File</span>.exists?(source_code)
+    file_not_found = &quot;The specified code, #{job_data['@code']},&quot; +
+      <span class="ruby-string">&quot; didn't exist. Please specify one that exists and try again&quot;</span>
+    abort(file_not_found)
+  end
+  dest_code = <span class="ruby-string">&quot;/tmp/thempicode&quot;</span>
+  keyname = job_data[<span class="ruby-string">&quot;@keyname&quot;</span>]
+  <span class="ruby-constant">CommonFunctions</span>.scp_to_shadow(source_code, dest_code, keyname)
+end</pre>
+          </div>
+        </div>
+      </div>
+    </div>
+  </div>
+  <div id="validator-badges">
+    <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+    <p><small>Generated with the <a href="http://deveiate.org/projects/Darkfish-Rdoc/">Darkfish
+      Rdoc Generator</a> 2</small>.</p>
+  </div>
+</body>
+</html>

data/doc/README.html ADDED Viewed

@@ -0,0 +1,135 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
+  <title>File: README [RDoc Documentation]</title>
+  <link type="text/css" media="screen" href="./rdoc.css" rel="stylesheet" />
+  <script src="./js/jquery.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/thickbox-compressed.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/quicksearch.js" type="text/javascript"
+    charset="utf-8"></script>
+  <script src="./js/darkfish.js" type="text/javascript"
+    charset="utf-8"></script>
+</head>
+<body class="file">
+  <div id="metadata">
+    <div id="home-metadata">
+      <div id="home-section" class="section">
+        <h3 class="section-header">
+          <a href="./index.html">Home</a>
+          <a href="./index.html#classes">Classes</a>
+          <a href="./index.html#methods">Methods</a>
+        </h3>
+      </div>
+    </div>
+    <div id="project-metadata">
+      <div id="fileindex-section" class="section project-section">
+        <h3 class="section-header">Files</h3>
+        <ul>
+          <li class="file"><a href="./LICENSE.html">LICENSE</a></li>
+          <li class="file"><a href="./README.html">README</a></li>
+        </ul>
+      </div>
+      <div id="classindex-section" class="section project-section">
+        <h3 class="section-header">Class Index
+          <span class="search-toggle"><img src="./images/find.png"
+            height="16" width="16" alt="[+]"
+            title="show/hide quicksearch" /></span></h3>
+        <form action="#" method="get" accept-charset="utf-8" class="initially-hidden">
+        <fieldset>
+          <legend>Quicksearch</legend>
+          <input type="text" name="quicksearch" value=""
+            class="quicksearch-field" />
+        </fieldset>
+        </form>
+        <ul class="link-list">
+          <li><a href="./AppControllerClient.html">AppControllerClient</a></li>
+          <li><a href="./CommonFunctions.html">CommonFunctions</a></li>
+          <li><a href="./Object.html">Object</a></li>
+        </ul>
+        <div id="no-class-search-results" style="display: none;">No matching classes.</div>
+      </div>
+    </div>
+  </div>
+  <div id="documentation">
+<h1>begin rdoc</h1>
+<p>Neptune: A Domain Specific Language for Deploying HPC  Software on Cloud
+Platforms</p>
+<p>Neptune provides programmers with a simple interface by which they can
+deploy MPI, X10, and MapReduce jobs to without needing to know the
+particulars of the underlying cloud platform. You only need to give Neptune
+your code, tell it how many machines to run on and where to put the output:
+Neptune handles everything else. No more writing configuration files,
+having to start up NFS on all your machines, yada yada yada. Neptune works
+together with supported cloud platforms (currently AppScale is recommended)
+and can deploy over anything AppScale can - Xen or KVM virtual machines as
+well as Eucalyptus and Amazon EC2. There’s nothing virtualization
+specific in there, so in theory any machine installed with the AppScale
+software should work fine.</p>
+<p>Although Neptune is designed to automate deploying HPC jobs, it also can be
+used to deploy other types of software. For example, Neptune has support
+for user-specified scaling of the underlying cloud platform: users can
+write Neptune jobs that manually add load balancers, application servers,
+or database servers to a running AppScale deployment.</p>
+<p>Sample Neptune job scripts can be found in samples. Test scripts will be
+added to test in the next release.</p>
+<p>Developed by Chris Bunch as part of the AppScale project. See <a
+href="LICENSE.html">LICENSE</a> for the specifics of the New BSD License by
+which Neptune is released.</p>
+<p>Check us out on the web:</p>
+<p><a href="http://code.google.com/p/appscale">code.google.com/p/appscale</a></p>
+<p><a href="http://appscale.cs.ucsb.edu">appscale.cs.ucsb.edu</a></p>
+<p>Contributors welcome! We would love to add support for other cloud
+platforms and test out Neptune more on non-virtualized deployments, as well
+as adding capabilities for other types of computation.</p>
+<p>An academic paper describing Neptune is in the works - check in for a link
+to that as it becomes available.</p>
+<h1>end</h1>
+  </div>
+  <div id="validator-badges">
+    <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+    <p><small>Generated with the <a href="http://deveiate.org/projects/Darkfish-Rdoc/">Darkfish
+      Rdoc Generator</a> 2</small>.</p>
+  </div>
+</body>
+</html>