torque_rm 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 06d6fce41705812d8811ac4cf5ed34b2e240dbc9
+  data.tar.gz: cdff027855a94198d26e746bd7bf4582b093b1a9
+SHA512:
+  metadata.gz: f98b672b55c96ef7fe2466d3cbf89e644ca429df8a105679c294a4e6c2f5e017ccdbd7d611878d2fa453fff602e9f0701682727d356879391c179c3d65deebc0
+  data.tar.gz: 32e1699707d447a6d01c9d49bc2d14f7f08c60dd7d6df751e88715d1d80ce6df68f4522df9f79bcb9ad4e3193ecdba152453eb4f91a7be6fde8daff59b5268e1

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/.rspec

@@ -0,0 +1 @@
+--color

data/Gemfile

@@ -0,0 +1,25 @@
+source "http://rubygems.org"
+# Add dependencies required to use your gem here.
+# Example:
+#   gem "activesupport", ">= 2.3.5"
+gem 'rye'
+gem 'parslet'
+gem "colorize"
+gem "terminal-table"
+gem 'sinatra'
+gem 'sinatra-contrib'
+gem 'shotgun'
+gem 'haml'
+gem 'sinatra-twitter-bootstrap', :require => 'sinatra/twitter-bootstrap'
+# gem "awesome_print"
+# gem 'sinatra-twitter-bootstrap', :require => 'sinatra/twitter-bootstrap'
+# Add dependencies to develop your gem here.
+# Include everything needed to run rake, tests, features, etc.
+group :development do
+  gem "rspec", "~> 2.8.0"
+  gem "rdoc", "~> 3.12"
+  gem "cucumber", ">= 0"
+  gem "bundler", "> 1.0.0"
+  gem "jeweler", "~> 1.8.4"
+  # gem "rcov", "> 0"
+end

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2013 Raoul Jean Pierre Bonnal
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.html

@@ -0,0 +1,230 @@
+<h1>torque_rm</h1>
+
+<p>Example of configuration:</p>
+
+<pre><code>TORQUE.qcommands_path='/usr/bin'
+TORQUE.server = "my_grid.remote.net"
+job = TORQUE::Qsub.new  m: "abe", name:"FirstTest", shell: '/bin/bash', ppn: 4
+job.script = "echo `hostname`; sleep 1000"
+job.submit
+stats=TORQUE::Qstat.new
+
+# get all stats on all jobs
+stats.query
+
+# select stats from only the submitted job
+stat.query job_id: job.id
+
+# it is also possible to get the stats from a single job
+job.stat
+</code></pre>
+
+<h2>HowItWorks</h2>
+
+<p>Torque<em>Rm uses Rye as interface with a local/remote computer/server, using ssh, so every command
+is a executed with ssh. Rye also guarantees a certain level of security disabling dangerous operations
+like moving, deleting files or traversing recursevely a directory. Obviously this feature can be disabled,
+but for now it is the default way Torque</em>Rm operates.</p>
+
+<h2>Job Configurations</h2>
+
+<h3>Where files are saved ?</h3>
+
+<p>By default torque_rm creates/transfers files in the user home directory <code>~</code>. It's possible to change
+this default behaviour setting an internal variable when a <code>job</code> is created on managed.
+Considering the example above configuring the wrinting directory before submitting the <code>job</code>:</p>
+
+<pre><code>job.root_directory = '/my/personal/path'
+</code></pre>
+
+<p>or</p>
+
+<pre><code>job.working_directory = '/my/personal/path'
+</code></pre>
+
+<p><code>root</code> has precedence on <code>working</code> directory. It is possible to configure the <code>root</code>/<code>working</code> directory
+at the time of creation of the new job using the usual hash key/value convention.</p>
+
+<pre><code>:root_directory =&gt; '/my/personal/path'
+</code></pre>
+
+<p>or </p>
+
+<pre><code>:working_directory =&gt; '/my/personal/path'
+</code></pre>
+
+<h2>Configuration</h2>
+
+<h3>Saving</h3>
+
+<p>It is possible to save in a configuration file <code>server</code> and <code>path</code>.
+User can sare a configuration file:</p>
+
+<pre><code>TORQUE.qcommands_path='/usr/bin'
+TORQUE.server = "my_grid.remote.net"
+TORQUE.save_config
+</code></pre>
+
+<p>it will create a YAML file <code>~/.torque_rm.yaml</code> in the user home directory.
+In case the user wants to save the configuration in a custom location </p>
+
+<pre><code>TORQUE.save_config "file_name"
+</code></pre>
+
+<h3>Loading</h3>
+
+<p>Read a configuration from the dafult location <code>~/.torque_rm.yaml</code></p>
+
+<pre><code>TORQUE.load_config
+</code></pre>
+
+<p>In case the user has a custom file</p>
+
+<pre><code>TORQUE.load_config "file_name"
+</code></pre>
+
+<h2>Define a job using a DSL</h2>
+
+<pre><code>rhostname = TORQUE::Qsub.new do |job|
+        job.m = "abe"
+        job.name = "FirstTest"
+        job.shell = '/bin/bash'
+        job.ppn = 4
+        job.script = "echo `hostname`; sleep 1000"
+      end
+rhostname.submit
+rhostname.stat
+</code></pre>
+
+<h2>Stats</h2>
+
+<p>When a job is submitted, is possible to get some stats</p>
+
+<pre><code>[{:job_id=&gt;"2750.sun.universe.space"@1387,
+  :job_name=&gt;"FirstTest"@1421,
+  :job_owner=&gt;"helios@sun.universe.space"@1447,
+  :job_state=&gt;"Q"@1489,
+  :queue=&gt;"bio"@1503,
+  :server=&gt;"sun.universe.space"@1520,
+  :checkpoint=&gt;"u"@1551,
+  :ctime=&gt;"Tue Aug 27 15:56:41 2013"@1565,
+  :error_path=&gt;
+   "sun.universe.space:/home/helios/FirstTest.e2750"@1607,
+  :hold_types=&gt;"n"@1689,
+  :join_path=&gt;"n"@1707,
+  :keep_files=&gt;"n"@1726,
+  :mail_points=&gt;"abe"@1746,
+  :mtime=&gt;"Tue Aug 27 15:56:41 2013"@1762,
+  :output_path=&gt;
+   "deep.space.nine:/home/helios/FirstTest.o2750"@1805,
+  :priority=&gt;"0"@1885,
+  :qtime=&gt;"Tue Aug 27 15:56:41 2013"@1899,
+  :rerunable=&gt;"True"@1940,
+  :resource_list_nodect=&gt;"1"@1972,
+  :resource_list_nodes=&gt;"1:ppn=4"@2000,
+  :shell_path_list=&gt;"/bin/bash"@2030,
+  :variable_list=&gt;
+   "PBS_O_QUEUE=bio,PBS_O_HOST=deep.space.nine,PBS_O_HOME=/home/helios,PBS_O_LANG=it_IT.UTF-8,PBS_O_LOGNAME=helios,PBS_O_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/helios/.rvm/bin,PBS_O_MAIL=/var/mail/helios,PBS_O_SHELL=/bin/bash,PBS_SERVER=spark,PBS_O_WORKDIR=/home/helios\n"@2060,
+  :etime=&gt;"Tue Aug 27 15:56:41 2013"@2456,
+  :submit_args=&gt;"FirstTest.qsub"@2499,
+  :fault_tolerant=&gt;"False"@2535,
+:submit_host=&gt;"sun.universe.space"@2559}]
+</code></pre>
+
+<p>directly from the job</p>
+
+<pre><code>job.stat
+</code></pre>
+
+<p>or by requesting the information using the specific object</p>
+
+<pre><code>stats=TORQUE::Qstat.new
+
+# select stats from only the submitted job
+stat.query job_id: job.id
+</code></pre>
+
+<p>it will return an hash but is also possible to return the raw string from TORQUE/PBS</p>
+
+<pre><code>stat.query job_id: job.id, type: :raw
+</code></pre>
+
+<p>Quering multiple jobs at the same time is possible as well (note the 's' after 'id'):</p>
+
+<pre><code>stat.query job_ids: ["2751.sun.universe.space","2752.sun.universe.space","2754.sun.universe.space","2755.sun.universe.space"]
+</code></pre>
+
+<h2>Delete Jobs</h2>
+
+<p>From a qsub object if submitted:</p>
+
+<pre><code>job.rm
+</code></pre>
+
+<p>Directly from PBS:</p>
+
+<pre><code>TORQUE::Qdel.rm("2750.sun.universe.space")
+</code></pre>
+
+<h2>REST Api</h2>
+
+<p>All objects returned from TORQUE<em>RM</em>REST are in JSON format.</p>
+
+<h3>Qstat</h3>
+
+<h4>Get</h4>
+
+<p><code>/qstat</code> Returns an Array of Hash with the following keys, if available <code>"job_id","job_name","job_owner","resources_used_cput","resources_used_mem","resources_used_vmem","resources_used_walltime","job_state","queue","server","checkpoint","ctime","error_path","exec_host","exec_port","hold_types","join_path","keep_files","mail_points","mail_users","mtime","output_path","priority","qtime","rerunable","resource_list","session_id","shell_path_list","variable_list","etime","exit_status","submit_args","start_time","start_count","fault_tolerant","comp_time","job_radix","total_runtime","submit_host"</code> .</p>
+
+<p><code>/qstat/fields</code> Returns an Array of String, the available fields for a single <code>qstat</code> query.</p>
+
+<p><code>/qstat/all/:field</code> Return an Array of Strings, each String is the <code>field</code> requested by the user. If the <code>field</code> requested is not valid, it will return 404</p>
+
+<h3>Job</h3>
+
+<h4>Get</h4>
+
+<p><code>/qstat/:job_id</code> Returns an Hash with main key <code>job</code> and the JSON conversion of a TORQUE::Qstat::Job object. If the <code>job_id</code> does not exist will return 404</p>
+
+<p><code>/qstat/job/fields</code> Returns and Hash <code>{"job_fields":["job_id","job_name","job_owner","resources_used_cput","resources_used_mem","resources_used_vmem","resources_used_walltime","job_state","queue","server","checkpoint","ctime","error_path","exec_host","exec_port","hold_types","join_path","keep_files","mail_points","mail_users","mtime","output_path","priority","qtime","rerunable","resource_list","session_id","shell_path_list","variable_list","etime","exit_status","submit_args","start_time","start_count","fault_tolerant","comp_time","job_radix","total_runtime","submit_host","is_runnig?","is_queued?","is_exited?","is_completed?","time","memory","node"]}</code> </p>
+
+<p><code>/qstat/:job_id/:field</code> Returns and Hash with a single key which is the required <code>field</code>. If the <code>job_id</code> or <code>field</code> do not exist will return 404</p>
+
+<h4>Post</h4>
+
+<p><code>/qstat/:job_id/del</code> Removes the <code>job_id</code> from the queue. If the <code>job_id</code> does not exist will return 404</p>
+
+<h3>Interacting from CLI</h3>
+
+<p>To perform a query or call a fucntion use <code>curl</code> setting the content type. Extension <code>.json</code> is not considering for now, why? Read <a href="http://stackoverflow.com/questions/477816/what-is-the-correct-json-content-type">What-is-the-correct-json-content-type</a> discussion on <a href="http://stackoverflow.com/">Stackoverflow</a></p>
+
+<p>curl -H "Content-type: application/json"  "http://localhost:4567/qstat"</p>
+
+<h3>WebApp</h3>
+
+<p>A web application is under development and it will let you interact with you TORQUE/PBS server in a more fancy way, be patient and give us your feedback or request functionalities. A list of supported html pages:</p>
+
+<ul>
+<li>get: /qstat</li>
+</ul>
+
+<h3>HowIsItMade</h3>
+
+<p><a href="http://www.sinatrarb.com/">Sinatra</a> is the low level framework used for developing the REST API and the web infrastructure. <a href="http://getbootstrap.com/">Twitter Bootstrap</a> is used to make the web app nice and fancy.</p>
+
+<h2>Contributing to torque_rm</h2>
+
+<ul>
+<li>Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.</li>
+<li>Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.</li>
+<li>Fork the project.</li>
+<li>Start a feature/bugfix branch.</li>
+<li>Commit and push until you are happy with your contribution.</li>
+<li>Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.</li>
+<li>Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.</li>
+</ul>
+
+<h2>Copyright</h2>
+
+<p>Copyright (c) 2013 Raoul Jean Pierre Bonnal. See LICENSE.txt for
+further details.</p>

data/README.md

@@ -0,0 +1,231 @@
+# torque_rm
+
+Example of configuration:
+
+    TORQUE.qcommands_path='/usr/bin'
+    TORQUE.server = "my_grid.remote.net"
+    job = TORQUE::Qsub.new  m: "abe", name:"FirstTest", shell: '/bin/bash', ppn: 4
+    job.script = "echo `hostname`; sleep 1000"
+    job.submit
+    stats=TORQUE::Qstat.new
+
+    # get all stats on all jobs
+    stats.query
+
+    # select stats from only the submitted job
+    stat.query job_id: job.id
+
+    # it is also possible to get the stats from a single job
+    job.stat
+
+## HowItWorks
+Torque_Rm uses Rye as interface with a local/remote computer/server, using ssh, so every command
+is a executed with ssh. Rye also guarantees a certain level of security disabling dangerous operations
+like moving, deleting files or traversing recursevely a directory. Obviously this feature can be disabled,
+but for now it is the default way Torque_Rm operates.
+
+## Job Configurations
+### Where files are saved ?
+By default torque_rm creates/transfers files in the user home directory `~`. It's possible to change
+this default behaviour setting an internal variable when a `job` is created on managed.
+Considering the example above configuring the wrinting directory before submitting the `job`:
+
+    job.root_directory = '/my/personal/path'
+
+or
+
+    job.working_directory = '/my/personal/path'
+
+`root` has precedence on `working` directory. It is possible to configure the `root`/`working` directory
+at the time of creation of the new job using the usual hash key/value convention.
+
+    :root_directory => '/my/personal/path'
+or 
+    
+    :working_directory => '/my/personal/path'
+
+## Configuration
+
+### byHand
+
+TORQUE_RM uses `yaml` configuration, user can create such a file in his/her home directory:
+
+    touch ~/.torque_rm.yaml
+
+and fill it with similar details:
+
+    ---
+    :hostname: deep.space.nine
+    :path: /usr/bin
+
+### Saving
+
+It is possible to save in a configuration file `server` and `path`.
+User can save a configuration file:
+
+    TORQUE.qcommands_path='/usr/bin'
+    TORQUE.server = "my_grid.remote.net"
+    TORQUE.save_config
+
+it will create a YAML file `~/.torque_rm.yaml` in the user home directory.
+In case the user wants to save the configuration in a custom location 
+
+    TORQUE.save_config "file_name"
+
+### Loading
+
+Read a configuration from the dafult location `~/.torque_rm.yaml`
+
+    TORQUE.load_config
+
+In case the user has a custom file
+    
+    TORQUE.load_config "file_name"
+
+
+## Define a job using a DSL 
+
+    rhostname = TORQUE::Qsub.new do |job|
+            job.m = "abe"
+            job.name = "FirstTest"
+            job.shell = '/bin/bash'
+            job.ppn = 4
+            job.script = "echo `hostname`; sleep 1000"
+          end
+    rhostname.submit
+    rhostname.stat
+
+## Stats
+
+When a job is submitted, is possible to get some stats
+
+    [{:job_id=>"2750.sun.universe.space"@1387,
+      :job_name=>"FirstTest"@1421,
+      :job_owner=>"helios@sun.universe.space"@1447,
+      :job_state=>"Q"@1489,
+      :queue=>"bio"@1503,
+      :server=>"sun.universe.space"@1520,
+      :checkpoint=>"u"@1551,
+      :ctime=>"Tue Aug 27 15:56:41 2013"@1565,
+      :error_path=>
+       "sun.universe.space:/home/helios/FirstTest.e2750"@1607,
+      :hold_types=>"n"@1689,
+      :join_path=>"n"@1707,
+      :keep_files=>"n"@1726,
+      :mail_points=>"abe"@1746,
+      :mtime=>"Tue Aug 27 15:56:41 2013"@1762,
+      :output_path=>
+       "deep.space.nine:/home/helios/FirstTest.o2750"@1805,
+      :priority=>"0"@1885,
+      :qtime=>"Tue Aug 27 15:56:41 2013"@1899,
+      :rerunable=>"True"@1940,
+      :resource_list_nodect=>"1"@1972,
+      :resource_list_nodes=>"1:ppn=4"@2000,
+      :shell_path_list=>"/bin/bash"@2030,
+      :variable_list=>
+       "PBS_O_QUEUE=bio,PBS_O_HOST=deep.space.nine,PBS_O_HOME=/home/helios,PBS_O_LANG=it_IT.UTF-8,PBS_O_LOGNAME=helios,PBS_O_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/helios/.rvm/bin,PBS_O_MAIL=/var/mail/helios,PBS_O_SHELL=/bin/bash,PBS_SERVER=spark,PBS_O_WORKDIR=/home/helios\n"@2060,
+      :etime=>"Tue Aug 27 15:56:41 2013"@2456,
+      :submit_args=>"FirstTest.qsub"@2499,
+      :fault_tolerant=>"False"@2535,
+    :submit_host=>"sun.universe.space"@2559}]
+
+directly from the job
+    
+    job.stat
+
+or by requesting the information using the specific object
+
+    stats=TORQUE::Qstat.new
+
+    # select stats from only the submitted job
+    stat.query job_id: job.id
+
+it will return an hash but is also possible to return the raw string from TORQUE/PBS
+
+    stat.query job_id: job.id, type: :raw   
+
+Quering multiple jobs at the same time is possible as well (note the 's' after 'id'):
+
+    stat.query job_ids: ["2751.sun.universe.space","2752.sun.universe.space","2754.sun.universe.space","2755.sun.universe.space"]
+
+
+## Delete Jobs
+
+From a qsub object if submitted:
+
+    job.rm
+
+Directly from PBS:
+
+    TORQUE::Qdel.rm("2750.sun.universe.space")
+
+## REST Api
+
+All objects returned from TORQUE_RM_REST are in JSON format.
+
+### Qstat
+#### Get
+
+`/qstat` Returns an Array of Hash with the following keys, if available `"job_id","job_name","job_owner","resources_used_cput","resources_used_mem","resources_used_vmem","resources_used_walltime","job_state","queue","server","checkpoint","ctime","error_path","exec_host","exec_port","hold_types","join_path","keep_files","mail_points","mail_users","mtime","output_path","priority","qtime","rerunable","resource_list","session_id","shell_path_list","variable_list","etime","exit_status","submit_args","start_time","start_count","fault_tolerant","comp_time","job_radix","total_runtime","submit_host"` .
+
+`/qstat/fields` Returns an Array of String, the available fields for a single `qstat` query.
+
+`/qstat/all/:field` Return an Array of Strings, each String is the `field` requested by the user. If the `field` requested is not valid, it will return 404
+
+### Job
+#### Get
+
+`/qstat/:job_id` Returns an Hash with main key `job` and the JSON conversion of a TORQUE::Qstat::Job object. If the `job_id` does not exist will return 404
+
+`/qstat/job/fields` Returns and Hash `{"job_fields":["job_id","job_name","job_owner","resources_used_cput","resources_used_mem","resources_used_vmem","resources_used_walltime","job_state","queue","server","checkpoint","ctime","error_path","exec_host","exec_port","hold_types","join_path","keep_files","mail_points","mail_users","mtime","output_path","priority","qtime","rerunable","resource_list","session_id","shell_path_list","variable_list","etime","exit_status","submit_args","start_time","start_count","fault_tolerant","comp_time","job_radix","total_runtime","submit_host","is_runnig?","is_queued?","is_exited?","is_completed?","time","memory","node"]}` 
+
+`/qstat/:job_id/:field` Returns and Hash with a single key which is the required `field`. If the `job_id` or `field` do not exist will return 404
+
+#### Post 
+
+`/qstat/:job_id/del` Removes the `job_id` from the queue. If the `job_id` does not exist will return 404
+
+
+### Interacting from CLI
+
+To perform a query or call a fucntion use `curl` setting the content type. Extension `.json` is not considering for now, why? Read [What-is-the-correct-json-content-type](http://stackoverflow.com/questions/477816/what-is-the-correct-json-content-type) discussion on [Stackoverflow](http://stackoverflow.com/)
+
+curl -H "Content-type: application/json"  "http://localhost:4567/qstat"
+
+### WebApp
+
+A web application is under development and it will let you interact with you TORQUE/PBS server in a more fancy way, be patient and give us your feedback or request functionalities. A list of supported html pages:
+
+* get: /qstat
+
+### HowIsItMade
+
+[Sinatra](http://www.sinatrarb.com/) is the low level framework used for developing the REST API and the web infrastructure. [Twitter Bootstrap](http://getbootstrap.com/) is used to make the web app nice and fancy.
+
+### Run the Web App as a Serv(er/ice)
+
+From the command line
+
+`torque_rm_rest`
+
+click [here](http://localhost:4567/)
+
+Remember to configure by hand the `~/.torque_rm.yaml` as documented at the beginning
+
+
+
+## Contributing to torque_rm
+ 
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
+* Fork the project.
+* Start a feature/bugfix branch.
+* Commit and push until you are happy with your contribution.
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+## Copyright
+
+Copyright (c) 2013 Raoul Jean Pierre Bonnal. See LICENSE.txt for
+further details.
+