osc-machete 1.1.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: c1517fce8efdd8af65a55f139e0b4b38f1c3481c
+  data.tar.gz: e4fbae834bb6adf6b369cb655b23c7a550f0a0fd
+SHA512:
+  metadata.gz: 88a1af6cddefc4380f7622b9bd70efa3e364f930b3f14fc9a4f5f9b80e097f8372bffc36ee9eeab2bcdcff13b26d6aa2ef76eba0dd5e4dfed5d5f6fe6aad8f84
+  data.tar.gz: 434158ca87d980ab77125c43bed516febd1d8678c4d897b4edf7e6b7969307adcb4c0d019083bd5235052a3aa825ae8c1de0da15388f6c29c5059cd169bf807e

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/CHANGELOG.md

@@ -0,0 +1,87 @@
+# Change Log
+
+All notable changes to this project will be documented in this file.
+This project adheres to [Semantic Versioning](http://semver.org/).
+
+## [Unreleased]
+
+## [1.1.1] - 2016-02-24
+
+### Fixed
+
+- Omit account string when submitting a job if using default account string that is an invalid project i.e. `appl`
+
+## [1.1.0] - 2016-02-18
+
+### Changed
+
+- Account string by default is specified as being the primary group name of the
+process running the app. This corresponds to OSC's convention that the primary
+group is the project of the user.
+- OSC::Machete::Job is updated to change the default account string used for all
+instances via setting OSC::Machete::Job.default_account_string
+- OSC::Machete::Job is updated to accept account_string as an argument to the
+initializer to use for that instance.
+
+## [1.0.1] - 2016-02-16
+
+### Fixed
+
+- use latest version of pbs gem and its custom Error classes to catch the common cases for qdel and qstat when the pbsid is unknown
+
+## [1.0.0] - 2016-02-03
+
+### Fixed
+
+- qstat would return nil if the job completed or if an error occurred with qstat; now qstat throws exception in error cases and returns a valid Status value otherwise
+- using qstat with a Ruby job would fail because Ruby pbsid's don't include the host; fixed by adding host arg and if thats omitted inspecting the script first or else assuming its Ruby if the host is omitted from the job id
+
+### Added
+
+- lib/osc/machete/status.rb: Status value object
+- lib/osc/machete/process.rb: Provides helper methods wrapping Etc and Process modules to inspect user info from the currently running process.
+
+
+
+### Changed
+
+lib/osc/machete/user.rb
+
+- uses Etc instead of the environment variables to determine the current user by default (but any username can be passed in)
+- provides information about the specified user from Etc and inspecting the system's groups file
+- new methods include User#groups, User#member_of_group?, and a factory method to get an instance from the uid: User.from_uid
+
+lib/osc/machete/job.rb
+
+- host can be past in as an argument to the initializer; if this is not provided, torque_helper internally will try to determine what OSC system the PBSID corresponds to, or try inspecting the script for PBS headers
+- Job#submit now throws ScriptMissingError or PBS::Error
+- Job#status now returns an OSC::Machete::Status object instead of a character
+- Job#delete now throws PBS::Error
+
+lib/osc/machete/torque_helper.rb _(still an internal class right now, not meant to be used directly)_
+
+- try to determine what OSC system the PBSID corresponds to, or try inspecting the script for PBS headers
+- returns OSC::Machete::Status for qsub, qstat, qdel
+- uses pbs gem instead of shelling out for qsub, qstat, qdel
+- throws PBS::Error for qsub, qstat, qdel in erroneous cases
+- handles mapping between Torque specific status values and the generic OSC::Machete::Status
+- if host not provided, tries to determine host from pbsid and job script
+
+### Removed
+
+- lib/osc/machete/simple_job.rb - module is now alias for OscMacheteRails in the os_machete_rails gem; but including SimpleJob no longer results in including Statutable and Submittable
+- lib/osc/machete/simple_job/statusable.rb - moved to osc_machete_rails
+- lib/osc/machete/simple_job/workflow.rb - moved to osc_machete_rails
+- lib/osc/machete/simple_job/submittable.rb - removed! use has_workflow_of instead
+- lib/osc/machete/staging.rb - removed!
+
+## 0.6.3 - 2015-11-23
+
+Previous release of osc-machete
+
+[Unreleased]: https://github.com/AweSim-OSC/osc-machete/compare/v1.1.1...master
+[1.1.1]: https://github.com/AweSim-OSC/osc-machete/compare/v1.1.0...v1.1.1
+[1.1.0]: https://github.com/AweSim-OSC/osc-machete/compare/v1.0.1...v1.1.0
+[1.0.1]: https://github.com/AweSim-OSC/osc-machete/compare/v1.0.0...v1.0.1
+[1.0.0]: https://github.com/AweSim-OSC/osc-machete/compare/v0.6.3...v1.0.0
+

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in osc-machete.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013-2016 Ohio Supercomputer Center
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,194 @@
+# OSC::Machete
+
+Ruby code to help with staging and checking the status of batch jobs.
+
+
+## Installation
+
+To use, add this line to your application's Gemfile:
+
+    gem 'osc-machete'
+
+And then execute:
+
+    $ bundle install
+
+
+## Usage
+
+Three main classes are provided: Job, Process, and User.
+The other are support classes for these three.
+
+
+### OSC::Machete::Job
+
+This is the main class and is a utility class for managing batch simulations. It
+uses pbs-ruby to submit jobs, check the status of jobs, and stop running jobs.
+
+
+Check the status of a job:
+
+```ruby
+s = OSC::Machete::Job.new(pbsid: "117711759.opt-batch.osc.edu").status
+#=> #<OSC::Machete::Status:0x002ba824829e50 @char="R">
+puts s #=> "Running"
+```
+* status returns an `OSC::Machete::Status` value object
+
+Setup dependencies, submit, and delete a job:
+
+```ruby
+solve_job = OSC::Machete::Job.new(script: path_to_solve_script)
+post_job = OSC::Machete::Job.new(script: path_to_post_script)
+
+# ensure that post_job doesn't start till solve_job ends (with any exit status)
+post_job.afterany(solve_job)
+
+# submit both jobs (can do it in any order, dependencies will be managed for you)
+post_job.submit
+solve_job.submit
+
+# if you want to qdel both jobs:
+solve_job.delete
+post_job.delete
+```
+
+* when submitting a job, if a shell script is not found, OSC::Machete::Job::ScriptMissingError error is raised
+* `Job#submit`, `Job#status`, `Job#delete` all raise a `PBS::Error` if something
+goes wrong with interacting with Torque.
+
+#### Account String for submitting jobs
+
+By default, the account string will be set as a command line argument to qsub
+using the `-A` flag, which means setting this in a PBS header in the shell
+scripts will not work. The default account_string is the primary group of the
+process, which in our case happens to be the user.
+
+If you need to change the default account_string, you can do so by providing an
+extra argument to the OSC::Machete::Job initializer:
+
+```ruby
+j = OSC::Machete::Job.new(script: path_to_script)
+j.account_string # nil - so when the job submits the primary group will be used
+
+j = OSC::Machete::Job.new(script: path_to_script, account_string: "PZS0530")
+j.account_string # "PZS0530" - so when the job submits "PZS0530" will be used
+```
+
+You can also set a class variable on the job object so that all future job
+objects are instantiated using the specified account string:
+
+```ruby
+OSC::Machete::Job.default_account_string = "PZS0530"
+
+j = OSC::Machete::Job.new(script: path_to_script)
+j.account_string # "PZS0530" - so when the job submits "PZS0530" will be used
+```
+
+### OSC::Machete::Status
+
+See [Martin Fowler on value objects](http://martinfowler.com/bliki/ValueObject.html)
+
+```ruby
+s = OSC::Machete::Job.new(pbsid: "117711759.opt-batch.osc.edu").status
+#=> #<OSC::Machete::Status:0x002ba824829e50 @char="R">
+puts s #=> "Running"
+
+s.passed? #=> true
+s.completed? #=> true
+s.failed? #=> false
+s.submitted? #=> true
+
+f = OSC::Machete::Status.failed #=> #<OSC::Machete::Status:0x002ba8274334d8 @char="F">
+f.failed? #=> true
+f.completed? #=> true
+```
+
+To get an array of all the possible values:
+
+```ruby
+irb(main):001:0> OSC::Machete::Status.values
+=> [#<OSC::Machete::Status:0x002ba201079918 @char="U">, #<OSC::Machete::Status:0x002ba2010798a0 @char=nil>, #<OSC::Machete::Status:0x002ba201079710 @char="C">, #<OSC::Machete::Status:0x002ba201079620 @char="F">, #<OSC::Machete::Status:0x002ba201079558 @char="H">, #<OSC::Machete::Status:0x002ba2010794e0 @char="Q">, #<OSC::Machete::Status:0x002ba2010793c8 @char="R">, #<OSC::Machete::Status:0x002ba201079328 @char="S">]
+irb(main):002:0> OSC::Machete::Status.values.map(&:to_s)
+=> ["Undetermined", "Not Submitted", "Passed", "Failed", "Held", "Queued", "Running", "Suspended"]
+irb(main):003:0>
+```
+
+### OSC::Machete::Process
+
+Gives information about the running process. Uses Ruby's Process library when it
+makes sense.
+
+Examles using pry:
+
+```
+[14] pry(main)> OSC::Machete::Process.new.groupname
+=> "PZS0562"
+[15] pry(main)> OSC::Machete::Process.new.username
+=> "efranz"
+[16] pry(main)> OSC::Machete::Process.new.home
+=> "/nfs/17/efranz"
+[17] pry(main)> OSC::Machete::Process.new.group_membership_changed?
+=> false
+```
+
+### OSC::Machete::User
+
+Gives informaiton about the specified user, by using Ruby's Etc library and
+inspecting the group membership file.
+
+Example using pry:
+
+```
+[18] pry(main)> OSC::Machete::User.new.member_of_group?("awsmdev")
+=> true
+[19] pry(main)> OSC::Machete::User.new.home
+=> "/nfs/17/efranz"
+[20] pry(main)> OSC::Machete::Process.new.home
+=> "/nfs/17/efranz"
+[21] pry(main)> OSC::Machete::User.new.groups
+=> [2959,
+ 3140,
+ 3141,
+ 3179,
+ 3285,
+ 3528,
+ 3572,
+ 4391,
+ 4497,
+ 4498,
+ 4511,
+ 4514,
+ 4517,
+ 4580,
+ 4807,
+ 4808]
+```
+
+### Example of using Machete directly via irb
+
+```sh
+-bash-3.2$ pry -rosc/machete
+[24] pry(main)> puts OSC::Machete::Job.new(pbsid: "17711768.opt-batch.osc.edu").status
+Running
+=> nil
+```
+
+Or you could write your own ruby script that  that does something using the gem:
+
+```sh
+-bash-3.2$ cat test.rb
+require 'osc/machete'
+
+pbsid = "17711768.opt-batch.osc.edu"
+j = OSC::Machete::Job.new pbsid: pbsid
+puts j.status
+```
+
+And then run it like this:
+
+```sh
+-bash-3.2$ ruby test.rb
+Running
+-bash-3.2$
+```

data/Rakefile

@@ -0,0 +1,23 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'test'
+end
+
+desc "Run tests"
+task :default do
+
+  puts "\nIf you want to run tests that submit simple jobs to batch system, " \
+    "set the environment variable LIVETEST.\n\n" unless ENV['LIVETEST']
+
+  Rake::Task['test'].invoke
+end
+
+task :console do
+  require 'irb'
+  require 'irb/completion'
+  require 'osc/machete'
+  ARGV.clear
+  IRB.start
+end

data/lib/osc/machete.rb

@@ -0,0 +1,18 @@
+# declares the module
+require "osc/machete/version"
+
+require "osc/machete/status"
+require "osc/machete/job"
+require "osc/machete/job_dir"
+require "osc/machete/location"
+require "osc/machete/torque_helper"
+require "osc/machete/user"
+require "osc/machete/process"
+
+# The OSC namespace module.
+module OSC
+  # The main osc-machete module.
+  module Machete
+    # Your code goes here...
+  end
+end

data/lib/osc/machete/job.rb

@@ -0,0 +1,239 @@
+require 'pathname'
+
+
+# Core object for working with batch jobs, including:
+#
+# * submitting jobs
+# * checking job status
+# * setting dependencies between jobs via a directed acyclic graph
+#
+# Create a new Job from a script:
+#
+#     job = Job.new(script: '/nfs/17/efranz/jobs/1/script.sh')
+#     job.submitted? #=> false
+#     job.path #=> '/nfs/17/efranz/jobs/1'
+#     job.script_name #=> 'script.sh'
+#     job.status #=> nil
+#     job.pbsid #=> nil
+#
+#     # PBS_O_WORKDIR will be set to the directory containing the script
+#     job.submit
+#
+#     job.submitted? #=> true
+#     job.status #=> "Q"
+#     job.pbsid #=> "3422735.oak-batch"
+#
+#     # if you know the pbs id you can instantiate a
+#     # Job object to ask for the status of it
+#     job2 = Job.new(pbsid: "3422735.oak-batch")
+#     job2.status #=> "Q"
+#
+#     # because the object was created with only the pbsid passed in,
+#     # path and script_name and dependency information is not available
+#     job2.path #=> nil
+#     job2.script_name #=> nil
+#
+#     # but an unknown pbsid results in status nil
+#     job3 = Job.new(pbsid: "12345.oak-batch")
+#     job3.status #=> nil
+#
+# Create two Job instances and form a dependency between them:
+#
+#     job1 = Job.new(script: '/nfs/17/efranz/jobs/1/script.sh')
+#     job2 = Job.new(script: '/nfs/17/efranz/jobs/1/post.sh')
+#
+#     job2.afterany(job1) # job2 runs after job1 completes with any exit status
+#
+#     job1.submit
+#     job2.submit
+#
+#     job1.status #=> "Q"
+#     job2.status #=> "H"
+#
+# @!attribute [r] pbsid
+#   @return [String, nil] the PBS job id, or nil if not set
+# @!attribute [r] script_path
+#   @return [String, nil] path of the job script, or nil if not set
+#
+class OSC::Machete::Job
+  attr_reader :pbsid, :script_path, :account_string
+
+  class << self
+    # set this to change the billable account that is used by default
+    attr_accessor :default_account_string
+  end
+
+  # Create Job instance to represent an unsubmitted batch job from the specified
+  # script, or an existing, already submitted batch job from the specified pbsid
+  #
+  # Takes params in options hash as single argument:
+  #
+  #     Job.new(script: '/path/to/job/dir/go.sh')
+  #
+  # or
+  #
+  #     opts = Hash.new(script: '/path/to/job/dir/go.sh')
+  #     Job.new(opts)
+  #
+  # Job class makes assumption that a job's PBS_O_WORKDIR will be
+  # in the directory containing the shell script that is run.
+  #
+  # @param [Hash] args the arguments to create the job
+  # @option args [String] :script  full path to script (optional)
+  # @option args [String, nil] :pbsid   pbsid of a job already submitted (optional)
+  # @option args [TorqueHelper, nil]  :torque_helper  override default torque helper (optional)
+  #                       NOTE: used for testing purposes
+  #                       we could use it also if we had different
+  #                       torque_helper classes for different systems
+  def initialize(args)
+    @script_path = Pathname.new(args[:script]).cleanpath unless args[:script].nil?
+    # @script_path = @script_path.expand_path would change this to absolute path
+
+    @pbsid =  args[:pbsid]
+    @host =   args[:host]
+    @torque = args[:torque_helper] || OSC::Machete::TorqueHelper.default
+    @account_string = args[:account_string] || self.class.default_account_string
+
+    @dependencies = {} # {:afterany => [Job, Job], :afterok => [Job]}
+  end
+
+  # @return [String, nil] script name or nil if instance wasn't initialized with a script
+  def script_name
+    Pathname.new(@script_path).basename.to_s if @script_path
+  end
+
+  # @return [String, nil] job directory or nil if instance wasn't initialized with a script
+  def path
+    Pathname.new(@script_path).dirname if @script_path
+  end
+
+  # Submit any dependent jobs that haven't been submitted
+  # then submit this job, specifying dependencies as required by Torque.
+  # Submitting includes cd-ing into the script's directory and qsub-ing from
+  # that location, ensuring that environment variable PBS_O_WORKDIR is
+  # set to the directory containing the script.
+  #
+  # @raise [ScriptMissingError] Raised when the path to the script does not exist or cannot be read.
+  def submit
+    return if submitted?
+    raise ScriptMissingError, "#{script_path} does not exist or cannot be read" unless script_path.file? && script_path.readable?
+
+    # submit any dependent jobs that have not yet been submitted
+    submit_dependencies
+
+    # cd into directory, submit job from there
+    # so that PBS_O_WORKDIR is set to location
+    # where job is run
+    #
+    #TODO: you can set PBS_O_WORKDIR via qsub args, is this necessary? there is
+    # another env var besides PBS_O_WORKDIR that is affected by the path of the
+    # current directory when the job is submitted
+    #
+    #TODO: what if you want to submit via piping to qsub i.e. without creating a file?
+    Dir.chdir(path.to_s) do
+      @pbsid = @torque.qsub script_name, depends_on: dependency_ids, host: @host, account_string: account_string
+    end
+  end
+
+  # Check whether the job jas been submitted.
+  #
+  # @return [Boolean] true if @pbsid is set
+  def submitted?
+    ! @pbsid.nil?
+  end
+
+  # Perform a qstat and return a char representing the status of the job.
+  #
+  # @return [Status] value object representing status of a job
+  def status
+    if @pbsid.nil?
+      OSC::Machete::Status.not_submitted
+    else
+      @torque.qstat @pbsid, host: @host
+    end
+  end
+
+  # Ensure Job starts only after the specified Job(s) complete
+  #
+  # @param [Job, Array<Job>] jobs Job(s) that this Job should depend on (wait for)
+  # @return [self] self so you can chain method calls
+  def afterany(jobs)
+    add_dependencies(:afterany, jobs)
+  end
+
+  # Ensure Job starts only after the specified Job(s) complete with successful
+  # return value.
+  #
+  # @param (see #afterany)
+  # @return (see #afterany)
+  def afterok(jobs)
+    add_dependencies(:afterok, jobs)
+  end
+
+  # Ensure Job starts only after the specified Job(s) start.
+  #
+  # @param (see #afterany)
+  # @return (see #afterany)
+  def after(jobs)
+    add_dependencies(:after, jobs)
+  end
+
+  # Ensure Job starts only after the specified Job(s) complete with error
+  # return value.
+  #
+  # @param (see #afterany)
+  # @return (see #afterany)
+  def afternotok(jobs)
+    add_dependencies(:afternotok, jobs)
+  end
+
+  # Kill the currently running batch job
+  #
+  # @param [Boolean] rmdir (false) if true, recursively remove the containing directory
+  #                                of the job script if killing the job succeeded
+  #
+  # @return [nil]
+  def delete(rmdir: false)
+    # FIXME: rethink this interface... should qdel be idempotent?
+    # After first call, no errors thrown after?
+
+    if pbsid
+
+      @torque.qdel(pbsid, host: @host)
+      # FIXME: removing a directory is always a dangerous action.
+      # I wonder if we can add more tests to make sure we don't delete
+      # something if the script name is munged
+
+      # recursively delete the directory after killing the job
+      Pathname.new(path).rmtree if path && rmdir && File.exists?(path)
+    end
+  end
+
+  # Error class thrown when script is not available.
+  class ScriptMissingError < StandardError; end
+
+  private
+
+  def submit_dependencies
+    #  assumes each dependency is a Job object
+    @dependencies.values.flatten.each { |j| j.submit }
+  end
+
+  # build a dictionary of ids for each dependency type
+  def dependency_ids
+    ids = {}
+
+    @dependencies.each do |type, jobs|
+      ids[type] = jobs.map(&:pbsid).compact
+    end
+
+    ids.keep_if { |k,v| ! v.empty? }
+  end
+
+  def add_dependencies(type, jobs)
+    @dependencies[type] = [] unless @dependencies.has_key?(type)
+    @dependencies[type].concat(Array(jobs))
+
+    self
+  end
+end