osc-reservations 1.0.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ab5a834ec9a8cff3089cb19f5ad11c3ae15ffc14
+  data.tar.gz: 526dc32a86f095ab039bb7a39f046d8566cdc9cb
+SHA512:
+  metadata.gz: acaca8c8eddeb61afb95eb33a70c5f119148a8f2f2496b31dbadf3baad6ba5220ae43353bd11e553c2292beb0e3e1fb36a8a1cb03ea97197e13abcb18b977f62
+  data.tar.gz: 0ef6c89a286cce588c33bd94e97aa2afbd18e03f664a50b92e23ed8412dfe28b2908feec94bda15639f72e79c7c4eb5496ca0174d0fd57325183e17f9c965e2f

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in osc-reservations.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015-2016 Ohio Supercomputer Center
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,103 @@
+# OSC::Reservations
+
+Ruby library that queries OSC systems for active batch reservations of the
+current user.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'osc-reservations'
+
+And then execute:
+
+    $ bundle install
+
+## Usage
+
+Currently only `OSC::Reservations::Query` is implemented, meaning you can query
+for available reservations or for a single reservation. At this time you are
+unable to submit a reservation to the system. Also it should be noted that the
+OSC scheduler is set up to display only the reservations owned by the current
+running user. Viewing reservations that the current user does not have access
+to is impossible at this time.
+
+### Simple Example (Oakley)
+
+This library has a set of OSC clusters pre-programmed into it. Querying a
+reservation of the Oakley cluster is as simple as
+
+```ruby
+q = OSC::Reservations::Query.oakley
+
+q.reservations
+```
+
+This will return an array of immutable `Reservation` objects containing the
+relevant information for each reservation allocated to the user. The structure
+of such objects are detailed in the corresponding yardoc pages.
+
+### Advance Example
+
+This library comes with a list of adapters to communicate with your
+corresponding batch scheduler. As of writing this the only adapter implemented
+is the `Adapters::OSCMoab` adapter. It provides an interface to OSC's Moab
+batch scheduler.
+
+Once you have chosen your desired adapter, you will need to define a `Batch`
+server object that the library will connect to when requesting/submitting the
+reservations. Certain adapters may have further requirements on the `Batch`
+object (e.g., `Adapters::OSCMoab` requires a `mrsvctl` property on the `Batch`
+object).
+
+A full implementation would look like
+
+```ruby
+adapter = OSC::Reservations::Adapters::OSCMoab.new
+batch = OSC::Reservations::Batch.new(
+  oak-batch.osc.edu,
+  mrsvctl: 'LD_LIBRARY_PATH=/usr/local/moab-6.1.11/lib /usr/local/moab-6.1.11/bin/mrsvctl'
+)
+
+q = OSC::Reservations::Query.new adapter, batch
+q.reservations
+```
+
+### Configure your own Batch Schedulers
+
+Currently the library will read from a defined configuration YAML file located
+at `conf/batch.yml`. An example entry may look like
+
+```yaml
+oakley:
+  adapter: 'OSC::Reservations::Adapters::OSCMoab'
+  server: 'oak-batch.osc.edu'
+  mrsvctl: 'LD_LIBRARY_PATH=/usr/local/moab-6.1.11/lib /usr/local/moab-6.1.11/bin/mrsvctl'
+```
+
+The `Query` object will then use these entries when defining its methods. For
+example, a developer can access this `oakley` batch scheduler through
+
+```ruby
+q = OSC::Reservations::Query.oakley
+```
+
+To predefine your own batch schedulers you will need to make a local YAML file
+with the necessary properties. Then you would be able to access and use this
+YAML file as shown below.
+
+```ruby
+OSC::Reservations.batch_config_path = "/path/to/config.yml"
+q = OSC::Reservations::Query.my_batch
+
+q.reservations
+```
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Test your changes (`rake test`)
+4. Commit your changes (`git commit -am 'Add some feature'`)
+5. Push to the branch (`git push origin my-new-feature`)
+6. Create new Pull Request

data/Rakefile

@@ -0,0 +1,7 @@
+require 'bundler/gem_tasks'
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs << "test"
+  t.pattern = "test/**/test_*.rb"
+end

data/config/batch.yml

@@ -0,0 +1,13 @@
+oakley:
+  adapter: &OSCMoab
+    type: 'OSC::Reservations::Adapters::OSCMoab'
+  batch:
+    server: 'oak-batch.osc.edu'
+    torque_name: 'oakley'
+    mrsvctl: 'LD_LIBRARY_PATH=/usr/local/moab-6.1.11/lib:$LD_LIBRARY_PATH MOABHOMEDIR=/var/spool/batch/moab /usr/local/moab-6.1.11/bin/mrsvctl'
+ruby:
+  adapter: *OSCMoab
+  batch:
+    server: 'ruby-batch.ten.osc.edu'
+    torque_name: 'ruby'
+    mrsvctl: 'LD_LIBRARY_PATH=/usr/local/moab-6.1.11/lib:$LD_LIBRARY_PATH MOABHOMEDIR=/var/spool/batch/moab /usr/local/moab-6.1.11/bin/mrsvctl'

data/config/websvcs08.osc.edu.yml

@@ -0,0 +1,13 @@
+oakley:
+  adapter: &OSCMoab
+    type: 'OSC::Reservations::Adapters::OSCMoab'
+  batch:
+    server: 'oak-batch.osc.edu'
+    torque_name: 'oakley'
+    mrsvctl: 'LD_LIBRARY_PATH=/usr/local/moab/8.1.1.2-2015080516-eb28ad0-el6/lib:$LD_LIBRARY_PATH MOABHOMEDIR=/var/spool/batch/moab /usr/local/moab/8.1.1.2-2015080516-eb28ad0-el6/bin/mrsvctl'
+ruby:
+  adapter: *OSCMoab
+  batch:
+    server: 'ruby-batch.ten.osc.edu'
+    torque_name: 'ruby'
+    mrsvctl: 'LD_LIBRARY_PATH=/usr/local/moab/8.1.1.2-2015080516-eb28ad0-el6/lib:$LD_LIBRARY_PATH MOABHOMEDIR=/var/spool/batch/moab /usr/local/moab/8.1.1.2-2015080516-eb28ad0-el6/bin/mrsvctl'

data/lib/osc/reservations.rb

@@ -0,0 +1,47 @@
+require 'yaml'
+require 'socket'
+
+require_relative 'reservations/version'
+require_relative 'reservations/batch'
+require_relative 'reservations/reservation'
+require_relative 'reservations/node'
+require_relative 'reservations/query'
+
+# The namespace used for OSC gems.
+module OSC
+  # The main namespace for OSC::Reservations. Provides the ability to submit
+  # and read back reservations to the local batch scheduler.
+  module Reservations
+      # Path to batch server configuration file.
+      CONFIG_ROOT = File.expand_path("../../../config", __FILE__)
+
+      # Default path to the batch config yaml file.
+      # @return [String] Path to the default batch config yaml file.
+      def self.default_batch_config_path
+        host_config = File.expand_path("#{CONFIG_ROOT}/#{Socket.gethostname}.yml")
+        default_config = File.expand_path("#{CONFIG_ROOT}/batch.yml")
+        File.file?(host_config) ? host_config : default_config
+      end
+
+      # Path to the batch config yaml file describing the batch servers for
+      # local batch schedulers.
+      # @return [String] Path to the batch config yaml file.
+      def self.batch_config_path
+        @batch_config_path ||= default_batch_config_path
+      end
+
+      # Set the path to the batch config yaml file.
+      # @param path [String] The path to the batch config yaml file.
+      def self.batch_config_path=(path)
+        @batch_config_path = File.expand_path(path)
+      end
+
+      # Hash generated from reading the batch config yaml file.
+      # @return [Hash] Batch configuration generated from config yaml file.
+      def self.batch_config
+        YAML.load_file(batch_config_path)
+      end
+  end
+end
+
+require_relative 'reservations/adapters/osc_moab'

data/lib/osc/reservations/adapter.rb

@@ -0,0 +1,38 @@
+module OSC
+  module Reservations
+    # Adapters are the glue between Reservations API and the scheduling
+    # service. This provides a scheduler independent interface.
+    class Adapter
+      # @param opts [Hash] The options to create an adapter with.
+      def initialize(opts = {})
+      end
+
+      # Queries the batch server for a given reservation.
+      # @param batch [Batch] The batch server to query for the reservation.
+      # @param id [String] The ID of the reservation.
+      # @return [Reservation, nil]
+      # @abstract This should be implemented by the adapter.
+      def query_reservation(batch, id)
+        raise NotImplementedError
+      end
+
+      # Queries the batch server for a list of reservations.
+      # @param batch [Batch] The batch server to query for reservations.
+      # @return [Array<Reservation>]
+      # @abstract This should be implemented by the adapter.
+      def query_reservations(batch)
+        raise NotImplementedError
+      end
+
+      # @!method submit_reservation(batch, reservation)
+      # Submits a given reservation to the batch server.
+      # @param batch [Batch] The batch server to submit the reservation at.
+      # @param reservation [Reservation] A reservation with necessary information.
+      # @return [Reservation]
+      # @abstract This should be implemented by the adapter.
+      def submit_reservation(batch, reservation)
+        raise NotImplementedError
+      end
+    end
+  end
+end

data/lib/osc/reservations/adapters/osc_moab.rb

@@ -0,0 +1,167 @@
+require 'nokogiri'
+require 'open3'
+require 'pbs'
+
+require_relative '../adapter'
+
+module OSC
+  module Reservations
+    # A namespace to hold all subclasses of {Adapter}.
+    module Adapters
+      # This adapter is designed for OSC systems using the Moab scheduler. This
+      # adapter requires the parameters <tt>torque_name</tt> and
+      # <tt>mrsvctl</tt> in the Batch object to work. Only tested on Torque >4.
+      class OSCMoab < Adapter
+        # Queries the batch server for a given reservation.
+        # @param batch [Batch] The batch server to query for the reservation.
+        # @param id [String] The ID of the reservation.
+        # @return [Reservation, nil]
+        # @raise [CommandLineError] if a command run from the shell exits with error
+        def query_reservation(batch, id)
+          cmd = "#{batch.mrsvctl} --host=#{batch.server} -q #{id} --xml"
+          begin
+            xml = get_xml(cmd, batch)
+          rescue CommandLineError
+            return
+          end
+
+          r_xml = xml.xpath("//rsv")
+          rsv = parse_rsv(r_xml)
+          query_nodes(rsv, batch)
+          query_node_users(rsv, batch)
+
+          rsv
+        end
+
+        # Queries the batch server for a list of reservations.
+        # @param batch [Batch] The batch server to query for the reservation.
+        # @return [Array<Reservation>]
+        # @raise [CommandLineError] if a command run from the shell exits with error
+        def query_reservations(batch)
+          cmd = "#{batch.mrsvctl} --host=#{batch.server} -q ALL --xml"
+          xml = get_xml(cmd, batch)
+          xml = filter(xml) # filter out "fake" reservations
+
+          rsv_list = []
+          xml.xpath("//rsv").each do |r_xml|
+            rsv_list << parse_rsv(r_xml)
+          end
+          query_nodes(rsv_list, batch)
+          query_node_users(rsv_list, batch)
+
+          rsv_list
+        end
+
+      # Submits a given reservation to the batch server.
+      # @param batch [Batch] The batch server to submit the reservation at.
+      # @param reservation [Reservation] A reservation with necessary information.
+      # @return [Reservation]
+      # @raise [CommandLineError] if a command run from the shell exits with error
+      def submit_reservation(batch, reservation)
+      end
+
+      private
+
+        # Query the nodes from the batch server
+        # @param rsvs [Array<Reservation>] An array of reservations to check.
+        # @param batch [Batch] The batch server object to use.
+        # @return [Array<Reservation>] The array of reservations on this batch server.
+        def query_nodes(rsvs, batch)
+          # make list of unique nodes for all reservations
+          nodes = [*rsvs].map(&:nodes).flatten.uniq
+
+          c = PBS::Conn.batch(batch.torque_name)
+          q = PBS::Query.new(conn: c, type: :node)
+          node_list = {}
+          nodes.each do |n|
+            node_hash = q.find(id: n)[0][:attribs]
+            np = node_hash[:np].to_i
+            props = node_hash[:properties].split(',')
+            jobs = node_hash.fetch(:jobs, '').split(',').map{|j| j.split('/')[1]}
+            users = []
+
+            node_list[n] = Node.new(
+              id: n,
+              np: np,
+              props: props,
+              jobs: jobs,
+              users: users
+            )
+          end
+
+          # replace node lists with lists of Node objects
+          [*rsvs].each{|r| r.nodes.map!{|n| node_list[n]}}
+        end
+
+        # Query what users are running jobs on the nodes for list of reservations
+        # @param rsvs [Array<Reservation>] An array of reservations to check.
+        # @param batch [Batch] The batch server object to use.
+        def query_node_users(rsvs, batch)
+          # make list of job ids
+          jobs = [*rsvs].map{|r| r.nodes.map(&:jobs)}.flatten.uniq
+
+          c = PBS::Conn.batch(batch.torque_name)
+          q = PBS::Query.new(conn: c, type: :job)
+          job_list = {}
+          jobs.each do |j|
+            job_list[j] = q.find(id: j)[0][:attribs][:Job_Owner].split('@')[0]
+          end
+
+          [*rsvs].each{|r| r.nodes.each{|n| n.users = n.jobs.map{|j| job_list[j]}.uniq}}
+        end
+
+        # Error when executing from command line
+        class CommandLineError < StandardError; end
+
+        # Get reservation list as xml
+        # @param cmd [String] Command used in terminal to get XML reservation list.
+        # @param batch [Batch] The batch server object to use.
+        # @return [Nokogiri::XML::Document] The XML document corresponding to the queried reservation list.
+        def get_xml(cmd, batch)
+          Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
+            exit_status = wait_thr.value
+            unless exit_status.success?
+              raise CommandLineError, "Bad exit status for: #{cmd}"
+            end
+
+            Nokogiri::XML(stdout.read)
+          end
+        end
+
+        # Filter out unwanted reservations in a query
+        # @param xml [Nokogiri::XML::Document] The XML document to filter for this adapter.
+        # @return [Nokogiri::XML::Document] The filtered XML document.
+        def filter(xml)
+          # Remove running jobs
+          xml.xpath("//rsv[@SubType='JobReservation']").remove
+
+          # Remove debug queue from Glenn cluster
+          # xml.xpath("//rsv[@SubType='StandingReservation']").remove
+
+          xml
+        end
+
+        # Parse a reservation from a "rsv" element in XML
+        # @param xml [Nokogiri::XML::Element] The XML element corresponding to a reservation.
+        # @return [Reservation] The reservation object.
+        def parse_rsv(xml)
+          id = xml.xpath("@Name").to_s
+          starttime = Time.at(xml.xpath("@starttime").to_s.to_i)
+          endtime = Time.at(xml.xpath("@endtime").to_s.to_i)
+          auser = xml.xpath("@AUser").to_s
+          users = xml.xpath("ACL[@type='USER']/@name").map { |v| v.value }
+          nodes = xml.xpath("@AllocNodeList").to_s.split(',')
+
+          Reservation.new(
+             id: id,
+             starttime: starttime,
+             endtime: endtime,
+             auser: auser,
+             users: users,
+             nodes: nodes
+          )
+        end
+      end
+    end
+  end
+end