ood_core 0.11.2 → 0.14.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ee8aed504e3bdb8ef0e768a82cae8632a683d16764248a16c6061350a88f9f86
-  data.tar.gz: 92c4f97a2f6f10e22085be6406ff5d3f9eebd1e0656a15f7b835f4fe4421d90a
+  metadata.gz: 52ba764b085dedb7eaeb06d95751f1804a50488e1859f980a7836d2d9032b95d
+  data.tar.gz: c2dc5edf395fe158960f33b80c554f3dc745f15e7ec1337b738683a0e1bbdc7f
 SHA512:
-  metadata.gz: 5b0f8422158f377f56a202a788a703ceb628dcfffc1e13a78c193ffa4e0f538108280a3c45af2dd404dd933142c1e78858cb7e8c2ffbdcd51e243d8d23afb6ca
-  data.tar.gz: 7599e841e9c9e56b1c3308e5baa8b58db919707d940947dfee8cc93fdf98e5262b4674aec551ad084428c0312e4725408cb13d589ea9c119b71d3eb66f94c2de
+  metadata.gz: 59915bae23a008a923c249d222e50548a7bee3438144068a29ae1cafdd489ca1229ee1a14f4f81e3fd065381f46f920bef24344fe633c7c578cb1f6a4f9a2a77
+  data.tar.gz: 8d2ca42c7f49158c8d321c21b79aff1c636df3c77bb7e71107db70371a34058d79c8a5ec32ca93883e7d3bcc7dc2202375144d23613f167ab089318d6270248c

data/CHANGELOG.md

@@ -6,6 +6,39 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+## [0.14.0] - 2020-10-01
+### Added 
+- Kubernetes adapter in PR [156](https://github.com/OSC/ood_core/pull/156)
+
+### Fixed
+- Catch Slurm times. [209](https://github.com/OSC/ood_core/pull/209)
+- LHA race condition in deleteing tmp files. [212](https://github.com/OSC/ood_core/pull/212)
+
+## [0.13.0] - 2020-08-10
+### Added
+- CloudyCluster CCQ Adapter
+
+## [0.12.0] - 2020-08-05
+### Added
+- qos option to Slurm and Torque [#205](https://github.com/OSC/ood_core/pull/205)
+- native hash returned in qstat for SGE adapter [#198](https://github.com/OSC/ood_core/pull/198)
+- option for specifying `submit_host` to submit jobs via ssh on other host [#204](https://github.com/OSC/ood_core/pull/204)
+
+### Fixed
+- SGE handle milliseconds instead of seconds when milliseconds used [#206](https://github.com/OSC/ood_core/issues/206)
+- Torque's native "hash" for job submission now handles env vars values with spaces [#202](https://github.com/OSC/ood_core/pull/202)
+
+## [0.11.4] - 2020-05-27
+### Fixed
+- Environment exports in SLURM while implementing [#158](https://github.com/OSC/ood_core/issues/158)
+  and [#109](https://github.com/OSC/ood_core/issues/109) in [#163](https://github.com/OSC/ood_core/pull/163)
+
+## [0.11.3] - 2020-05-11
+### Fixed
+- LinuxhHost Adapter to work with any login shell ([#188](https://github.com/OSC/ood_core/pull/188))
+- LinuxhHost Adapter needs to display long lines in pstree to successfully parse
+  output ([#188](https://github.com/OSC/ood_core/pull/188))
+
 ## [0.11.2] - 2020-04-23
 ### Fixed
 - fix signature of `LinuxHost#info_where_owner`
@@ -222,7 +255,12 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 - Initial release!
 
-[Unreleased]: https://github.com/OSC/ood_core/compare/v0.11.2...HEAD
+[Unreleased]: https://github.com/OSC/ood_core/compare/v0.14.0...HEAD
+[0.14.0]: https://github.com/OSC/ood_core/compare/v0.13.0...v0.14.0
+[0.13.0]: https://github.com/OSC/ood_core/compare/v0.12.0...v0.13.0
+[0.12.0]: https://github.com/OSC/ood_core/compare/v0.11.4...v0.12.0
+[0.11.4]: https://github.com/OSC/ood_core/compare/v0.11.3...v0.11.4
+[0.11.3]: https://github.com/OSC/ood_core/compare/v0.11.2...v0.11.3
 [0.11.2]: https://github.com/OSC/ood_core/compare/v0.11.1...v0.11.2
 [0.11.1]: https://github.com/OSC/ood_core/compare/v0.11.0...v0.11.1
 [0.11.0]: https://github.com/OSC/ood_core/compare/v0.10.0...v0.11.0

data/README.md

@@ -4,12 +4,13 @@
 ![GitHub Release](https://img.shields.io/github/release/osc/ood_core.svg)
 ![GitHub License](https://img.shields.io/github/license/osc/ood_core.svg)
 
-Welcome to your new gem! In this directory, you'll find the files you need to
-be able to package up your Ruby library into a gem. Put your Ruby code in the
-file `lib/ood_core`. To experiment with that code, run `bin/console` for an
-interactive prompt.
+- Website: http://openondemand.org/
+- Website repo with JOSS publication: https://github.com/OSC/Open-OnDemand
+- Documentation: https://osc.github.io/ood-documentation/latest/
+- Main code repo: https://github.com/OSC/ondemand
+- Core library repo: https://github.com/OSC/ood_core
 
-TODO: Delete this and the text above, and describe your gem
+OnDemand core library with adapters for each batch scheduler.
 
 ## Installation
 

data/lib/ood_core/job/adapters/ccq.rb

@@ -0,0 +1,267 @@
+require "ood_core/job/adapters/helper"
+require "tempfile"
+
+module OodCore
+  module Job
+    class Factory
+      using Refinements::HashExtensions
+
+      # Build the Cloudy Cluster adapter from a configuration
+      # @param config [#to_h] the configuration for job adapter
+      # @option config [Object] :image (nil) The default VM image to use
+      # @option config [Object] :cloud (gcp) The cloud provider being used [gcp,aws]
+      # @option config [Object] :scheduler (nil) The name of the scheduler to use
+      # @option config [Object] :sge_root (nil) Path to SGE root, note that
+      # @option config [#to_h] :bin (nil) Path to CC client binaries
+      # @option config [#to_h] :bin_overrides ({}) Optional overrides to CC client executables
+      def self.build_ccq(config)
+        Adapters::CCQ.new(config.to_h.symbolize_keys)
+      end
+    end
+
+    module Adapters
+
+      class PromptError < StandardError; end
+
+      class CCQ < Adapter
+        using Refinements::ArrayExtensions
+
+        attr_reader :image, :cloud, :scheduler, :bin, :bin_overrides, :jobid_regex
+
+        def initialize(config)
+          @image = config.fetch(:image, nil)
+          @cloud = config.fetch(:cloud, gcp_provider)
+          @scheduler = config.fetch(:scheduler, nil)
+          @bin = config.fetch(:bin, '/opt/CloudyCluster/srv/CCQ')
+          @bin_overrides = config.fetch(:bin_overrides, {})
+          @jobid_regex = config.fetch(:jobid_regex, "job id is: (?<job_id>\\d+) you")
+        end
+
+        # Submit a job with the attributes defined in the job template instance
+        # @param script [Script] script object that describes the script and
+        #   attributes for the submitted job
+        # @param after [#to_s, Array<#to_s>] not used
+        # @param afterok [#to_s, Array<#to_s>] not used
+        # @param afternotok [#to_s, Array<#to_s>] not used
+        # @param afterany [#to_s, Array<#to_s>] not used
+        # @return [String] the job id returned after successfully submitting a
+        #   job
+        # @see Adapter#submit
+        def submit(script, after: [], afterok: [], afternotok: [], afterany: [])
+          script_file = make_script_file(script.content)
+          args = []
+
+          # cluster configuration args
+          args.concat ["-s", scheduler] unless scheduler.nil?
+          args.concat [image_arg, image] unless image.nil?
+
+          args.concat ["-o", script.output_path.to_s] unless script.output_path.nil?
+          args.concat ["-e", script.error_path.to_s] unless script.error_path.nil?
+          args.concat ["-tl", seconds_to_duration(script.wall_time)] unless script.wall_time.nil?
+          args.concat ["-js", script_file.path.to_s]
+
+          args.concat script.native if script.native
+
+          output = call("ccqsub", args: args)
+          parse_job_id_from_ccqsub(output)
+        ensure
+          script_file.close
+        end
+
+        # Retrieve info for all jobs from the resource manager
+        # @return [Array<Info>] information describing submitted jobs
+        def info_all(attrs: nil)
+          args = []
+          args.concat ["-s", scheduler] unless scheduler.nil?
+
+          stat_output = call("ccqstat", args: args)
+          info_from_ccqstat(stat_output)
+        end
+
+        # Retrieve job info from the resource manager
+        # @param id [#to_s] the id of the job
+        # @return [Info] information describing submitted job
+        def info(id)
+          args = []
+          args.concat ["-s", scheduler] unless scheduler.nil?
+          args.concat ["-ji", id]
+
+          stat_output = call("ccqstat", args: args)
+
+          # WARNING: code path differs here than info_all because the output
+          # from ccqstat -ji $JOBID is much more data than just the 4
+          # columns that ccqstat gives.
+          info_from_ccqstat_extended(stat_output)
+        end
+
+        # Retrieve job status from resource manager
+        # @param id [#to_s] the id of the job
+        # @return [Status] status of job
+        # @see Adapter#status
+        def status(id)
+          info(id).status
+        end
+
+        # This adapter does not implement hold and will always raise
+        #   an exception.
+        # @param id [#to_s] the id of the job
+        # @raise [JobAdapterError] always
+        # @return [void]
+        def hold(_)
+          raise NotImplementedError, "subclass did not define #hold"
+        end
+
+        # This adapter does not implement release and will always raise
+        #   an exception.
+        # @param id [#to_s] the id of the job
+        # @raise [JobAdapterError] always
+        # @return [void]
+        def release(_)
+          raise NotImplementedError, "subclass did not define #release"
+        end
+
+        # Delete the submitted job
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def delete(id)
+          call("ccqdel", args: [id])
+        end
+
+        def directive_prefix
+          '#CC'
+        end
+
+        private
+
+        # Mapping of state codes
+        STATE_MAP =
+        {
+          'Error'         => :suspended,  # not running, but infrastructure still possibly exists
+          'CreatingCG'    => :queued,     # creating control group
+          'Pending'       => :queued,     # in queue
+          'Submitted'     => :queued,     #
+          'Provisioning'  => :queued,     # node is being provisioned
+          'Running'       => :running,    #
+          'Completed'     => :completed,  #
+        }.freeze
+
+        def gcp_provider
+          'gcp'
+        end
+
+        def aws_provider
+          'aws'
+        end
+
+        def image_arg
+          if cloud == gcp_provider
+            '-gcpgi'
+          else
+            '-awsami'
+          end
+        end
+
+        def call(cmd, args: [], env: {}, stdin: "")
+          cmd = OodCore::Job::Adapters::Helper.bin_path(cmd, bin, bin_overrides)
+          args = args.map(&:to_s)
+          env = env.to_h
+          o, e, s = Open3.capture3(env, cmd, *args, stdin_data: stdin.to_s)
+          s.success? ? o : interpret_and_raise(e, cmd)
+        end
+
+        # helper function to interpret an error the command had given and
+        # raise a different error.
+        def interpret_and_raise(error, command)
+          # a special case with CCQ that prompts the user for username & password
+          # so let's be helpful and tell the user what to do.
+          if error.end_with?("EOFError: EOF when reading a line\n")
+            raise(
+              PromptError,
+              "The #{command} command was prompted. You need to generate the certificate " + 
+              "manually in a shell by running 'ccqstat'\nand entering your username/password"
+            )
+          else
+            raise(JobAdapterError, e.message)
+          end
+        end
+
+        # Convert seconds to duration
+        def seconds_to_duration(seconds)
+          format("%02d:%02d:%02d", seconds / 3600, seconds / 60 % 60, seconds % 60)
+        end
+
+        # helper to make a script file. We can't pipe it into ccq so we have to
+        # write a file.
+        def make_script_file(content)
+          file = Tempfile.new(tmp_file_name)
+          file.write(content.to_s)
+          file.flush
+          file
+        end
+
+        def tmp_file_name
+          'ccq_ood_script_'
+        end
+
+        def parse_job_id_from_ccqsub(output)
+          match_data = /#{jobid_regex}/.match(output)
+          # match_data could be nil, OR re-configured jobid_regex could be looking for a different named group
+          job_id = match_data&.named_captures&.fetch('job_id', nil)
+          throw JobAdapterError.new "Could not extract job id out of ccqsub output '#{output}'" if job_id.nil?
+          job_id
+        end
+
+        # parse an Ood::Job::Info object from extended ccqstat output
+        def info_from_ccqstat_extended(data)
+          raw = extended_data_to_hash(data)
+          data_hash = { native: raw }
+          data_hash[:status] = get_state(raw['status'])
+          data_hash[:id] = raw['name']
+          data_hash[:job_name] = raw['jobName']
+          data_hash[:job_owner] = raw['userName']
+          data_hash[:submit_host] = raw['submitHostInstanceId']
+          data_hash[:dispatch_time] = raw['startTime'].to_i
+          data_hash[:submission_time] = raw['dateSubmitted'].to_i
+          data_hash[:queue_name] = raw['criteriaPriority']
+
+          Info.new(data_hash)
+        end
+
+        # extended data is just lines of 'key: value' value, so parse
+        # it and stick it all in a hash.
+        def extended_data_to_hash(data)
+          Hash[data.to_s.scan(/(\w+): (\S+)/)]
+        end
+
+        def info_from_ccqstat(data)
+          infos = []
+
+          data.to_s.each_line do |line|
+            words = line.split(/\s/).reject(&:empty?)
+            next if !words.empty? && words[0] == "Id" # just skip the header
+
+            infos << Info.new(line_to_hash(words)) if words.size == 5
+          end
+
+          infos
+        end
+
+        def line_to_hash(words)
+          return unless words.size == 5
+
+          data_hash = {}
+          data_hash[:id] = words[0]
+          data_hash[:job_name] = words[1]
+          data_hash[:job_owner] = words[2]
+          data_hash[:status] = get_state(words[4])
+
+          data_hash
+        end
+
+        def get_state(state)
+          STATE_MAP.fetch(state, :undetermined)
+        end
+      end
+    end
+  end
+end

data/lib/ood_core/job/adapters/helper.rb

@@ -12,7 +12,26 @@ module OodCore
         def self.bin_path(cmd, bin_default, bin_overrides)
           bin_overrides.fetch(cmd.to_s) { Pathname.new(bin_default.to_s).join(cmd.to_s).to_s }
         end
+        
+        # Gets a command that submits command on another host via ssh
+        # @param submit_host [String] where to submit the command
+        # @param cmd [String] the desired command to execute on another host
+        # @param cmd_args [Array] arguments to the command specified above
+        # @param strict_host_checking [Bool] whether to use strict_host_checking
+        # @param env [Hash] env variables to be set w/ssh
+        #
+        # @return cmd [String] command wrapped in ssh if submit_host is present
+        # @return args [Array] command arguments including ssh_flags and original command
+        def self.ssh_wrap(submit_host, cmd, cmd_args, strict_host_checking = true, env = {})
+          return cmd, cmd_args if submit_host.to_s.empty?
+
+          check_host = strict_host_checking ? "yes" : "no"
+          args = ['-o', 'BatchMode=yes', '-o', 'UserKnownHostsFile=/dev/null', '-o', "StrictHostKeyChecking=#{check_host}", "#{submit_host}"]
+          env.each{|key, value| args.push("export #{key}=#{value};")}
+
+          return 'ssh', args + [cmd] + cmd_args
+        end
       end
     end
   end
-end
+end

data/lib/ood_core/job/adapters/kubernetes.rb

@@ -0,0 +1,193 @@
+require "ood_core/refinements/hash_extensions"
+require "ood_core/refinements/array_extensions"
+
+module OodCore
+  module Job
+    class Factory
+      using Refinements::HashExtensions
+
+      def self.build_kubernetes(config)
+        batch = Adapters::Kubernetes::Batch.new(config.to_h.symbolize_keys, Adapters::Kubernetes::Helper.new)
+        Adapters::Kubernetes.new(batch)
+      end
+    end
+
+    module Adapters
+      class Kubernetes < Adapter
+
+        using Refinements::ArrayExtensions
+        using Refinements::HashExtensions
+
+        require "ood_core/job/adapters/kubernetes/batch"
+
+        attr_reader :batch
+
+        def initialize(batch)
+          @batch = batch
+        end
+
+        # Submit a job with the attributes defined in the job template instance
+        # @abstract Subclass is expected to implement {#submit}
+        # @raise [NotImplementedError] if subclass did not define {#submit}
+        # @example Submit job template to cluster
+        #   solver_id = job_adapter.submit(solver_script)
+        #   #=> "1234.server"
+        # @example Submit job that depends on previous job
+        #   post_id = job_adapter.submit(
+        #     post_script,
+        #     afterok: solver_id
+        #   )
+        #   #=> "1235.server"
+        # @param script [Script] script object that describes the
+        #   script and attributes for the submitted job
+        # @param after [#to_s, Array<#to_s>] this job may be scheduled for execution
+        #   at any point after dependent jobs have started execution
+        # @param afterok [#to_s, Array<#to_s>] this job may be scheduled for
+        #   execution only after dependent jobs have terminated with no errors
+        # @param afternotok [#to_s, Array<#to_s>] this job may be scheduled for
+        #   execution only after dependent jobs have terminated with errors
+        # @param afterany [#to_s, Array<#to_s>] this job may be scheduled for
+        #   execution after dependent jobs have terminated
+        # @return [String] the job id returned after successfully submitting a job
+        def submit(script, after: [], afterok: [], afternotok: [], afterany: [])
+          raise ArgumentError, 'Must specify the script' if script.nil?
+
+          batch.submit(script)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+
+
+        # Retrieve info for all jobs from the resource manager
+        # @abstract Subclass is expected to implement {#info_all}
+        # @raise [NotImplementedError] if subclass did not define {#info_all}
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @return [Array<Info>] information describing submitted jobs
+        def info_all(attrs: nil)
+          batch.info_all(attrs: attrs)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+
+        # Retrieve info for all jobs for a given owner or owners from the
+        # resource manager
+        # @param owner [#to_s, Array<#to_s>] the owner(s) of the jobs
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @return [Array<Info>] information describing submitted jobs
+        def info_where_owner(owner, attrs: nil)
+          owner = Array.wrap(owner).map(&:to_s)
+
+          # must at least have job_owner to filter by job_owner
+          attrs = Array.wrap(attrs) | [:job_owner] unless attrs.nil?
+
+          info_all(attrs: attrs).select { |info| owner.include? info.job_owner }
+        end
+
+        # Iterate over each job Info object
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @yield [Info] of each job to block
+        # @return [Enumerator] if no block given
+        def info_all_each(attrs: nil)
+          return to_enum(:info_all_each, attrs: attrs) unless block_given?
+
+          info_all(attrs: attrs).each do |job|
+            yield job
+          end
+        end
+
+        # Iterate over each job Info object
+        # @param owner [#to_s, Array<#to_s>] the owner(s) of the jobs
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @yield [Info] of each job to block
+        # @return [Enumerator] if no block given
+        def info_where_owner_each(owner, attrs: nil)
+          return to_enum(:info_where_owner_each, owner, attrs: attrs) unless block_given?
+
+          info_where_owner(owner, attrs: attrs).each do |job|
+            yield job
+          end
+        end
+
+        # Whether the adapter supports job arrays
+        # @return [Boolean] - assumes true; but can be overridden by adapters that
+        #   explicitly do not
+        def supports_job_arrays?
+          false
+        end
+
+        # Retrieve job info from the resource manager
+        # @abstract Subclass is expected to implement {#info}
+        # @raise [NotImplementedError] if subclass did not define {#info}
+        # @param id [#to_s] the id of the job
+        # @return [Info] information describing submitted job
+        def info(id)
+          batch.info(id.to_s)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+
+        # Retrieve job status from resource manager
+        # @note Optimized slightly over retrieving complete job information from server
+        # @abstract Subclass is expected to implement {#status}
+        # @raise [NotImplementedError] if subclass did not define {#status}
+        # @param id [#to_s] the id of the job
+        # @return [Status] status of job
+        def status(id)
+          info(id).status
+        end
+
+        # Put the submitted job on hold
+        # @abstract Subclass is expected to implement {#hold}
+        # @raise [NotImplementedError] if subclass did not define {#hold}
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def hold(id)
+          raise NotImplementedError, 'subclass did not define #hold'
+        end
+
+        # Release the job that is on hold
+        # @abstract Subclass is expected to implement {#release}
+        # @raise [NotImplementedError] if subclass did not define {#release}
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def release(id)
+          raise NotImplementedError, 'subclass did not define #release'
+        end
+
+        # Delete the submitted job.
+        #
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def delete(id)
+          batch.delete(id.to_s)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+      end
+    end
+  end
+end