ood_core 0.12.0 → 0.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bb944d43beb0aced99e13efb2ef10bf33f9666c705c50ca5ae1727751de43073
-  data.tar.gz: 6e3cd66160be3bbd63124d6f2ddc794bce4ae64e385977828faba5dfd28ff838
+  metadata.gz: 392bbf30609185792c59bc9b83bad0d2d6861885db4264a4da39356cb64624b6
+  data.tar.gz: 76df75234a646147a8fed14367d6467ffabe289bfa8264ff02614b60f6463652
 SHA512:
-  metadata.gz: 176e331a856c1e6958c444426d5c1b41aa881e90a69dca507b07f5463eb81355689e8391e0bf27823fc42a9484789f623ffd566b9d6c414c9cf741a7cafd1def
-  data.tar.gz: 15481101ad3120d3e8457612f2b8a8be4f1e268b38538b18b710f27887836f7a47eac3bf2e89d8f73745ae96ae78e21cd5ba5afefb4161cf95f435d6f2fdf001
+  metadata.gz: fd127307d7048a220bdd5bed28302e90708bb0a32c9c6905136ddb3664b8f0c5035510c528f255161c4d40c98a772d6200fa60c938aff10ea4a157fdf56d8d1f
+  data.tar.gz: f3c1ea0622da5b4387ee123843da2e0e955afd40d28e0f25e425fea010865896cdebf87171e2264ffbffbd743be8d1533971a25566251dcb8da5748191dcbeea

data/.github/workflows/test.yml

@@ -0,0 +1,30 @@
+name: Unit Tests
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: checkout
+        uses: actions/checkout@v2
+
+      - name: Setup Ruby using Bundler
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: "2.7.1"
+          bundler-cache: true
+          bundler: "2.1.4"
+
+      - name: install gems
+        run: bundle install
+
+      - name: test
+        run: bundle exec rake spec

data/CHANGELOG.md

@@ -6,6 +6,63 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+## [0.16.0] - 2021-04-20
+### Fixed
+- tmux 2.7+ bug in the linux host adapter in [2.5.8](https://github.com/OSC/ood_core/pull/258)
+  and [259](https://github.com/OSC/ood_core/pull/259).
+
+### Changed
+
+- Changed how k8s configmaps in are defined in [251](https://github.com/OSC/ood_core/pull/251).
+  The data structure now expects a key called files which is an array of objects that hold
+  filename, data, mount_path, sub_path and init_mount_path.
+  [255](https://github.com/OSC/ood_core/pull/255) also relates to this interface change.
+
+### Added
+
+- The k8s adapter can now specify environment variables and creates defaults 
+  in [252](https://github.com/OSC/ood_core/pull/252).
+- The k8s adapter can now specify image pull secrets in [253](https://github.com/OSC/ood_core/pull/253).
+
+## [0.15.1] - 2021-02-25
+### Fixed
+- kubernetes adapter uses the full module for helpers in [245](https://github.com/OSC/ood_core/pull/245).
+
+### Changed
+- kubernetes pods spawn with runAsNonRoot set to true in [247](https://github.com/OSC/ood_core/pull/247).
+- kubernetes pods can spawn with supplemental groups along with some other in security defaults in
+  [246](https://github.com/OSC/ood_core/pull/246).
+
+## [0.15.0] - 2021-01-26
+### Fixed
+- ccq adapter now accepts job names with spaces in [210](https://github.com/OSC/ood_core/pull/209)
+- k8s correctly handles having no mount volumes in [239](https://github.com/OSC/ood_core/pull/239)
+
+### Added
+- k8s adapter now applies account metadata to resources in [216](https://github.com/OSC/ood_core/pull/216) and
+  [231](https://github.com/OSC/ood_core/pull/231)
+- k8s adapter can now prefix namespaces in [218](https://github.com/OSC/ood_core/pull/218)
+- k8s adapter now applies time limits to pods in [224](https://github.com/OSC/ood_core/pull/224)
+
+### Changed
+- testing automation is now done in github actions in [221](https://github.com/OSC/ood_core/pull/218)
+- update bunlder to 2.1.4 and ruby to 2.7 in [235](https://github.com/OSC/ood_core/pull/218) updated bundler and ruby
+- k8s adapter more appropriately labels unschedulable pods as queued in [230](https://github.com/OSC/ood_core/pull/230)
+- k8s adapter now uses the script#ood_connection_info API instead of script#native in
+  [222](https://github.com/OSC/ood_core/pull/222)
+
+## [0.14.0] - 2020-10-01
+### Added 
+- Kubernetes adapter in PR [156](https://github.com/OSC/ood_core/pull/156)
+
+### Fixed
+- Catch Slurm times. [209](https://github.com/OSC/ood_core/pull/209)
+- LHA race condition in deleteing tmp files. [212](https://github.com/OSC/ood_core/pull/212)
+
+## [0.13.0] - 2020-08-10
+### Added
+- CloudyCluster CCQ Adapter
+
 ## [0.12.0] - 2020-08-05
 ### Added
 - qos option to Slurm and Torque [#205](https://github.com/OSC/ood_core/pull/205)
@@ -243,7 +300,12 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 - Initial release!
 
-[Unreleased]: https://github.com/OSC/ood_core/compare/v0.12.0...HEAD
+[Unreleased]: https://github.com/OSC/ood_core/compare/v0.16.0...HEAD
+[0.15.2]: https://github.com/OSC/ood_core/compare/v0.15.1...v0.16.0
+[0.15.1]: https://github.com/OSC/ood_core/compare/v0.15.0...v0.15.1
+[0.15.0]: https://github.com/OSC/ood_core/compare/v0.14.0...v0.15.0
+[0.14.0]: https://github.com/OSC/ood_core/compare/v0.13.0...v0.14.0
+[0.13.0]: https://github.com/OSC/ood_core/compare/v0.12.0...v0.13.0
 [0.12.0]: https://github.com/OSC/ood_core/compare/v0.11.4...v0.12.0
 [0.11.4]: https://github.com/OSC/ood_core/compare/v0.11.3...v0.11.4
 [0.11.3]: https://github.com/OSC/ood_core/compare/v0.11.2...v0.11.3

data/README.md

@@ -1,12 +1,12 @@
 # OodCore
 
-[![Build Status](https://travis-ci.org/OSC/ood_core.svg?branch=master)](https://travis-ci.org/OSC/ood_core)
+[![Build Status](https://github.com/osc/ood_core/workflows/Unit%20Tests/badge.svg)](https://github.com/OSC/ood_core/actions?query=workflow%3A%22Unit+Tests%22)
 ![GitHub Release](https://img.shields.io/github/release/osc/ood_core.svg)
 ![GitHub License](https://img.shields.io/github/license/osc/ood_core.svg)
 
 - Website: http://openondemand.org/
 - Website repo with JOSS publication: https://github.com/OSC/Open-OnDemand
-- Documentation: https://osc.github.io/ood-documentation/master/
+- Documentation: https://osc.github.io/ood-documentation/latest/
 - Main code repo: https://github.com/OSC/ondemand
 - Core library repo: https://github.com/OSC/ood_core
 

data/lib/ood_core/job/adapters/ccq.rb

@@ -0,0 +1,274 @@
+require "ood_core/job/adapters/helper"
+require "tempfile"
+
+module OodCore
+  module Job
+    class Factory
+      using Refinements::HashExtensions
+
+      # Build the Cloudy Cluster adapter from a configuration
+      # @param config [#to_h] the configuration for job adapter
+      # @option config [Object] :image (nil) The default VM image to use
+      # @option config [Object] :cloud (gcp) The cloud provider being used [gcp,aws]
+      # @option config [Object] :scheduler (nil) The name of the scheduler to use
+      # @option config [Object] :sge_root (nil) Path to SGE root, note that
+      # @option config [#to_h] :bin (nil) Path to CC client binaries
+      # @option config [#to_h] :bin_overrides ({}) Optional overrides to CC client executables
+      def self.build_ccq(config)
+        Adapters::CCQ.new(config.to_h.symbolize_keys)
+      end
+    end
+
+    module Adapters
+
+      class PromptError < StandardError; end
+
+      class CCQ < Adapter
+        using Refinements::ArrayExtensions
+
+        attr_reader :image, :cloud, :scheduler, :bin, :bin_overrides, :jobid_regex
+
+        def initialize(config)
+          @image = config.fetch(:image, nil)
+          @cloud = config.fetch(:cloud, gcp_provider)
+          @scheduler = config.fetch(:scheduler, nil)
+          @bin = config.fetch(:bin, '/opt/CloudyCluster/srv/CCQ')
+          @bin_overrides = config.fetch(:bin_overrides, {})
+          @jobid_regex = config.fetch(:jobid_regex, "job id is: (?<job_id>\\d+) you")
+        end
+
+        # Submit a job with the attributes defined in the job template instance
+        # @param script [Script] script object that describes the script and
+        #   attributes for the submitted job
+        # @param after [#to_s, Array<#to_s>] not used
+        # @param afterok [#to_s, Array<#to_s>] not used
+        # @param afternotok [#to_s, Array<#to_s>] not used
+        # @param afterany [#to_s, Array<#to_s>] not used
+        # @return [String] the job id returned after successfully submitting a
+        #   job
+        # @see Adapter#submit
+        def submit(script, after: [], afterok: [], afternotok: [], afterany: [])
+          script_file = make_script_file(script.content)
+          args = []
+
+          # cluster configuration args
+          args.concat ["-s", scheduler] unless scheduler.nil?
+          args.concat [image_arg, image] unless image.nil?
+
+          args.concat ["-o", script.output_path.to_s] unless script.output_path.nil?
+          args.concat ["-e", script.error_path.to_s] unless script.error_path.nil?
+          args.concat ["-tl", seconds_to_duration(script.wall_time)] unless script.wall_time.nil?
+          args.concat ["-js", script_file.path.to_s]
+
+          args.concat script.native if script.native
+
+          output = call("ccqsub", args: args)
+          parse_job_id_from_ccqsub(output)
+        ensure
+          script_file.close
+        end
+
+        # Retrieve info for all jobs from the resource manager
+        # @return [Array<Info>] information describing submitted jobs
+        def info_all(attrs: nil)
+          args = []
+          args.concat ["-s", scheduler] unless scheduler.nil?
+
+          stat_output = call("ccqstat", args: args)
+          info_from_ccqstat(stat_output)
+        end
+
+        # Retrieve job info from the resource manager
+        # @param id [#to_s] the id of the job
+        # @return [Info] information describing submitted job
+        def info(id)
+          args = []
+          args.concat ["-s", scheduler] unless scheduler.nil?
+          args.concat ["-ji", id]
+
+          stat_output = call("ccqstat", args: args)
+
+          # WARNING: code path differs here than info_all because the output
+          # from ccqstat -ji $JOBID is much more data than just the 4
+          # columns that ccqstat gives.
+          info_from_ccqstat_extended(stat_output)
+        end
+
+        # Retrieve job status from resource manager
+        # @param id [#to_s] the id of the job
+        # @return [Status] status of job
+        # @see Adapter#status
+        def status(id)
+          info(id).status
+        end
+
+        # This adapter does not implement hold and will always raise
+        #   an exception.
+        # @param id [#to_s] the id of the job
+        # @raise [JobAdapterError] always
+        # @return [void]
+        def hold(_)
+          raise NotImplementedError, "subclass did not define #hold"
+        end
+
+        # This adapter does not implement release and will always raise
+        #   an exception.
+        # @param id [#to_s] the id of the job
+        # @raise [JobAdapterError] always
+        # @return [void]
+        def release(_)
+          raise NotImplementedError, "subclass did not define #release"
+        end
+
+        # Delete the submitted job
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def delete(id)
+          call("ccqdel", args: [id])
+        end
+
+        def directive_prefix
+          '#CC'
+        end
+
+        private
+
+        # Mapping of state codes
+        STATE_MAP =
+        {
+          'Error'         => :suspended,  # not running, but infrastructure still possibly exists
+          'CreatingCG'    => :queued,     # creating control group
+          'Pending'       => :queued,     # in queue
+          'Submitted'     => :queued,     #
+          'Provisioning'  => :queued,     # node is being provisioned
+          'Running'       => :running,    #
+          'Completed'     => :completed,  #
+        }.freeze
+
+        def gcp_provider
+          'gcp'
+        end
+
+        def aws_provider
+          'aws'
+        end
+
+        def image_arg
+          if cloud == gcp_provider
+            '-gcpgi'
+          else
+            '-awsami'
+          end
+        end
+
+        def call(cmd, args: [], env: {}, stdin: "")
+          cmd = OodCore::Job::Adapters::Helper.bin_path(cmd, bin, bin_overrides)
+          args = args.map(&:to_s)
+          env = env.to_h
+          o, e, s = Open3.capture3(env, cmd, *args, stdin_data: stdin.to_s)
+          s.success? ? o : interpret_and_raise(e, cmd)
+        end
+
+        # helper function to interpret an error the command had given and
+        # raise a different error.
+        def interpret_and_raise(error, command)
+          # a special case with CCQ that prompts the user for username & password
+          # so let's be helpful and tell the user what to do.
+          if error.end_with?("EOFError: EOF when reading a line\n")
+            raise(
+              PromptError,
+              "The #{command} command was prompted. You need to generate the certificate " + 
+              "manually in a shell by running 'ccqstat'\nand entering your username/password"
+            )
+          else
+            raise(JobAdapterError, e.message)
+          end
+        end
+
+        # Convert seconds to duration
+        def seconds_to_duration(seconds)
+          format("%02d:%02d:%02d", seconds / 3600, seconds / 60 % 60, seconds % 60)
+        end
+
+        # helper to make a script file. We can't pipe it into ccq so we have to
+        # write a file.
+        def make_script_file(content)
+          file = Tempfile.new(tmp_file_name)
+          file.write(content.to_s)
+          file.flush
+          file
+        end
+
+        def tmp_file_name
+          'ccq_ood_script_'
+        end
+
+        def ccqstat_regex
+          /^(?<id>\S+)\s+(?<name>.+)\s+(?<username>\S+)\s+(?<scheduler>\S+)\s+(?<status>\S+)\s*$/
+        end
+
+        def parse_job_id_from_ccqsub(output)
+          match_data = /#{jobid_regex}/.match(output)
+          # match_data could be nil, OR re-configured jobid_regex could be looking for a different named group
+          job_id = match_data&.named_captures&.fetch('job_id', nil)
+          throw JobAdapterError.new "Could not extract job id out of ccqsub output '#{output}'" if job_id.nil?
+          job_id
+        end
+
+        # parse an Ood::Job::Info object from extended ccqstat output
+        def info_from_ccqstat_extended(data)
+          raw = extended_data_to_hash(data)
+          data_hash = { native: raw }
+          data_hash[:status] = get_state(raw['status'])
+          data_hash[:id] = raw['name']
+          data_hash[:job_name] = raw['jobName']
+          data_hash[:job_owner] = raw['userName']
+          data_hash[:submit_host] = raw['submitHostInstanceId']
+          data_hash[:dispatch_time] = raw['startTime'].to_i
+          data_hash[:submission_time] = raw['dateSubmitted'].to_i
+          data_hash[:queue_name] = raw['criteriaPriority']
+
+          Info.new(data_hash)
+        end
+
+        # extended data is just lines of 'key: value' value, so parse
+        # it and stick it all in a hash.
+        def extended_data_to_hash(data)
+          Hash[data.to_s.scan(/(\w+): (\S+)/)]
+        end
+
+        def info_from_ccqstat(data)
+          infos = []
+
+          data.to_s.lines.drop(1).each do |line|
+            match_data = ccqstat_regex.match(line)
+            infos << Info.new(ccqstat_match_to_hash(match_data)) if valid_ccqstat_match?(match_data)
+          end
+
+          infos
+        end
+
+        def ccqstat_match_to_hash(match)
+          data_hash = {}
+          data_hash[:id] = match.named_captures.fetch('id', nil)
+          data_hash[:job_owner] = match.named_captures.fetch('username', nil)
+          data_hash[:status] = get_state(match.named_captures.fetch('status', nil))
+
+          # The regex leaves trailing empty spaces. There's no way to tell if they're _actually_
+          # a part of the job name or not, so we assume they're not and add the rstrip.
+          data_hash[:job_name] = match.named_captures.fetch('name', nil).to_s.rstrip
+
+          data_hash
+        end
+
+        def valid_ccqstat_match?(match)
+          !match.nil? && !match.named_captures.fetch('id', nil).nil?
+        end
+
+        def get_state(state)
+          STATE_MAP.fetch(state, :undetermined)
+        end
+      end
+    end
+  end
+end

data/lib/ood_core/job/adapters/kubernetes.rb

@@ -0,0 +1,193 @@
+require "ood_core/refinements/hash_extensions"
+require "ood_core/refinements/array_extensions"
+
+module OodCore
+  module Job
+    class Factory
+      using Refinements::HashExtensions
+
+      def self.build_kubernetes(config)
+        batch = Adapters::Kubernetes::Batch.new(config.to_h.symbolize_keys)
+        Adapters::Kubernetes.new(batch)
+      end
+    end
+
+    module Adapters
+      class Kubernetes < Adapter
+
+        using Refinements::ArrayExtensions
+        using Refinements::HashExtensions
+
+        require "ood_core/job/adapters/kubernetes/batch"
+
+        attr_reader :batch
+
+        def initialize(batch)
+          @batch = batch
+        end
+
+        # Submit a job with the attributes defined in the job template instance
+        # @abstract Subclass is expected to implement {#submit}
+        # @raise [NotImplementedError] if subclass did not define {#submit}
+        # @example Submit job template to cluster
+        #   solver_id = job_adapter.submit(solver_script)
+        #   #=> "1234.server"
+        # @example Submit job that depends on previous job
+        #   post_id = job_adapter.submit(
+        #     post_script,
+        #     afterok: solver_id
+        #   )
+        #   #=> "1235.server"
+        # @param script [Script] script object that describes the
+        #   script and attributes for the submitted job
+        # @param after [#to_s, Array<#to_s>] this job may be scheduled for execution
+        #   at any point after dependent jobs have started execution
+        # @param afterok [#to_s, Array<#to_s>] this job may be scheduled for
+        #   execution only after dependent jobs have terminated with no errors
+        # @param afternotok [#to_s, Array<#to_s>] this job may be scheduled for
+        #   execution only after dependent jobs have terminated with errors
+        # @param afterany [#to_s, Array<#to_s>] this job may be scheduled for
+        #   execution after dependent jobs have terminated
+        # @return [String] the job id returned after successfully submitting a job
+        def submit(script, after: [], afterok: [], afternotok: [], afterany: [])
+          raise ArgumentError, 'Must specify the script' if script.nil?
+
+          batch.submit(script)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+
+
+        # Retrieve info for all jobs from the resource manager
+        # @abstract Subclass is expected to implement {#info_all}
+        # @raise [NotImplementedError] if subclass did not define {#info_all}
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @return [Array<Info>] information describing submitted jobs
+        def info_all(attrs: nil)
+          batch.info_all(attrs: attrs)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+
+        # Retrieve info for all jobs for a given owner or owners from the
+        # resource manager
+        # @param owner [#to_s, Array<#to_s>] the owner(s) of the jobs
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @return [Array<Info>] information describing submitted jobs
+        def info_where_owner(owner, attrs: nil)
+          owner = Array.wrap(owner).map(&:to_s)
+
+          # must at least have job_owner to filter by job_owner
+          attrs = Array.wrap(attrs) | [:job_owner] unless attrs.nil?
+
+          info_all(attrs: attrs).select { |info| owner.include? info.job_owner }
+        end
+
+        # Iterate over each job Info object
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @yield [Info] of each job to block
+        # @return [Enumerator] if no block given
+        def info_all_each(attrs: nil)
+          return to_enum(:info_all_each, attrs: attrs) unless block_given?
+
+          info_all(attrs: attrs).each do |job|
+            yield job
+          end
+        end
+
+        # Iterate over each job Info object
+        # @param owner [#to_s, Array<#to_s>] the owner(s) of the jobs
+        # @param attrs [Array<symbol>] defaults to nil (and all attrs are provided) 
+        #   This array specifies only attrs you want, in addition to id and status.
+        #   If an array, the Info object that is returned to you is not guarenteed
+        #   to have a value for any attr besides the ones specified and id and status.
+        #
+        #   For certain adapters this may speed up the response since
+        #   adapters can get by without populating the entire Info object
+        # @yield [Info] of each job to block
+        # @return [Enumerator] if no block given
+        def info_where_owner_each(owner, attrs: nil)
+          return to_enum(:info_where_owner_each, owner, attrs: attrs) unless block_given?
+
+          info_where_owner(owner, attrs: attrs).each do |job|
+            yield job
+          end
+        end
+
+        # Whether the adapter supports job arrays
+        # @return [Boolean] - assumes true; but can be overridden by adapters that
+        #   explicitly do not
+        def supports_job_arrays?
+          false
+        end
+
+        # Retrieve job info from the resource manager
+        # @abstract Subclass is expected to implement {#info}
+        # @raise [NotImplementedError] if subclass did not define {#info}
+        # @param id [#to_s] the id of the job
+        # @return [Info] information describing submitted job
+        def info(id)
+          batch.info(id.to_s)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+
+        # Retrieve job status from resource manager
+        # @note Optimized slightly over retrieving complete job information from server
+        # @abstract Subclass is expected to implement {#status}
+        # @raise [NotImplementedError] if subclass did not define {#status}
+        # @param id [#to_s] the id of the job
+        # @return [Status] status of job
+        def status(id)
+          info(id).status
+        end
+
+        # Put the submitted job on hold
+        # @abstract Subclass is expected to implement {#hold}
+        # @raise [NotImplementedError] if subclass did not define {#hold}
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def hold(id)
+          raise NotImplementedError, 'subclass did not define #hold'
+        end
+
+        # Release the job that is on hold
+        # @abstract Subclass is expected to implement {#release}
+        # @raise [NotImplementedError] if subclass did not define {#release}
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def release(id)
+          raise NotImplementedError, 'subclass did not define #release'
+        end
+
+        # Delete the submitted job.
+        #
+        # @param id [#to_s] the id of the job
+        # @return [void]
+        def delete(id)
+          batch.delete(id.to_s)
+        rescue Batch::Error => e
+          raise JobAdapterError, e.message
+        end
+      end
+    end
+  end
+end