ood_core 0.17.4 → 0.18.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -1
- data/RAILS-LICENSE +20 -0
- data/lib/ood_core/cluster.rb +14 -0
- data/lib/ood_core/job/adapters/kubernetes/batch.rb +60 -56
- data/lib/ood_core/job/adapters/kubernetes/helper.rb +8 -3
- data/lib/ood_core/job/adapters/kubernetes/resources.rb +17 -9
- data/lib/ood_core/job/adapters/kubernetes/templates/pod.yml.erb +7 -4
- data/lib/ood_core/job/adapters/linux_host/launcher.rb +15 -11
- data/lib/ood_core/job/adapters/linux_host/templates/script_wrapper.erb.sh +1 -0
- data/lib/ood_core/refinements/hash_extensions.rb +33 -0
- data/lib/ood_core/version.rb +1 -1
- data/ood_core.gemspec +0 -1
- metadata +3 -22
    
        checksums.yaml
    CHANGED
    
    | @@ -1,7 +1,7 @@ | |
| 1 1 | 
             
            ---
         | 
| 2 2 | 
             
            SHA256:
         | 
| 3 | 
            -
              metadata.gz:  | 
| 4 | 
            -
              data.tar.gz:  | 
| 3 | 
            +
              metadata.gz: a110306c66de3f349e7a5569cedc3ea02dfedd5dfaa360b352f8689af113f98b
         | 
| 4 | 
            +
              data.tar.gz: ae608343e63bb98e6383fea71af53943d7136bdf72d8bf46b653f0c19801fcec
         | 
| 5 5 | 
             
            SHA512:
         | 
| 6 | 
            -
              metadata.gz:  | 
| 7 | 
            -
              data.tar.gz:  | 
| 6 | 
            +
              metadata.gz: 8a6b9928561a6dba1b84cbb2ac58d389b84e8317589648c483382c166c81982859fb74f68f76297a25319faed06712c6256abdf1c6a5e0175be939aa0392f283
         | 
| 7 | 
            +
              data.tar.gz: 21396c77e39329f9d7b6112c7900dd7ffa51d695b137d15089c487799ed16e3f74aea1f1dfab9958e2928fb98f49db098f865906c53abf667a8ed64ceda5dc53
         | 
    
        data/CHANGELOG.md
    CHANGED
    
    | @@ -7,6 +7,43 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. | |
| 7 7 |  | 
| 8 8 | 
             
            ## [Unreleased]
         | 
| 9 9 |  | 
| 10 | 
            +
            ## [0.18.1] - 10-18-2021
         | 
| 11 | 
            +
             | 
| 12 | 
            +
            ### Fixed
         | 
| 13 | 
            +
             | 
| 14 | 
            +
            - Fixed kubernetes initialization in [331](https://github.com/OSC/ood_core/pull/331).
         | 
| 15 | 
            +
             | 
| 16 | 
            +
            ## [0.18.0] - 10-18-2021
         | 
| 17 | 
            +
             | 
| 18 | 
            +
            ### Fixed
         | 
| 19 | 
            +
             | 
| 20 | 
            +
            - Fixed LHA crashing on strange bash output in [322](https://github.com/OSC/ood_core/pull/322).
         | 
| 21 | 
            +
             | 
| 22 | 
            +
            ### Added
         | 
| 23 | 
            +
             | 
| 24 | 
            +
            - All adapters now respond to #{adapter}? methods like slurm?, pbspro?, kubernetes? and so on
         | 
| 25 | 
            +
              in [326](https://github.com/OSC/ood_core/pull/326).
         | 
| 26 | 
            +
             | 
| 27 | 
            +
            ### Changed
         | 
| 28 | 
            +
             | 
| 29 | 
            +
            - The kubernetes adapter now expects to set context statically in [324](https://github.com/OSC/ood_core/pull/324).
         | 
| 30 | 
            +
              And can now accept context as a part of it's interface. It will now also always send --context when using OIDC
         | 
| 31 | 
            +
              and that context defaults to the clustername in [327](https://github.com/OSC/ood_core/pull/327).
         | 
| 32 | 
            +
            - Removed the activesupport dependency in [329](https://github.com/OSC/ood_core/pull/329).
         | 
| 33 | 
            +
             | 
| 34 | 
            +
            ## [0.17.6] - 8-24-2021
         | 
| 35 | 
            +
             | 
| 36 | 
            +
            ### Added
         | 
| 37 | 
            +
             | 
| 38 | 
            +
            - kubernetes now allows for arbitrary labels to be set in [317](https://github.com/OSC/ood_core/pull/317).
         | 
| 39 | 
            +
            - kubernetes now allows for limits and requests to be different in [318](https://github.com/OSC/ood_core/pull/318).
         | 
| 40 | 
            +
             | 
| 41 | 
            +
            ## [0.17.5] - 8-20-2021
         | 
| 42 | 
            +
             | 
| 43 | 
            +
            ### Fixed
         | 
| 44 | 
            +
             | 
| 45 | 
            +
            - kubernetes jobs delete without waiting in [314](https://github.com/OSC/ood_core/pull/314).
         | 
| 46 | 
            +
             | 
| 10 47 | 
             
            ## [0.17.4] - 7-29-2021
         | 
| 11 48 |  | 
| 12 49 | 
             
            Functionally the same as [0.17.3] but with some CI updates.
         | 
| @@ -374,7 +411,11 @@ Functionally the same as [0.17.3] but with some CI updates. | |
| 374 411 | 
             
            ### Added
         | 
| 375 412 | 
             
            - Initial release!
         | 
| 376 413 |  | 
| 377 | 
            -
            [Unreleased]: https://github.com/OSC/ood_core/compare/v0. | 
| 414 | 
            +
            [Unreleased]: https://github.com/OSC/ood_core/compare/v0.18.1...HEAD
         | 
| 415 | 
            +
            [0.18.1]: https://github.com/OSC/ood_core/compare/v0.18.0...v0.18.1
         | 
| 416 | 
            +
            [0.18.0]: https://github.com/OSC/ood_core/compare/v0.17.8...v0.18.0
         | 
| 417 | 
            +
            [0.17.6]: https://github.com/OSC/ood_core/compare/v0.17.5...v0.17.6
         | 
| 418 | 
            +
            [0.17.5]: https://github.com/OSC/ood_core/compare/v0.17.4...v0.17.5
         | 
| 378 419 | 
             
            [0.17.4]: https://github.com/OSC/ood_core/compare/v0.17.3...v0.17.4
         | 
| 379 420 | 
             
            [0.17.3]: https://github.com/OSC/ood_core/compare/v0.17.2...v0.17.3
         | 
| 380 421 | 
             
            [0.17.2]: https://github.com/OSC/ood_core/compare/v0.17.1...v0.17.2
         | 
    
        data/RAILS-LICENSE
    ADDED
    
    | @@ -0,0 +1,20 @@ | |
| 1 | 
            +
            Copyright (c) 2005-2021 David Heinemeier Hansson
         | 
| 2 | 
            +
             | 
| 3 | 
            +
            Permission is hereby granted, free of charge, to any person obtaining
         | 
| 4 | 
            +
            a copy of this software and associated documentation files (the
         | 
| 5 | 
            +
            "Software"), to deal in the Software without restriction, including
         | 
| 6 | 
            +
            without limitation the rights to use, copy, modify, merge, publish,
         | 
| 7 | 
            +
            distribute, sublicense, and/or sell copies of the Software, and to
         | 
| 8 | 
            +
            permit persons to whom the Software is furnished to do so, subject to
         | 
| 9 | 
            +
            the following conditions:
         | 
| 10 | 
            +
             | 
| 11 | 
            +
            The above copyright notice and this permission notice shall be
         | 
| 12 | 
            +
            included in all copies or substantial portions of the Software.
         | 
| 13 | 
            +
             | 
| 14 | 
            +
            THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
         | 
| 15 | 
            +
            EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
         | 
| 16 | 
            +
            MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
         | 
| 17 | 
            +
            NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
         | 
| 18 | 
            +
            LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
         | 
| 19 | 
            +
            OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
         | 
| 20 | 
            +
            WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
         | 
    
        data/lib/ood_core/cluster.rb
    CHANGED
    
    | @@ -63,6 +63,20 @@ module OodCore | |
| 63 63 | 
             
                  @errors          = c.fetch(:errors, [])  .to_a
         | 
| 64 64 | 
             
                end
         | 
| 65 65 |  | 
| 66 | 
            +
             | 
| 67 | 
            +
                # programatically define some methods like slurm? or torque?
         | 
| 68 | 
            +
                Dir.entries("#{__dir__}/job/adapters").select do |f|
         | 
| 69 | 
            +
                  File.file?("#{__dir__}/job/adapters/#{f}") && File.extname(f) == '.rb'
         | 
| 70 | 
            +
                end.map do |f|
         | 
| 71 | 
            +
                  File.basename(f, '.rb')
         | 
| 72 | 
            +
                end.reject do |f|
         | 
| 73 | 
            +
                  ['helper', 'drmaa' ].include?(f)
         | 
| 74 | 
            +
                end.each do |adapter|
         | 
| 75 | 
            +
                  define_method(:"#{adapter}?")  do
         | 
| 76 | 
            +
                    job_config.fetch(:adapter, nil) == adapter
         | 
| 77 | 
            +
                  end
         | 
| 78 | 
            +
                end
         | 
| 79 | 
            +
             | 
| 66 80 | 
             
                # Metadata that provides extra information about this cluster
         | 
| 67 81 | 
             
                # @return [OpenStruct] the metadata
         | 
| 68 82 | 
             
                def metadata
         | 
| @@ -11,15 +11,15 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 11 11 | 
             
              class Error < StandardError; end
         | 
| 12 12 | 
             
              class NotFoundError < StandardError; end
         | 
| 13 13 |  | 
| 14 | 
            -
              attr_reader :config_file, :bin, :cluster, :mounts
         | 
| 15 | 
            -
              attr_reader :all_namespaces, : | 
| 14 | 
            +
              attr_reader :config_file, :bin, :cluster, :context, :mounts
         | 
| 15 | 
            +
              attr_reader :all_namespaces, :helper
         | 
| 16 16 | 
             
              attr_reader :username_prefix, :namespace_prefix
         | 
| 17 17 | 
             
              attr_reader :auto_supplemental_groups
         | 
| 18 18 |  | 
| 19 19 | 
             
              def initialize(options = {})
         | 
| 20 20 | 
             
                options = options.to_h.symbolize_keys
         | 
| 21 21 |  | 
| 22 | 
            -
                @config_file = options.fetch(:config_file, default_config_file)
         | 
| 22 | 
            +
                @config_file = options.fetch(:config_file, self.class.default_config_file)
         | 
| 23 23 | 
             
                @bin = options.fetch(:bin, '/usr/bin/kubectl')
         | 
| 24 24 | 
             
                @cluster = options.fetch(:cluster, 'open-ondemand')
         | 
| 25 25 | 
             
                @mounts = options.fetch(:mounts, []).map { |m| m.to_h.symbolize_keys }
         | 
| @@ -28,15 +28,10 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 28 28 | 
             
                @namespace_prefix = options.fetch(:namespace_prefix, '')
         | 
| 29 29 | 
             
                @auto_supplemental_groups = options.fetch(:auto_supplemental_groups, false)
         | 
| 30 30 |  | 
| 31 | 
            -
                 | 
| 32 | 
            -
                @ | 
| 31 | 
            +
                tmp_ctx = options.fetch(:context, nil)
         | 
| 32 | 
            +
                @context = tmp_ctx.nil? && oidc_auth?(options.fetch(:auth, {}).symbolize_keys) ? @cluster : tmp_ctx
         | 
| 33 33 |  | 
| 34 | 
            -
                 | 
| 35 | 
            -
                  make_kubectl_config(options)
         | 
| 36 | 
            -
                rescue
         | 
| 37 | 
            -
                  # FIXME could use a log here
         | 
| 38 | 
            -
                  # means you couldn't 'kubectl set config'
         | 
| 39 | 
            -
                end
         | 
| 34 | 
            +
                @helper = OodCore::Job::Adapters::Kubernetes::Helper.new
         | 
| 40 35 | 
             
              end
         | 
| 41 36 |  | 
| 42 37 | 
             
              def resource_file(resource_type = 'pod')
         | 
| @@ -61,10 +56,10 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 61 56 | 
             
              end
         | 
| 62 57 |  | 
| 63 58 | 
             
              def info_all(attrs: nil)
         | 
| 64 | 
            -
                cmd = if all_namespaces
         | 
| 65 | 
            -
                        "#{base_cmd}  | 
| 59 | 
            +
                cmd = if @all_namespaces
         | 
| 60 | 
            +
                        "#{base_cmd} -o json get pods --all-namespaces"
         | 
| 66 61 | 
             
                      else
         | 
| 67 | 
            -
                        "#{namespaced_cmd}  | 
| 62 | 
            +
                        "#{namespaced_cmd} -o json get pods"
         | 
| 68 63 | 
             
                      end
         | 
| 69 64 |  | 
| 70 65 | 
             
                output = call(cmd)
         | 
| @@ -117,6 +112,32 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 117 112 | 
             
                safe_call("delete", "configmap", configmap_name(id))
         | 
| 118 113 | 
             
              end
         | 
| 119 114 |  | 
| 115 | 
            +
              class << self
         | 
| 116 | 
            +
                def default_config_file
         | 
| 117 | 
            +
                  (ENV['KUBECONFIG'] || "#{Dir.home}/.kube/config")
         | 
| 118 | 
            +
                end
         | 
| 119 | 
            +
             | 
| 120 | 
            +
                def default_auth
         | 
| 121 | 
            +
                  {
         | 
| 122 | 
            +
                    type: 'managed'
         | 
| 123 | 
            +
                  }.symbolize_keys
         | 
| 124 | 
            +
                end
         | 
| 125 | 
            +
             | 
| 126 | 
            +
                def default_server
         | 
| 127 | 
            +
                  {
         | 
| 128 | 
            +
                    endpoint: 'https://localhost:8080',
         | 
| 129 | 
            +
                    cert_authority_file: nil
         | 
| 130 | 
            +
                  }.symbolize_keys
         | 
| 131 | 
            +
                end
         | 
| 132 | 
            +
             | 
| 133 | 
            +
                def configure_kube!(config)
         | 
| 134 | 
            +
                  k = self.new(config)
         | 
| 135 | 
            +
                  # TODO: probably shouldn't be using send here
         | 
| 136 | 
            +
                  k.send(:set_cluster, config.fetch(:server, default_server).to_h.symbolize_keys)
         | 
| 137 | 
            +
                  k.send(:configure_auth, config.fetch(:auth, default_auth).to_h.symbolize_keys)
         | 
| 138 | 
            +
                end
         | 
| 139 | 
            +
              end
         | 
| 140 | 
            +
             | 
| 120 141 | 
             
              private
         | 
| 121 142 |  | 
| 122 143 | 
             
              def safe_call(verb, resource, id)
         | 
| @@ -125,7 +146,7 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 125 146 | 
             
                  when "get"
         | 
| 126 147 | 
             
                    call_json_output('get', resource, id)
         | 
| 127 148 | 
             
                  when "delete"
         | 
| 128 | 
            -
                    call("#{namespaced_cmd} delete #{resource} #{id}")
         | 
| 149 | 
            +
                    call("#{namespaced_cmd} delete #{resource} #{id} --wait=false")
         | 
| 129 150 | 
             
                  end
         | 
| 130 151 | 
             
                rescue NotFoundError
         | 
| 131 152 | 
             
                  {}
         | 
| @@ -227,6 +248,7 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 227 248 | 
             
              # and  id=my-pod-id
         | 
| 228 249 | 
             
              def call_json_output(verb, resource, id, stdin: nil)
         | 
| 229 250 | 
             
                cmd = "#{formatted_ns_cmd} #{verb} #{resource} #{id}"
         | 
| 251 | 
            +
             | 
| 230 252 | 
             
                data = call(cmd, stdin: stdin)
         | 
| 231 253 | 
             
                data = data.empty? ? '{}' : data
         | 
| 232 254 | 
             
                json_data = JSON.parse(data, symbolize_names: true)
         | 
| @@ -250,27 +272,6 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 250 272 | 
             
                "#{namespace_prefix}#{username}"
         | 
| 251 273 | 
             
              end
         | 
| 252 274 |  | 
| 253 | 
            -
              def context
         | 
| 254 | 
            -
                cluster
         | 
| 255 | 
            -
              end
         | 
| 256 | 
            -
             | 
| 257 | 
            -
              def default_config_file
         | 
| 258 | 
            -
                (ENV['KUBECONFIG'] || "#{Dir.home}/.kube/config")
         | 
| 259 | 
            -
              end
         | 
| 260 | 
            -
             | 
| 261 | 
            -
              def default_auth
         | 
| 262 | 
            -
                {
         | 
| 263 | 
            -
                  type: 'managaged'
         | 
| 264 | 
            -
                }.symbolize_keys
         | 
| 265 | 
            -
              end
         | 
| 266 | 
            -
             | 
| 267 | 
            -
              def default_server
         | 
| 268 | 
            -
                {
         | 
| 269 | 
            -
                  endpoint: 'https://localhost:8080',
         | 
| 270 | 
            -
                  cert_authority_file: nil
         | 
| 271 | 
            -
                }.symbolize_keys
         | 
| 272 | 
            -
              end
         | 
| 273 | 
            -
             | 
| 274 275 | 
             
              def formatted_ns_cmd
         | 
| 275 276 | 
             
                "#{namespaced_cmd} -o json"
         | 
| 276 277 | 
             
              end
         | 
| @@ -281,7 +282,7 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 281 282 |  | 
| 282 283 | 
             
              def base_cmd
         | 
| 283 284 | 
             
                base = "#{bin} --kubeconfig=#{config_file}"
         | 
| 284 | 
            -
                base << " --context=#{context}" if  | 
| 285 | 
            +
                base << " --context=#{context}" if context?
         | 
| 285 286 | 
             
                base
         | 
| 286 287 | 
             
              end
         | 
| 287 288 |  | 
| @@ -309,28 +310,30 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 309 310 | 
             
                nil
         | 
| 310 311 | 
             
              end
         | 
| 311 312 |  | 
| 312 | 
            -
              def make_kubectl_config(config)
         | 
| 313 | 
            -
                set_cluster(config.fetch(:server, default_server).to_h.symbolize_keys)
         | 
| 314 | 
            -
                configure_auth(config.fetch(:auth, default_auth).to_h.symbolize_keys)
         | 
| 315 | 
            -
              end
         | 
| 316 | 
            -
             | 
| 317 313 | 
             
              def configure_auth(auth)
         | 
| 318 | 
            -
                 | 
| 319 | 
            -
             | 
| 320 | 
            -
             | 
| 321 | 
            -
                case type
         | 
| 322 | 
            -
                when 'gke'
         | 
| 314 | 
            +
                if managed_auth?(auth)
         | 
| 315 | 
            +
                  return
         | 
| 316 | 
            +
                elsif gke_auth?(auth)
         | 
| 323 317 | 
             
                  set_gke_config(auth)
         | 
| 324 | 
            -
                 | 
| 325 | 
            -
                  set_context
         | 
| 318 | 
            +
                elsif oidc_auth?(auth)
         | 
| 319 | 
            +
                  set_context if context?
         | 
| 326 320 | 
             
                end
         | 
| 327 321 | 
             
              end
         | 
| 328 322 |  | 
| 329 | 
            -
              def  | 
| 330 | 
            -
                 | 
| 323 | 
            +
              def context?
         | 
| 324 | 
            +
                !@context.nil?
         | 
| 325 | 
            +
              end
         | 
| 326 | 
            +
             | 
| 327 | 
            +
              def gke_auth?(auth = {})
         | 
| 328 | 
            +
                auth.fetch(:type, nil) == 'gke'
         | 
| 329 | 
            +
              end
         | 
| 330 | 
            +
             | 
| 331 | 
            +
              def oidc_auth?(auth = {})
         | 
| 332 | 
            +
                auth.fetch(:type, nil) == 'oidc'
         | 
| 331 333 | 
             
              end
         | 
| 332 334 |  | 
| 333 | 
            -
              def  | 
| 335 | 
            +
              def managed_auth?(auth = {})
         | 
| 336 | 
            +
                type = auth.fetch(:type, nil)
         | 
| 334 337 | 
             
                if type.nil?
         | 
| 335 338 | 
             
                  true # maybe should be false?
         | 
| 336 339 | 
             
                else
         | 
| @@ -359,23 +362,24 @@ class OodCore::Job::Adapters::Kubernetes::Batch | |
| 359 362 | 
             
                # gke cluster name can probably can differ from what ood calls the cluster
         | 
| 360 363 | 
             
                cmd = "gcloud container clusters get-credentials #{locale} #{cluster}"
         | 
| 361 364 | 
             
                env = { 'KUBECONFIG' => config_file }
         | 
| 362 | 
            -
                call(cmd, env)
         | 
| 365 | 
            +
                call(cmd, env: env)
         | 
| 363 366 | 
             
              end
         | 
| 364 367 |  | 
| 365 368 | 
             
              def set_context
         | 
| 366 | 
            -
                 | 
| 369 | 
            +
                # can't really use base_cmd, bc it may use --context flag
         | 
| 370 | 
            +
                cmd = "#{bin} --kubeconfig=#{config_file} config set-context #{context}"
         | 
| 367 371 | 
             
                cmd << " --cluster=#{cluster} --namespace=#{namespace}"
         | 
| 368 372 | 
             
                cmd << " --user=#{k8s_username}"
         | 
| 369 373 |  | 
| 370 374 | 
             
                call(cmd)
         | 
| 371 | 
            -
                use_context
         | 
| 372 375 | 
             
              end
         | 
| 373 376 |  | 
| 374 377 | 
             
              def set_cluster(config)
         | 
| 375 378 | 
             
                server = config.fetch(:endpoint)
         | 
| 376 379 | 
             
                cert = config.fetch(:cert_authority_file, nil)
         | 
| 377 380 |  | 
| 378 | 
            -
                 | 
| 381 | 
            +
                # shouldn't use context here either
         | 
| 382 | 
            +
                cmd = "#{bin} --kubeconfig=#{config_file} config set-cluster #{cluster}"
         | 
| 379 383 | 
             
                cmd << " --server=#{server}"
         | 
| 380 384 | 
             
                cmd << " --certificate-authority=#{cert}" unless cert.nil?
         | 
| 381 385 |  | 
| @@ -4,7 +4,9 @@ class OodCore::Job::Adapters::Kubernetes::Helper | |
| 4 4 | 
             
              require_relative 'k8s_job_info'
         | 
| 5 5 | 
             
              require 'resolv'
         | 
| 6 6 | 
             
              require 'base64'
         | 
| 7 | 
            -
              require ' | 
| 7 | 
            +
              require 'ood_core/refinements/hash_extensions'
         | 
| 8 | 
            +
             | 
| 9 | 
            +
              using OodCore::Refinements::HashExtensions
         | 
| 8 10 |  | 
| 9 11 | 
             
              class K8sDataError < StandardError; end
         | 
| 10 12 |  | 
| @@ -49,14 +51,17 @@ class OodCore::Job::Adapters::Kubernetes::Helper | |
| 49 51 | 
             
                  command: parse_command(container[:command]),
         | 
| 50 52 | 
             
                  port: container[:port],
         | 
| 51 53 | 
             
                  env: default_env.merge(env),
         | 
| 52 | 
            -
                   | 
| 53 | 
            -
                   | 
| 54 | 
            +
                  memory_limit: container[:memory_limit] || container[:memory],
         | 
| 55 | 
            +
                  memory_request: container[:memory_request] || container[:memory],
         | 
| 56 | 
            +
                  cpu_limit: container[:cpu_limit] || container[:cpu],
         | 
| 57 | 
            +
                  cpu_request: container[:cpu_request] || container[:cpu],
         | 
| 54 58 | 
             
                  working_dir: container[:working_dir],
         | 
| 55 59 | 
             
                  restart_policy: container[:restart_policy],
         | 
| 56 60 | 
             
                  image_pull_policy: container[:image_pull_policy],
         | 
| 57 61 | 
             
                  image_pull_secret: container[:image_pull_secret],
         | 
| 58 62 | 
             
                  supplemental_groups: container[:supplemental_groups],
         | 
| 59 63 | 
             
                  startup_probe: container[:startup_probe],
         | 
| 64 | 
            +
                  labels: container[:labels],
         | 
| 60 65 | 
             
                )
         | 
| 61 66 | 
             
              end
         | 
| 62 67 |  | 
| @@ -55,14 +55,16 @@ module OodCore::Job::Adapters::Kubernetes::Resources | |
| 55 55 | 
             
              end
         | 
| 56 56 |  | 
| 57 57 | 
             
              class Container
         | 
| 58 | 
            -
                attr_accessor :name, :image, :command, :port, :env, : | 
| 58 | 
            +
                attr_accessor :name, :image, :command, :port, :env, :working_dir,
         | 
| 59 | 
            +
                              :memory_limit, :memory_request, :cpu_limit, :cpu_request, 
         | 
| 59 60 | 
             
                              :restart_policy, :image_pull_policy, :image_pull_secret, :supplemental_groups,
         | 
| 60 | 
            -
                              :startup_probe
         | 
| 61 | 
            +
                              :startup_probe, :labels
         | 
| 61 62 |  | 
| 62 63 | 
             
                def initialize(
         | 
| 63 | 
            -
                    name, image, command: [], port: nil, env: {}, | 
| 64 | 
            +
                    name, image, command: [], port: nil, env: {},
         | 
| 65 | 
            +
                    memory_limit: nil, memory_request: nil, cpu_limit: nil, cpu_request: nil,
         | 
| 64 66 | 
             
                    working_dir: "", restart_policy: "Never", image_pull_policy: nil, image_pull_secret: nil, supplemental_groups: [],
         | 
| 65 | 
            -
                    startup_probe: {}
         | 
| 67 | 
            +
                    startup_probe: {}, labels: {}
         | 
| 66 68 | 
             
                  )
         | 
| 67 69 | 
             
                  raise ArgumentError, "containers need valid names and images" unless name && image
         | 
| 68 70 |  | 
| @@ -71,14 +73,17 @@ module OodCore::Job::Adapters::Kubernetes::Resources | |
| 71 73 | 
             
                  @command = command.nil? ? [] : command
         | 
| 72 74 | 
             
                  @port = port&.to_i
         | 
| 73 75 | 
             
                  @env = env.nil? ? {} : env
         | 
| 74 | 
            -
                  @ | 
| 75 | 
            -
                  @ | 
| 76 | 
            +
                  @memory_limit = memory_limit.nil? ? "4Gi" : memory_limit
         | 
| 77 | 
            +
                  @memory_request = memory_request.nil? ? "4Gi" : memory_request
         | 
| 78 | 
            +
                  @cpu_limit = cpu_limit.nil? ? "1" : cpu_limit
         | 
| 79 | 
            +
                  @cpu_request = cpu_request.nil? ? "1" : cpu_request
         | 
| 76 80 | 
             
                  @working_dir = working_dir.nil? ? "" : working_dir
         | 
| 77 81 | 
             
                  @restart_policy = restart_policy.nil? ? "Never" : restart_policy
         | 
| 78 82 | 
             
                  @image_pull_policy = image_pull_policy.nil? ? "IfNotPresent" : image_pull_policy
         | 
| 79 83 | 
             
                  @image_pull_secret = image_pull_secret
         | 
| 80 84 | 
             
                  @supplemental_groups = supplemental_groups.nil? ? [] : supplemental_groups
         | 
| 81 85 | 
             
                  @startup_probe = TCPProbe.new(@port, startup_probe)
         | 
| 86 | 
            +
                  @labels = labels.nil? ? {} : labels
         | 
| 82 87 | 
             
                end
         | 
| 83 88 |  | 
| 84 89 | 
             
                def ==(other)
         | 
| @@ -87,14 +92,17 @@ module OodCore::Job::Adapters::Kubernetes::Resources | |
| 87 92 | 
             
                    command == other.command &&
         | 
| 88 93 | 
             
                    port == other.port &&
         | 
| 89 94 | 
             
                    env == other.env &&
         | 
| 90 | 
            -
                     | 
| 91 | 
            -
                     | 
| 95 | 
            +
                    memory_limit == other.memory_limit &&
         | 
| 96 | 
            +
                    memory_request == other.memory_request &&
         | 
| 97 | 
            +
                    cpu_limit == other.cpu_limit &&
         | 
| 98 | 
            +
                    cpu_request == other.cpu_request &&
         | 
| 92 99 | 
             
                    working_dir == other.working_dir &&
         | 
| 93 100 | 
             
                    restart_policy == other.restart_policy &&
         | 
| 94 101 | 
             
                    image_pull_policy == other.image_pull_policy &&
         | 
| 95 102 | 
             
                    image_pull_secret == other.image_pull_secret &&
         | 
| 96 103 | 
             
                    supplemental_groups == other.supplemental_groups &&
         | 
| 97 | 
            -
                    startup_probe.to_h == other.startup_probe.to_h
         | 
| 104 | 
            +
                    startup_probe.to_h == other.startup_probe.to_h &&
         | 
| 105 | 
            +
                    labels.to_h == other.labels.to_h
         | 
| 98 106 | 
             
                end
         | 
| 99 107 | 
             
              end
         | 
| 100 108 |  | 
| @@ -10,6 +10,9 @@ metadata: | |
| 10 10 | 
             
                <%- if !script.accounting_id.nil? && script.accounting_id != "" -%>
         | 
| 11 11 | 
             
                account: <%= script.accounting_id %>
         | 
| 12 12 | 
             
                <%- end -%>
         | 
| 13 | 
            +
                <%- spec.container.labels.each_pair do |key, value| -%>
         | 
| 14 | 
            +
                <%= key %>: "<%= value %>"
         | 
| 15 | 
            +
                <%- end -%>
         | 
| 13 16 | 
             
              annotations:
         | 
| 14 17 | 
             
                <%- unless script.wall_time.nil? -%>
         | 
| 15 18 | 
             
                pod.kubernetes.io/lifetime: <%= helper.seconds_to_duration(script.wall_time) %>
         | 
| @@ -88,14 +91,14 @@ spec: | |
| 88 91 | 
             
                <%- end # configmap mounts? and all_mounts not empty -%>
         | 
| 89 92 | 
             
                resources:
         | 
| 90 93 | 
             
                  limits:
         | 
| 91 | 
            -
                    memory: "<%= spec.container. | 
| 92 | 
            -
                    cpu: "<%= spec.container. | 
| 94 | 
            +
                    memory: "<%= spec.container.memory_limit %>"
         | 
| 95 | 
            +
                    cpu: "<%= spec.container.cpu_limit %>"
         | 
| 93 96 | 
             
                    <%- unless script.gpus_per_node.nil? -%>
         | 
| 94 97 | 
             
                    <%= gpu_type %>: <%= script.gpus_per_node %>
         | 
| 95 98 | 
             
                    <%- end -%>
         | 
| 96 99 | 
             
                  requests:
         | 
| 97 | 
            -
                    memory: "<%= spec.container. | 
| 98 | 
            -
                    cpu: "<%= spec.container. | 
| 100 | 
            +
                    memory: "<%= spec.container.memory_request %>"
         | 
| 101 | 
            +
                    cpu: "<%= spec.container.cpu_request %>"
         | 
| 99 102 | 
             
                    <%- unless script.gpus_per_node.nil? -%>
         | 
| 100 103 | 
             
                    <%= gpu_type %>: <%= script.gpus_per_node %>
         | 
| 101 104 | 
             
                    <%- end -%>
         | 
| @@ -61,7 +61,7 @@ class OodCore::Job::Adapters::LinuxHost::Launcher | |
| 61 61 |  | 
| 62 62 | 
             
                session_name = unique_session_name
         | 
| 63 63 | 
             
                output = call(*cmd, stdin: wrapped_script(script, session_name))
         | 
| 64 | 
            -
                hostname = output | 
| 64 | 
            +
                hostname = parse_hostname(output)
         | 
| 65 65 |  | 
| 66 66 | 
             
                "#{session_name}@#{hostname}"
         | 
| 67 67 | 
             
              end
         | 
| @@ -242,22 +242,20 @@ class OodCore::Job::Adapters::LinuxHost::Launcher | |
| 242 242 | 
             
              def list_remote_tmux_session(destination_host)
         | 
| 243 243 | 
             
                # Note that the tmux variable substitution looks like Ruby string sub,
         | 
| 244 244 | 
             
                # these must either be single quoted strings or Ruby-string escaped as well
         | 
| 245 | 
            -
                format_str = Shellwords.escape(
         | 
| 246 | 
            -
                  ['#{session_name}', '#{session_created}', '#{pane_pid}'].join(UNIT_SEPARATOR)
         | 
| 247 | 
            -
                )
         | 
| 245 | 
            +
                format_str = Shellwords.escape(['#{session_name}', '#{session_created}', '#{pane_pid}'].join(UNIT_SEPARATOR))
         | 
| 248 246 | 
             
                keys = [:session_name, :session_created, :session_pid]
         | 
| 249 247 | 
             
                cmd = ssh_cmd(destination_host, ['tmux', 'list-panes', '-aF', format_str])
         | 
| 250 | 
            -
             | 
| 251 | 
            -
                call(*cmd).split(
         | 
| 252 | 
            -
                  "\n"
         | 
| 253 | 
            -
                ).map do |line|
         | 
| 248 | 
            +
                
         | 
| 249 | 
            +
                call(*cmd).split("\n").map do |line|
         | 
| 254 250 | 
             
                  Hash[keys.zip(line.split(UNIT_SEPARATOR))].tap do |session_hash|
         | 
| 255 251 | 
             
                    session_hash[:destination_host] = destination_host
         | 
| 256 252 | 
             
                    session_hash[:id] = "#{session_hash[:session_name]}@#{destination_host}"
         | 
| 257 253 | 
             
                  end
         | 
| 258 | 
            -
                end.select | 
| 259 | 
            -
                   | 
| 260 | 
            -
             | 
| 254 | 
            +
                end.select do |session_hash| 
         | 
| 255 | 
            +
                  session_hash.compact.length >= 5 && 
         | 
| 256 | 
            +
                    !session_hash[:session_name].nil? && 
         | 
| 257 | 
            +
                    session_hash[:session_name].start_with?(session_name_label)
         | 
| 258 | 
            +
                end
         | 
| 261 259 | 
             
              rescue Error => e
         | 
| 262 260 | 
             
                interpret_and_raise(e)
         | 
| 263 261 | 
             
                []
         | 
| @@ -287,4 +285,10 @@ class OodCore::Job::Adapters::LinuxHost::Launcher | |
| 287 285 | 
             
                  raise error
         | 
| 288 286 | 
             
                end
         | 
| 289 287 | 
             
              end
         | 
| 288 | 
            +
             | 
| 289 | 
            +
              def parse_hostname(output)
         | 
| 290 | 
            +
                output.split($/).map do |line|
         | 
| 291 | 
            +
                  line.match(/^(([\w+]|[a-zA-Z0-9][\w*-]*\.))*$/)
         | 
| 292 | 
            +
                end.compact.last.to_s
         | 
| 293 | 
            +
              end
         | 
| 290 294 | 
             
            end
         | 
| @@ -2,6 +2,8 @@ module OodCore | |
| 2 2 | 
             
              # Namespace for Ruby refinements
         | 
| 3 3 | 
             
              module Refinements
         | 
| 4 4 | 
             
                # This module provides refinements for manipulating the Ruby {Hash} class.
         | 
| 5 | 
            +
                # Some elements have been taken from Rails (https://github.com/rails/rails)
         | 
| 6 | 
            +
                # and it's LICENSE has been added as RAILS-LICENSE in the root directory of this project.
         | 
| 5 7 | 
             
                module HashExtensions
         | 
| 6 8 | 
             
                  refine Hash do
         | 
| 7 9 | 
             
                    # Symbolize the keys in a {Hash}
         | 
| @@ -28,6 +30,37 @@ module OodCore | |
| 28 30 | 
             
                    def compact
         | 
| 29 31 | 
             
                      self.select { |_, value| !value.nil? }
         | 
| 30 32 | 
             
                    end
         | 
| 33 | 
            +
             | 
| 34 | 
            +
                    # Returns a new hash with +self+ and +other_hash+ merged recursively.
         | 
| 35 | 
            +
                    #
         | 
| 36 | 
            +
                    #   h1 = { a: true, b: { c: [1, 2, 3] } }
         | 
| 37 | 
            +
                    #   h2 = { a: false, b: { x: [3, 4, 5] } }
         | 
| 38 | 
            +
                    #
         | 
| 39 | 
            +
                    #   h1.deep_merge(h2) # => { a: false, b: { c: [1, 2, 3], x: [3, 4, 5] } }
         | 
| 40 | 
            +
                    #
         | 
| 41 | 
            +
                    # Like with Hash#merge in the standard library, a block can be provided
         | 
| 42 | 
            +
                    # to merge values:
         | 
| 43 | 
            +
                    #
         | 
| 44 | 
            +
                    #   h1 = { a: 100, b: 200, c: { c1: 100 } }
         | 
| 45 | 
            +
                    #   h2 = { b: 250, c: { c1: 200 } }
         | 
| 46 | 
            +
                    #   h1.deep_merge(h2) { |key, this_val, other_val| this_val + other_val }
         | 
| 47 | 
            +
                    #   # => { a: 100, b: 450, c: { c1: 300 } }
         | 
| 48 | 
            +
                    def deep_merge(other_hash, &block)
         | 
| 49 | 
            +
                      dup.deep_merge!(other_hash, &block)
         | 
| 50 | 
            +
                    end
         | 
| 51 | 
            +
             | 
| 52 | 
            +
                    # Same as +deep_merge+, but modifies +self+.
         | 
| 53 | 
            +
                    def deep_merge!(other_hash, &block)
         | 
| 54 | 
            +
                      merge!(other_hash) do |key, this_val, other_val|
         | 
| 55 | 
            +
                        if this_val.is_a?(Hash) && other_val.is_a?(Hash)
         | 
| 56 | 
            +
                          this_val.deep_merge(other_val, &block)
         | 
| 57 | 
            +
                        elsif block_given?
         | 
| 58 | 
            +
                          block.call(key, this_val, other_val)
         | 
| 59 | 
            +
                        else
         | 
| 60 | 
            +
                          other_val
         | 
| 61 | 
            +
                        end
         | 
| 62 | 
            +
                      end
         | 
| 63 | 
            +
                    end
         | 
| 31 64 | 
             
                  end
         | 
| 32 65 | 
             
                end
         | 
| 33 66 | 
             
              end
         | 
    
        data/lib/ood_core/version.rb
    CHANGED
    
    
    
        data/ood_core.gemspec
    CHANGED
    
    | @@ -25,7 +25,6 @@ Gem::Specification.new do |spec| | |
| 25 25 | 
             
              spec.add_runtime_dependency "ood_support", "~> 0.0.2"
         | 
| 26 26 | 
             
              spec.add_runtime_dependency "ffi", "~> 1.9", ">= 1.9.6"
         | 
| 27 27 | 
             
              spec.add_development_dependency "bundler", "~> 2.1"
         | 
| 28 | 
            -
              spec.add_runtime_dependency "activesupport", ">= 5.2", "< 6.0"
         | 
| 29 28 | 
             
              spec.add_development_dependency "rake", "~> 13.0.1"
         | 
| 30 29 | 
             
              spec.add_development_dependency "rspec", "~> 3.0"
         | 
| 31 30 | 
             
              spec.add_development_dependency "pry", "~> 0.10"
         | 
    
        metadata
    CHANGED
    
    | @@ -1,7 +1,7 @@ | |
| 1 1 | 
             
            --- !ruby/object:Gem::Specification
         | 
| 2 2 | 
             
            name: ood_core
         | 
| 3 3 | 
             
            version: !ruby/object:Gem::Version
         | 
| 4 | 
            -
              version: 0. | 
| 4 | 
            +
              version: 0.18.1
         | 
| 5 5 | 
             
            platform: ruby
         | 
| 6 6 | 
             
            authors:
         | 
| 7 7 | 
             
            - Eric Franz
         | 
| @@ -10,7 +10,7 @@ authors: | |
| 10 10 | 
             
            autorequire: 
         | 
| 11 11 | 
             
            bindir: exe
         | 
| 12 12 | 
             
            cert_chain: []
         | 
| 13 | 
            -
            date: 2021- | 
| 13 | 
            +
            date: 2021-10-18 00:00:00.000000000 Z
         | 
| 14 14 | 
             
            dependencies:
         | 
| 15 15 | 
             
            - !ruby/object:Gem::Dependency
         | 
| 16 16 | 
             
              name: ood_support
         | 
| @@ -60,26 +60,6 @@ dependencies: | |
| 60 60 | 
             
                - - "~>"
         | 
| 61 61 | 
             
                  - !ruby/object:Gem::Version
         | 
| 62 62 | 
             
                    version: '2.1'
         | 
| 63 | 
            -
            - !ruby/object:Gem::Dependency
         | 
| 64 | 
            -
              name: activesupport
         | 
| 65 | 
            -
              requirement: !ruby/object:Gem::Requirement
         | 
| 66 | 
            -
                requirements:
         | 
| 67 | 
            -
                - - ">="
         | 
| 68 | 
            -
                  - !ruby/object:Gem::Version
         | 
| 69 | 
            -
                    version: '5.2'
         | 
| 70 | 
            -
                - - "<"
         | 
| 71 | 
            -
                  - !ruby/object:Gem::Version
         | 
| 72 | 
            -
                    version: '6.0'
         | 
| 73 | 
            -
              type: :runtime
         | 
| 74 | 
            -
              prerelease: false
         | 
| 75 | 
            -
              version_requirements: !ruby/object:Gem::Requirement
         | 
| 76 | 
            -
                requirements:
         | 
| 77 | 
            -
                - - ">="
         | 
| 78 | 
            -
                  - !ruby/object:Gem::Version
         | 
| 79 | 
            -
                    version: '5.2'
         | 
| 80 | 
            -
                - - "<"
         | 
| 81 | 
            -
                  - !ruby/object:Gem::Version
         | 
| 82 | 
            -
                    version: '6.0'
         | 
| 83 63 | 
             
            - !ruby/object:Gem::Dependency
         | 
| 84 64 | 
             
              name: rake
         | 
| 85 65 | 
             
              requirement: !ruby/object:Gem::Requirement
         | 
| @@ -165,6 +145,7 @@ files: | |
| 165 145 | 
             
            - CHANGELOG.md
         | 
| 166 146 | 
             
            - Gemfile
         | 
| 167 147 | 
             
            - LICENSE.txt
         | 
| 148 | 
            +
            - RAILS-LICENSE
         | 
| 168 149 | 
             
            - README.md
         | 
| 169 150 | 
             
            - Rakefile
         | 
| 170 151 | 
             
            - bin/console
         |