hybrid_platforms_conductor 33.5.1 → 33.7.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a1e882e48d05f919ad5818c9251e045f941d78a9eacc16824f3fd49e787d91b5
-  data.tar.gz: b685a6a1ca1f6f40714da2443ad7de95d78e90489f15a4b8499e153f4d84945a
+  metadata.gz: 143014298b9351f34fdfa3d4e69e0c91a918c3f6c6ed8faa9348cc36380e43e2
+  data.tar.gz: aa78731557e9650d06780a4b1505b28c206476b111d9f42171e4cab4cc46b829
 SHA512:
-  metadata.gz: f9386d953b2e652fd453656a21a935470099ba3f935a82229829d5777408e93149833883d7acd277da6cf2f08ba5ca49c8c62d43d3207b34113c347299335e5f
-  data.tar.gz: 9066d78f77fed46560b1884bade1074055da61547bd196c2a6479cf9412ba30f9d7518f91e05244307b703ebd241a620985e59f93a7911cd353543af26ae3699
+  metadata.gz: b13deb407182b0ef9bbb3ea5cb513938acf57b9773aa28c4c43088d972b1b668f1ded6c46067bfe8c9544c67bb2ce596750c7f211389bd1ffc32039aef1e5511
+  data.tar.gz: f5f11ed1bc6d675478778c6fb66af5a90e7e06e0a61f73b6683782f960492339002f93f3e6ce9e37ad4bc2f690a31e2529044fda6a8787a8db15e81611323518

data/CHANGELOG.md

@@ -1,3 +1,50 @@
+# [v33.7.2](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.7.1...v33.7.2) (2021-07-15 14:37:59)
+
+## Global changes
+### Patches
+
+* [[Hotfix(provisioner_docker)] Increase Futex timeout to cope with image building time](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/52a03adbfc176f49800793acb3ab28026eb56289)
+
+## Changes for provisioner_docker
+### Patches
+
+* [[Hotfix(provisioner_docker)] Increase Futex timeout to cope with image building time](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/52a03adbfc176f49800793acb3ab28026eb56289)
+
+# [v33.7.1](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.7.0...v33.7.1) (2021-07-09 17:17:18)
+
+## Global changes
+### Patches
+
+* [[Bugfix(platform_handler_serverless_chef)] [#93] Make sure chef runs use colors in their output](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/14352425115cf46b92e4c747b2e34e3734314288)
+
+## Changes for platform_handler_serverless_chef
+### Patches
+
+* [[Bugfix(platform_handler_serverless_chef)] [#93] Make sure chef runs use colors in their output](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/14352425115cf46b92e4c747b2e34e3734314288)
+
+# [v33.7.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.6.0...v33.7.0) (2021-07-09 16:32:25)
+
+### Features
+
+* [[Feature] [#91] Expose log_debug? method to the config DSL to adapt configuration in case we are in debug mode](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/87f6f51df2d20c7861f9ddd59a8d7f68c27cdd74)
+
+# [v33.6.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.5.1...v33.6.0) (2021-07-07 15:45:12)
+
+## Global changes
+### Patches
+
+* [[Feature(connector_ssh)] [Feature(cmdb_host_keys)] [#78] Way to configure the SSH port being used with the ssh_port metadata](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/f1be319eae30eedcf7f901d3ce0a14f6f4d1f2ea)
+
+## Changes for cmdb_host_keys
+### Features
+
+* [[Feature(connector_ssh)] [Feature(cmdb_host_keys)] [#78] Way to configure the SSH port being used with the ssh_port metadata](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/f1be319eae30eedcf7f901d3ce0a14f6f4d1f2ea)
+
+## Changes for connector_ssh
+### Features
+
+* [[Feature(connector_ssh)] [Feature(cmdb_host_keys)] [#78] Way to configure the SSH port being used with the ssh_port metadata](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/f1be319eae30eedcf7f901d3ce0a14f6f4d1f2ea)
+
 # [v33.5.1](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.5.0...v33.5.1) (2021-07-07 12:01:32)
 
 ### Patches

data/README.md

@@ -84,9 +84,11 @@ Here are the various plugin categories:
 * **[Actions](docs/plugins/action)** implement a given **action**. For example: bash code execution, ruby code execution, file transfer...
 * **[Cmdbs](docs/plugins/cmdb)** parse **metadata** from various sources. For example: a database, Chef/Ansible inventory files, a configuration management database such as Consul...
 * **[Connectors](docs/plugins/connector)** give ways to execute commands and transfer files on **nodes**. For example: using an SSH connection, a CLI for a Cloud provider...
+* **[Logs](docs/plugins/log)** handle any deployment log storage strategy. They take deployment logs and send them wherever the plugins want.
 * **[Platform handlers](docs/plugins/platform_handler)** handle any **platform** of a given **platform type**. They read **nodes**' inventory and **services** from **platforms**, and provide **actions** to deploy a **service** on a **node**.
 * **[Provisioners](docs/plugins/provisioner)** provision (create, destroy, start, stop...) **nodes**. For example: using OpenShift, Proxmox, Docker, Podman...
 * **[Reports](docs/plugins/report)** gather **platforms** and **nodes**' inventory information and publish them to some medium. For example: on command line, as a JSON file, on a content management system like Confluence or Mediawiki...
+* **[Secrets readers](docs/plugins/secrets_reader)** reads secrets from various sources (vaults, APIs, local files...).
 * **[Tests](docs/plugins/test)** define tests to be performed on **platforms** and **nodes**.
 * **[Tests reports](docs/plugins/test_report)** publish tests results to some medium. For example: on command line, as a JSON file, on a content management system like Confluence or Mediawiki...
 

data/docs/plugins/cmdb/host_keys.md

@@ -21,7 +21,9 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
-| `hostname` | `String` | Used to query the IP from DNS records |
+| `host_ip` | `String` | Used to perform the `ssh-keyscan` |
+| `hostname` | `String` | Used in place of the `host_ip` in case `host_ip` is not available |
+| `ssh_port` | `Integer` | Port on which the `ssh-keyscan` will be performed (default: 22) |
 
 ## Used environment variables
 

data/docs/plugins/connector/ssh.md

@@ -70,6 +70,7 @@ end
 | `host_keys` | `Array<String>` | The node's host keys used to generate a `known_hosts` file with those to avoid user confirmations when connecting. |
 | `hostname` | `String` | Host name used to connect in case no IP address can be found in metadata. |
 | `private_ips` | `Array<String>` | IP list to connect in case `host_ip` is not defined in metadata. |
+| `ssh_port` | `Integer` | Port used to connect to SSH (default: 22). |
 | `ssh_session_exec` | `Boolean` | If set to `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
 
 ## Used environment variables

data/lib/hybrid_platforms_conductor/config.rb

@@ -34,6 +34,8 @@ module HybridPlatformsConductor
     end
     @mixin_initializers = []
 
+    expose :log_debug?
+
     # Directory of the definition of the platforms
     #   String
     attr_reader :hybrid_platforms_dir

data/lib/hybrid_platforms_conductor/hpc_plugins/cmdb/host_keys.rb

@@ -24,7 +24,7 @@ module HybridPlatformsConductor
         # * Hash<Symbol, Symbol or Array<Symbol> >: The list of necessary properties (or single one) that should be set, per property name (:others can also be used here)
         def property_dependencies
           {
-            host_keys: %i[hostname host_ip]
+            host_keys: %i[hostname host_ip ssh_port]
           }
         end
 
@@ -41,19 +41,20 @@ module HybridPlatformsConductor
         #     Nodes for which the property can't be fetched can be ommitted.
         def get_host_keys(_nodes, metadata)
           updated_metadata = {}
-          # Get the list of nodes, per hostname (just in case several nodes share the same hostname)
-          # Hash<String, Array<String> >
+          # Get the list of nodes, per [hostname, port] (just in case several nodes share the same hostname and port)
+          # Hash<[String, Integer], Array<String> >
           hostnames = Hash.new { |hash, key| hash[key] = [] }
           metadata.each do |node, node_metadata|
+            ssh_port = node_metadata[:ssh_port] || 22
             if node_metadata[:host_ip]
-              hostnames[node_metadata[:host_ip]] << node
+              hostnames[[node_metadata[:host_ip], ssh_port]] << node
             elsif node_metadata[:hostname]
-              hostnames[node_metadata[:hostname]] << node
+              hostnames[[node_metadata[:hostname], ssh_port]] << node
             end
           end
           unless hostnames.empty?
-            host_keys_for(*hostnames.keys).each do |hostname, ip|
-              hostnames[hostname].each do |node|
+            host_keys_for(*hostnames.keys).each do |host_id, ip|
+              hostnames[host_id].each do |node|
                 updated_metadata[node] = ip
               end
             end
@@ -71,7 +72,7 @@ module HybridPlatformsConductor
         # Discover the host keys associated to a list of hosts.
         #
         # Parameters::
-        # * *hosts* (Array<String>): The hosts to check for
+        # * *hosts* (Array<[String, Integer]>): The hosts to check for ([hostname, port])
         # Result::
         # * Hash<String, Array<String> >: The corresponding host keys, per host name
         def host_keys_for(*hosts)
@@ -82,9 +83,9 @@ module HybridPlatformsConductor
             parallel: true,
             nbr_threads_max: MAX_THREADS_SSH_KEY_SCAN,
             progress: log_debug? ? 'Gather host keys' : nil
-          ) do |host|
+          ) do |(host, ssh_port)|
             exit_status, stdout, _stderr = @cmd_runner.run_cmd(
-              "ssh-keyscan #{host}",
+              "ssh-keyscan -p #{ssh_port} #{host}",
               timeout: TIMEOUT_SSH_KEYSCAN,
               log_to_stdout: log_debug?,
               no_exception: true
@@ -97,9 +98,9 @@ module HybridPlatformsConductor
                   found_keys << "#{type} #{key}"
                 end
               end
-              results[host] = found_keys.sort unless found_keys.empty?
+              results[[host, ssh_port]] = found_keys.sort unless found_keys.empty?
             else
-              log_warn "Unable to get host key for #{host}. Ignoring it. Accessing #{host} might require manual acceptance of its host key."
+              log_warn "Unable to get host key for #{host} (port #{ssh_port}). Ignoring it. Accessing #{host} might require manual acceptance of its host key."
             end
           end
           results

data/lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb

@@ -384,16 +384,18 @@ module HybridPlatformsConductor
 
           # Add each node
           # Query for the metadata of all nodes at once
-          @nodes_handler.prefetch_metadata_of nodes, %i[private_ips hostname host_ip description]
+          @nodes_handler.prefetch_metadata_of nodes, %i[private_ips hostname host_ip description ssh_port]
           nodes.sort.each do |node|
             # Generate the conf for the node
             connection, connection_user, gateway, gateway_user = connection_info_for(node, no_exception: true)
             if connection.nil?
               config_content << "# #{node} - Not connectable using SSH - #{@nodes_handler.get_description_of(node) || ''}\n"
             else
+              ssh_port = @nodes_handler.get_ssh_port_of(node)
               config_content << "# #{node} - #{connection} - #{@nodes_handler.get_description_of(node) || ''}\n"
               config_content << "Host #{ssh_aliases_for(node).join(' ')}\n"
               config_content << "  Hostname #{connection}\n"
+              config_content << "  Port #{ssh_port}\n" unless ssh_port.nil?
               config_content << "  User \"#{connection_user}\"\n" if connection_user != @ssh_user
               config_content << "  ProxyCommand #{ssh_exec} -q -W %h:%p #{gateway_user}@#{gateway}\n" unless gateway.nil?
               if @passwords.key?(node)
@@ -658,7 +660,7 @@ module HybridPlatformsConductor
             existing_users = File.exist?(control_master_users_file) ? File.read(control_master_users_file).split("\n") : []
             ssh_url = "hpc.#{node}"
             connection, connection_user, _gateway, _gateway_user = connection_info_for(node)
-            control_path_file = control_master_file(connection, '22', connection_user)
+            control_path_file = control_master_file(connection, @nodes_handler.get_ssh_port_of(node) || 22, connection_user)
             if existing_users.empty?
               # Make sure there is no stale one.
               if File.exist?(control_path_file)

data/lib/hybrid_platforms_conductor/hpc_plugins/platform_handler/serverless_chef.rb

@@ -241,18 +241,25 @@ module HybridPlatformsConductor
             '--json-attributes', "nodes/#{node}.json"
           ]
           client_options << '--why-run' if use_why_run
+          # client_options.concat ['--log_level', 'debug'] if log_debug?
+          # Force setting of TERM variable and usage of unbuffer to get colored output from chef-client even if executed through a non-interactive SSH session.
+          client_env = {
+            'SSL_CERT_DIR' => '/etc/ssl/certs',
+            'TERM' => 'xterm-256color'
+          }
           if @nodes_handler.get_use_local_chef_of(node)
             # Just run the chef-client directly from the packaged repository
-            sudo_prefix = @cmd_runner.root? ? '' : 'sudo '
+            sudo_prefix = @cmd_runner.root? ? '' : 'sudo -E '
             [
               {
                 bash: [
                   'set -e',
                   "cd #{package_dir}"
                 ] +
-                  gems_to_install.map { |(gem_name, gem_version)| "#{sudo_prefix}SSL_CERT_DIR=/etc/ssl/certs /opt/chef-workstation/bin/chef gem install #{gem_name} --version \"#{gem_version}\"" } +
+                  client_env.map { |var_name, value| "export #{var_name}=#{value}" } +
+                  gems_to_install.map { |(gem_name, gem_version)| "#{sudo_prefix}/opt/chef-workstation/bin/chef gem install #{gem_name} --version \"#{gem_version}\"" } +
                   [
-                    "#{sudo_prefix}SSL_CERT_DIR=/etc/ssl/certs /opt/chef-workstation/bin/chef-client #{client_options.join(' ')}"
+                    "#{sudo_prefix}/opt/chef-workstation/bin/chef-client #{client_options.join(' ')}"
                   ]
               }
             ]
@@ -270,7 +277,7 @@ module HybridPlatformsConductor
                 remote_bash: [
                   'set -e',
                   'set -o pipefail',
-                  "if [ -n \"$(command -v apt)\" ]; then #{sudo}apt update && #{sudo}apt install -y curl build-essential ; else #{sudo}yum groupinstall 'Development Tools' && #{sudo}yum install -y curl ; fi",
+                  "if [ -n \"$(command -v apt)\" ]; then #{sudo}apt update && #{sudo}apt install -y curl build-essential expect ; else #{sudo}yum groupinstall 'Development Tools' && #{sudo}yum install -y curl expect ; fi",
                   'mkdir -p ./hpc_deploy',
                   'rm -rf ./hpc_deploy/tmp',
                   'mkdir -p ./hpc_deploy/tmp',
@@ -281,16 +288,19 @@ module HybridPlatformsConductor
               },
               {
                 scp: { package_dir => './hpc_deploy' },
-                remote_bash: [
-                  'set -e',
-                  "cd ./hpc_deploy/#{package_name}"
-                ] +
-                  gems_to_install.map { |(gem_name, gem_version)| "#{sudo}SSL_CERT_DIR=/etc/ssl/certs /opt/chef/embedded/bin/gem install #{gem_name} --version \"#{gem_version}\"" } +
-                  [
-                    "#{sudo}SSL_CERT_DIR=/etc/ssl/certs /opt/chef/bin/chef-client #{client_options.join(' ')}",
-                    'cd ..'
+                remote_bash: {
+                  commands: [
+                    'set -e',
+                    "cd ./hpc_deploy/#{package_name}"
                   ] +
-                  (log_debug? ? [] : ["#{sudo}rm -rf ./hpc_deploy/#{package_name}"])
+                    gems_to_install.map { |(gem_name, gem_version)| "#{sudo}/opt/chef/embedded/bin/gem install #{gem_name} --version \"#{gem_version}\"" } +
+                    [
+                      "#{sudo}unbuffer /opt/chef/bin/chef-client #{client_options.join(' ')}",
+                      'cd ..'
+                    ] +
+                    (log_debug? ? [] : ["#{sudo}rm -rf ./hpc_deploy/#{package_name}"]),
+                  env: client_env
+                }
               }
             ]
           end

data/lib/hybrid_platforms_conductor/hpc_plugins/provisioner/docker.rb

@@ -21,7 +21,7 @@ module HybridPlatformsConductor
             ::Docker.validate_version!
             docker_ok = true
           rescue
-            log_error "[ #{@node}/#{@environment} ] - Docker is not installed correctly. Please install it. Error: #{$ERROR_INFO}"
+            log_error "Docker is not installed correctly. Please install it. Error: #{$ERROR_INFO}"
           end
           docker_ok
         end
@@ -40,7 +40,7 @@ module HybridPlatformsConductor
           docker_image = nil
           image_futex_file = "#{Dir.tmpdir}/hpc_docker_image_futexes/#{image_tag}"
           FileUtils.mkdir_p File.dirname(image_futex_file)
-          Futex.new(image_futex_file).open do
+          Futex.new(image_futex_file, timeout: 600).open do
             docker_image = ::Docker::Image.all.find { |search_image| !search_image.info['RepoTags'].nil? && search_image.info['RepoTags'].include?("#{image_tag}:latest") }
             unless docker_image
               log_debug "[ #{@node}/#{@environment} ] - Creating Docker image #{image_tag}..."

data/lib/hybrid_platforms_conductor/hpc_plugins/provisioner/proxmox.rb

@@ -2,6 +2,7 @@ require 'json'
 require 'proxmox'
 require 'digest'
 require 'hybrid_platforms_conductor/actions_executor'
+require 'hybrid_platforms_conductor/credentials'
 require 'hybrid_platforms_conductor/provisioner'
 
 module HybridPlatformsConductor

data/lib/hybrid_platforms_conductor/hpc_plugins/test/check_deploy_and_idempotence.rb

@@ -26,7 +26,13 @@ module HybridPlatformsConductor
             # Check that we can connect with root
             ssh_ok = false
             begin
-              Net::SSH.start(instance.ip, 'root', password: 'root_pwd', auth_methods: ['password'], verify_host_key: :never) do |ssh|
+              Net::SSH.start(
+                instance.ip,
+                'root',
+                password: 'root_pwd',
+                auth_methods: ['password'],
+                verify_host_key: :never
+              ) do |ssh|
                 ssh_ok = ssh.exec!('echo Works').strip == 'Works'
               end
             rescue
@@ -53,14 +59,22 @@ module HybridPlatformsConductor
                 #   System is booting up. See pam_nologin(8)
                 #   Authentication failed.
                 instance.stop
-                instance.with_running_instance(port: 22) do
+                ssh_port = @nodes_handler.get_ssh_port_of(@node) || 22
+                instance.with_running_instance(port: ssh_port) do
 
                   unless @nodes_handler.get_root_access_allowed_of(@node)
                     # ===== Deploy removes root access
                     # Check that we can't connect with root
                     ssh_ok = false
                     begin
-                      Net::SSH.start(instance.ip, 'root', password: 'root_pwd', auth_methods: ['password'], verify_host_key: :never) do |ssh|
+                      Net::SSH.start(
+                        instance.ip,
+                        'root',
+                        password: 'root_pwd',
+                        auth_methods: ['password'],
+                        verify_host_key: :never,
+                        port: ssh_port
+                      ) do |ssh|
                         ssh_ok = ssh.exec!('echo Works').strip == 'Works'
                       end
                     rescue

data/lib/hybrid_platforms_conductor/hpc_plugins/test/deploy_removes_root_access.rb

@@ -18,7 +18,13 @@ module HybridPlatformsConductor
             # Check that we can connect with root
             ssh_ok = false
             begin
-              Net::SSH.start(instance.ip, 'root', password: 'root_pwd', auth_methods: ['password'], verify_host_key: :never) do |ssh|
+              Net::SSH.start(
+                instance.ip,
+                'root',
+                password: 'root_pwd',
+                auth_methods: ['password'],
+                verify_host_key: :never
+              ) do |ssh|
                 ssh_ok = ssh.exec!('echo Works').strip == 'Works'
               end
             rescue
@@ -29,17 +35,31 @@ module HybridPlatformsConductor
               deployer.nbr_retries_on_error = 3
               deployer.deploy_on @node
               # As sshd is certainly being restarted, start and stop the container to reload it.
-              deployer.restart @node
-              # Check that we can't connect with root
-              ssh_ok = false
-              begin
-                Net::SSH.start(instance.ip, 'root', password: 'root_pwd', auth_methods: ['password'], verify_host_key: :never) do |ssh|
-                  ssh_ok = ssh.exec!('echo Works').strip == 'Works'
+              # As it's possible sshd has to be restarted because of a change in its conf, restart the container.
+              # Otherwise you'll get the following error upon reconnection:
+              #   System is booting up. See pam_nologin(8)
+              #   Authentication failed.
+              instance.stop
+              ssh_port = @nodes_handler.get_ssh_port_of(@node) || 22
+              instance.with_running_instance(port: ssh_port) do
+                # Check that we can't connect with root
+                ssh_ok = false
+                begin
+                  Net::SSH.start(
+                    instance.ip,
+                    'root',
+                    password: 'root_pwd',
+                    auth_methods: ['password'],
+                    verify_host_key: :never,
+                    port: ssh_port
+                  ) do |ssh|
+                    ssh_ok = ssh.exec!('echo Works').strip == 'Works'
+                  end
+                rescue
+                  nil
                 end
-              rescue
-                nil
+                assert_equal ssh_ok, false, 'Root can still connect on the image after deployment'
               end
-              assert_equal ssh_ok, false, 'Root can still connect on the image after deployment'
             end
           end
         end

data/lib/hybrid_platforms_conductor/hpc_plugins/test/idempotence.rb

@@ -27,7 +27,7 @@ module HybridPlatformsConductor
               #   System is booting up. See pam_nologin(8)
               #   Authentication failed.
               instance.stop
-              instance.with_running_instance(port: 22) do
+              instance.with_running_instance(port: @nodes_handler.get_ssh_port_of(@node) || 22) do
                 # Now that the node has been deployed, use the a_testadmin user for the check-node (as root has no more access)
                 deployer.instance_variable_get(:@actions_executor).connector(:ssh).ssh_user = 'a_testadmin'
                 deployer.instance_variable_get(:@actions_executor).connector(:ssh).passwords.delete(@node)

data/lib/hybrid_platforms_conductor/version.rb

@@ -1,5 +1,5 @@
 module HybridPlatformsConductor
 
-  VERSION = '33.5.1'
+  VERSION = '33.7.2'
 
 end

data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/connections_spec.rb

@@ -132,6 +132,36 @@ describe HybridPlatformsConductor::ActionsExecutor do
         end
       end
 
+      it 'creates SSH master to several nodes differing only by the SSH port' do
+        with_test_platform(
+          {
+            nodes: {
+              'node1' => { meta: { host_ip: '192.168.42.1', ssh_port: 6661 } },
+              'node2' => { meta: { host_ip: '192.168.42.1', ssh_port: 6662 } },
+              'node3' => { meta: { host_ip: '192.168.42.1', ssh_port: 6663 } }
+            }
+          }
+        ) do
+          with_cmd_runner_mocked(
+            [
+              ['which env', proc { [0, "/usr/bin/env\n", ''] }],
+              ['ssh -V 2>&1', proc { [0, "OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u  20 Dec 2019\n", ''] }]
+            ] + ssh_expected_commands_for(
+              {
+                'node1' => { connection: '192.168.42.1', user: 'test_user', port: 6661 },
+                'node2' => { connection: '192.168.42.1', user: 'test_user', port: 6662 },
+                'node3' => { connection: '192.168.42.1', user: 'test_user', port: 6663 }
+              }
+            )
+          ) do
+            test_connector.ssh_user = 'test_user'
+            test_connector.with_connection_to(%w[node1 node2 node3]) do |connected_nodes|
+              expect(connected_nodes.sort).to eq %w[node1 node2 node3].sort
+            end
+          end
+        end
+      end
+
       it 'creates SSH master to several nodes with ssh connections transformed' do
         with_test_platform(
           { nodes: {

data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/global_helpers_spec.rb

@@ -175,6 +175,16 @@ describe HybridPlatformsConductor::ActionsExecutor do
         end
       end
 
+      it 'generates a simple config for a node with host_ip and a given SSH port' do
+        with_test_platform({ nodes: { 'node' => { meta: { host_ip: '192.168.42.42', ssh_port: 666 } } } }) do
+          expect(ssh_config_for('node')).to eq <<~EO_SSH_CONFIG
+            Host hpc.node
+              Hostname 192.168.42.42
+              Port 666
+          EO_SSH_CONFIG
+        end
+      end
+
       it 'generates a simple config for several nodes' do
         with_test_platform(
           {

data/spec/hybrid_platforms_conductor_test/api/config_spec.rb

@@ -29,6 +29,17 @@ describe HybridPlatformsConductor::Config do
     end
   end
 
+  it 'can check if we are in debug mode' do
+    with_platforms(
+      <<~EO_CONFIG
+        os_image :image_1, '/path/to/image_1' if log_debug?
+        os_image :image_2, '/path/to/image_2'
+      EO_CONFIG
+    ) do
+      expect(test_config.known_os_images.sort).to eq %i[image_2].sort
+    end
+  end
+
   it 'returns the tests provisioner correctly' do
     with_platforms 'tests_provisioner :test_provisioner' do
       expect(test_config.tests_provisioner_id).to eq :test_provisioner

data/spec/hybrid_platforms_conductor_test/api/nodes_handler/cmdbs/host_keys_spec.rb

@@ -4,7 +4,7 @@ describe HybridPlatformsConductor::NodesHandler do
 
     it 'makes sure to have hostname or host_ip set to compute host_keys' do
       with_test_platform({}) do
-        expect(cmdb(:host_keys).property_dependencies[:host_keys].sort).to eq %i[hostname host_ip].sort
+        expect(cmdb(:host_keys).property_dependencies[:host_keys].sort).to eq %i[hostname host_ip ssh_port].sort
       end
     end
 
@@ -17,17 +17,27 @@ describe HybridPlatformsConductor::NodesHandler do
     it 'returns host keys when hostname is set' do
       with_test_platform({ nodes: { 'test_node' => {} } }) do
         with_cmd_runner_mocked [
-          ['ssh-keyscan my_host.my_domain', proc { [0, "my_host.my_domain ssh-rsa fake_host_key\n", ''] }]
+          ['ssh-keyscan -p 22 my_host.my_domain', proc { [0, "my_host.my_domain ssh-rsa fake_host_key\n", ''] }]
         ] do
           expect(cmdb(:host_keys).get_host_keys(['test_node'], { 'test_node' => { hostname: 'my_host.my_domain' } })).to eq('test_node' => ['ssh-rsa fake_host_key'])
         end
       end
     end
 
+    it 'returns host keys for the correct ssh port' do
+      with_test_platform({ nodes: { 'test_node' => {} } }) do
+        with_cmd_runner_mocked [
+          ['ssh-keyscan -p 666 my_host.my_domain', proc { [0, "my_host.my_domain ssh-rsa fake_host_key\n", ''] }]
+        ] do
+          expect(cmdb(:host_keys).get_host_keys(['test_node'], { 'test_node' => { hostname: 'my_host.my_domain', ssh_port: 666 } })).to eq('test_node' => ['ssh-rsa fake_host_key'])
+        end
+      end
+    end
+
     it 'returns host keys when host_ip is set' do
       with_test_platform({ nodes: { 'test_node' => {} } }) do
         with_cmd_runner_mocked [
-          ['ssh-keyscan 192.168.42.42', proc { [0, "192.168.42.42 ssh-rsa fake_host_key\n", ''] }]
+          ['ssh-keyscan -p 22 192.168.42.42', proc { [0, "192.168.42.42 ssh-rsa fake_host_key\n", ''] }]
         ] do
           expect(cmdb(:host_keys).get_host_keys(['test_node'], { 'test_node' => { host_ip: '192.168.42.42' } })).to eq('test_node' => ['ssh-rsa fake_host_key'])
         end
@@ -37,7 +47,7 @@ describe HybridPlatformsConductor::NodesHandler do
     it 'returns several host keys' do
       with_test_platform({ nodes: { 'test_node' => {} } }) do
         with_cmd_runner_mocked [
-          ['ssh-keyscan 192.168.42.42', proc do
+          ['ssh-keyscan -p 22 192.168.42.42', proc do
             [0, <<~EO_STDOUT, '']
               192.168.42.42 ssh-rsa fake_host_key_rsa
               192.168.42.42 ssh-ed25519 fake_host_key_ed25519
@@ -55,7 +65,7 @@ describe HybridPlatformsConductor::NodesHandler do
     it 'returns several host keys and ignores comments from ssh-keyscan' do
       with_test_platform({ nodes: { 'test_node' => {} } }) do
         with_cmd_runner_mocked [
-          ['ssh-keyscan 192.168.42.42', proc do
+          ['ssh-keyscan -p 22 192.168.42.42', proc do
             [0, <<~EO_STDOUT, '']
               # That's a comment
               192.168.42.42 ssh-rsa fake_host_key_rsa
@@ -78,7 +88,7 @@ describe HybridPlatformsConductor::NodesHandler do
     it 'returns host keys sorted' do
       with_test_platform({ nodes: { 'test_node' => {} } }) do
         with_cmd_runner_mocked [
-          ['ssh-keyscan 192.168.42.42', proc do
+          ['ssh-keyscan -p 22 192.168.42.42', proc do
             [0, <<~EO_STDOUT, '']
               192.168.42.42 ssh-dsa fake_host_key_dsa
               192.168.42.42 ssh-rsa fake_host_key_rsa
@@ -100,7 +110,7 @@ describe HybridPlatformsConductor::NodesHandler do
     it 'does not return host keys when ssh-keyscan can\'t retrieve them' do
       with_test_platform({ nodes: { 'test_node' => {} } }) do
         with_cmd_runner_mocked [
-          ['ssh-keyscan 192.168.42.42', proc { [0, '', ''] }]
+          ['ssh-keyscan -p 22 192.168.42.42', proc { [0, '', ''] }]
         ] do
           expect(cmdb(:host_keys).get_host_keys(['test_node'], { 'test_node' => { host_ip: '192.168.42.42' } })).to eq({})
         end
@@ -119,9 +129,9 @@ describe HybridPlatformsConductor::NodesHandler do
         }
       ) do
         with_cmd_runner_mocked [
-          ['ssh-keyscan 192.168.42.1', proc { [0, "192.168.42.1 ssh-rsa fake_host_key_1\n", ''] }],
-          ['ssh-keyscan 192.168.42.2', proc { [0, '', ''] }],
-          ['ssh-keyscan my_host_4.my_domain', proc { [0, "my_host_4.my_domain ssh-rsa fake_host_key_4\n", ''] }]
+          ['ssh-keyscan -p 22 192.168.42.1', proc { [0, "192.168.42.1 ssh-rsa fake_host_key_1\n", ''] }],
+          ['ssh-keyscan -p 22 192.168.42.2', proc { [0, '', ''] }],
+          ['ssh-keyscan -p 22 my_host_4.my_domain', proc { [0, "my_host_4.my_domain ssh-rsa fake_host_key_4\n", ''] }]
         ] do
           expect(
             cmdb(:host_keys).get_host_keys(
@@ -141,6 +151,35 @@ describe HybridPlatformsConductor::NodesHandler do
       end
     end
 
+    it 'returns different host keys for hosts having the same IPs but different SSH ports' do
+      with_test_platform(
+        {
+          nodes: {
+            'test_node1' => {},
+            'test_node2' => {}
+          }
+        }
+      ) do
+        with_cmd_runner_mocked [
+          ['ssh-keyscan -p 6661 192.168.42.1', proc { [0, "192.168.42.1 ssh-rsa fake_host_key_1\n", ''] }],
+          ['ssh-keyscan -p 6662 192.168.42.1', proc { [0, "192.168.42.1 ssh-rsa fake_host_key_2\n", ''] }]
+        ] do
+          expect(
+            cmdb(:host_keys).get_host_keys(
+              ['test_node'],
+              {
+                'test_node1' => { host_ip: '192.168.42.1', ssh_port: 6661 },
+                'test_node2' => { host_ip: '192.168.42.1', ssh_port: 6662 }
+              }
+            )
+          ).to eq(
+            'test_node1' => ['ssh-rsa fake_host_key_1'],
+            'test_node2' => ['ssh-rsa fake_host_key_2']
+          )
+        end
+      end
+    end
+
   end
 
 end

data/spec/hybrid_platforms_conductor_test/api/platform_handlers/serverless_chef/services_deployment_spec.rb

@@ -46,7 +46,7 @@ describe HybridPlatformsConductor::HpcPlugins::PlatformHandler::ServerlessChef d
           remote_bash: [
             'set -e',
             'set -o pipefail',
-            "if [ -n \"$(command -v apt)\" ]; then #{sudo}apt update && #{sudo}apt install -y curl build-essential ; else #{sudo}yum groupinstall 'Development Tools' && #{sudo}yum install -y curl ; fi",
+            "if [ -n \"$(command -v apt)\" ]; then #{sudo}apt update && #{sudo}apt install -y curl build-essential expect ; else #{sudo}yum groupinstall 'Development Tools' && #{sudo}yum install -y curl expect ; fi",
             'mkdir -p ./hpc_deploy',
             'rm -rf ./hpc_deploy/tmp',
             'mkdir -p ./hpc_deploy/tmp',
@@ -57,16 +57,22 @@ describe HybridPlatformsConductor::HpcPlugins::PlatformHandler::ServerlessChef d
         },
         {
           scp: { "#{repository}/dist/#{env}/#{policy}" => './hpc_deploy' },
-          remote_bash: [
-            'set -e',
-            "cd ./hpc_deploy/#{policy}"
-          ] +
-            gems_install_cmds.map { |gem_install_cmd| "#{sudo}SSL_CERT_DIR=/etc/ssl/certs /opt/chef/embedded/bin/#{gem_install_cmd}" } +
-            [
-              "#{sudo}SSL_CERT_DIR=/etc/ssl/certs /opt/chef/bin/chef-client --local-mode --chef-license accept --json-attributes nodes/#{node}.json#{check_mode ? ' --why-run' : ''}",
-              'cd ..',
-              "#{sudo}rm -rf ./hpc_deploy/#{policy}"
-            ]
+          remote_bash: {
+            commands: [
+              'set -e',
+              "cd ./hpc_deploy/#{policy}"
+            ] +
+              gems_install_cmds.map { |gem_install_cmd| "#{sudo}/opt/chef/embedded/bin/#{gem_install_cmd}" } +
+              [
+                "#{sudo}unbuffer /opt/chef/bin/chef-client --local-mode --chef-license accept --json-attributes nodes/#{node}.json#{check_mode ? ' --why-run' : ''}",
+                'cd ..',
+                "#{sudo}rm -rf ./hpc_deploy/#{policy}"
+              ],
+            env: {
+              'SSL_CERT_DIR' => '/etc/ssl/certs',
+              'TERM' => 'xterm-256color'
+            }
+          }
         }
       ]
     end
@@ -323,7 +329,9 @@ describe HybridPlatformsConductor::HpcPlugins::PlatformHandler::ServerlessChef d
               bash: [
                 'set -e',
                 "cd #{repository}/dist/prod/test_policy_1",
-                'sudo SSL_CERT_DIR=/etc/ssl/certs /opt/chef-workstation/bin/chef-client --local-mode --chef-license accept --json-attributes nodes/local.json'
+                'export SSL_CERT_DIR=/etc/ssl/certs',
+                'export TERM=xterm-256color',
+                'sudo -E /opt/chef-workstation/bin/chef-client --local-mode --chef-license accept --json-attributes nodes/local.json'
               ]
             }
           ]
@@ -352,10 +360,12 @@ describe HybridPlatformsConductor::HpcPlugins::PlatformHandler::ServerlessChef d
               bash: [
                 'set -e',
                 "cd #{repository}/dist/prod/test_policy_1",
-                'sudo SSL_CERT_DIR=/etc/ssl/certs /opt/chef-workstation/bin/chef gem install my_gem_1 --version "0.0.1"',
-                'sudo SSL_CERT_DIR=/etc/ssl/certs /opt/chef-workstation/bin/chef gem install my_gem_2 --version "0.0.2"',
-                'sudo SSL_CERT_DIR=/etc/ssl/certs /opt/chef-workstation/bin/chef gem install my_gem_3 --version "~> 1.3"',
-                'sudo SSL_CERT_DIR=/etc/ssl/certs /opt/chef-workstation/bin/chef-client --local-mode --chef-license accept --json-attributes nodes/local.json'
+                'export SSL_CERT_DIR=/etc/ssl/certs',
+                'export TERM=xterm-256color',
+                'sudo -E /opt/chef-workstation/bin/chef gem install my_gem_1 --version "0.0.1"',
+                'sudo -E /opt/chef-workstation/bin/chef gem install my_gem_2 --version "0.0.2"',
+                'sudo -E /opt/chef-workstation/bin/chef gem install my_gem_3 --version "~> 1.3"',
+                'sudo -E /opt/chef-workstation/bin/chef-client --local-mode --chef-license accept --json-attributes nodes/local.json'
               ]
             }
           ]

data/spec/hybrid_platforms_conductor_test/helpers/connector_ssh_helpers.rb

@@ -11,6 +11,7 @@ module HybridPlatformsConductorTest
       # * *nodes_connections* (Hash<String, Hash<Symbol,Object> >): Nodes' connections info, per node name:
       #   * *connection* (String): Connection string (fqdn, IP...) used by SSH
       #   * *ip* (String): IP used by SSH (can be different from connection in case of transformed SSH) [default: connection]
+      #   * *port* (Integer): SSH port used [default: 22]
       #   * *user* (String): User used by SSH
       #   * *times* (Integer): Number of times this connection should be used [default: 1]
       #   * *control_master_create_error* (String or nil): Error to simulate during the SSH ControlMaster creation, or nil for none [default: nil]
@@ -37,6 +38,7 @@ module HybridPlatformsConductorTest
       )
         nodes_connections.map do |node, node_connection_info|
           node_connection_info[:times] = 1 unless node_connection_info.key?(:times)
+          node_connection_info[:port] = 22 unless node_connection_info.key?(:port)
           ssh_commands_once = []
           ssh_commands_per_connection = []
           if with_strict_host_key_checking
@@ -44,7 +46,7 @@ module HybridPlatformsConductorTest
             ssh_commands_once.concat(
               [
                 [
-                  "ssh-keyscan #{ip}",
+                  "ssh-keyscan -p #{node_connection_info[:port]} #{ip}",
                   proc { [0, "#{ip} ssh-rsa fake_host_key_for_#{ip}", ''] }
                 ]
               ]
@@ -67,7 +69,7 @@ module HybridPlatformsConductorTest
                 %r{^xterm -e '.+/ssh -o ControlMaster=yes -o ControlPersist=yes hpc\.#{Regexp.escape(node)}'$}
               end,
               proc do
-                control_file = test_actions_executor.connector(:ssh).send(:control_master_file, node_connection_info[:connection], '22', node_connection_info[:user])
+                control_file = test_actions_executor.connector(:ssh).send(:control_master_file, node_connection_info[:connection], node_connection_info[:port].to_s, node_connection_info[:user])
                 # Fail if the ControlMaster file already exists, as would SSH do if the file is stalled
                 if File.exist?(control_file)
                   [255, '', "Control file #{control_file} already exists"]
@@ -97,7 +99,7 @@ module HybridPlatformsConductorTest
               %r{^.+/ssh -O exit hpc\.#{Regexp.escape(node)} 2>&1 \| grep -v 'Exit request sent\.'$},
               proc do
                 # Really mock the control file deletion
-                File.unlink(test_actions_executor.connector(:ssh).send(:control_master_file, node_connection_info[:connection], '22', node_connection_info[:user]))
+                File.unlink(test_actions_executor.connector(:ssh).send(:control_master_file, node_connection_info[:connection], node_connection_info[:port].to_s, node_connection_info[:user]))
                 [1, '', '']
               end,
               { optional: with_control_master_destroy_optional }

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hybrid_platforms_conductor
 version: !ruby/object:Gem::Version
-  version: 33.5.1
+  version: 33.7.2
 platform: ruby
 authors:
 - Muriel Salvan
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-07-07 00:00:00.000000000 Z
+date: 2021-07-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: range_operators