hybrid_platforms_conductor 32.16.4 → 33.0.1

data/docs/plugins.md

@@ -10,9 +10,11 @@ Following are all possible plugin types and the plugins shipped by default with
   * [`action`](#action)
   * [`cmdb`](#cmdb)
   * [`connector`](#connector)
+  * [`log`](#log)
   * [`platform_handler`](#platform_handler)
   * [`provisioner`](#provisioner)
   * [`report`](#report)
+  * [`secrets_reader`](#secrets_reader)
   * [`test`](#test)
   * [`test_report`](#test_report)
 
@@ -79,6 +81,30 @@ Plugins shipped by default:
 * [`local`](plugins/connector/local.md)
 * [`ssh`](plugins/connector/ssh.md)
 
+<a name="log"></a>
+## Logs
+
+Save deployment logs to a given medium (files, log servers...).
+
+Corresponding plugin type: `log`.
+
+These plugins give ways for the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb) to save logs output from services deployments on nodes.
+Information that can be saved is:
+* The deployment stdout.
+* The deployment stderr.
+* The deployment exit status.
+* The list of services that have been deployed.
+* Some deployment metadata (like git commits information that have been deployed).
+
+Examples of log plugins are:
+* Remote file system: Save logs on the node's remote file system (useful for local debugging).
+* Log servers: Send logs to log servers.
+
+Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/log/my_log_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
+
+Plugins shipped by default:
+* [`remote_fs`](plugins/log/remote_fs.md)
+
 <a name="platform_handler"></a>
 ## Platform Handlers
 
@@ -152,6 +178,26 @@ Plugins shipped by default:
 * [`mediawiki`](plugins/report/mediawiki.md)
 * [`stdout`](plugins/report/stdout.md)
 
+<a name="secrets_reader"></a>
+## Secrets readers
+
+Secrets reader are responsible for fetching secrets (passwords, private keys, API tokens...) needed during deployment from various sources (command line, environment, vaults, secrets servers...).
+
+Corresponding plugin type: `secrets_reader`.
+
+These plugins add new ways to retrieve secrets used by the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb)
+
+Examples of secrets readers are:
+* Command-line: Give secrets from a local file.
+* Vault: Get secrets from vaults (encrypted databases).
+* Secrets servers: Query secrets servers to retrieve secrets.
+
+Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/my_secrets_reader_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
+
+Plugins shipped by default:
+* [`cli`](plugins/secrets_reader/cli.md)
+* [`thycotic`](plugins/secrets_reader/thycotic.md)
+
 <a name="test"></a>
 ## Tests
 
@@ -182,6 +228,7 @@ Plugins shipped by default:
 * [`executables`](plugins/test/executables.md)
 * [`file_system_hdfs`](plugins/test/file_system_hdfs.md)
 * [`file_system`](plugins/test/file_system.md)
+* [`github_ci`](plugins/test/github_ci.md)
 * [`hostname`](plugins/test/hostname.md)
 * [`idempotence`](plugins/test/idempotence.md)
 * [`ip`](plugins/test/ip.md)

data/docs/plugins/connector/ssh.md

@@ -70,7 +70,7 @@ end
 | `host_keys` | `Array<String>` | The node's host keys used to generate a `known_hosts` file with those to avoid user confirmations when connecting. |
 | `hostname` | `String` | Host name used to connect in case no IP address can be found in metadata. |
 | `private_ips` | `Array<String>` | IP list to connect in case `host_ip` is not defined in metadata. |
-| `ssh_session_exec` | `String` | If set to the string `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
+| `ssh_session_exec` | `Boolean` | If set to `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
 
 ## Used environment variables
 

data/docs/plugins/log/remote_fs.md

@@ -0,0 +1,26 @@
+# Log plugin: `remote_fs`
+
+The `remote_fs` log plugin saves deployment logs in each remote node that has been deployed, in the `/var/log/deployments` directory.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/secrets_reader/cli.md

@@ -0,0 +1,31 @@
+# Secrets reader plugin: `cli`
+
+The `cli` secrets reader plugin reads secrets from a local JSON file that can be given through the `--secrets` command-line parameter.
+
+Example:
+```bash
+./bin/deploy --node my_node --secrets /path/to/my_secrets.json
+```
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/secrets_reader/thycotic.md

@@ -0,0 +1,46 @@
+# Secrets reader plugin: `thycotic`
+
+The `thycotic` secrets reader plugin retrieves secrets from a [Thycotic secrets server](https://thycotic.com/products/secret-server-vdo/), using its SOAP API.
+
+It is configured using the `secrets_from_thycotic` (see below) config DSL and uses the `thycotic` credential ID to authenticate.
+
+## Config DSL extension
+
+### `secrets_from_thycotic`
+
+Define a Thycotic URL and Thycotic secret ID to fetch from a Thycotic server.
+The Thycotic secret should contain a JSON file that will be retrieved locally to be used as a secrets source. The local copy will then be removed after deployment.
+
+Can be applied to subset of nodes using the [`for_nodes` DSL method](/docs/config_dsl.md#for_nodes).
+
+It takes the following parameters:
+* **thycotic_url** (`String`): The Thycotic server URL.
+* **secret_id** (`Integer`): The Thycotic secret ID containing the secrets file to be used as secrets.
+
+Example:
+```ruby
+secrets_from_thycotic(
+  thycotic_url: 'https://my-thycotic-server.my-domain.com/SecretServer',
+  secret_id: 1107
+)
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+| `thycotic` | Used to authenticate on the Thycotic server's SOAP API |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/bitbucket_conf.md

@@ -12,7 +12,7 @@ Define a Bitbucket installation to be targeted.
 It takes the following parameters:
 * **url** (`String`): URL to the Bitbucket server
 * **project** (`String`): Project name from the Bitbucket server, storing repositories
-* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or :all for all [default: :all]
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
 * **checks** (`Hash<Symbol, Object>`): Checks definition to be perform on those repositories [default: {}]
   * **branch_permissions** (`Array< Hash<Symbol, Object> >`): List of branch permissions to check [optional]
     * **type** (`String`): Type of branch permissions to check. Examples of values are 'fast-forward-only', 'no-deletes', 'pull-request-only'.

data/docs/plugins/test/check_deploy_and_idempotence.md

@@ -49,7 +49,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
-| `root_access_allowed` | `String` | If set to `true`, then skip the test for `root` access being disabled after deployment |
+| `root_access_allowed` | `Boolean` | If set to `true`, then skip the test for `root` access being disabled after deployment |
 
 ## Used environment variables
 

data/docs/plugins/test/connection.md

@@ -16,6 +16,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/deploy_removes_root_access.md

@@ -17,7 +17,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
-| `root_access_allowed` | `String` | If set to `true`, then skip this test |
+| `root_access_allowed` | `Boolean` | If set to `true`, then skip this test |
 
 ## Used environment variables
 

data/docs/plugins/test/file_system.md

@@ -38,6 +38,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/github_ci.md

@@ -0,0 +1,48 @@
+# Test plugin: `github_ci`
+
+The `github_ci` test plugin checks that the `master` branch of Github repositories has a successful CI result from its [Github Actions](https://github.com/features/actions).
+
+## Config DSL extension
+
+### `github_repos`
+
+Define Github repositories to be targeted.
+
+It takes the following parameters:
+* **url** (`String`): URL to the Github API [default: `'https://api.github.com'`]
+* **user** (`String`): User or organization name, storing repositories
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
+
+Example:
+```ruby
+github_repos(
+  # Github's user containing repositories
+  user: 'My-Github-User',
+  # List of repositories to check
+  repos: [
+    'my-platform-repo',
+    'my-chef-repo',
+    'my-hpc-plugins'
+  ]
+)
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+| `github` | Used to connect to the Github API. Password should be the Github API token. |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/hostname.md

@@ -16,6 +16,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/ip.md

@@ -16,6 +16,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 | `private_ips` | `Array<String>` | List of possible private IPs the node should have |
 
 ## Used environment variables

data/docs/plugins/test/jenkins_ci_conf.md

@@ -12,7 +12,7 @@ It takes the following parameters:
 * **url** (`String`): URL to the Bitbucket server
 * **project** (`String`): Project name from the Bitbucket server, storing repositories
 * **jenkins_ci_url** (`String` or `nil`): Corresponding Jenkins CI URL, or nil if none.
-* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or :all for all [default: :all]
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
 
 Example:
 ```ruby

data/docs/plugins/test/jenkins_ci_masters_ok.md

@@ -12,7 +12,7 @@ It takes the following parameters:
 * **url** (`String`): URL to the Bitbucket server
 * **project** (`String`): Project name from the Bitbucket server, storing repositories
 * **jenkins_ci_url** (`String` or `nil`): Corresponding Jenkins CI URL, or nil if none.
-* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or :all for all [default: :all]
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
 
 Example:
 ```ruby

data/docs/plugins/test/local_users.md

@@ -37,6 +37,7 @@ check_local_users_do_not_exist %w[olduser1 olduser2]
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/mounts.md

@@ -44,6 +44,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/orphan_files.md

@@ -27,6 +27,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/ports.md

@@ -39,6 +39,7 @@ check_closed_ports 25, 110
 | Metadata | Type | Usage
 | --- | --- | --- |
 | `host_ip` | `String` | Host IP address to be tested for port listening |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/spectre.md

@@ -15,6 +15,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/vulnerabilities.md

@@ -54,6 +54,7 @@ None
 | Metadata | Type | Usage
 | --- | --- | --- |
 | `image` | `String` | The name of the OS image to be used. The [configuration](../../config_dsl.md) should define the image and point it to a directory containing a `oval.json` that will contain definition of OVAL files to be checked for this OS(see above). |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/lib/hybrid_platforms_conductor/action.rb

@@ -16,8 +16,8 @@ module HybridPlatformsConductor
     # * *actions_executor* (ActionsExecutor): Actions Executor to be used. [default: ActionsExecutor.new]
     # * *action_info* (Object or nil): Action info needed to setup the action, or nil if none [default: nil]
     def initialize(
-      logger: Logger.new(STDOUT),
-      logger_stderr: Logger.new(STDERR),
+      logger: Logger.new($stdout),
+      logger_stderr: Logger.new($stderr),
       config: Config.new,
       cmd_runner: CmdRunner.new,
       actions_executor: ActionsExecutor.new,
@@ -27,7 +27,7 @@ module HybridPlatformsConductor
       @cmd_runner = cmd_runner
       @actions_executor = actions_executor
       @action_info = action_info
-      setup(@action_info) if self.respond_to?(:setup)
+      setup(@action_info) if respond_to?(:setup)
     end
 
     # Do we need a connector to execute this action on a node?
@@ -53,7 +53,7 @@ module HybridPlatformsConductor
       @timeout = timeout
       @stdout_io = stdout_io
       @stderr_io = stderr_io
-      @connector.prepare_for(@node, @timeout, @stdout_io, @stderr_io) if @connector
+      @connector&.prepare_for(@node, @timeout, @stdout_io, @stderr_io)
     end
 
     private

data/lib/hybrid_platforms_conductor/actions_executor.rb

@@ -35,7 +35,7 @@ module HybridPlatformsConductor
     # * *config* (Config): Config to be used. [default = Config.new]
     # * *cmd_runner* (CmdRunner): Command runner to be used. [default = CmdRunner.new]
     # * *nodes_handler* (NodesHandler): Nodes handler to be used. [default = NodesHandler.new]
-    def initialize(logger: Logger.new(STDOUT), logger_stderr: Logger.new(STDERR), config: Config.new, cmd_runner: CmdRunner.new, nodes_handler: NodesHandler.new)
+    def initialize(logger: Logger.new($stdout), logger_stderr: Logger.new($stderr), config: Config.new, cmd_runner: CmdRunner.new, nodes_handler: NodesHandler.new)
       init_loggers(logger, logger_stderr)
       @config = config
       @cmd_runner = cmd_runner
@@ -74,17 +74,17 @@ module HybridPlatformsConductor
       end
       # Display options connectors might have
       @connector_plugins.each do |connector_name, connector|
-        if connector.respond_to?(:options_parse)
-          options_parser.separator ''
-          options_parser.separator "Connector #{connector_name} options:"
-          connector.options_parse(options_parser)
-        end
+        next unless connector.respond_to?(:options_parse)
+
+        options_parser.separator ''
+        options_parser.separator "Connector #{connector_name} options:"
+        connector.options_parse(options_parser)
       end
     end
 
     # Validate that parsed parameters are valid
     def validate_params
-      @connector_plugins.values.each do |connector|
+      @connector_plugins.each_value do |connector|
         connector.validate_params if connector.respond_to?(:validate_params)
       end
     end
@@ -102,7 +102,14 @@ module HybridPlatformsConductor
     # * *progress_name* (String): Name to display on the progress bar [default: 'Executing actions']
     # Result::
     # * Hash<String, [Integer or Symbol, String, String]>: Exit status code (or Symbol in case of error or dry run), standard output and error for each node.
-    def execute_actions(actions_per_nodes, timeout: nil, concurrent: false, log_to_dir: "#{@config.hybrid_platforms_dir}/run_logs", log_to_stdout: true, progress_name: 'Executing actions')
+    def execute_actions(
+      actions_per_nodes,
+      timeout: nil,
+      concurrent: false,
+      log_to_dir: "#{@config.hybrid_platforms_dir}/run_logs",
+      log_to_stdout: true,
+      progress_name: 'Executing actions'
+    )
       # Keep a list of nodes that will need remote access
       nodes_needing_connectors = []
       # Compute the ordered list of actions per selected node
@@ -117,6 +124,7 @@ module HybridPlatformsConductor
           nodes_actions_set.each do |action_type, action_info|
             raise 'Cannot have concurrent executions for interactive sessions' if concurrent && action_type == :interactive && action_info
             raise "Unknown action type #{action_type}" unless @action_plugins.key?(action_type)
+
             action = @action_plugins[action_type].new(
               logger: @logger,
               logger_stderr: @logger_stderr,
@@ -137,7 +145,7 @@ module HybridPlatformsConductor
           actions_per_node[node].concat(resolved_nodes_actions)
         end
       end
-      result = Hash[actions_per_node.keys.map { |node| [node, nil] }]
+      result = actions_per_node.keys.map { |node| [node, nil] }.to_h
       with_connections_prepared_to(nodes_needing_connectors, no_exception: true) do |connected_nodes|
         missing_nodes = []
         connected_nodes.each do |node, connector|
@@ -151,7 +159,7 @@ module HybridPlatformsConductor
         # Prepare the result (stdout or nil per node)
         unless accessible_nodes.empty?
           # If we run in parallel then clone the connectors, so that each node has its own instance for thread-safe code.
-          connected_nodes = Hash[connected_nodes.map { |node, connector| [node, connector.clone] }] if concurrent
+          connected_nodes = connected_nodes.transform_values(&:clone) if concurrent
           @nodes_handler.for_each_node_in(
             accessible_nodes,
             parallel: concurrent,
@@ -185,10 +193,11 @@ module HybridPlatformsConductor
     #     * *connected_nodes* (Hash<String, Connector or Symbol>): Prepared connectors (or Symbol in case of failure with no_exception), per node name
     def with_connections_prepared_to(nodes, no_exception: false)
       # Make sure every node needing connectors finds a connector
-      nodes_needing_connectors = Hash[nodes.map { |node| [node, nil] }]
-      @connector_plugins.each do |connector_name, connector|
+      nodes_needing_connectors = nodes.map { |node| [node, nil] }.to_h
+      @connector_plugins.each_value do |connector|
         nodes_without_connectors = nodes_needing_connectors.select { |_node, selected_connector| selected_connector.nil? }.keys
         break if nodes_without_connectors.empty?
+
         (connector.connectable_nodes_from(nodes_without_connectors) & nodes_without_connectors).each do |node|
           nodes_needing_connectors[node] = connector if nodes_needing_connectors[node].nil?
         end
@@ -206,23 +215,20 @@ module HybridPlatformsConductor
         if connector_name.nil?
           # All plugins have been prepared.
           # Call our client code.
-          yield Hash[nodes_needing_connectors.map do |node, selected_connector|
-            [
-              node,
-              selected_connector.nil? ? :no_connector : selected_connector
-            ]
-          end]
+          yield(nodes_needing_connectors.transform_values do |selected_connector|
+            selected_connector.nil? ? :no_connector : selected_connector
+          end)
         else
           connector = @connector_plugins[connector_name]
           selected_nodes = nodes_needing_connectors.select { |_node, selected_connector| selected_connector == connector }.keys
           if selected_nodes.empty?
-            preparation_code.call(remaining_plugins_to_prepare[1..-1])
+            preparation_code.call(remaining_plugins_to_prepare[1..])
           else
             connector.with_connection_to(selected_nodes, no_exception: no_exception) do |connected_nodes|
               (selected_nodes - connected_nodes).each do |node_in_error|
                 nodes_needing_connectors[node_in_error] = :connection_error
               end
-              preparation_code.call(remaining_plugins_to_prepare[1..-1])
+              preparation_code.call(remaining_plugins_to_prepare[1..])
             end
           end
         end
@@ -262,8 +268,6 @@ module HybridPlatformsConductor
         if log_to_file
           FileUtils.mkdir_p(File.dirname(log_to_file))
           File.open(log_to_file, 'w')
-        else
-          nil
         end
       stdout_queue = Queue.new
       stderr_queue = Queue.new
@@ -277,28 +281,26 @@ module HybridPlatformsConductor
           (log_to_stdout ? [@logger_stderr] : []) +
           (file_output.nil? ? [] : [file_output])
       ) do
-        begin
-          log_debug "[#{node}] - Execute #{actions.size} actions on #{node}..."
-          actions.each do |action|
-            action.prepare_for(node, connector, remaining_timeout, stdout_queue, stderr_queue)
-            start_time = Time.now
-            action.execute
-            remaining_timeout -= Time.now - start_time unless remaining_timeout.nil?
-          end
-        rescue ConnectionError
-          exit_status = :connection_error
-          stderr_queue << "#{$!}\n"
-        rescue CmdRunner::UnexpectedExitCodeError
-          exit_status = :failed_command
-          stderr_queue << "#{$!}\n"
-        rescue CmdRunner::TimeoutError
-          # Error has already been logged in stderr
-          exit_status = :timeout
-        rescue
-          log_error "Uncaught exception while executing actions on #{node}: #{$!}\n#{$!.backtrace.join("\n")}"
-          stderr_queue << "#{$!}\n"
-          exit_status = :failed_action
+        log_debug "[#{node}] - Execute #{actions.size} actions on #{node}..."
+        actions.each do |action|
+          action.prepare_for(node, connector, remaining_timeout, stdout_queue, stderr_queue)
+          start_time = Time.now
+          action.execute
+          remaining_timeout -= Time.now - start_time unless remaining_timeout.nil?
         end
+      rescue ConnectionError
+        exit_status = :connection_error
+        stderr_queue << "#{$ERROR_INFO}\n"
+      rescue CmdRunner::UnexpectedExitCodeError
+        exit_status = :failed_command
+        stderr_queue << "#{$ERROR_INFO}\n"
+      rescue CmdRunner::TimeoutError
+        # Error has already been logged in stderr
+        exit_status = :timeout
+      rescue
+        log_error "Uncaught exception while executing actions on #{node}: #{$ERROR_INFO}\n#{$ERROR_INFO.backtrace.join("\n")}"
+        stderr_queue << "#{$ERROR_INFO}\n"
+        exit_status = :failed_action
       end
       [exit_status, stdout, stderr]
     end