hybrid_platforms_conductor 32.18.0 → 33.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b724f6cb69633133e800d381ee1d0bfdd1a1107d7fc388fe1dcbcf423480ca9d
-  data.tar.gz: 3252bc11e083295c2694df60a9af790d8b01cbe851baebbe1d8e75b656e0acd3
+  metadata.gz: bedd336d4e713f15ac516cd61d50cf964c2a25051fbb624e1ca75196ccd9dd39
+  data.tar.gz: 2d75b29868194072733dd273f6d2f17ca001cc167f446105b72381e1620c57a1
 SHA512:
-  metadata.gz: 43793a4b3f8ec9a1353b4c075289dfcd77574b64a481754db68e1affa1c599500a3c29d8fa12b31c13dd69b47c5a2840f52aef80045fbec8be7a363ca4eb90b7
-  data.tar.gz: 0451da97325093d7a59684de1ffc77856f4baf5c3a4c76853e6df4492c7cfb1d52cbf9405c0d44b1d8af643e61f64569fc305ed61204be26f081591bdccb6d43
+  metadata.gz: f4333f83879592e3089ce6cf0e40609efab2435fe9bbecae699462871bab7c3df6b5e44da476b823dfde6f0f1868d1cbd5ca621a076ca147355950b1610b6e38
+  data.tar.gz: e0f21e5ccec863d19c93fa1f36bcf51ce1be1a7b1e7bcdbcf83329914df3b3c852eb43b8af2e178b68f94305584c406007de0d47acc713f8170e7b7bba3da621

data/CHANGELOG.md

@@ -1,3 +1,39 @@
+# [v33.0.4](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.0.3...v33.0.4) (2021-06-18 10:09:57)
+
+## Global changes
+### Patches
+
+* [[Fix(cmd_runner)] [#69] Make sure commands are run in unbundled env](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/0795c1ffdcf1b355c9dc91423bb6b7d88918fcf7)
+
+## Changes for cmd_runner
+### Patches
+
+* [[Fix(cmd_runner)] [#69] Make sure commands are run in unbundled env](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/0795c1ffdcf1b355c9dc91423bb6b7d88918fcf7)
+
+# [v33.0.3](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.0.2...v33.0.3) (2021-06-17 12:42:32)
+
+### Patches
+
+* [[#49] Corrected minor warnings and added tests that no warning occurs anymore in rspec](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/57e9936da05431ca469a98d7f079211d9278574f)
+
+# [v33.0.2](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.0.1...v33.0.2) (2021-06-17 11:15:29)
+
+### Patches
+
+* [[#49] Corrected warnings to prepare migration to Ruby 3](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/8f3e758a881dcd988540d660b2df8a38fe39d1ca)
+
+# [v33.0.1](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.0.0...v33.0.1) (2021-06-16 16:22:41)
+
+### Patches
+
+* [[#49] Added lint checks in CI and corrected all non acceptable lint errors](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/b3188532e679f845f66966e7c8054ce8db63be26)
+
+# [v33.0.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.18.0...v33.0.0) (2021-06-15 16:10:47)
+
+### Breaking changes
+
+* [[Breaking] Add secrets reader plugins with 2 default plugins: cli and thycotic](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/2cfacebe8cfac57de40ef003877da5b99aca5b5e)
+
 # [v32.18.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.17.1...v32.18.0) (2021-06-14 15:01:02)
 
 ## Global changes

data/README.md

@@ -216,13 +216,13 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
 All executables also have the `--debug` switch to display more verbose and debugging information.

data/bin/check-node

@@ -12,7 +12,6 @@ executable = HybridPlatformsConductor::Executable.new(
   end
 end
 nodes_handler = executable.nodes_handler
-actions_executor = executable.actions_executor
 deployer = executable.deployer
 
 executable.parse_options!

data/bin/deploy

@@ -2,7 +2,6 @@
 require 'hybrid_platforms_conductor/executable'
 
 executable = HybridPlatformsConductor::Executable.new
-actions_executor = executable.actions_executor
 deployer = executable.deployer
 
 executable.parse_options!

data/bin/get_impacted_nodes

@@ -22,7 +22,6 @@ executable = HybridPlatformsConductor::Executable.new(nodes_selection_options: f
 end
 nodes_handler = executable.nodes_handler
 platforms_handler = executable.platforms_handler
-cmd_runner = executable.cmd_runner
 executable.parse_options!
 raise 'No platform specified. Please use --platform option' if platform.nil?
 

data/bin/last_deploys

@@ -8,12 +8,12 @@ possible_string_sorts = %i[
   node
   user
 ]
-possible_sorts = Hash[possible_string_sorts.map do |property_name|
+possible_sorts = possible_string_sorts.map do |property_name|
   [
     property_name,
     proc { |node, deploy_info| [deploy_info.key?(:error) || !deploy_info.key?(property_name) ? '' : deploy_info[property_name], node] }
   ]
-end]
+end.to_h
 
 sort_by = :node
 sort_desc = false
@@ -21,7 +21,7 @@ executable = HybridPlatformsConductor::Executable.new(deploy_options: false) do
   opts.on('-r', '--sort-by SORT', "Specify a sort. Possible values are: #{possible_sorts.keys.sort.join(', ')}. Each value can append _desc to specify a reverse sorting. Defaults to #{sort_by}.") do |sort_name|
     if sort_name =~ /^(.+)_desc$/
       sort_desc = true
-      sort_name = $1
+      sort_name = Regexp.last_match(1)
     end
     sort_by = sort_name.to_sym
   end
@@ -32,8 +32,8 @@ deployer = executable.deployer
 executable.parse_options!
 raise "Unknown sort name: #{sort_by}. Should be one of #{possible_sorts.keys.join(', ')}." unless possible_sorts.key?(sort_by)
 
-sorted_deploy_info = Hash[
-  deployer.deployment_info_from(nodes_handler.select_nodes(executable.selected_nodes.empty? ? [{ all: true }] : executable.selected_nodes)).
+sorted_deploy_info = deployer.
+  deployment_info_from(nodes_handler.select_nodes(executable.selected_nodes.empty? ? [{ all: true }] : executable.selected_nodes)).
   map do |node, deploy_info|
     decorated_deploy_info = deploy_info.merge(node: node)
     if deploy_info.key?(:deployment_info)
@@ -42,8 +42,9 @@ sorted_deploy_info = Hash[
     end
     decorated_deploy_info[:services] = deploy_info[:services].join(', ') if deploy_info.key?(:services)
     [node, decorated_deploy_info]
-  end
-].sort_by(&possible_sorts[sort_by])
+  end.
+  to_h.
+  sort_by(&possible_sorts[sort_by])
 sorted_deploy_info.reverse! if sort_desc
 info_displayed = {
   node: 'Node',

data/bin/nodes_to_deploy

@@ -21,7 +21,7 @@ deployer = executable.deployer
 
 executable.parse_options!
 
-nodes = (nodes_handler.select_nodes(executable.selected_nodes.empty? ? [{ all: true }] : executable.selected_nodes))
+nodes = nodes_handler.select_nodes(executable.selected_nodes.empty? ? [{ all: true }] : executable.selected_nodes)
 
 unless ignore_schedule
   # Select nodes to be deployed, based first on deployment schedule
@@ -67,7 +67,7 @@ unless ignore_deploy_info
       node_impacted = false
       # Loop over all possible repositories concerned by this deployment
       repo_idx = 0
-      while node_deploy_info[:deployment_info].key?("repo_name_#{repo_idx}".to_sym) do
+      while node_deploy_info[:deployment_info].key?("repo_name_#{repo_idx}".to_sym)
         repo_name = node_deploy_info[:deployment_info]["repo_name_#{repo_idx}".to_sym]
         commit_id = node_deploy_info[:deployment_info]["commit_id_#{repo_idx}".to_sym]
         impacted_nodes = cache_impacted_nodes.dig(repo_name, commit_id)

data/bin/setup

@@ -7,10 +7,10 @@ platforms_handler = executable.platforms_handler
 executable.parse_options!
 
 platforms_handler.known_platforms.each do |platform|
-  if platform.respond_to?(:setup)
-    executable.out "===== Setup platform #{platform.name}..."
-    platform.setup
-    executable.out "===== Platform #{platform.name} setup successfully."
-    executable.out ''
-  end
+  next unless platform.respond_to?(:setup)
+
+  executable.out "===== Setup platform #{platform.name}..."
+  platform.setup
+  executable.out "===== Platform #{platform.name} setup successfully."
+  executable.out ''
 end

data/bin/topograph

@@ -17,7 +17,7 @@ executable.parse_options!
 
 # Initialize the topograph
 from_nodes, to_nodes = topographer.resolve_from_to
-topographer.get_json_files
+topographer.json_files
 
 # The list of clusters to group nodes into 1 graphviz node to simplify the graph.
 known_nodes_lists = nodes_handler.known_nodes_lists

data/docs/config_dsl.md

@@ -2,7 +2,7 @@
 
 The DSL used in configuration files is comprised of Ruby methods that can be called directly in the main `hpc_config.rb` file.
 
-This DSL can also be completed by plugins. Check [the plugins documentations](plugins) to know about DSL extensions brought by plugins.
+This DSL can also be completed by plugins. Check [the plugins documentations](plugins.md) to know about DSL extensions brought by plugins.
 
 # Table of Contents
   * [`<platform_type>_platform`](#platform_type_platform)
@@ -13,6 +13,7 @@ This DSL can also be completed by plugins. Check [the plugins documentations](pl
   * [`hybrid_platforms_dir`](#hybrid_platforms_dir)
   * [`tests_provisioner`](#tests_provisioner)
   * [`expect_tests_to_fail`](#expect_tests_to_fail)
+  * [`read_secrets_from`](#read_secrets_from)
   * [`send_logs_to`](#send_logs_to)
   * [`retry_deploy_for_errors_on_stdout`](#retry_deploy_for_errors_on_stdout)
   * [`retry_deploy_for_errors_on_stderr`](#retry_deploy_for_errors_on_stderr)
@@ -201,6 +202,27 @@ for_nodes('/tst/') do
 end
 ```
 
+<a name="read_secrets_from"></a>
+## `read_secrets_from`
+
+Set the list of [secrets reader plugins](plugins.md#secrets_reader) to use.
+By default (if no plugins is specifically set) the [secrets reader plugin `cli`](plugins/secrets_reader/cli.md) is being used.
+
+Takes the list of secrets reader plugin names, as symbols, as a parameter.
+
+Can be applied to subset of nodes using the [`for_nodes` DSL method](#for_nodes).
+
+Examples:
+```ruby
+# By default, get secrets from the command-line
+read_secrets_from :cli
+
+# All our production nodes also have their secrets stored on a secured Thycotic server
+for_nodes('/prd/') do
+  read_secrets_from :thycotic
+end
+```
+
 <a name="send_logs_to"></a>
 ## `send_logs_to`
 

data/docs/executables.md

@@ -132,25 +132,22 @@ The Deployer options are used to drive a deployment (be it in why-run mode or no
 
 ```
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
-* `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...).
-  The location can be:
-  * A local file path (for example /path/to/file.json).
-  * A Thycotic Secret Server URL followed by a secret id (for example https://portal.muc.msp.my_company.net/SecretServer:8845).
 * `--parallel`: Specify that the deployment process should perform concurrently on the different nodes it has to deploy to.
 * `--timeout SECS`: Specify the timeout (in seconds) to apply while deploying. This can be set only in why-run mode.
 * `--why-run`: Specify the why-run mode. The why-run mode is used to simulate a deployment on the nodes, and report what a real deployment would have changed on the node.
 * `--retries-on-error NBR`: Specify the number of retries deploys can do in case of non-deterministic errors.
   Non-deterministic errors are matched using a set of strings or regular expressions that can be configured in the `hpc_config.rb` file of any platform, using the `retry_deploy_for_errors_on_stdout` and `retry_deploy_for_errors_on_stderr` properties:
   For example:
+
 ```ruby
 retry_deploy_for_errors_on_stdout [
   'This is a raw string error that will be matched against stdout',
@@ -161,6 +158,8 @@ retry_deploy_for_errors_on_stderr [
 ]
 ```
 
+* `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...). This option is used by the [`cli` secrets reader plugin](plugins/secrets_reader/cli.md). See [secrets reader plugins](plugins.md#secrets_reader) for more info about secrets retrieval.
+
 ## JSON dump options
 
 The JSON dump options drive the way nodes' JSON information is being dumped.

data/docs/executables/check-node.md

@@ -65,11 +65,11 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
 ## Examples

data/docs/executables/deploy.md

@@ -82,13 +82,13 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
 ## Examples

data/docs/executables/dump_nodes_json.md

@@ -54,13 +54,13 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
 
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
+
 JSON dump options:
     -k, --skip-run                   Skip the actual gathering of dumps in run_logs. If set, the current run_logs content will be used.
     -j, --json-dir DIRECTORY         Specify the output directory in which JSON files are being written. Defaults to nodes_json.

data/docs/executables/test.md

@@ -93,11 +93,11 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
 
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
+
 Tests runner options:
     -i, --tests-list FILE_NAME       Specify a tests file name. The file should contain a list of tests name (1 per line). Can be used several times.
     -k, --skip-run                   Skip running the check-node commands for real, and just analyze existing run logs.

data/docs/executables/topograph.md

@@ -39,12 +39,12 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
 
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
+
 JSON dump options:
     -j, --json-dir DIRECTORY         Specify the output directory in which JSON files are being written. Defaults to nodes_json.
 

data/docs/plugins.md

@@ -14,6 +14,7 @@ Following are all possible plugin types and the plugins shipped by default with
   * [`platform_handler`](#platform_handler)
   * [`provisioner`](#provisioner)
   * [`report`](#report)
+  * [`secrets_reader`](#secrets_reader)
   * [`test`](#test)
   * [`test_report`](#test_report)
 
@@ -177,6 +178,26 @@ Plugins shipped by default:
 * [`mediawiki`](plugins/report/mediawiki.md)
 * [`stdout`](plugins/report/stdout.md)
 
+<a name="secrets_reader"></a>
+## Secrets readers
+
+Secrets reader are responsible for fetching secrets (passwords, private keys, API tokens...) needed during deployment from various sources (command line, environment, vaults, secrets servers...).
+
+Corresponding plugin type: `secrets_reader`.
+
+These plugins add new ways to retrieve secrets used by the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb)
+
+Examples of secrets readers are:
+* Command-line: Give secrets from a local file.
+* Vault: Get secrets from vaults (encrypted databases).
+* Secrets servers: Query secrets servers to retrieve secrets.
+
+Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/my_secrets_reader_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
+
+Plugins shipped by default:
+* [`cli`](plugins/secrets_reader/cli.md)
+* [`thycotic`](plugins/secrets_reader/thycotic.md)
+
 <a name="test"></a>
 ## Tests
 

data/docs/plugins/secrets_reader/cli.md

@@ -0,0 +1,31 @@
+# Secrets reader plugin: `cli`
+
+The `cli` secrets reader plugin reads secrets from a local JSON file that can be given through the `--secrets` command-line parameter.
+
+Example:
+```bash
+./bin/deploy --node my_node --secrets /path/to/my_secrets.json
+```
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/secrets_reader/thycotic.md

@@ -0,0 +1,46 @@
+# Secrets reader plugin: `thycotic`
+
+The `thycotic` secrets reader plugin retrieves secrets from a [Thycotic secrets server](https://thycotic.com/products/secret-server-vdo/), using its SOAP API.
+
+It is configured using the `secrets_from_thycotic` (see below) config DSL and uses the `thycotic` credential ID to authenticate.
+
+## Config DSL extension
+
+### `secrets_from_thycotic`
+
+Define a Thycotic URL and Thycotic secret ID to fetch from a Thycotic server.
+The Thycotic secret should contain a JSON file that will be retrieved locally to be used as a secrets source. The local copy will then be removed after deployment.
+
+Can be applied to subset of nodes using the [`for_nodes` DSL method](/docs/config_dsl.md#for_nodes).
+
+It takes the following parameters:
+* **thycotic_url** (`String`): The Thycotic server URL.
+* **secret_id** (`Integer`): The Thycotic secret ID containing the secrets file to be used as secrets.
+
+Example:
+```ruby
+secrets_from_thycotic(
+  thycotic_url: 'https://my-thycotic-server.my-domain.com/SecretServer',
+  secret_id: 1107
+)
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+| `thycotic` | Used to authenticate on the Thycotic server's SOAP API |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None