hybrid_platforms_conductor 32.16.3 → 33.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 22f6bf2ee85b7db6649e517503f00dd71f0604d4afb3a12bee347820af4d90e2
-  data.tar.gz: 7804ffa80bb56697c98970b1d781a5e3be12b27bc8c38a3df9eb26b666791ed8
+  metadata.gz: ae17e600c52586447071f6f463ee1525754327c0be4a147bec7d3ebcb13b018d
+  data.tar.gz: 9e4dc83a6d37fc9c17b99fd9a16eae20b73673dd492dde52679a159ed922a3d5
 SHA512:
-  metadata.gz: a89d17b28222ca1e32a132d42167a9b7b78e59b8ddb281f2805200e2a6a356e6547ee316fdacedc2c00d53ee14956027ea17b0a0424bd7fd8ad3260ba071d873
-  data.tar.gz: b29e79b9d21b8b9019307918c05c73c8d422c5fb942e6f3dc0c9fd317c293430e7a4658ed5551503f9bbf9fdf7bd17394904e0e7aabcb664101b555309c6b912
+  metadata.gz: d9b89fc3f6ed1fe6e87f23aa1519b32dc62bdcf861b0332088909b1ac8945adf273425c97860c0df5131069921c6adfed60d265cbfe9c1d50a6e4cd5099cc420
+  data.tar.gz: d9b3d6d66250552aaec66672b2b9fc97843eb39fc5578a34b74c44f9a4511bccf7978ef59559a4a21b276557e6293418e3350b8151308000d478e22aba84ab56

data/CHANGELOG.md

@@ -1,3 +1,45 @@
+# [v33.0.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.18.0...v33.0.0) (2021-06-15 16:10:47)
+
+### Breaking changes
+
+* [[Breaking] Add secrets reader plugins with 2 default plugins: cli and thycotic](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/2cfacebe8cfac57de40ef003877da5b99aca5b5e)
+
+# [v32.18.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.17.1...v32.18.0) (2021-06-14 15:01:02)
+
+## Global changes
+### Patches
+
+* [[Feature(log_remote_fs)] [#60] Add new plugins type log with a first log plugin remote_fs to extend deployment logs save functionality](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/20187c6da577c932b5205204187af883140995fe)
+
+## Changes for log_remote_fs
+### Features
+
+* [[Feature(log_remote_fs)] [#60] Add new plugins type log with a first log plugin remote_fs to extend deployment logs save functionality](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/20187c6da577c932b5205204187af883140995fe)
+
+# [v32.17.1](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.17.0...v32.17.1) (2021-06-03 16:20:09)
+
+### Patches
+
+* [[Hotfix] Don't run remote nodes tests on local nodes](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/fe8e34a4d74f21d4903c14b1e156b9730f4b5fee)
+
+# [v32.17.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.16.4...v32.17.0) (2021-06-02 12:57:44)
+
+## Global changes
+### Patches
+
+* [[Feature(test_github_ci)] [#61] Add the github_ci test plugin to check for CI/CD of projects on Github](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/a0082efe5c138ca88ca91a3c10f777f47fca5034)
+
+## Changes for test_github_ci
+### Features
+
+* [[Feature(test_github_ci)] [#61] Add the github_ci test plugin to check for CI/CD of projects on Github](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/a0082efe5c138ca88ca91a3c10f777f47fca5034)
+
+# [v32.16.4](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.16.3...v32.16.4) (2021-06-01 13:25:19)
+
+### Patches
+
+* [[Hotfix] Fixed boolean values incorrectly used as strings](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/a107f3e0fe4c1512ce7607a303fdf0753d92ddac)
+
 # [v32.16.3](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.16.2...v32.16.3) (2021-06-01 11:19:50)
 
 ## Global changes

data/README.md

@@ -216,13 +216,13 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
 All executables also have the `--debug` switch to display more verbose and debugging information.
@@ -263,6 +263,7 @@ participant PlatformHandler as Platform Handler
 participant PlatformRepo as Platform repository (ie Chef)
 participant Connector as Connector (ie SSH)
 participant Node as Provisioned node (my_node)
+participant Log as Log
 
 Deploy->>+CMDB: Get services to be deployed on my_node
 CMDB->>+PlatformHandler: Get my_node metadata from the platform
@@ -276,6 +277,8 @@ Deploy->>+Connector: Connect to my_node to execute actions
 Connector->>+Node: Execute actions through SSH to deploy my_web_app on my_node
 Node-->>-Connector: my_web_app is deployed successfully
 Connector-->>-Deploy: Close connection
+Deploy->>+Log: Save deployment logs
+Log-->>-Deploy: Deployment logs saved
 ```
 </details>
 <!-- Mermaid generator - Section end -->

data/bin/last_deploys

@@ -36,7 +36,10 @@ sorted_deploy_info = Hash[
   deployer.deployment_info_from(nodes_handler.select_nodes(executable.selected_nodes.empty? ? [{ all: true }] : executable.selected_nodes)).
   map do |node, deploy_info|
     decorated_deploy_info = deploy_info.merge(node: node)
-    decorated_deploy_info[:date] = deploy_info[:date].strftime('%F %T') if deploy_info.key?(:date)
+    if deploy_info.key?(:deployment_info)
+      decorated_deploy_info.merge!(deploy_info[:deployment_info])
+      decorated_deploy_info[:date] = deploy_info[:date].strftime('%F %T') if deploy_info.key?(:date)
+    end
     decorated_deploy_info[:services] = deploy_info[:services].join(', ') if deploy_info.key?(:services)
     [node, decorated_deploy_info]
   end

data/bin/nodes_to_deploy

@@ -60,16 +60,16 @@ unless ignore_deploy_info
     if node_deploy_info.key?(:error)
       executable.log_warn "[ #{node} ] - Deployment in error: #{node_deploy_info[:error]}"
       true
-    elsif !node_deploy_info.key?(:exit_status) || node_deploy_info[:exit_status] != '0'
+    elsif !node_deploy_info.key?(:exit_status) || node_deploy_info[:exit_status] != 0
       executable.log_warn "[ #{node} ] - Last deployment did not end successfully: #{node_deploy_info[:exit_status]}"
       true
-    elsif node_deploy_info.key?(:repo_name_0)
+    elsif node_deploy_info[:deployment_info].key?(:repo_name_0)
       node_impacted = false
       # Loop over all possible repositories concerned by this deployment
       repo_idx = 0
-      while node_deploy_info.key?("repo_name_#{repo_idx}".to_sym) do
-        repo_name = node_deploy_info["repo_name_#{repo_idx}".to_sym]
-        commit_id = node_deploy_info["commit_id_#{repo_idx}".to_sym]
+      while node_deploy_info[:deployment_info].key?("repo_name_#{repo_idx}".to_sym) do
+        repo_name = node_deploy_info[:deployment_info]["repo_name_#{repo_idx}".to_sym]
+        commit_id = node_deploy_info[:deployment_info]["commit_id_#{repo_idx}".to_sym]
         impacted_nodes = cache_impacted_nodes.dig(repo_name, commit_id)
         if impacted_nodes.nil?
           begin

data/docs/config_dsl.md

@@ -2,7 +2,7 @@
 
 The DSL used in configuration files is comprised of Ruby methods that can be called directly in the main `hpc_config.rb` file.
 
-This DSL can also be completed by plugins. Check [the plugins documentations](plugins) to know about DSL extensions brought by plugins.
+This DSL can also be completed by plugins. Check [the plugins documentations](plugins.md) to know about DSL extensions brought by plugins.
 
 # Table of Contents
   * [`<platform_type>_platform`](#platform_type_platform)
@@ -13,6 +13,8 @@ This DSL can also be completed by plugins. Check [the plugins documentations](pl
   * [`hybrid_platforms_dir`](#hybrid_platforms_dir)
   * [`tests_provisioner`](#tests_provisioner)
   * [`expect_tests_to_fail`](#expect_tests_to_fail)
+  * [`read_secrets_from`](#read_secrets_from)
+  * [`send_logs_to`](#send_logs_to)
   * [`retry_deploy_for_errors_on_stdout`](#retry_deploy_for_errors_on_stdout)
   * [`retry_deploy_for_errors_on_stderr`](#retry_deploy_for_errors_on_stderr)
   * [`packaging_timeout`](#packaging_timeout)
@@ -200,6 +202,48 @@ for_nodes('/tst/') do
 end
 ```
 
+<a name="read_secrets_from"></a>
+## `read_secrets_from`
+
+Set the list of [secrets reader plugins](plugins.md#secrets_reader) to use.
+By default (if no plugins is specifically set) the [secrets reader plugin `cli`](plugins/secrets_reader/cli.md) is being used.
+
+Takes the list of secrets reader plugin names, as symbols, as a parameter.
+
+Can be applied to subset of nodes using the [`for_nodes` DSL method](#for_nodes).
+
+Examples:
+```ruby
+# By default, get secrets from the command-line
+read_secrets_from :cli
+
+# All our production nodes also have their secrets stored on a secured Thycotic server
+for_nodes('/prd/') do
+  read_secrets_from :thycotic
+end
+```
+
+<a name="send_logs_to"></a>
+## `send_logs_to`
+
+Set the list of [log plugins](plugins.md#log) to use to save logs.
+By default (if no plugins is specifically set) the [log plugin `remote_fs`](plugins/log/remote_fs.md) is being used.
+
+Takes the list of log plugin names, as symbols, as a parameter.
+
+Can be applied to subset of nodes using the [`for_nodes` DSL method](#for_nodes).
+
+Examples:
+```ruby
+# By default, everything gets logged on the nodes
+send_logs_to :remote_fs
+
+# All our production nodes also have their logs uploaded on our logs servers
+for_nodes('/prd/') do
+  send_logs_to :datadog_log_server, :loggly
+end
+```
+
 <a name="retry_deploy_for_errors_on_stdout"></a>
 ## `retry_deploy_for_errors_on_stdout`
 

data/docs/executables.md

@@ -132,25 +132,22 @@ The Deployer options are used to drive a deployment (be it in why-run mode or no
 
 ```
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
-* `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...).
-  The location can be:
-  * A local file path (for example /path/to/file.json).
-  * A Thycotic Secret Server URL followed by a secret id (for example https://portal.muc.msp.my_company.net/SecretServer:8845).
 * `--parallel`: Specify that the deployment process should perform concurrently on the different nodes it has to deploy to.
 * `--timeout SECS`: Specify the timeout (in seconds) to apply while deploying. This can be set only in why-run mode.
 * `--why-run`: Specify the why-run mode. The why-run mode is used to simulate a deployment on the nodes, and report what a real deployment would have changed on the node.
 * `--retries-on-error NBR`: Specify the number of retries deploys can do in case of non-deterministic errors.
   Non-deterministic errors are matched using a set of strings or regular expressions that can be configured in the `hpc_config.rb` file of any platform, using the `retry_deploy_for_errors_on_stdout` and `retry_deploy_for_errors_on_stderr` properties:
   For example:
+
 ```ruby
 retry_deploy_for_errors_on_stdout [
   'This is a raw string error that will be matched against stdout',
@@ -161,6 +158,8 @@ retry_deploy_for_errors_on_stderr [
 ]
 ```
 
+* `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...). This option is used by the [`cli` secrets reader plugin](plugins/secrets_reader/cli.md). See [secrets reader plugins](plugins.md#secrets_reader) for more info about secrets retrieval.
+
 ## JSON dump options
 
 The JSON dump options drive the way nodes' JSON information is being dumped.

data/docs/executables/check-node.md

@@ -65,11 +65,11 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
 ## Examples

data/docs/executables/deploy.md

@@ -82,13 +82,13 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
 ```
 
 ## Examples

data/docs/executables/dump_nodes_json.md

@@ -54,13 +54,13 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
     -W, --why-run                    Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
 
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
+
 JSON dump options:
     -k, --skip-run                   Skip the actual gathering of dumps in run_logs. If set, the current run_logs content will be used.
     -j, --json-dir DIRECTORY         Specify the output directory in which JSON files are being written. Defaults to nodes_json.

data/docs/executables/test.md

@@ -93,11 +93,11 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
 
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
+
 Tests runner options:
     -i, --tests-list FILE_NAME       Specify a tests file name. The file should contain a list of tests name (1 per line). Can be used several times.
     -k, --skip-run                   Skip running the check-node commands for real, and just analyze existing run logs.

data/docs/executables/topograph.md

@@ -39,12 +39,12 @@ Connector ssh options:
         --ssh-gateways-conf
 
 Deployer options:
-    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
-                                     * Local path to a JSON file
-                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
     -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
         --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
 
+Secrets reader cli options:
+    -e, --secrets JSON_FILE          Specify a secrets location from a local JSON file. Can be specified several times.
+
 JSON dump options:
     -j, --json-dir DIRECTORY         Specify the output directory in which JSON files are being written. Defaults to nodes_json.
 

data/docs/plugins.md

@@ -10,9 +10,11 @@ Following are all possible plugin types and the plugins shipped by default with
   * [`action`](#action)
   * [`cmdb`](#cmdb)
   * [`connector`](#connector)
+  * [`log`](#log)
   * [`platform_handler`](#platform_handler)
   * [`provisioner`](#provisioner)
   * [`report`](#report)
+  * [`secrets_reader`](#secrets_reader)
   * [`test`](#test)
   * [`test_report`](#test_report)
 
@@ -79,6 +81,30 @@ Plugins shipped by default:
 * [`local`](plugins/connector/local.md)
 * [`ssh`](plugins/connector/ssh.md)
 
+<a name="log"></a>
+## Logs
+
+Save deployment logs to a given medium (files, log servers...).
+
+Corresponding plugin type: `log`.
+
+These plugins give ways for the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb) to save logs output from services deployments on nodes.
+Information that can be saved is:
+* The deployment stdout.
+* The deployment stderr.
+* The deployment exit status.
+* The list of services that have been deployed.
+* Some deployment metadata (like git commits information that have been deployed).
+
+Examples of log plugins are:
+* Remote file system: Save logs on the node's remote file system (useful for local debugging).
+* Log servers: Send logs to log servers.
+
+Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/log/my_log_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
+
+Plugins shipped by default:
+* [`remote_fs`](plugins/log/remote_fs.md)
+
 <a name="platform_handler"></a>
 ## Platform Handlers
 
@@ -152,6 +178,26 @@ Plugins shipped by default:
 * [`mediawiki`](plugins/report/mediawiki.md)
 * [`stdout`](plugins/report/stdout.md)
 
+<a name="secrets_reader"></a>
+## Secrets readers
+
+Secrets reader are responsible for fetching secrets (passwords, private keys, API tokens...) needed during deployment from various sources (command line, environment, vaults, secrets servers...).
+
+Corresponding plugin type: `secrets_reader`.
+
+These plugins add new ways to retrieve secrets used by the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb)
+
+Examples of secrets readers are:
+* Command-line: Give secrets from a local file.
+* Vault: Get secrets from vaults (encrypted databases).
+* Secrets servers: Query secrets servers to retrieve secrets.
+
+Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/my_secrets_reader_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
+
+Plugins shipped by default:
+* [`cli`](plugins/secrets_reader/cli.md)
+* [`thycotic`](plugins/secrets_reader/thycotic.md)
+
 <a name="test"></a>
 ## Tests
 
@@ -182,6 +228,7 @@ Plugins shipped by default:
 * [`executables`](plugins/test/executables.md)
 * [`file_system_hdfs`](plugins/test/file_system_hdfs.md)
 * [`file_system`](plugins/test/file_system.md)
+* [`github_ci`](plugins/test/github_ci.md)
 * [`hostname`](plugins/test/hostname.md)
 * [`idempotence`](plugins/test/idempotence.md)
 * [`ip`](plugins/test/ip.md)

data/docs/plugins/connector/ssh.md

@@ -70,7 +70,7 @@ end
 | `host_keys` | `Array<String>` | The node's host keys used to generate a `known_hosts` file with those to avoid user confirmations when connecting. |
 | `hostname` | `String` | Host name used to connect in case no IP address can be found in metadata. |
 | `private_ips` | `Array<String>` | IP list to connect in case `host_ip` is not defined in metadata. |
-| `ssh_session_exec` | `String` | If set to the string `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
+| `ssh_session_exec` | `Boolean` | If set to `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
 
 ## Used environment variables
 

data/docs/plugins/log/remote_fs.md

@@ -0,0 +1,26 @@
+# Log plugin: `remote_fs`
+
+The `remote_fs` log plugin saves deployment logs in each remote node that has been deployed, in the `/var/log/deployments` directory.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None