hybrid_platforms_conductor 32.16.2 → 32.18.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f78c78a80235a940f793af89344e6a972609f92d79ddce8acc6735b0e6b54ceb
-  data.tar.gz: 9c0bfef24e47646c61dd79899af40f33f063f90e73a626ed9c2617c7f7933593
+  metadata.gz: b724f6cb69633133e800d381ee1d0bfdd1a1107d7fc388fe1dcbcf423480ca9d
+  data.tar.gz: 3252bc11e083295c2694df60a9af790d8b01cbe851baebbe1d8e75b656e0acd3
 SHA512:
-  metadata.gz: 60a24969416d8d674cad078642e575c1b22e3b4457dcabb5a436bf62a597d13388bf042851008d5a3b01a7de82ee6003c6856b90821adddc1436bf2a0ecbfdd1
-  data.tar.gz: 83cdf6e30fefad4d1d97cb1e1af84278e71aff6eadac4fe5d27232f70d6537a35a48d0e30494213a616796177fd1e6aa36f26bc6ac86d313b651ca2023ade1a0
+  metadata.gz: 43793a4b3f8ec9a1353b4c075289dfcd77574b64a481754db68e1affa1c599500a3c29d8fa12b31c13dd69b47c5a2840f52aef80045fbec8be7a363ca4eb90b7
+  data.tar.gz: 0451da97325093d7a59684de1ffc77856f4baf5c3a4c76853e6df4492c7cfb1d52cbf9405c0d44b1d8af643e61f64569fc305ed61204be26f081591bdccb6d43

data/CHANGELOG.md

@@ -1,3 +1,51 @@
+# [v32.18.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.17.1...v32.18.0) (2021-06-14 15:01:02)
+
+## Global changes
+### Patches
+
+* [[Feature(log_remote_fs)] [#60] Add new plugins type log with a first log plugin remote_fs to extend deployment logs save functionality](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/20187c6da577c932b5205204187af883140995fe)
+
+## Changes for log_remote_fs
+### Features
+
+* [[Feature(log_remote_fs)] [#60] Add new plugins type log with a first log plugin remote_fs to extend deployment logs save functionality](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/20187c6da577c932b5205204187af883140995fe)
+
+# [v32.17.1](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.17.0...v32.17.1) (2021-06-03 16:20:09)
+
+### Patches
+
+* [[Hotfix] Don't run remote nodes tests on local nodes](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/fe8e34a4d74f21d4903c14b1e156b9730f4b5fee)
+
+# [v32.17.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.16.4...v32.17.0) (2021-06-02 12:57:44)
+
+## Global changes
+### Patches
+
+* [[Feature(test_github_ci)] [#61] Add the github_ci test plugin to check for CI/CD of projects on Github](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/a0082efe5c138ca88ca91a3c10f777f47fca5034)
+
+## Changes for test_github_ci
+### Features
+
+* [[Feature(test_github_ci)] [#61] Add the github_ci test plugin to check for CI/CD of projects on Github](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/a0082efe5c138ca88ca91a3c10f777f47fca5034)
+
+# [v32.16.4](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.16.3...v32.16.4) (2021-06-01 13:25:19)
+
+### Patches
+
+* [[Hotfix] Fixed boolean values incorrectly used as strings](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/a107f3e0fe4c1512ce7607a303fdf0753d92ddac)
+
+# [v32.16.3](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.16.2...v32.16.3) (2021-06-01 11:19:50)
+
+## Global changes
+### Patches
+
+* [[Hotfix(platform_handler_serverless_chef)] Fix Chef packaging missing licenses accept](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/69eafc1b23f77de65ba0dd68e5adb7fafb58157e)
+
+## Changes for platform_handler_serverless_chef
+### Patches
+
+* [[Hotfix(platform_handler_serverless_chef)] Fix Chef packaging missing licenses accept](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/69eafc1b23f77de65ba0dd68e5adb7fafb58157e)
+
 # [v32.16.2](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.16.1...v32.16.2) (2021-06-01 09:52:22)
 
 ## Global changes

data/README.md

@@ -263,6 +263,7 @@ participant PlatformHandler as Platform Handler
 participant PlatformRepo as Platform repository (ie Chef)
 participant Connector as Connector (ie SSH)
 participant Node as Provisioned node (my_node)
+participant Log as Log
 
 Deploy->>+CMDB: Get services to be deployed on my_node
 CMDB->>+PlatformHandler: Get my_node metadata from the platform
@@ -276,6 +277,8 @@ Deploy->>+Connector: Connect to my_node to execute actions
 Connector->>+Node: Execute actions through SSH to deploy my_web_app on my_node
 Node-->>-Connector: my_web_app is deployed successfully
 Connector-->>-Deploy: Close connection
+Deploy->>+Log: Save deployment logs
+Log-->>-Deploy: Deployment logs saved
 ```
 </details>
 <!-- Mermaid generator - Section end -->

data/bin/last_deploys

@@ -36,7 +36,10 @@ sorted_deploy_info = Hash[
   deployer.deployment_info_from(nodes_handler.select_nodes(executable.selected_nodes.empty? ? [{ all: true }] : executable.selected_nodes)).
   map do |node, deploy_info|
     decorated_deploy_info = deploy_info.merge(node: node)
-    decorated_deploy_info[:date] = deploy_info[:date].strftime('%F %T') if deploy_info.key?(:date)
+    if deploy_info.key?(:deployment_info)
+      decorated_deploy_info.merge!(deploy_info[:deployment_info])
+      decorated_deploy_info[:date] = deploy_info[:date].strftime('%F %T') if deploy_info.key?(:date)
+    end
     decorated_deploy_info[:services] = deploy_info[:services].join(', ') if deploy_info.key?(:services)
     [node, decorated_deploy_info]
   end

data/bin/nodes_to_deploy

@@ -60,16 +60,16 @@ unless ignore_deploy_info
     if node_deploy_info.key?(:error)
       executable.log_warn "[ #{node} ] - Deployment in error: #{node_deploy_info[:error]}"
       true
-    elsif !node_deploy_info.key?(:exit_status) || node_deploy_info[:exit_status] != '0'
+    elsif !node_deploy_info.key?(:exit_status) || node_deploy_info[:exit_status] != 0
       executable.log_warn "[ #{node} ] - Last deployment did not end successfully: #{node_deploy_info[:exit_status]}"
       true
-    elsif node_deploy_info.key?(:repo_name_0)
+    elsif node_deploy_info[:deployment_info].key?(:repo_name_0)
       node_impacted = false
       # Loop over all possible repositories concerned by this deployment
       repo_idx = 0
-      while node_deploy_info.key?("repo_name_#{repo_idx}".to_sym) do
-        repo_name = node_deploy_info["repo_name_#{repo_idx}".to_sym]
-        commit_id = node_deploy_info["commit_id_#{repo_idx}".to_sym]
+      while node_deploy_info[:deployment_info].key?("repo_name_#{repo_idx}".to_sym) do
+        repo_name = node_deploy_info[:deployment_info]["repo_name_#{repo_idx}".to_sym]
+        commit_id = node_deploy_info[:deployment_info]["commit_id_#{repo_idx}".to_sym]
         impacted_nodes = cache_impacted_nodes.dig(repo_name, commit_id)
         if impacted_nodes.nil?
           begin

data/docs/config_dsl.md

@@ -13,6 +13,7 @@ This DSL can also be completed by plugins. Check [the plugins documentations](pl
   * [`hybrid_platforms_dir`](#hybrid_platforms_dir)
   * [`tests_provisioner`](#tests_provisioner)
   * [`expect_tests_to_fail`](#expect_tests_to_fail)
+  * [`send_logs_to`](#send_logs_to)
   * [`retry_deploy_for_errors_on_stdout`](#retry_deploy_for_errors_on_stdout)
   * [`retry_deploy_for_errors_on_stderr`](#retry_deploy_for_errors_on_stderr)
   * [`packaging_timeout`](#packaging_timeout)
@@ -200,6 +201,27 @@ for_nodes('/tst/') do
 end
 ```
 
+<a name="send_logs_to"></a>
+## `send_logs_to`
+
+Set the list of [log plugins](plugins.md#log) to use to save logs.
+By default (if no plugins is specifically set) the [log plugin `remote_fs`](plugins/log/remote_fs.md) is being used.
+
+Takes the list of log plugin names, as symbols, as a parameter.
+
+Can be applied to subset of nodes using the [`for_nodes` DSL method](#for_nodes).
+
+Examples:
+```ruby
+# By default, everything gets logged on the nodes
+send_logs_to :remote_fs
+
+# All our production nodes also have their logs uploaded on our logs servers
+for_nodes('/prd/') do
+  send_logs_to :datadog_log_server, :loggly
+end
+```
+
 <a name="retry_deploy_for_errors_on_stdout"></a>
 ## `retry_deploy_for_errors_on_stdout`
 

data/docs/plugins.md

@@ -10,6 +10,7 @@ Following are all possible plugin types and the plugins shipped by default with
   * [`action`](#action)
   * [`cmdb`](#cmdb)
   * [`connector`](#connector)
+  * [`log`](#log)
   * [`platform_handler`](#platform_handler)
   * [`provisioner`](#provisioner)
   * [`report`](#report)
@@ -79,6 +80,30 @@ Plugins shipped by default:
 * [`local`](plugins/connector/local.md)
 * [`ssh`](plugins/connector/ssh.md)
 
+<a name="log"></a>
+## Logs
+
+Save deployment logs to a given medium (files, log servers...).
+
+Corresponding plugin type: `log`.
+
+These plugins give ways for the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb) to save logs output from services deployments on nodes.
+Information that can be saved is:
+* The deployment stdout.
+* The deployment stderr.
+* The deployment exit status.
+* The list of services that have been deployed.
+* Some deployment metadata (like git commits information that have been deployed).
+
+Examples of log plugins are:
+* Remote file system: Save logs on the node's remote file system (useful for local debugging).
+* Log servers: Send logs to log servers.
+
+Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/log/my_log_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
+
+Plugins shipped by default:
+* [`remote_fs`](plugins/log/remote_fs.md)
+
 <a name="platform_handler"></a>
 ## Platform Handlers
 
@@ -182,6 +207,7 @@ Plugins shipped by default:
 * [`executables`](plugins/test/executables.md)
 * [`file_system_hdfs`](plugins/test/file_system_hdfs.md)
 * [`file_system`](plugins/test/file_system.md)
+* [`github_ci`](plugins/test/github_ci.md)
 * [`hostname`](plugins/test/hostname.md)
 * [`idempotence`](plugins/test/idempotence.md)
 * [`ip`](plugins/test/ip.md)

data/docs/plugins/connector/ssh.md

@@ -70,7 +70,7 @@ end
 | `host_keys` | `Array<String>` | The node's host keys used to generate a `known_hosts` file with those to avoid user confirmations when connecting. |
 | `hostname` | `String` | Host name used to connect in case no IP address can be found in metadata. |
 | `private_ips` | `Array<String>` | IP list to connect in case `host_ip` is not defined in metadata. |
-| `ssh_session_exec` | `String` | If set to the string `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
+| `ssh_session_exec` | `Boolean` | If set to `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
 
 ## Used environment variables
 

data/docs/plugins/log/remote_fs.md

@@ -0,0 +1,26 @@
+# Log plugin: `remote_fs`
+
+The `remote_fs` log plugin saves deployment logs in each remote node that has been deployed, in the `/var/log/deployments` directory.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/bitbucket_conf.md

@@ -12,7 +12,7 @@ Define a Bitbucket installation to be targeted.
 It takes the following parameters:
 * **url** (`String`): URL to the Bitbucket server
 * **project** (`String`): Project name from the Bitbucket server, storing repositories
-* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or :all for all [default: :all]
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
 * **checks** (`Hash<Symbol, Object>`): Checks definition to be perform on those repositories [default: {}]
   * **branch_permissions** (`Array< Hash<Symbol, Object> >`): List of branch permissions to check [optional]
     * **type** (`String`): Type of branch permissions to check. Examples of values are 'fast-forward-only', 'no-deletes', 'pull-request-only'.

data/docs/plugins/test/check_deploy_and_idempotence.md

@@ -49,7 +49,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
-| `root_access_allowed` | `String` | If set to `true`, then skip the test for `root` access being disabled after deployment |
+| `root_access_allowed` | `Boolean` | If set to `true`, then skip the test for `root` access being disabled after deployment |
 
 ## Used environment variables
 

data/docs/plugins/test/connection.md

@@ -16,6 +16,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/deploy_removes_root_access.md

@@ -17,7 +17,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
-| `root_access_allowed` | `String` | If set to `true`, then skip this test |
+| `root_access_allowed` | `Boolean` | If set to `true`, then skip this test |
 
 ## Used environment variables
 

data/docs/plugins/test/file_system.md

@@ -38,6 +38,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/github_ci.md

@@ -0,0 +1,48 @@
+# Test plugin: `github_ci`
+
+The `github_ci` test plugin checks that the `master` branch of Github repositories has a successful CI result from its [Github Actions](https://github.com/features/actions).
+
+## Config DSL extension
+
+### `github_repos`
+
+Define Github repositories to be targeted.
+
+It takes the following parameters:
+* **url** (`String`): URL to the Github API [default: `'https://api.github.com'`]
+* **user** (`String`): User or organization name, storing repositories
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
+
+Example:
+```ruby
+github_repos(
+  # Github's user containing repositories
+  user: 'My-Github-User',
+  # List of repositories to check
+  repos: [
+    'my-platform-repo',
+    'my-chef-repo',
+    'my-hpc-plugins'
+  ]
+)
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+| `github` | Used to connect to the Github API. Password should be the Github API token. |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/hostname.md

@@ -16,6 +16,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/ip.md

@@ -16,6 +16,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 | `private_ips` | `Array<String>` | List of possible private IPs the node should have |
 
 ## Used environment variables

data/docs/plugins/test/jenkins_ci_conf.md

@@ -12,7 +12,7 @@ It takes the following parameters:
 * **url** (`String`): URL to the Bitbucket server
 * **project** (`String`): Project name from the Bitbucket server, storing repositories
 * **jenkins_ci_url** (`String` or `nil`): Corresponding Jenkins CI URL, or nil if none.
-* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or :all for all [default: :all]
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
 
 Example:
 ```ruby

data/docs/plugins/test/jenkins_ci_masters_ok.md

@@ -12,7 +12,7 @@ It takes the following parameters:
 * **url** (`String`): URL to the Bitbucket server
 * **project** (`String`): Project name from the Bitbucket server, storing repositories
 * **jenkins_ci_url** (`String` or `nil`): Corresponding Jenkins CI URL, or nil if none.
-* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or :all for all [default: :all]
+* **repos** (`Array<String>` or `Symbol`): List of repository names from this project, or `:all` for all [default: `:all`]
 
 Example:
 ```ruby

data/docs/plugins/test/local_users.md

@@ -37,6 +37,7 @@ check_local_users_do_not_exist %w[olduser1 olduser2]
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/mounts.md

@@ -44,6 +44,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/orphan_files.md

@@ -27,6 +27,7 @@ end
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/ports.md

@@ -39,6 +39,7 @@ check_closed_ports 25, 110
 | Metadata | Type | Usage
 | --- | --- | --- |
 | `host_ip` | `String` | Host IP address to be tested for port listening |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables
 

data/docs/plugins/test/spectre.md

@@ -15,6 +15,7 @@ None
 
 | Metadata | Type | Usage
 | --- | --- | --- |
+| `local_node` | `Boolean` | Skip this test for nodes having this metadata set to `true` |
 
 ## Used environment variables