hybrid_platforms_conductor 32.18.0 → 33.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/README.md +3 -3
- data/docs/config_dsl.md +23 -1
- data/docs/executables.md +6 -7
- data/docs/executables/check-node.md +3 -3
- data/docs/executables/deploy.md +3 -3
- data/docs/executables/dump_nodes_json.md +3 -3
- data/docs/executables/test.md +3 -3
- data/docs/executables/topograph.md +3 -3
- data/docs/plugins.md +21 -0
- data/docs/plugins/secrets_reader/cli.md +31 -0
- data/docs/plugins/secrets_reader/thycotic.md +46 -0
- data/lib/hybrid_platforms_conductor/deployer.rb +96 -36
- data/lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/cli.rb +75 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/my_secrets_reader_plugin.rb.sample +46 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/thycotic.rb +87 -0
- data/lib/hybrid_platforms_conductor/plugins.rb +1 -0
- data/lib/hybrid_platforms_conductor/secrets_reader.rb +31 -0
- data/lib/hybrid_platforms_conductor/version.rb +1 -1
- data/spec/hybrid_platforms_conductor_test.rb +5 -0
- data/spec/hybrid_platforms_conductor_test/api/deployer/config_dsl_spec.rb +22 -0
- data/spec/hybrid_platforms_conductor_test/api/deployer/secrets_reader_plugins/cli_spec.rb +63 -0
- data/spec/hybrid_platforms_conductor_test/api/deployer/secrets_reader_plugins/thycotic_spec.rb +253 -0
- data/spec/hybrid_platforms_conductor_test/executables/options/deployer_spec.rb +0 -182
- data/spec/hybrid_platforms_conductor_test/helpers/deployer_test_helpers.rb +249 -13
- data/spec/hybrid_platforms_conductor_test/test_secrets_reader_plugin.rb +45 -0
- metadata +13 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ae17e600c52586447071f6f463ee1525754327c0be4a147bec7d3ebcb13b018d
|
4
|
+
data.tar.gz: 9e4dc83a6d37fc9c17b99fd9a16eae20b73673dd492dde52679a159ed922a3d5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d9b89fc3f6ed1fe6e87f23aa1519b32dc62bdcf861b0332088909b1ac8945adf273425c97860c0df5131069921c6adfed60d265cbfe9c1d50a6e4cd5099cc420
|
7
|
+
data.tar.gz: d9b3d6d66250552aaec66672b2b9fc97843eb39fc5578a34b74c44f9a4511bccf7978ef59559a4a21b276557e6293418e3350b8151308000d478e22aba84ab56
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,9 @@
|
|
1
|
+
# [v33.0.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.18.0...v33.0.0) (2021-06-15 16:10:47)
|
2
|
+
|
3
|
+
### Breaking changes
|
4
|
+
|
5
|
+
* [[Breaking] Add secrets reader plugins with 2 default plugins: cli and thycotic](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/2cfacebe8cfac57de40ef003877da5b99aca5b5e)
|
6
|
+
|
1
7
|
# [v32.18.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.17.1...v32.18.0) (2021-06-14 15:01:02)
|
2
8
|
|
3
9
|
## Global changes
|
data/README.md
CHANGED
@@ -216,13 +216,13 @@ Connector ssh options:
|
|
216
216
|
--ssh-gateways-conf
|
217
217
|
|
218
218
|
Deployer options:
|
219
|
-
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
220
|
-
* Local path to a JSON file
|
221
|
-
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
222
219
|
-p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
|
223
220
|
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
|
224
221
|
-W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
|
225
222
|
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
223
|
+
|
224
|
+
Secrets reader cli options:
|
225
|
+
-e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
|
226
226
|
```
|
227
227
|
|
228
228
|
All executables also have the `--debug` switch to display more verbose and debugging information.
|
data/docs/config_dsl.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
The DSL used in configuration files is comprised of Ruby methods that can be called directly in the main `hpc_config.rb` file.
|
4
4
|
|
5
|
-
This DSL can also be completed by plugins. Check [the plugins documentations](plugins) to know about DSL extensions brought by plugins.
|
5
|
+
This DSL can also be completed by plugins. Check [the plugins documentations](plugins.md) to know about DSL extensions brought by plugins.
|
6
6
|
|
7
7
|
# Table of Contents
|
8
8
|
* [`<platform_type>_platform`](#platform_type_platform)
|
@@ -13,6 +13,7 @@ This DSL can also be completed by plugins. Check [the plugins documentations](pl
|
|
13
13
|
* [`hybrid_platforms_dir`](#hybrid_platforms_dir)
|
14
14
|
* [`tests_provisioner`](#tests_provisioner)
|
15
15
|
* [`expect_tests_to_fail`](#expect_tests_to_fail)
|
16
|
+
* [`read_secrets_from`](#read_secrets_from)
|
16
17
|
* [`send_logs_to`](#send_logs_to)
|
17
18
|
* [`retry_deploy_for_errors_on_stdout`](#retry_deploy_for_errors_on_stdout)
|
18
19
|
* [`retry_deploy_for_errors_on_stderr`](#retry_deploy_for_errors_on_stderr)
|
@@ -201,6 +202,27 @@ for_nodes('/tst/') do
|
|
201
202
|
end
|
202
203
|
```
|
203
204
|
|
205
|
+
<a name="read_secrets_from"></a>
|
206
|
+
## `read_secrets_from`
|
207
|
+
|
208
|
+
Set the list of [secrets reader plugins](plugins.md#secrets_reader) to use.
|
209
|
+
By default (if no plugins is specifically set) the [secrets reader plugin `cli`](plugins/secrets_reader/cli.md) is being used.
|
210
|
+
|
211
|
+
Takes the list of secrets reader plugin names, as symbols, as a parameter.
|
212
|
+
|
213
|
+
Can be applied to subset of nodes using the [`for_nodes` DSL method](#for_nodes).
|
214
|
+
|
215
|
+
Examples:
|
216
|
+
```ruby
|
217
|
+
# By default, get secrets from the command-line
|
218
|
+
read_secrets_from :cli
|
219
|
+
|
220
|
+
# All our production nodes also have their secrets stored on a secured Thycotic server
|
221
|
+
for_nodes('/prd/') do
|
222
|
+
read_secrets_from :thycotic
|
223
|
+
end
|
224
|
+
```
|
225
|
+
|
204
226
|
<a name="send_logs_to"></a>
|
205
227
|
## `send_logs_to`
|
206
228
|
|
data/docs/executables.md
CHANGED
@@ -132,25 +132,22 @@ The Deployer options are used to drive a deployment (be it in why-run mode or no
|
|
132
132
|
|
133
133
|
```
|
134
134
|
Deployer options:
|
135
|
-
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
136
|
-
* Local path to a JSON file
|
137
|
-
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
138
135
|
-p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
|
139
136
|
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
|
140
137
|
-W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
|
141
138
|
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
139
|
+
|
140
|
+
Secrets reader cli options:
|
141
|
+
-e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
|
142
142
|
```
|
143
143
|
|
144
|
-
* `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...).
|
145
|
-
The location can be:
|
146
|
-
* A local file path (for example /path/to/file.json).
|
147
|
-
* A Thycotic Secret Server URL followed by a secret id (for example https://portal.muc.msp.my_company.net/SecretServer:8845).
|
148
144
|
* `--parallel`: Specify that the deployment process should perform concurrently on the different nodes it has to deploy to.
|
149
145
|
* `--timeout SECS`: Specify the timeout (in seconds) to apply while deploying. This can be set only in why-run mode.
|
150
146
|
* `--why-run`: Specify the why-run mode. The why-run mode is used to simulate a deployment on the nodes, and report what a real deployment would have changed on the node.
|
151
147
|
* `--retries-on-error NBR`: Specify the number of retries deploys can do in case of non-deterministic errors.
|
152
148
|
Non-deterministic errors are matched using a set of strings or regular expressions that can be configured in the `hpc_config.rb` file of any platform, using the `retry_deploy_for_errors_on_stdout` and `retry_deploy_for_errors_on_stderr` properties:
|
153
149
|
For example:
|
150
|
+
|
154
151
|
```ruby
|
155
152
|
retry_deploy_for_errors_on_stdout [
|
156
153
|
'This is a raw string error that will be matched against stdout',
|
@@ -161,6 +158,8 @@ retry_deploy_for_errors_on_stderr [
|
|
161
158
|
]
|
162
159
|
```
|
163
160
|
|
161
|
+
* `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...). This option is used by the [`cli` secrets reader plugin](plugins/secrets_reader/cli.md). See [secrets reader plugins](plugins.md#secrets_reader) for more info about secrets retrieval.
|
162
|
+
|
164
163
|
## JSON dump options
|
165
164
|
|
166
165
|
The JSON dump options drive the way nodes' JSON information is being dumped.
|
@@ -65,11 +65,11 @@ Connector ssh options:
|
|
65
65
|
--ssh-gateways-conf
|
66
66
|
|
67
67
|
Deployer options:
|
68
|
-
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
69
|
-
* Local path to a JSON file
|
70
|
-
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
71
68
|
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
|
72
69
|
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
70
|
+
|
71
|
+
Secrets reader cli options:
|
72
|
+
-e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
|
73
73
|
```
|
74
74
|
|
75
75
|
## Examples
|
data/docs/executables/deploy.md
CHANGED
@@ -82,13 +82,13 @@ Connector ssh options:
|
|
82
82
|
--ssh-gateways-conf
|
83
83
|
|
84
84
|
Deployer options:
|
85
|
-
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
86
|
-
* Local path to a JSON file
|
87
|
-
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
88
85
|
-p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
|
89
86
|
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
|
90
87
|
-W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
|
91
88
|
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
89
|
+
|
90
|
+
Secrets reader cli options:
|
91
|
+
-e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
|
92
92
|
```
|
93
93
|
|
94
94
|
## Examples
|
@@ -54,13 +54,13 @@ Connector ssh options:
|
|
54
54
|
--ssh-gateways-conf
|
55
55
|
|
56
56
|
Deployer options:
|
57
|
-
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
58
|
-
* Local path to a JSON file
|
59
|
-
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
60
57
|
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
|
61
58
|
-W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
|
62
59
|
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
63
60
|
|
61
|
+
Secrets reader cli options:
|
62
|
+
-e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
|
63
|
+
|
64
64
|
JSON dump options:
|
65
65
|
-k, --skip-run Skip the actual gathering of dumps in run_logs. If set, the current run_logs content will be used.
|
66
66
|
-j, --json-dir DIRECTORY Specify the output directory in which JSON files are being written. Defaults to nodes_json.
|
data/docs/executables/test.md
CHANGED
@@ -93,11 +93,11 @@ Connector ssh options:
|
|
93
93
|
--ssh-gateways-conf
|
94
94
|
|
95
95
|
Deployer options:
|
96
|
-
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
97
|
-
* Local path to a JSON file
|
98
|
-
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
99
96
|
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
100
97
|
|
98
|
+
Secrets reader cli options:
|
99
|
+
-e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
|
100
|
+
|
101
101
|
Tests runner options:
|
102
102
|
-i, --tests-list FILE_NAME Specify a tests file name. The file should contain a list of tests name (1 per line). Can be used several times.
|
103
103
|
-k, --skip-run Skip running the check-node commands for real, and just analyze existing run logs.
|
@@ -39,12 +39,12 @@ Connector ssh options:
|
|
39
39
|
--ssh-gateways-conf
|
40
40
|
|
41
41
|
Deployer options:
|
42
|
-
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
43
|
-
* Local path to a JSON file
|
44
|
-
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
45
42
|
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
|
46
43
|
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
47
44
|
|
45
|
+
Secrets reader cli options:
|
46
|
+
-e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
|
47
|
+
|
48
48
|
JSON dump options:
|
49
49
|
-j, --json-dir DIRECTORY Specify the output directory in which JSON files are being written. Defaults to nodes_json.
|
50
50
|
|
data/docs/plugins.md
CHANGED
@@ -14,6 +14,7 @@ Following are all possible plugin types and the plugins shipped by default with
|
|
14
14
|
* [`platform_handler`](#platform_handler)
|
15
15
|
* [`provisioner`](#provisioner)
|
16
16
|
* [`report`](#report)
|
17
|
+
* [`secrets_reader`](#secrets_reader)
|
17
18
|
* [`test`](#test)
|
18
19
|
* [`test_report`](#test_report)
|
19
20
|
|
@@ -177,6 +178,26 @@ Plugins shipped by default:
|
|
177
178
|
* [`mediawiki`](plugins/report/mediawiki.md)
|
178
179
|
* [`stdout`](plugins/report/stdout.md)
|
179
180
|
|
181
|
+
<a name="secrets_reader"></a>
|
182
|
+
## Secrets readers
|
183
|
+
|
184
|
+
Secrets reader are responsible for fetching secrets (passwords, private keys, API tokens...) needed during deployment from various sources (command line, environment, vaults, secrets servers...).
|
185
|
+
|
186
|
+
Corresponding plugin type: `secrets_reader`.
|
187
|
+
|
188
|
+
These plugins add new ways to retrieve secrets used by the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb)
|
189
|
+
|
190
|
+
Examples of secrets readers are:
|
191
|
+
* Command-line: Give secrets from a local file.
|
192
|
+
* Vault: Get secrets from vaults (encrypted databases).
|
193
|
+
* Secrets servers: Query secrets servers to retrieve secrets.
|
194
|
+
|
195
|
+
Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/my_secrets_reader_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
|
196
|
+
|
197
|
+
Plugins shipped by default:
|
198
|
+
* [`cli`](plugins/secrets_reader/cli.md)
|
199
|
+
* [`thycotic`](plugins/secrets_reader/thycotic.md)
|
200
|
+
|
180
201
|
<a name="test"></a>
|
181
202
|
## Tests
|
182
203
|
|
@@ -0,0 +1,31 @@
|
|
1
|
+
# Secrets reader plugin: `cli`
|
2
|
+
|
3
|
+
The `cli` secrets reader plugin reads secrets from a local JSON file that can be given through the `--secrets` command-line parameter.
|
4
|
+
|
5
|
+
Example:
|
6
|
+
```bash
|
7
|
+
./bin/deploy --node my_node --secrets /path/to/my_secrets.json
|
8
|
+
```
|
9
|
+
|
10
|
+
## Config DSL extension
|
11
|
+
|
12
|
+
None
|
13
|
+
|
14
|
+
## Used credentials
|
15
|
+
|
16
|
+
| Credential | Usage
|
17
|
+
| --- | --- |
|
18
|
+
|
19
|
+
## Used Metadata
|
20
|
+
|
21
|
+
| Metadata | Type | Usage
|
22
|
+
| --- | --- | --- |
|
23
|
+
|
24
|
+
## Used environment variables
|
25
|
+
|
26
|
+
| Variable | Usage
|
27
|
+
| --- | --- |
|
28
|
+
|
29
|
+
## External tools dependencies
|
30
|
+
|
31
|
+
None
|
@@ -0,0 +1,46 @@
|
|
1
|
+
# Secrets reader plugin: `thycotic`
|
2
|
+
|
3
|
+
The `thycotic` secrets reader plugin retrieves secrets from a [Thycotic secrets server](https://thycotic.com/products/secret-server-vdo/), using its SOAP API.
|
4
|
+
|
5
|
+
It is configured using the `secrets_from_thycotic` (see below) config DSL and uses the `thycotic` credential ID to authenticate.
|
6
|
+
|
7
|
+
## Config DSL extension
|
8
|
+
|
9
|
+
### `secrets_from_thycotic`
|
10
|
+
|
11
|
+
Define a Thycotic URL and Thycotic secret ID to fetch from a Thycotic server.
|
12
|
+
The Thycotic secret should contain a JSON file that will be retrieved locally to be used as a secrets source. The local copy will then be removed after deployment.
|
13
|
+
|
14
|
+
Can be applied to subset of nodes using the [`for_nodes` DSL method](/docs/config_dsl.md#for_nodes).
|
15
|
+
|
16
|
+
It takes the following parameters:
|
17
|
+
* **thycotic_url** (`String`): The Thycotic server URL.
|
18
|
+
* **secret_id** (`Integer`): The Thycotic secret ID containing the secrets file to be used as secrets.
|
19
|
+
|
20
|
+
Example:
|
21
|
+
```ruby
|
22
|
+
secrets_from_thycotic(
|
23
|
+
thycotic_url: 'https://my-thycotic-server.my-domain.com/SecretServer',
|
24
|
+
secret_id: 1107
|
25
|
+
)
|
26
|
+
```
|
27
|
+
|
28
|
+
## Used credentials
|
29
|
+
|
30
|
+
| Credential | Usage
|
31
|
+
| --- | --- |
|
32
|
+
| `thycotic` | Used to authenticate on the Thycotic server's SOAP API |
|
33
|
+
|
34
|
+
## Used Metadata
|
35
|
+
|
36
|
+
| Metadata | Type | Usage
|
37
|
+
| --- | --- | --- |
|
38
|
+
|
39
|
+
## Used environment variables
|
40
|
+
|
41
|
+
| Variable | Usage
|
42
|
+
| --- | --- |
|
43
|
+
|
44
|
+
## External tools dependencies
|
45
|
+
|
46
|
+
None
|
@@ -11,7 +11,6 @@ require 'hybrid_platforms_conductor/logger_helpers'
|
|
11
11
|
require 'hybrid_platforms_conductor/nodes_handler'
|
12
12
|
require 'hybrid_platforms_conductor/services_handler'
|
13
13
|
require 'hybrid_platforms_conductor/plugins'
|
14
|
-
require 'hybrid_platforms_conductor/thycotic'
|
15
14
|
|
16
15
|
module HybridPlatformsConductor
|
17
16
|
|
@@ -27,6 +26,12 @@ module HybridPlatformsConductor
|
|
27
26
|
# Array< Hash<Symbol, Object> >
|
28
27
|
attr_reader :deployment_logs
|
29
28
|
|
29
|
+
# List of secrets reader plugins. Each info has the following properties:
|
30
|
+
# * *nodes_selectors_stack* (Array<Object>): Stack of nodes selectors impacted by this rule.
|
31
|
+
# * *secrets_readers* (Array<Symbol>): List of log plugins to be used to store deployment logs.
|
32
|
+
# Array< Hash<Symbol, Object> >
|
33
|
+
attr_reader :secrets_readers
|
34
|
+
|
30
35
|
# Integer: Timeout (in seconds) for packaging repositories
|
31
36
|
attr_reader :packaging_timeout_secs
|
32
37
|
|
@@ -34,6 +39,7 @@ module HybridPlatformsConductor
|
|
34
39
|
def init_deployer_config
|
35
40
|
@packaging_timeout_secs = 60
|
36
41
|
@deployment_logs = []
|
42
|
+
@secrets_readers = []
|
37
43
|
end
|
38
44
|
|
39
45
|
# Set the packaging timeout
|
@@ -55,6 +61,17 @@ module HybridPlatformsConductor
|
|
55
61
|
}
|
56
62
|
end
|
57
63
|
|
64
|
+
# Set the secrets readers
|
65
|
+
#
|
66
|
+
# Parameters::
|
67
|
+
# * *secrets_readers* (Symbol or Array<Symbol>): The list of (or single) secrets readers plugins to be used
|
68
|
+
def read_secrets_from(*secrets_readers)
|
69
|
+
@secrets_readers << {
|
70
|
+
nodes_selectors_stack: current_nodes_selectors_stack,
|
71
|
+
secrets_readers: secrets_readers.flatten
|
72
|
+
}
|
73
|
+
end
|
74
|
+
|
58
75
|
end
|
59
76
|
|
60
77
|
include LoggerHelpers
|
@@ -73,10 +90,6 @@ module HybridPlatformsConductor
|
|
73
90
|
# Boolean
|
74
91
|
attr_accessor :concurrent_execution
|
75
92
|
|
76
|
-
# The list of JSON secrets
|
77
|
-
# Array<Hash>
|
78
|
-
attr_accessor :secrets
|
79
|
-
|
80
93
|
# Are we deploying in a local environment?
|
81
94
|
# Boolean
|
82
95
|
attr_accessor :local_environment
|
@@ -110,7 +123,21 @@ module HybridPlatformsConductor
|
|
110
123
|
@nodes_handler = nodes_handler
|
111
124
|
@actions_executor = actions_executor
|
112
125
|
@services_handler = services_handler
|
113
|
-
@
|
126
|
+
@override_secrets = nil
|
127
|
+
@secrets_readers = Plugins.new(
|
128
|
+
:secrets_reader,
|
129
|
+
logger: @logger,
|
130
|
+
logger_stderr: @logger_stderr,
|
131
|
+
init_plugin: proc do |plugin_class|
|
132
|
+
plugin_class.new(
|
133
|
+
logger: @logger,
|
134
|
+
logger_stderr: @logger_stderr,
|
135
|
+
config: @config,
|
136
|
+
cmd_runner: @cmd_runner,
|
137
|
+
nodes_handler: @nodes_handler
|
138
|
+
)
|
139
|
+
end
|
140
|
+
)
|
114
141
|
@provisioners = Plugins.new(:provisioner, logger: @logger, logger_stderr: @logger_stderr)
|
115
142
|
@log_plugins = Plugins.new(
|
116
143
|
:log,
|
@@ -144,30 +171,6 @@ module HybridPlatformsConductor
|
|
144
171
|
def options_parse(options_parser, parallel_switch: true, why_run_switch: false, timeout_options: true)
|
145
172
|
options_parser.separator ''
|
146
173
|
options_parser.separator 'Deployer options:'
|
147
|
-
options_parser.on(
|
148
|
-
'-e', '--secrets SECRETS_LOCATION',
|
149
|
-
'Specify a secrets location. Can be specified several times. Location can be:',
|
150
|
-
'* Local path to a JSON file',
|
151
|
-
'* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.'
|
152
|
-
) do |secrets_location|
|
153
|
-
@secrets << JSON.parse(
|
154
|
-
if secrets_location =~ /^(https?:\/\/.+):(\d+)$/
|
155
|
-
url = $1
|
156
|
-
secret_id = $2
|
157
|
-
secret = nil
|
158
|
-
Thycotic.with_thycotic(url, @logger, @logger_stderr) do |thycotic|
|
159
|
-
secret_file_item_id = thycotic.get_secret(secret_id).dig(:secret, :items, :secret_item, :id)
|
160
|
-
raise "Unable to fetch secret file ID #{secrets_location}" if secret_file_item_id.nil?
|
161
|
-
secret = thycotic.download_file_attachment_by_item_id(secret_id, secret_file_item_id)
|
162
|
-
raise "Unable to fetch secret file attachment from #{secrets_location}" if secret.nil?
|
163
|
-
end
|
164
|
-
secret
|
165
|
-
else
|
166
|
-
raise "Missing secret file: #{secrets_location}" unless File.exist?(secrets_location)
|
167
|
-
File.read(secrets_location)
|
168
|
-
end
|
169
|
-
)
|
170
|
-
end
|
171
174
|
options_parser.on('-p', '--parallel', 'Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)') do
|
172
175
|
@concurrent_execution = true
|
173
176
|
end if parallel_switch
|
@@ -180,6 +183,14 @@ module HybridPlatformsConductor
|
|
180
183
|
options_parser.on('--retries-on-error NBR', "Number of retries in case of non-deterministic errors (defaults to #{@nbr_retries_on_error})") do |nbr_retries|
|
181
184
|
@nbr_retries_on_error = nbr_retries.to_i
|
182
185
|
end
|
186
|
+
# Display options secrets readers might have
|
187
|
+
@secrets_readers.each do |secret_reader_name, secret_reader|
|
188
|
+
if secret_reader.respond_to?(:options_parse)
|
189
|
+
options_parser.separator ''
|
190
|
+
options_parser.separator "Secrets reader #{secret_reader_name} options:"
|
191
|
+
secret_reader.options_parse(options_parser)
|
192
|
+
end
|
193
|
+
end
|
183
194
|
end
|
184
195
|
|
185
196
|
# Validate that parsed parameters are valid
|
@@ -190,6 +201,16 @@ module HybridPlatformsConductor
|
|
190
201
|
# String: File used as a Futex for packaging
|
191
202
|
PACKAGING_FUTEX_FILE = "#{Dir.tmpdir}/hpc_packaging"
|
192
203
|
|
204
|
+
# Override the secrets with a given JSON.
|
205
|
+
# When using this method with a secrets Hash, further deployments will not query secrets readers, but will use those secrets directly.
|
206
|
+
# Useful to override secrets in test conditions when using dummy secrets for example.
|
207
|
+
#
|
208
|
+
# Parameters::
|
209
|
+
# * *secrets* (Hash or nil): Secrets to take into account in place of secrets readers, or nil to cancel a previous overriding and use secrets readers instead.
|
210
|
+
def override_secrets(secrets)
|
211
|
+
@override_secrets = secrets
|
212
|
+
end
|
213
|
+
|
193
214
|
# Deploy on a given list of nodes selectors.
|
194
215
|
# The workflow is the following:
|
195
216
|
# 1. Package the services to be deployed, considering the nodes, services and context (options, secrets, environment...)
|
@@ -209,10 +230,20 @@ module HybridPlatformsConductor
|
|
209
230
|
|
210
231
|
# Get the secrets to be deployed
|
211
232
|
secrets = {}
|
212
|
-
@
|
213
|
-
secrets
|
214
|
-
|
215
|
-
|
233
|
+
if @override_secrets
|
234
|
+
secrets = @override_secrets
|
235
|
+
else
|
236
|
+
services_to_deploy.each do |node, services|
|
237
|
+
# If there is no config for secrets, just use cli
|
238
|
+
(@config.secrets_readers.empty? ? [{ secrets_readers: %i[cli] }] : @nodes_handler.select_confs_for_node(node, @config.secrets_readers)).inject([]) do |secrets_readers, secrets_readers_info|
|
239
|
+
secrets_readers + secrets_readers_info[:secrets_readers]
|
240
|
+
end.sort.uniq.each do |secrets_reader|
|
241
|
+
services.each do |service|
|
242
|
+
node_secrets = @secrets_readers[secrets_reader].secrets_for(node, service)
|
243
|
+
conflicting_path = safe_merge(secrets, node_secrets)
|
244
|
+
raise "Secret set at path #{conflicting_path.join('->')} by #{secrets_reader} for service #{service} on node #{node} has conflicting values (#{log_debug? ? "#{node_secrets.dig(*conflicting_path)} != #{secrets.dig(*conflicting_path)}" : 'set debug for value details'})." unless conflicting_path.nil?
|
245
|
+
end
|
246
|
+
end
|
216
247
|
end
|
217
248
|
end
|
218
249
|
|
@@ -370,7 +401,7 @@ module HybridPlatformsConductor
|
|
370
401
|
deployer.local_environment = true
|
371
402
|
# Ignore secrets that might have been given: in Docker containers we always use dummy secrets
|
372
403
|
dummy_secrets_file = "#{@config.hybrid_platforms_dir}/dummy_secrets.json"
|
373
|
-
deployer.
|
404
|
+
deployer.override_secrets(File.exist?(dummy_secrets_file) ? JSON.parse(File.read(dummy_secrets_file)) : {})
|
374
405
|
yield deployer, instance
|
375
406
|
end
|
376
407
|
rescue
|
@@ -433,6 +464,35 @@ module HybridPlatformsConductor
|
|
433
464
|
|
434
465
|
private
|
435
466
|
|
467
|
+
# Safe-merge 2 hashes.
|
468
|
+
# Safe-merging is done by:
|
469
|
+
# * Merging values that are hashes.
|
470
|
+
# * Reporting errors when values conflict.
|
471
|
+
# When values are conflicting, the initial hash won't modify those conflicting values and will stop the merge.
|
472
|
+
#
|
473
|
+
# Parameters::
|
474
|
+
# * *hash* (Hash): Hash to be modified merging hash_to_merge
|
475
|
+
# * *hash_to_merge* (Hash): Hash to be merged into hash
|
476
|
+
# Result::
|
477
|
+
# * nil or Array<Object>: nil in case of success, or the keys path leading to a conflicting value in case of error
|
478
|
+
def safe_merge(hash, hash_to_merge)
|
479
|
+
conflicting_path = nil
|
480
|
+
hash_to_merge.each do |key, value_to_merge|
|
481
|
+
if hash.key?(key)
|
482
|
+
if hash[key].is_a?(Hash) && value_to_merge.is_a?(Hash)
|
483
|
+
sub_conflicting_path = safe_merge(hash[key], value_to_merge)
|
484
|
+
conflicting_path = [key] + sub_conflicting_path unless sub_conflicting_path.nil?
|
485
|
+
elsif hash[key] != value_to_merge
|
486
|
+
conflicting_path = [key]
|
487
|
+
end
|
488
|
+
else
|
489
|
+
hash[key] = value_to_merge
|
490
|
+
end
|
491
|
+
break unless conflicting_path.nil?
|
492
|
+
end
|
493
|
+
conflicting_path
|
494
|
+
end
|
495
|
+
|
436
496
|
# Get the list of retriable errors a node got from deployment logs.
|
437
497
|
# Useful to know if an error is non-deterministic (due to external and temporary factors).
|
438
498
|
#
|
@@ -485,7 +545,7 @@ module HybridPlatformsConductor
|
|
485
545
|
Hash[services.map do |node, node_services|
|
486
546
|
image_id = @nodes_handler.get_image_of(node)
|
487
547
|
sudo = (ssh_user == 'root' ? '' : "#{@nodes_handler.sudo_on(node)} ")
|
488
|
-
# Install
|
548
|
+
# Install corporate certificates if present
|
489
549
|
certificate_actions =
|
490
550
|
if @local_environment && ENV['hpc_certificates']
|
491
551
|
if File.exist?(ENV['hpc_certificates'])
|