hybrid_platforms_conductor 32.18.0 → 33.0.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (28) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +6 -0
  3. data/README.md +3 -3
  4. data/docs/config_dsl.md +23 -1
  5. data/docs/executables.md +6 -7
  6. data/docs/executables/check-node.md +3 -3
  7. data/docs/executables/deploy.md +3 -3
  8. data/docs/executables/dump_nodes_json.md +3 -3
  9. data/docs/executables/test.md +3 -3
  10. data/docs/executables/topograph.md +3 -3
  11. data/docs/plugins.md +21 -0
  12. data/docs/plugins/secrets_reader/cli.md +31 -0
  13. data/docs/plugins/secrets_reader/thycotic.md +46 -0
  14. data/lib/hybrid_platforms_conductor/deployer.rb +96 -36
  15. data/lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/cli.rb +75 -0
  16. data/lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/my_secrets_reader_plugin.rb.sample +46 -0
  17. data/lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/thycotic.rb +87 -0
  18. data/lib/hybrid_platforms_conductor/plugins.rb +1 -0
  19. data/lib/hybrid_platforms_conductor/secrets_reader.rb +31 -0
  20. data/lib/hybrid_platforms_conductor/version.rb +1 -1
  21. data/spec/hybrid_platforms_conductor_test.rb +5 -0
  22. data/spec/hybrid_platforms_conductor_test/api/deployer/config_dsl_spec.rb +22 -0
  23. data/spec/hybrid_platforms_conductor_test/api/deployer/secrets_reader_plugins/cli_spec.rb +63 -0
  24. data/spec/hybrid_platforms_conductor_test/api/deployer/secrets_reader_plugins/thycotic_spec.rb +253 -0
  25. data/spec/hybrid_platforms_conductor_test/executables/options/deployer_spec.rb +0 -182
  26. data/spec/hybrid_platforms_conductor_test/helpers/deployer_test_helpers.rb +249 -13
  27. data/spec/hybrid_platforms_conductor_test/test_secrets_reader_plugin.rb +45 -0
  28. metadata +13 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: b724f6cb69633133e800d381ee1d0bfdd1a1107d7fc388fe1dcbcf423480ca9d
4
- data.tar.gz: 3252bc11e083295c2694df60a9af790d8b01cbe851baebbe1d8e75b656e0acd3
3
+ metadata.gz: ae17e600c52586447071f6f463ee1525754327c0be4a147bec7d3ebcb13b018d
4
+ data.tar.gz: 9e4dc83a6d37fc9c17b99fd9a16eae20b73673dd492dde52679a159ed922a3d5
5
5
  SHA512:
6
- metadata.gz: 43793a4b3f8ec9a1353b4c075289dfcd77574b64a481754db68e1affa1c599500a3c29d8fa12b31c13dd69b47c5a2840f52aef80045fbec8be7a363ca4eb90b7
7
- data.tar.gz: 0451da97325093d7a59684de1ffc77856f4baf5c3a4c76853e6df4492c7cfb1d52cbf9405c0d44b1d8af643e61f64569fc305ed61204be26f081591bdccb6d43
6
+ metadata.gz: d9b89fc3f6ed1fe6e87f23aa1519b32dc62bdcf861b0332088909b1ac8945adf273425c97860c0df5131069921c6adfed60d265cbfe9c1d50a6e4cd5099cc420
7
+ data.tar.gz: d9b3d6d66250552aaec66672b2b9fc97843eb39fc5578a34b74c44f9a4511bccf7978ef59559a4a21b276557e6293418e3350b8151308000d478e22aba84ab56
data/CHANGELOG.md CHANGED
@@ -1,3 +1,9 @@
1
+ # [v33.0.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.18.0...v33.0.0) (2021-06-15 16:10:47)
2
+
3
+ ### Breaking changes
4
+
5
+ * [[Breaking] Add secrets reader plugins with 2 default plugins: cli and thycotic](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/2cfacebe8cfac57de40ef003877da5b99aca5b5e)
6
+
1
7
  # [v32.18.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v32.17.1...v32.18.0) (2021-06-14 15:01:02)
2
8
 
3
9
  ## Global changes
data/README.md CHANGED
@@ -216,13 +216,13 @@ Connector ssh options:
216
216
  --ssh-gateways-conf
217
217
 
218
218
  Deployer options:
219
- -e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
220
- * Local path to a JSON file
221
- * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
222
219
  -p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
223
220
  -t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
224
221
  -W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
225
222
  --retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
223
+
224
+ Secrets reader cli options:
225
+ -e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
226
226
  ```
227
227
 
228
228
  All executables also have the `--debug` switch to display more verbose and debugging information.
data/docs/config_dsl.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  The DSL used in configuration files is comprised of Ruby methods that can be called directly in the main `hpc_config.rb` file.
4
4
 
5
- This DSL can also be completed by plugins. Check [the plugins documentations](plugins) to know about DSL extensions brought by plugins.
5
+ This DSL can also be completed by plugins. Check [the plugins documentations](plugins.md) to know about DSL extensions brought by plugins.
6
6
 
7
7
  # Table of Contents
8
8
  * [`<platform_type>_platform`](#platform_type_platform)
@@ -13,6 +13,7 @@ This DSL can also be completed by plugins. Check [the plugins documentations](pl
13
13
  * [`hybrid_platforms_dir`](#hybrid_platforms_dir)
14
14
  * [`tests_provisioner`](#tests_provisioner)
15
15
  * [`expect_tests_to_fail`](#expect_tests_to_fail)
16
+ * [`read_secrets_from`](#read_secrets_from)
16
17
  * [`send_logs_to`](#send_logs_to)
17
18
  * [`retry_deploy_for_errors_on_stdout`](#retry_deploy_for_errors_on_stdout)
18
19
  * [`retry_deploy_for_errors_on_stderr`](#retry_deploy_for_errors_on_stderr)
@@ -201,6 +202,27 @@ for_nodes('/tst/') do
201
202
  end
202
203
  ```
203
204
 
205
+ <a name="read_secrets_from"></a>
206
+ ## `read_secrets_from`
207
+
208
+ Set the list of [secrets reader plugins](plugins.md#secrets_reader) to use.
209
+ By default (if no plugins is specifically set) the [secrets reader plugin `cli`](plugins/secrets_reader/cli.md) is being used.
210
+
211
+ Takes the list of secrets reader plugin names, as symbols, as a parameter.
212
+
213
+ Can be applied to subset of nodes using the [`for_nodes` DSL method](#for_nodes).
214
+
215
+ Examples:
216
+ ```ruby
217
+ # By default, get secrets from the command-line
218
+ read_secrets_from :cli
219
+
220
+ # All our production nodes also have their secrets stored on a secured Thycotic server
221
+ for_nodes('/prd/') do
222
+ read_secrets_from :thycotic
223
+ end
224
+ ```
225
+
204
226
  <a name="send_logs_to"></a>
205
227
  ## `send_logs_to`
206
228
 
data/docs/executables.md CHANGED
@@ -132,25 +132,22 @@ The Deployer options are used to drive a deployment (be it in why-run mode or no
132
132
 
133
133
  ```
134
134
  Deployer options:
135
- -e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
136
- * Local path to a JSON file
137
- * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
138
135
  -p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
139
136
  -t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
140
137
  -W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
141
138
  --retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
139
+
140
+ Secrets reader cli options:
141
+ -e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
142
142
  ```
143
143
 
144
- * `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...).
145
- The location can be:
146
- * A local file path (for example /path/to/file.json).
147
- * A Thycotic Secret Server URL followed by a secret id (for example https://portal.muc.msp.my_company.net/SecretServer:8845).
148
144
  * `--parallel`: Specify that the deployment process should perform concurrently on the different nodes it has to deploy to.
149
145
  * `--timeout SECS`: Specify the timeout (in seconds) to apply while deploying. This can be set only in why-run mode.
150
146
  * `--why-run`: Specify the why-run mode. The why-run mode is used to simulate a deployment on the nodes, and report what a real deployment would have changed on the node.
151
147
  * `--retries-on-error NBR`: Specify the number of retries deploys can do in case of non-deterministic errors.
152
148
  Non-deterministic errors are matched using a set of strings or regular expressions that can be configured in the `hpc_config.rb` file of any platform, using the `retry_deploy_for_errors_on_stdout` and `retry_deploy_for_errors_on_stderr` properties:
153
149
  For example:
150
+
154
151
  ```ruby
155
152
  retry_deploy_for_errors_on_stdout [
156
153
  'This is a raw string error that will be matched against stdout',
@@ -161,6 +158,8 @@ retry_deploy_for_errors_on_stderr [
161
158
  ]
162
159
  ```
163
160
 
161
+ * `--secrets SECRETS_LOCATION`: Specify a JSON file storing secrets that can be used by the deployment process. Secrets are values that are needed for deployment but that should not be part of the platforms repositories (such as passwords, API keys, SSL certificates...). This option is used by the [`cli` secrets reader plugin](plugins/secrets_reader/cli.md). See [secrets reader plugins](plugins.md#secrets_reader) for more info about secrets retrieval.
162
+
164
163
  ## JSON dump options
165
164
 
166
165
  The JSON dump options drive the way nodes' JSON information is being dumped.
@@ -65,11 +65,11 @@ Connector ssh options:
65
65
  --ssh-gateways-conf
66
66
 
67
67
  Deployer options:
68
- -e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
69
- * Local path to a JSON file
70
- * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
71
68
  -t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
72
69
  --retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
70
+
71
+ Secrets reader cli options:
72
+ -e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
73
73
  ```
74
74
 
75
75
  ## Examples
@@ -82,13 +82,13 @@ Connector ssh options:
82
82
  --ssh-gateways-conf
83
83
 
84
84
  Deployer options:
85
- -e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
86
- * Local path to a JSON file
87
- * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
88
85
  -p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
89
86
  -t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
90
87
  -W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
91
88
  --retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
89
+
90
+ Secrets reader cli options:
91
+ -e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
92
92
  ```
93
93
 
94
94
  ## Examples
@@ -54,13 +54,13 @@ Connector ssh options:
54
54
  --ssh-gateways-conf
55
55
 
56
56
  Deployer options:
57
- -e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
58
- * Local path to a JSON file
59
- * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
60
57
  -t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
61
58
  -W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
62
59
  --retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
63
60
 
61
+ Secrets reader cli options:
62
+ -e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
63
+
64
64
  JSON dump options:
65
65
  -k, --skip-run Skip the actual gathering of dumps in run_logs. If set, the current run_logs content will be used.
66
66
  -j, --json-dir DIRECTORY Specify the output directory in which JSON files are being written. Defaults to nodes_json.
@@ -93,11 +93,11 @@ Connector ssh options:
93
93
  --ssh-gateways-conf
94
94
 
95
95
  Deployer options:
96
- -e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
97
- * Local path to a JSON file
98
- * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
99
96
  --retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
100
97
 
98
+ Secrets reader cli options:
99
+ -e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
100
+
101
101
  Tests runner options:
102
102
  -i, --tests-list FILE_NAME Specify a tests file name. The file should contain a list of tests name (1 per line). Can be used several times.
103
103
  -k, --skip-run Skip running the check-node commands for real, and just analyze existing run logs.
@@ -39,12 +39,12 @@ Connector ssh options:
39
39
  --ssh-gateways-conf
40
40
 
41
41
  Deployer options:
42
- -e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
43
- * Local path to a JSON file
44
- * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
45
42
  -t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
46
43
  --retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
47
44
 
45
+ Secrets reader cli options:
46
+ -e, --secrets JSON_FILE Specify a secrets location from a local JSON file. Can be specified several times.
47
+
48
48
  JSON dump options:
49
49
  -j, --json-dir DIRECTORY Specify the output directory in which JSON files are being written. Defaults to nodes_json.
50
50
 
data/docs/plugins.md CHANGED
@@ -14,6 +14,7 @@ Following are all possible plugin types and the plugins shipped by default with
14
14
  * [`platform_handler`](#platform_handler)
15
15
  * [`provisioner`](#provisioner)
16
16
  * [`report`](#report)
17
+ * [`secrets_reader`](#secrets_reader)
17
18
  * [`test`](#test)
18
19
  * [`test_report`](#test_report)
19
20
 
@@ -177,6 +178,26 @@ Plugins shipped by default:
177
178
  * [`mediawiki`](plugins/report/mediawiki.md)
178
179
  * [`stdout`](plugins/report/stdout.md)
179
180
 
181
+ <a name="secrets_reader"></a>
182
+ ## Secrets readers
183
+
184
+ Secrets reader are responsible for fetching secrets (passwords, private keys, API tokens...) needed during deployment from various sources (command line, environment, vaults, secrets servers...).
185
+
186
+ Corresponding plugin type: `secrets_reader`.
187
+
188
+ These plugins add new ways to retrieve secrets used by the [`Deployer`](../lib/hybrid_platforms_conductor/deployer.rb)
189
+
190
+ Examples of secrets readers are:
191
+ * Command-line: Give secrets from a local file.
192
+ * Vault: Get secrets from vaults (encrypted databases).
193
+ * Secrets servers: Query secrets servers to retrieve secrets.
194
+
195
+ Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/secrets_reader/my_secrets_reader_plugin.rb.sample) to know more about the API that needs to be implemented by such plugins.
196
+
197
+ Plugins shipped by default:
198
+ * [`cli`](plugins/secrets_reader/cli.md)
199
+ * [`thycotic`](plugins/secrets_reader/thycotic.md)
200
+
180
201
  <a name="test"></a>
181
202
  ## Tests
182
203
 
@@ -0,0 +1,31 @@
1
+ # Secrets reader plugin: `cli`
2
+
3
+ The `cli` secrets reader plugin reads secrets from a local JSON file that can be given through the `--secrets` command-line parameter.
4
+
5
+ Example:
6
+ ```bash
7
+ ./bin/deploy --node my_node --secrets /path/to/my_secrets.json
8
+ ```
9
+
10
+ ## Config DSL extension
11
+
12
+ None
13
+
14
+ ## Used credentials
15
+
16
+ | Credential | Usage
17
+ | --- | --- |
18
+
19
+ ## Used Metadata
20
+
21
+ | Metadata | Type | Usage
22
+ | --- | --- | --- |
23
+
24
+ ## Used environment variables
25
+
26
+ | Variable | Usage
27
+ | --- | --- |
28
+
29
+ ## External tools dependencies
30
+
31
+ None
@@ -0,0 +1,46 @@
1
+ # Secrets reader plugin: `thycotic`
2
+
3
+ The `thycotic` secrets reader plugin retrieves secrets from a [Thycotic secrets server](https://thycotic.com/products/secret-server-vdo/), using its SOAP API.
4
+
5
+ It is configured using the `secrets_from_thycotic` (see below) config DSL and uses the `thycotic` credential ID to authenticate.
6
+
7
+ ## Config DSL extension
8
+
9
+ ### `secrets_from_thycotic`
10
+
11
+ Define a Thycotic URL and Thycotic secret ID to fetch from a Thycotic server.
12
+ The Thycotic secret should contain a JSON file that will be retrieved locally to be used as a secrets source. The local copy will then be removed after deployment.
13
+
14
+ Can be applied to subset of nodes using the [`for_nodes` DSL method](/docs/config_dsl.md#for_nodes).
15
+
16
+ It takes the following parameters:
17
+ * **thycotic_url** (`String`): The Thycotic server URL.
18
+ * **secret_id** (`Integer`): The Thycotic secret ID containing the secrets file to be used as secrets.
19
+
20
+ Example:
21
+ ```ruby
22
+ secrets_from_thycotic(
23
+ thycotic_url: 'https://my-thycotic-server.my-domain.com/SecretServer',
24
+ secret_id: 1107
25
+ )
26
+ ```
27
+
28
+ ## Used credentials
29
+
30
+ | Credential | Usage
31
+ | --- | --- |
32
+ | `thycotic` | Used to authenticate on the Thycotic server's SOAP API |
33
+
34
+ ## Used Metadata
35
+
36
+ | Metadata | Type | Usage
37
+ | --- | --- | --- |
38
+
39
+ ## Used environment variables
40
+
41
+ | Variable | Usage
42
+ | --- | --- |
43
+
44
+ ## External tools dependencies
45
+
46
+ None
@@ -11,7 +11,6 @@ require 'hybrid_platforms_conductor/logger_helpers'
11
11
  require 'hybrid_platforms_conductor/nodes_handler'
12
12
  require 'hybrid_platforms_conductor/services_handler'
13
13
  require 'hybrid_platforms_conductor/plugins'
14
- require 'hybrid_platforms_conductor/thycotic'
15
14
 
16
15
  module HybridPlatformsConductor
17
16
 
@@ -27,6 +26,12 @@ module HybridPlatformsConductor
27
26
  # Array< Hash<Symbol, Object> >
28
27
  attr_reader :deployment_logs
29
28
 
29
+ # List of secrets reader plugins. Each info has the following properties:
30
+ # * *nodes_selectors_stack* (Array<Object>): Stack of nodes selectors impacted by this rule.
31
+ # * *secrets_readers* (Array<Symbol>): List of log plugins to be used to store deployment logs.
32
+ # Array< Hash<Symbol, Object> >
33
+ attr_reader :secrets_readers
34
+
30
35
  # Integer: Timeout (in seconds) for packaging repositories
31
36
  attr_reader :packaging_timeout_secs
32
37
 
@@ -34,6 +39,7 @@ module HybridPlatformsConductor
34
39
  def init_deployer_config
35
40
  @packaging_timeout_secs = 60
36
41
  @deployment_logs = []
42
+ @secrets_readers = []
37
43
  end
38
44
 
39
45
  # Set the packaging timeout
@@ -55,6 +61,17 @@ module HybridPlatformsConductor
55
61
  }
56
62
  end
57
63
 
64
+ # Set the secrets readers
65
+ #
66
+ # Parameters::
67
+ # * *secrets_readers* (Symbol or Array<Symbol>): The list of (or single) secrets readers plugins to be used
68
+ def read_secrets_from(*secrets_readers)
69
+ @secrets_readers << {
70
+ nodes_selectors_stack: current_nodes_selectors_stack,
71
+ secrets_readers: secrets_readers.flatten
72
+ }
73
+ end
74
+
58
75
  end
59
76
 
60
77
  include LoggerHelpers
@@ -73,10 +90,6 @@ module HybridPlatformsConductor
73
90
  # Boolean
74
91
  attr_accessor :concurrent_execution
75
92
 
76
- # The list of JSON secrets
77
- # Array<Hash>
78
- attr_accessor :secrets
79
-
80
93
  # Are we deploying in a local environment?
81
94
  # Boolean
82
95
  attr_accessor :local_environment
@@ -110,7 +123,21 @@ module HybridPlatformsConductor
110
123
  @nodes_handler = nodes_handler
111
124
  @actions_executor = actions_executor
112
125
  @services_handler = services_handler
113
- @secrets = []
126
+ @override_secrets = nil
127
+ @secrets_readers = Plugins.new(
128
+ :secrets_reader,
129
+ logger: @logger,
130
+ logger_stderr: @logger_stderr,
131
+ init_plugin: proc do |plugin_class|
132
+ plugin_class.new(
133
+ logger: @logger,
134
+ logger_stderr: @logger_stderr,
135
+ config: @config,
136
+ cmd_runner: @cmd_runner,
137
+ nodes_handler: @nodes_handler
138
+ )
139
+ end
140
+ )
114
141
  @provisioners = Plugins.new(:provisioner, logger: @logger, logger_stderr: @logger_stderr)
115
142
  @log_plugins = Plugins.new(
116
143
  :log,
@@ -144,30 +171,6 @@ module HybridPlatformsConductor
144
171
  def options_parse(options_parser, parallel_switch: true, why_run_switch: false, timeout_options: true)
145
172
  options_parser.separator ''
146
173
  options_parser.separator 'Deployer options:'
147
- options_parser.on(
148
- '-e', '--secrets SECRETS_LOCATION',
149
- 'Specify a secrets location. Can be specified several times. Location can be:',
150
- '* Local path to a JSON file',
151
- '* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.'
152
- ) do |secrets_location|
153
- @secrets << JSON.parse(
154
- if secrets_location =~ /^(https?:\/\/.+):(\d+)$/
155
- url = $1
156
- secret_id = $2
157
- secret = nil
158
- Thycotic.with_thycotic(url, @logger, @logger_stderr) do |thycotic|
159
- secret_file_item_id = thycotic.get_secret(secret_id).dig(:secret, :items, :secret_item, :id)
160
- raise "Unable to fetch secret file ID #{secrets_location}" if secret_file_item_id.nil?
161
- secret = thycotic.download_file_attachment_by_item_id(secret_id, secret_file_item_id)
162
- raise "Unable to fetch secret file attachment from #{secrets_location}" if secret.nil?
163
- end
164
- secret
165
- else
166
- raise "Missing secret file: #{secrets_location}" unless File.exist?(secrets_location)
167
- File.read(secrets_location)
168
- end
169
- )
170
- end
171
174
  options_parser.on('-p', '--parallel', 'Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)') do
172
175
  @concurrent_execution = true
173
176
  end if parallel_switch
@@ -180,6 +183,14 @@ module HybridPlatformsConductor
180
183
  options_parser.on('--retries-on-error NBR', "Number of retries in case of non-deterministic errors (defaults to #{@nbr_retries_on_error})") do |nbr_retries|
181
184
  @nbr_retries_on_error = nbr_retries.to_i
182
185
  end
186
+ # Display options secrets readers might have
187
+ @secrets_readers.each do |secret_reader_name, secret_reader|
188
+ if secret_reader.respond_to?(:options_parse)
189
+ options_parser.separator ''
190
+ options_parser.separator "Secrets reader #{secret_reader_name} options:"
191
+ secret_reader.options_parse(options_parser)
192
+ end
193
+ end
183
194
  end
184
195
 
185
196
  # Validate that parsed parameters are valid
@@ -190,6 +201,16 @@ module HybridPlatformsConductor
190
201
  # String: File used as a Futex for packaging
191
202
  PACKAGING_FUTEX_FILE = "#{Dir.tmpdir}/hpc_packaging"
192
203
 
204
+ # Override the secrets with a given JSON.
205
+ # When using this method with a secrets Hash, further deployments will not query secrets readers, but will use those secrets directly.
206
+ # Useful to override secrets in test conditions when using dummy secrets for example.
207
+ #
208
+ # Parameters::
209
+ # * *secrets* (Hash or nil): Secrets to take into account in place of secrets readers, or nil to cancel a previous overriding and use secrets readers instead.
210
+ def override_secrets(secrets)
211
+ @override_secrets = secrets
212
+ end
213
+
193
214
  # Deploy on a given list of nodes selectors.
194
215
  # The workflow is the following:
195
216
  # 1. Package the services to be deployed, considering the nodes, services and context (options, secrets, environment...)
@@ -209,10 +230,20 @@ module HybridPlatformsConductor
209
230
 
210
231
  # Get the secrets to be deployed
211
232
  secrets = {}
212
- @secrets.each do |secret_json|
213
- secrets.merge!(secret_json) do |key, value1, value2|
214
- raise "Secret #{key} has conflicting values between different secret JSON files." if value1 != value2
215
- value1
233
+ if @override_secrets
234
+ secrets = @override_secrets
235
+ else
236
+ services_to_deploy.each do |node, services|
237
+ # If there is no config for secrets, just use cli
238
+ (@config.secrets_readers.empty? ? [{ secrets_readers: %i[cli] }] : @nodes_handler.select_confs_for_node(node, @config.secrets_readers)).inject([]) do |secrets_readers, secrets_readers_info|
239
+ secrets_readers + secrets_readers_info[:secrets_readers]
240
+ end.sort.uniq.each do |secrets_reader|
241
+ services.each do |service|
242
+ node_secrets = @secrets_readers[secrets_reader].secrets_for(node, service)
243
+ conflicting_path = safe_merge(secrets, node_secrets)
244
+ raise "Secret set at path #{conflicting_path.join('->')} by #{secrets_reader} for service #{service} on node #{node} has conflicting values (#{log_debug? ? "#{node_secrets.dig(*conflicting_path)} != #{secrets.dig(*conflicting_path)}" : 'set debug for value details'})." unless conflicting_path.nil?
245
+ end
246
+ end
216
247
  end
217
248
  end
218
249
 
@@ -370,7 +401,7 @@ module HybridPlatformsConductor
370
401
  deployer.local_environment = true
371
402
  # Ignore secrets that might have been given: in Docker containers we always use dummy secrets
372
403
  dummy_secrets_file = "#{@config.hybrid_platforms_dir}/dummy_secrets.json"
373
- deployer.secrets = File.exist?(dummy_secrets_file) ? [JSON.parse(File.read(dummy_secrets_file))] : []
404
+ deployer.override_secrets(File.exist?(dummy_secrets_file) ? JSON.parse(File.read(dummy_secrets_file)) : {})
374
405
  yield deployer, instance
375
406
  end
376
407
  rescue
@@ -433,6 +464,35 @@ module HybridPlatformsConductor
433
464
 
434
465
  private
435
466
 
467
+ # Safe-merge 2 hashes.
468
+ # Safe-merging is done by:
469
+ # * Merging values that are hashes.
470
+ # * Reporting errors when values conflict.
471
+ # When values are conflicting, the initial hash won't modify those conflicting values and will stop the merge.
472
+ #
473
+ # Parameters::
474
+ # * *hash* (Hash): Hash to be modified merging hash_to_merge
475
+ # * *hash_to_merge* (Hash): Hash to be merged into hash
476
+ # Result::
477
+ # * nil or Array<Object>: nil in case of success, or the keys path leading to a conflicting value in case of error
478
+ def safe_merge(hash, hash_to_merge)
479
+ conflicting_path = nil
480
+ hash_to_merge.each do |key, value_to_merge|
481
+ if hash.key?(key)
482
+ if hash[key].is_a?(Hash) && value_to_merge.is_a?(Hash)
483
+ sub_conflicting_path = safe_merge(hash[key], value_to_merge)
484
+ conflicting_path = [key] + sub_conflicting_path unless sub_conflicting_path.nil?
485
+ elsif hash[key] != value_to_merge
486
+ conflicting_path = [key]
487
+ end
488
+ else
489
+ hash[key] = value_to_merge
490
+ end
491
+ break unless conflicting_path.nil?
492
+ end
493
+ conflicting_path
494
+ end
495
+
436
496
  # Get the list of retriable errors a node got from deployment logs.
437
497
  # Useful to know if an error is non-deterministic (due to external and temporary factors).
438
498
  #
@@ -485,7 +545,7 @@ module HybridPlatformsConductor
485
545
  Hash[services.map do |node, node_services|
486
546
  image_id = @nodes_handler.get_image_of(node)
487
547
  sudo = (ssh_user == 'root' ? '' : "#{@nodes_handler.sudo_on(node)} ")
488
- # Install My_company corporate certificates if present
548
+ # Install corporate certificates if present
489
549
  certificate_actions =
490
550
  if @local_environment && ENV['hpc_certificates']
491
551
  if File.exist?(ENV['hpc_certificates'])