secret_config 0.7.0 → 0.10.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +17 -756
- data/Rakefile +7 -7
- data/bin/{secret_config → secret-config} +1 -1
- data/lib/secret_config.rb +31 -2
- data/lib/secret_config/cli.rb +154 -97
- data/lib/secret_config/config.rb +44 -0
- data/lib/secret_config/parser.rb +76 -0
- data/lib/secret_config/providers/file.rb +17 -4
- data/lib/secret_config/providers/ssm.rb +9 -2
- data/lib/secret_config/registry.rb +46 -33
- data/lib/secret_config/setting_interpolator.rb +14 -27
- data/lib/secret_config/string_interpolator.rb +10 -9
- data/lib/secret_config/utils.rb +1 -1
- data/lib/secret_config/version.rb +1 -1
- data/test/config/application.yml +35 -5
- data/test/parser_test.rb +82 -0
- data/test/providers/file_test.rb +5 -5
- data/test/providers/ssm_test.rb +37 -12
- data/test/registry_test.rb +56 -26
- data/test/secret_config_test.rb +35 -4
- data/test/setting_interpolator_test.rb +43 -43
- data/test/test_helper.rb +6 -6
- data/test/utils_test.rb +4 -4
- metadata +9 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3379ac282a4a14c4b577024fbd417e94aaa498c974b5371026d5142515050810
|
4
|
+
data.tar.gz: f70b58807cfa0bbe3d668d975b1dba7ca4ee67126f7fe0b380a2b3b6e66c4cae
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 291fd4a292787ad879136b41511faebd7e9ad4e18caf071618d93c0b6ade144cfd4deab3e01313e81b130858d12037141a83bb99a9d516df316b57519beb12a9
|
7
|
+
data.tar.gz: 7eec020e42b794a3230dcff53bd6dde3ff67ae7bf4cc4dd8477d6ef13d7be4afead65cefaa337ac74abf8f85f1b9d30f1fb69c3821391c45e92aaa22b04c3a50
|
data/README.md
CHANGED
@@ -5,774 +5,35 @@ Centralized Configuration and Secrets Management for Ruby and Rails applications
|
|
5
5
|
|
6
6
|
Securely store configuration information centrally, supporting multiple tenants of the same application.
|
7
7
|
|
8
|
-
|
8
|
+
Checkout https://config.rocketjob.io/
|
9
9
|
|
10
|
-
|
11
|
-
* Settings
|
12
|
-
* Passwords
|
13
|
-
* Encryption keys and certificates
|
10
|
+
## Documentation
|
14
11
|
|
15
|
-
|
12
|
+
* [Guide](https://config.rocketjob.io/)
|
16
13
|
|
17
|
-
|
18
|
-
* File
|
19
|
-
* Development and testing use.
|
20
|
-
* Environment Variables
|
21
|
-
* Environment Variables take precedence and can be used to override any setting.
|
22
|
-
* AWS System Manager Parameter Store
|
23
|
-
* Encrypt and securely store secrets such as passwords centrally.
|
14
|
+
## Support
|
24
15
|
|
25
|
-
|
26
|
-
*
|
27
|
-
* float
|
28
|
-
* string
|
29
|
-
* boolean
|
30
|
-
* symbol
|
31
|
-
* json
|
16
|
+
* Questions? Join the chat room on Gitter for [rocketjob support](https://gitter.im/rocketjob/support)
|
17
|
+
* [Report bugs](https://github.com/rocketjob/secret_config/issues)
|
32
18
|
|
33
|
-
|
34
|
-
* base64
|
19
|
+
## v0.10 Upgrade Notes
|
35
20
|
|
36
|
-
|
21
|
+
String interpolation has been changed to use `$` instead of `%`. Please change
|
22
|
+
all interpolated strings to use `$` before upgrading.
|
37
23
|
|
38
|
-
|
24
|
+
Example: `%{date}` needs to be changed to `${date}`
|
39
25
|
|
40
|
-
|
41
|
-
* Environment variables force all config into a single level.
|
42
|
-
* Reduces the number of environment variables.
|
43
|
-
* In a large application the number of secrets can grow dramatically.
|
44
|
-
* Replaces sensitive data stored in local yaml or configuration files.
|
45
|
-
* Including securing and managing encryption keys.
|
46
|
-
* When encryption keys change, such as during a key rotation, config files don't have to be changed.
|
47
|
-
* Removes security concerns with placing passwords in the clear into environment variables.
|
48
|
-
* AWS System Manager Parameter Store does not charge for parameters.
|
49
|
-
* Still recommend using a custom KMS key that charges only $1 per month.
|
50
|
-
* Amounts as of 4/2019. Confirm what AWS charges you for these services.
|
51
|
-
* AWS Secrets Manager charges for every secret being managed, which can accumulate quickly with large projects.
|
52
|
-
* Configure multiple distinct application instances to support multiple tenants.
|
53
|
-
* For example, use separate databases with unique credentials for each tenant.
|
54
|
-
* Separation of responsibilities is achieved since operations can manage production configuration.
|
55
|
-
* Developers do not need to be involved with production configuration such as host names and passwords.
|
56
|
-
* All values are encrypted by default when stored in the AWS Parameter Store.
|
57
|
-
* Prevents accidentally not encrypting sensitive data.
|
58
|
-
|
59
|
-
## Introduction
|
26
|
+
## v0.9 Upgrade Notes
|
60
27
|
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
The in-memory copy of the registry can be refreshed at any time by calling `SecretConfig.refresh!`. It can be refreshed
|
65
|
-
via a process signal, or by calling it through an event, or via a messaging system.
|
66
|
-
|
67
|
-
It is suggested that any programmatic lookup to values stored in Secret Config are called every time a value is
|
68
|
-
being used, rather than creating a local copy of the value. This ensures that a refresh of the registry will take effect
|
69
|
-
immediately for any code reading from Secret Config.
|
70
|
-
|
71
|
-
## API
|
72
|
-
|
73
|
-
When Secret Config starts up it reads all configuration entries immediately for all keys under the configured path.
|
74
|
-
This means that once Secret Config has initialized all calls to Secret Config are extremely fast.
|
75
|
-
|
76
|
-
Secret Config supports the following programmatic interface:
|
77
|
-
|
78
|
-
### Read values
|
79
|
-
|
80
|
-
Fetch the value for the supplied key, returning nil if not found:
|
81
|
-
|
82
|
-
~~~ruby
|
83
|
-
# Key is present:
|
84
|
-
SecretConfig["logger/level"]
|
85
|
-
# => "info"
|
86
|
-
|
87
|
-
# Key is missing:
|
88
|
-
SecretConfig["logger/blah"]
|
89
|
-
# => nil
|
90
|
-
~~~
|
91
|
-
|
92
|
-
Fetch the value for the supplied key, raising `SecretConfig::MissingMandatoryKey` if not found:
|
93
|
-
|
94
|
-
~~~ruby
|
95
|
-
# Key is present:
|
96
|
-
SecretConfig.fetch("logger/level")
|
97
|
-
# => "info"
|
98
|
-
|
99
|
-
# Key is missing:
|
100
|
-
SecretConfig.fetch("logger/blah")
|
101
|
-
# => SecretConfig::MissingMandatoryKey (Missing configuration value for /development/logger/blah)
|
102
|
-
~~~
|
103
|
-
|
104
|
-
A default value can be supplied when the key is not found in the registry:
|
105
|
-
|
106
|
-
~~~ruby
|
107
|
-
SecretConfig.fetch("logger/level", default: "info")
|
108
|
-
# => "info"
|
109
|
-
~~~
|
110
|
-
|
111
|
-
Since AWS SSM Parameter store and environment variables only support string values,
|
112
|
-
it is neccessary to convert the string back to the type required by the program.
|
113
|
-
|
114
|
-
The following types are supported:
|
115
|
-
`:integer`
|
116
|
-
`:float`
|
117
|
-
`:string`
|
118
|
-
`:boolean`
|
119
|
-
`:symbol`
|
120
|
-
`:json`
|
121
|
-
|
122
|
-
~~~ruby
|
123
|
-
# Without type conversion:
|
124
|
-
SecretConfig.fetch("symmetric_encryption/version")
|
125
|
-
# => "0"
|
126
|
-
|
127
|
-
# With type conversion:
|
128
|
-
SecretConfig.fetch("symmetric_encryption/version", type: :integer)
|
129
|
-
# => 0
|
130
|
-
~~~
|
131
|
-
|
132
|
-
When storing binary data, it should be encoded with strict base64 encoding. To automatically convert it back to binary
|
133
|
-
specify the encoding as `:base64`
|
134
|
-
|
135
|
-
~~~ruby
|
136
|
-
# Return a value that was stored in Base64 encoding format:
|
137
|
-
SecretConfig.fetch("symmetric_encryption/iv")
|
138
|
-
# => "FW+/wLubAYM+ZU0bWQj59Q=="
|
139
|
-
|
140
|
-
# Base64 decode a value that was stored in Base64 encoding format:
|
141
|
-
SecretConfig.fetch("symmetric_encryption/iv", encoding: :base64)
|
142
|
-
# => "\x15o\xBF\xC0\xBB\x9B\x01\x83>eM\eY\b\xF9\xF5"
|
143
|
-
~~~
|
144
|
-
|
145
|
-
### Key presence
|
146
|
-
|
147
|
-
Returns whether a key is present in the registry:
|
148
|
-
|
149
|
-
~~~ruby
|
150
|
-
SecretConfig.key?("logger/level")
|
151
|
-
# => true
|
152
|
-
~~~
|
153
|
-
|
154
|
-
### Write values
|
155
|
-
|
156
|
-
When Secret Config is configured to use the AWS SSM Parameter store, its values can be modified:
|
157
|
-
|
158
|
-
~~~ruby
|
159
|
-
SecretConfig["logger/level"] = "debug"
|
160
|
-
~~~
|
161
|
-
|
162
|
-
~~~ruby
|
163
|
-
SecretConfig.set("logger/level", "debug")
|
164
|
-
~~~
|
165
|
-
|
166
|
-
### Configuration
|
167
|
-
|
168
|
-
Returns a Hash copy of the configuration as a tree:
|
169
|
-
|
170
|
-
~~~ruby
|
171
|
-
SecretConfig.configuration
|
172
|
-
~~~
|
173
|
-
|
174
|
-
### Refresh Configuration
|
175
|
-
|
176
|
-
Tell Secret Config to refresh its in-memory copy of the configuration settings.
|
177
|
-
|
178
|
-
~~~ruby
|
179
|
-
SecretConfig.refresh!
|
180
|
-
~~~
|
181
|
-
|
182
|
-
Example, refresh the registry any time a SIGUSR2 is raised, add the following code on startup:
|
183
|
-
|
184
|
-
~~~ruby
|
185
|
-
Signal.trap('USR2') do
|
186
|
-
SecretConfig.refresh!
|
187
|
-
end
|
188
|
-
~~~
|
189
|
-
|
190
|
-
Then to make the process refresh it registry:
|
191
|
-
~~~shell
|
192
|
-
kill -SIGUSR2 1234
|
193
|
-
~~~
|
194
|
-
|
195
|
-
Where `1234` above is the process PID.
|
196
|
-
|
197
|
-
## Development and Test use
|
198
|
-
|
199
|
-
In the development environment create the file `config/application.yml` within which to store local development credentials.
|
200
|
-
Depending on your team setup you may want to use the same file for all developers so can check it into you change control system.
|
201
|
-
|
202
|
-
For example: `config/application.yml`
|
203
|
-
|
204
|
-
~~~yaml
|
205
|
-
development:
|
206
|
-
mysql:
|
207
|
-
database: secret_config_development
|
208
|
-
username: secret_config
|
209
|
-
password: secret_configrules
|
210
|
-
host: 127.0.0.1
|
211
|
-
|
212
|
-
mongo:
|
213
|
-
database: secret_config_development
|
214
|
-
primary: 127.0.0.1:27017
|
215
|
-
secondary: 127.0.0.1:27018
|
216
|
-
|
217
|
-
secrets:
|
218
|
-
secret_key_base: somereallylongstring
|
219
|
-
|
220
|
-
test:
|
221
|
-
mysql:
|
222
|
-
database: secret_config_test
|
223
|
-
username: secret_config
|
224
|
-
password: secret_configrules
|
225
|
-
host: 127.0.0.1
|
226
|
-
|
227
|
-
mongo:
|
228
|
-
database: secret_config_test
|
229
|
-
primary: 127.0.0.1:27017
|
230
|
-
secondary: 127.0.0.1:27018
|
231
|
-
|
232
|
-
secrets:
|
233
|
-
secret_key_base: somereallylongteststring
|
234
|
-
~~~
|
235
|
-
|
236
|
-
Note how the hierarchical nature of configuration values is maintained. Typical environment variable approaches have
|
237
|
-
to flatten everything into a single level.
|
238
|
-
|
239
|
-
Note: Do not put any production credentials into this file.
|
240
|
-
|
241
|
-
### Environment Variables
|
242
|
-
|
243
|
-
Any of the above values can be overridden with an environment variable, unless explicitly configured `SecretConfig.check_env_var = false`.
|
244
|
-
|
245
|
-
To overwrite any of these settings with an environment variable:
|
246
|
-
|
247
|
-
* Join the keys together with an '_'
|
248
|
-
* Convert to uppercase
|
249
|
-
|
250
|
-
For example, `mysql/host` can be overridden with the env var:
|
251
|
-
|
252
|
-
export MYSQL_HOST=test.server
|
253
|
-
|
254
|
-
### Applying to existing config files
|
255
|
-
|
256
|
-
Go through all the configuration files and look for sensitive data such as passwords:
|
257
|
-
|
258
|
-
Example, an unchanged common `database.yml`:
|
259
|
-
|
260
|
-
~~~yaml
|
261
|
-
defaults: &defaults
|
262
|
-
encoding: utf8
|
263
|
-
adapter: mysql2
|
264
|
-
|
265
|
-
development:
|
266
|
-
<<: *defaults
|
267
|
-
database: secure_config_development
|
268
|
-
username: jack
|
269
|
-
password: jackrules
|
270
|
-
host: localhost
|
271
|
-
|
272
|
-
test:
|
273
|
-
<<: *defaults
|
274
|
-
database: secure_config_test
|
275
|
-
username: tester
|
276
|
-
password: khjsdjhdsjhdsr32
|
277
|
-
host: test.server
|
278
|
-
|
279
|
-
production:
|
280
|
-
<<: *defaults
|
281
|
-
database: secure_config_production
|
282
|
-
username: product
|
283
|
-
password: donotexpose45
|
284
|
-
host: production.server
|
285
|
-
~~~
|
286
|
-
|
287
|
-
Replace the sensitive data with a `SecureConfig.fetch`:
|
288
|
-
|
289
|
-
Updated `database.yml`:
|
290
|
-
|
291
|
-
~~~yaml
|
292
|
-
configuration: &configuration
|
293
|
-
database: <%= SecretConfig.fetch("mysql/database") %>
|
294
|
-
username: <%= SecretConfig.fetch("mysql/username") %>
|
295
|
-
password: <%= SecretConfig.fetch("mysql/password") %>
|
296
|
-
host: <%= SecretConfig.fetch("mysql/host") %>
|
297
|
-
encoding: utf8
|
298
|
-
adapter: mysql2
|
299
|
-
|
300
|
-
development:
|
301
|
-
<<: *configuration
|
302
|
-
|
303
|
-
test:
|
304
|
-
<<: *configuration
|
305
|
-
|
306
|
-
production:
|
307
|
-
<<: *configuration
|
308
|
-
~~~
|
309
|
-
|
310
|
-
Since the secrets are externalized the configuration between environments is simpler.
|
311
|
-
|
312
|
-
### Replacing custom config files
|
313
|
-
|
314
|
-
When writing new components or gems, instead of requiring a proprietary config file, refer
|
315
|
-
to the settings programmatically:
|
316
|
-
|
317
|
-
For example, somewhere in your codebase you need a persistent http connection:
|
318
|
-
|
319
|
-
~~~ruby
|
320
|
-
def http_client
|
321
|
-
@http_client ||=
|
322
|
-
PersistentHTTP.new(
|
323
|
-
name: 'HTTPClient',
|
324
|
-
url: SecretConfig.fetch('http_client/url'),
|
325
|
-
logger: logger,
|
326
|
-
pool_size: SecretConfig.fetch('http_client/pool_size', type: :integer, default: 10),
|
327
|
-
warn_timeout: SecretConfig.fetch('http_client/warn_timeout', type: :float, default: 0.25),
|
328
|
-
open_timeout: SecretConfig.fetch('http_client/open_timeout', type: :float, default: 30),
|
329
|
-
read_timeout: SecretConfig.fetch('http_client/read_timeout', type: :float, default: 30),
|
330
|
-
force_retry: true
|
331
|
-
)
|
332
|
-
end
|
333
|
-
~~~
|
334
|
-
|
335
|
-
Then the application that uses the above library / gem just needs to add the relevant entries to their
|
336
|
-
`application.yml` file:
|
337
|
-
|
338
|
-
~~~yaml
|
339
|
-
http_client:
|
340
|
-
url: https://test.example.com
|
341
|
-
pool_size: 20
|
342
|
-
read_timeout: 300
|
343
|
-
~~~
|
344
|
-
|
345
|
-
This avoids a custom config file just for the above library.
|
346
|
-
|
347
|
-
Additionally the values can be overridden with environment variables at any time:
|
348
|
-
|
349
|
-
export HTTP_CLIENT_URL=https://production.example.com
|
350
|
-
|
351
|
-
## Configuration
|
352
|
-
|
353
|
-
Add the following line to Gemfile
|
354
|
-
|
355
|
-
gem "secret_config"
|
356
|
-
|
357
|
-
Out of the box Secret Config will look in the local file system for the file `config/application.yml`
|
358
|
-
as covered above. By default it will use env var `RAILS_ENV` to define the path to look under for settings.
|
359
|
-
|
360
|
-
The default settings are great for getting started in development and test, but should not be used in production.
|
361
|
-
|
362
|
-
To ensure Secret Config is configured and available for use within any of the config files, add
|
363
|
-
the following lines to the very top of `application.rb` under the line `class Application < Rails::Application`:
|
364
|
-
|
365
|
-
~~~ruby
|
366
|
-
module MyApp
|
367
|
-
class Application < Rails::Application
|
368
|
-
|
369
|
-
# Add the following lines to configure Secret Config:
|
370
|
-
if Rails.env.development? || Rails.env.test?
|
371
|
-
# Use 'config/application.yml'
|
372
|
-
config.secret_config.use :file
|
373
|
-
else
|
374
|
-
# Read configuration from AWS SSM Parameter Store
|
375
|
-
config.secret_config.use :ssm, path: "/#{Rails.env}/my_app"
|
376
|
-
end
|
377
|
-
|
378
|
-
# ....
|
379
|
-
end
|
380
|
-
end
|
381
|
-
~~~
|
382
|
-
|
383
|
-
`path` is the path from which the configuration data will be read. This path uniquely identifies the
|
384
|
-
configuration for this instance of the application. In the example above it uses the rails env and application name
|
385
|
-
by default. This can be overridden using the `SECRET_CONFIG_PATH` environment variable when needed.
|
386
|
-
|
387
|
-
By placing the secret config configuration as the very first configuration item, it allows any subsequent
|
388
|
-
configuration item to access the centralized configuration in AWS System Manager Parameter Store.
|
389
|
-
|
390
|
-
The environment variable `SECRET_CONFIG_PROVIDER` can be used to override the provider when needed.
|
391
|
-
For example:
|
392
|
-
`export SECRET_CONFIG_PROVIDER=ssm`
|
393
|
-
Or,
|
394
|
-
`export SECRET_CONFIG_PROVIDER=file`
|
395
|
-
|
396
|
-
If we need 2 completely separate instances of the application running in a single AWS account then we could use
|
397
|
-
multiple paths. For example:
|
398
|
-
|
399
|
-
/production1/my_application
|
400
|
-
/production2/my_application
|
401
|
-
|
402
|
-
/production/instance1/my_application
|
403
|
-
/production/instance2/my_application
|
404
|
-
|
405
|
-
The `path` is completely flexible, but must be unique for every AWS account under which the application will run.
|
406
|
-
The same `path` can be used in different AWS accounts though. It is also not replicated across regions.
|
407
|
-
|
408
|
-
When writing settings to the parameter store, it is recommended to use a custom KMS key to encrypt the values.
|
409
|
-
To supply the key to encrypt the values with, add the `key_id` parameter:
|
410
|
-
|
411
|
-
~~~ruby
|
412
|
-
module MyApp
|
413
|
-
class Application < Rails::Application
|
414
|
-
|
415
|
-
# Add the following lines to configure Secret Config:
|
416
|
-
if Rails.env.development? || Rails.env.test?
|
417
|
-
# Use 'config/application.yml'
|
418
|
-
config.secret_config.use :file
|
419
|
-
else
|
420
|
-
# Read configuration from AWS SSM Parameter Store
|
421
|
-
config.secret_config.use :ssm,
|
422
|
-
path: "/#{Rails.env}/my_app",
|
423
|
-
key_id: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
|
424
|
-
end
|
425
|
-
|
426
|
-
# ....
|
427
|
-
end
|
428
|
-
end
|
429
|
-
~~~
|
430
|
-
|
431
|
-
Note: The relevant KMS key must be created first prior to using it here.
|
432
|
-
|
433
|
-
The `key_id` is only used when writing settings to the AWS Parameter store and can be left off when that instance
|
434
|
-
will only read from the parameter store.
|
435
|
-
|
436
|
-
### Shared configuration for development and test
|
437
|
-
|
438
|
-
When running multiple engines or private "gems" inside the same code repository, the development and test
|
439
|
-
configuration file `application.yml` can be shared. Update the lines above to:
|
440
|
-
|
441
|
-
~~~ruby
|
442
|
-
module MyApp
|
443
|
-
class Application < Rails::Application
|
444
|
-
|
445
|
-
# Add the following lines:
|
446
|
-
if Rails.env.development? || Rails.env.test?
|
447
|
-
# Use 'config/application.yml'
|
448
|
-
config.secret_config.use :file
|
449
|
-
else
|
450
|
-
# Read configuration from AWS SSM Parameter Store
|
451
|
-
config.secret_config.use :ssm, path: "/#{Rails.env}/my_app"
|
452
|
-
end
|
453
|
-
|
454
|
-
# ....
|
455
|
-
end
|
456
|
-
end
|
457
|
-
~~~
|
458
|
-
|
459
|
-
Where `file_name` is the full path and filename for where `application.yml` is located.
|
460
|
-
|
461
|
-
### Authorization
|
462
|
-
|
463
|
-
The following policy needs to be added to the IAM Group under which the application will be running:
|
464
|
-
|
465
|
-
~~~json
|
466
|
-
{
|
467
|
-
"Version": "2012-10-17",
|
468
|
-
"Statement": [
|
469
|
-
{
|
470
|
-
"Sid": "VisualEditor0",
|
471
|
-
"Effect": "Allow",
|
472
|
-
"Action": [
|
473
|
-
"ssm:GetParametersByPath",
|
474
|
-
"ssm:PutParameter",
|
475
|
-
"ssm:DeleteParameter",
|
476
|
-
],
|
477
|
-
"Resource": "*"
|
478
|
-
}
|
479
|
-
]
|
480
|
-
}
|
481
|
-
~~~
|
482
|
-
|
483
|
-
The above policy restricts read and write access to just the Parameter Store capabilities of AWS System Manager.
|
484
|
-
|
485
|
-
These additional Actions are not used by Secret Config, but may be useful for anyone using the AWS Console directly
|
486
|
-
to view and modify parameters:
|
487
|
-
- `ssm:DescribeParameters`
|
488
|
-
- `ssm:GetParameterHistory`
|
489
|
-
- `ssm:GetParameters`
|
490
|
-
- `ssm:GetParameter`
|
491
|
-
|
492
|
-
## String Interpolation
|
493
|
-
|
494
|
-
Values supplied for config settings can be replaced inline with date, time, hostname, pid and random values.
|
495
|
-
|
496
|
-
For example to include the `hostname` in the log file name setting:
|
497
|
-
|
498
|
-
~~~yaml
|
499
|
-
development:
|
500
|
-
logger:
|
501
|
-
level: info
|
502
|
-
file_name: /var/log/my_application_%{hostname}.log
|
503
|
-
~~~
|
504
|
-
|
505
|
-
Available interpolations:
|
28
|
+
Note that the command line program name has changed from `secret_config` to `secret-config`.
|
29
|
+
Be careful that the arguments have also changed. The arguments are now consistent across operations.
|
30
|
+
The command line examples below have also been updated to reflect the changes.
|
506
31
|
|
507
|
-
|
508
|
-
* Current date in the format of "%Y%m%d" (CCYYMMDD)
|
509
|
-
* %{date:format}
|
510
|
-
* Current date in the supplied format. See strftime
|
511
|
-
* %{time}
|
512
|
-
* Current date and time down to ms in the format of "%Y%m%d%Y%H%M%S%L" (CCYYMMDDHHMMSSmmm)
|
513
|
-
* %{time:format}
|
514
|
-
* Current date and time in the supplied format. See strftime
|
515
|
-
* %{env:name}
|
516
|
-
* Extract value from the named environment variable.
|
517
|
-
* %{hostname}
|
518
|
-
* Full name of this host.
|
519
|
-
* %{hostname:short}
|
520
|
-
* Short name of this host. Everything up to the first period.
|
521
|
-
* %{pid}
|
522
|
-
* Process Id for this process.
|
523
|
-
* %{random}
|
524
|
-
* URL safe Random 32 byte value.
|
525
|
-
* %{random:size}
|
526
|
-
* URL safe Random value of `size` bytes.
|
527
|
-
|
528
|
-
#### Notes:
|
529
|
-
|
530
|
-
* To prevent interpolation use %%{...}
|
531
|
-
* %% is not touched, only %{...} is searched for.
|
532
|
-
* Since these interpolations are only evaluated at load time and
|
533
|
-
every time the registry is refreshed there is no runtime overhead when keys are fetched.
|
534
|
-
|
535
|
-
## Command Line Interface
|
536
|
-
|
537
|
-
Secret Config has a command line interface for exporting, importing and copying between paths in the registry.
|
538
|
-
|
539
|
-
~~~
|
540
|
-
secret_config [options]
|
541
|
-
-e, --export [FILE_NAME] Export configuration to a file or stdout if no file_name supplied.
|
542
|
-
-i, --import [FILE_NAME] Import configuration from a file or stdin if no file_name supplied.
|
543
|
-
-C, --copy SOURCE_PATH Import configuration from a file or stdin if no file_name supplied.
|
544
|
-
-D, --diff [FILE_NAME] Compare configuration from a file or stdin if no file_name supplied.
|
545
|
-
-c, --console Start interactive console.
|
546
|
-
-p, --path PATH Path to import from / export to.
|
547
|
-
-P, --provider PROVIDER Provider to use. [ssm | file]. Default: ssm
|
548
|
-
-U, --no-filter Do not filter passwords and keys.
|
549
|
-
-d, --prune During import delete all existing keys for which there is no key in the import file.
|
550
|
-
-k, --key_id KEY_ID AWS KMS Key id or Key Alias to use when importing configuration values. Default: AWS Default key.
|
551
|
-
-r, --region REGION AWS Region to use. Default: AWS_REGION env var.
|
552
|
-
-R, --random_size INTEGER Size to use when generating random values. Whenever $random is encountered during an import. Default: 32
|
553
|
-
-v, --version Display Symmetric Encryption version.
|
554
|
-
-h, --help Prints this help.
|
555
|
-
~~~
|
556
|
-
|
557
|
-
### CLI Examples
|
558
|
-
|
559
|
-
#### Import from a file into SSM parameters
|
560
|
-
|
561
|
-
To get started it is useful to create a YAML file with all the relevant settings and then import
|
562
|
-
it into AWS SSM Parameter store. This file is the same as `applcation.yml` except that each file
|
563
|
-
is just for one environment. I.e. It does not contain the `test` or `development` root level entries.
|
564
|
-
|
565
|
-
For example: `production.yml`
|
566
|
-
|
567
|
-
~~~yaml
|
568
|
-
mysql:
|
569
|
-
database: secret_config_production
|
570
|
-
username: secret_config
|
571
|
-
password: secret_configrules
|
572
|
-
host: mysql_server.example.net
|
573
|
-
|
574
|
-
mongo:
|
575
|
-
database: secret_config_production
|
576
|
-
primary: mongo_primary.example.net:27017
|
577
|
-
secondary: mongo_secondary.example.net:27017
|
578
|
-
|
579
|
-
secrets:
|
580
|
-
secret_key_base: somereallylongproductionstring
|
581
|
-
~~~
|
582
|
-
|
583
|
-
Import a yaml file, into a path in AWS SSM Parameter Store:
|
584
|
-
|
585
|
-
secret_config --import production.yml --path /production/my_application
|
586
|
-
|
587
|
-
Import a yaml file, into a path in AWS SSM Parameter Store, using a custom KMS key to encrypt the values:
|
588
|
-
|
589
|
-
secret_config --import production.yml --path /production/my_application --key_id "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
|
590
|
-
|
591
|
-
#### Diff
|
592
|
-
|
593
|
-
Before importing a new config file into the AWS SSM Parameter store, a diff can be performed to determine
|
594
|
-
what the differences are that will be applied when the import is run with the `--prune` option.
|
595
|
-
|
596
|
-
secret_config --diff production.yml --path /production/my_application
|
597
|
-
|
598
|
-
Key:
|
599
|
-
|
600
|
-
+ Adding a new key to the registry.
|
601
|
-
- The key will be removed from the registry during the import if --prune is specified.
|
602
|
-
* The value for that key will change during an import.
|
603
|
-
|
604
|
-
#### Export SSM parameters
|
605
|
-
|
606
|
-
In AWS SSM Parameter store it can be difficult to
|
607
|
-
Export the values from a specific path into a yaml or json file so that they are easier to read.
|
608
|
-
|
609
|
-
Export from a path in AWS SSM Parameter Store to a yaml file, where passwords are filtered:
|
610
|
-
|
611
|
-
secret_config --export production.yml --path /production/my_application
|
612
|
-
|
613
|
-
Export from a path in AWS SSM Parameter Store to a yaml file, _without_ filtering out passwords:
|
614
|
-
|
615
|
-
secret_config --export production.yml --path /production/my_application --no-filter
|
616
|
-
|
617
|
-
Export from a path in AWS SSM Parameter Store to a json file, where passwords are filtered:
|
618
|
-
|
619
|
-
secret_config --export production.json --path /production/my_application
|
620
|
-
|
621
|
-
#### Copy values between paths in AWS SSM parameter store
|
622
|
-
|
623
|
-
It can be useful to keep a "master" copy of the values for an environment or stack in a custom path
|
624
|
-
in AWS Parameter Store. Then for each stack or environment that is spun up, copy the "master" / "common" values
|
625
|
-
into the new path. Once copied the values specific to that path can be updated accordingly.
|
626
|
-
|
627
|
-
Copy configuration from one path in AWS SSM Parameter Store to another path in AWS SSM Parameter Store:
|
628
|
-
|
629
|
-
secret_config --copy /production/my_application --path /tenant73/my_application
|
630
|
-
|
631
|
-
#### Generating random passwords
|
632
|
-
|
633
|
-
In the multi-tenant example above, we may want to generate a secure random password for each tenant.
|
634
|
-
In the source file or registry, set the value to `$random`, this will ensure that during the `import` or `copy`
|
635
|
-
that the destination will receive a secure random value.
|
636
|
-
|
637
|
-
By default the length of the randomized value is 32 bytes, use `--random_size` to adjust the length of
|
638
|
-
the randomized string.
|
639
|
-
|
640
|
-
## Docker
|
641
|
-
|
642
|
-
Secret Config is at its best when the application is containerized. By externalizing the configuration the same
|
643
|
-
docker container can be tested in one or more environments and then deployed directly to production without
|
644
|
-
any changes. The only difference being the path that container uses to read its configuration from.
|
645
|
-
|
646
|
-
Another important benefit is that the docker image does not contain any production or test credentials since
|
647
|
-
these are all stored in AWS SSM Parameter Store.
|
648
|
-
|
649
|
-
When a Ruby / Rails application is using Secret Config for its configuration settings, it only requires the
|
650
|
-
following environment variables when starting up the container in for example AWS ECS or AWS Fargate:
|
651
|
-
|
652
|
-
~~~shell
|
653
|
-
export SECRET_CONFIG_PATH=/production/my_application
|
654
|
-
~~~
|
655
|
-
|
656
|
-
For rails applications, typically the `RAILS_ENV` is also needed, but not required for Secret Config.
|
657
|
-
|
658
|
-
~~~shell
|
659
|
-
export RAILS_ENV=production
|
660
|
-
~~~
|
661
|
-
|
662
|
-
### Logging
|
663
|
-
|
664
|
-
When using Semantic Logger, the following code could be added to `application.rb` to facilitate configuration
|
665
|
-
of the logging output via Secret Config:
|
666
|
-
|
667
|
-
~~~ruby
|
668
|
-
# Logging
|
669
|
-
config.log_level = config.secret_config.fetch("logger/level", default: :info, type: :symbol)
|
670
|
-
config.semantic_logger.backtrace_level = config.secret_config.fetch("logger/backtrace_level", default: :error, type: :symbol)
|
671
|
-
config.semantic_logger.application = config.secret_config.fetch("logger/application", default: "my_app")
|
672
|
-
config.semantic_logger.environment = config.secret_config.fetch("logger/environment", default: Rails.env)
|
673
|
-
~~~
|
674
|
-
|
675
|
-
In any environment the log level can be changed, for example set `logger/level` to `debug`. And it can be changed
|
676
|
-
in the AWS SSM Parameter Store, or directly with the environment variable `export LOGGER_LEVEL=debug`
|
677
|
-
|
678
|
-
`logger/environment` can be used to identify which tenant the log messages are emanating from. By default it is just
|
679
|
-
the rails environment. For example set `logger/environment` to `tenant73`.
|
680
|
-
|
681
|
-
Additionally the following code can be used with containers to send log output to standard out:
|
682
|
-
|
683
|
-
~~~ruby
|
684
|
-
destination = config.secret_config.fetch("logger/destination", default: :file, type: :symbol)
|
685
|
-
if destination == :stdout
|
686
|
-
STDOUT.sync = true
|
687
|
-
config.rails_semantic_logger.add_file_appender = false
|
688
|
-
config.semantic_logger.add_appender(
|
689
|
-
io: STDOUT,
|
690
|
-
level: config.log_level,
|
691
|
-
formatter: config.secret_config.fetch("logger/formatter", default: :default, type: :symbol)
|
692
|
-
)
|
693
|
-
end
|
694
|
-
~~~
|
695
|
-
|
696
|
-
Specifically for docker containers it is necessary to turn off file logging and turn on logging to standard out
|
697
|
-
so that AWS Cloud Watch can pick up the log data.
|
698
|
-
|
699
|
-
To start with `logger/destination` of `stdout` will work with regular non-colorized output. When feeding the
|
700
|
-
log output into something that can process JSON, set `logger/formatter` to `json`.
|
701
|
-
|
702
|
-
The benefit with the above approach is that a developer can pull the exact same container image that is running
|
703
|
-
in production and configure it to run locally on their laptop. For example, set `logger/destination` to `file`.
|
704
|
-
|
705
|
-
The above code can be modified as necessary to add any Semantic Logger appender to write directly to external
|
706
|
-
centralized logging systems, instead of writing to standard out or local files.
|
707
|
-
|
708
|
-
### Email Server and Assets
|
709
|
-
|
710
|
-
An example of how to setup the email server and the assets for html emails. Add to `application.rb`:
|
711
|
-
|
712
|
-
~~~ruby
|
713
|
-
# Emails
|
714
|
-
application_url = config.secret_config.fetch("emails/asset_host")
|
715
|
-
uri = URI.parse(application_url)
|
716
|
-
|
717
|
-
config.action_mailer.default_url_options = {host: uri.host, protocol: uri.scheme}
|
718
|
-
config.action_mailer.asset_host = application_url
|
719
|
-
config.action_mailer.smtp_settings = {address: config.secret_config.fetch("emails/smtp/address", default: "localhost")}
|
720
|
-
config.action_mailer.raise_delivery_errors = config.secret_config.fetch("emails/raise_delivery_errors", default: true, type: :boolean)
|
721
|
-
~~~
|
722
|
-
|
723
|
-
### Symmetric Encryption
|
724
|
-
|
725
|
-
An example of how to setup Symmetric Encryption. Add to `application.rb`:
|
726
|
-
|
727
|
-
~~~ruby
|
728
|
-
# Encryption
|
729
|
-
config.symmetric_encryption.cipher =
|
730
|
-
SymmetricEncryption::Cipher.new(
|
731
|
-
key: config.secret_config.fetch('symmetric_encryption/key', encoding: :base64),
|
732
|
-
iv: config.secret_config.fetch('symmetric_encryption/iv', encoding: :base64),
|
733
|
-
version: config.secret_config.fetch('symmetric_encryption/version', type: :integer),
|
734
|
-
)
|
735
|
-
|
736
|
-
# Also support one prior encryption key version during key rotation
|
737
|
-
if config.secret_config.key?('symmetric_encryption/old/key')
|
738
|
-
SymmetricEncryption.secondary_ciphers = [
|
739
|
-
SymmetricEncryption::Cipher.new(
|
740
|
-
key: config.secret_config.fetch('symmetric_encryption/old/key', encoding: :base64),
|
741
|
-
iv: config.secret_config.fetch('symmetric_encryption/old/iv', encoding: :base64),
|
742
|
-
version: config.secret_config.fetch('symmetric_encryption/old/version', type: :integer),
|
743
|
-
),
|
744
|
-
]
|
745
|
-
end
|
746
|
-
~~~
|
747
|
-
|
748
|
-
Using this approach the file `config/symmetric-encryption.yml` can be removed once the keys have been moved to
|
749
|
-
the registry.
|
750
|
-
|
751
|
-
To extract existing keys from the config file so that they can be imported into the registry,
|
752
|
-
run the code below inside a console in each of the respective environments.
|
753
|
-
|
754
|
-
~~~ruby
|
755
|
-
require "yaml"
|
756
|
-
require "base64"
|
757
|
-
|
758
|
-
def se_config(cipher)
|
759
|
-
{
|
760
|
-
"key" => Base64.strict_encode64(cipher.send(:key)),
|
761
|
-
"iv" => Base64.strict_encode64(cipher.iv),
|
762
|
-
"version" => cipher.version
|
763
|
-
}
|
764
|
-
end
|
765
|
-
|
766
|
-
config = { "symmetric_encryption" => se_config(SymmetricEncryption.cipher) }
|
767
|
-
if cipher = SymmetricEncryption.secondary_ciphers.first
|
768
|
-
config["symmetric_encryption"]["old"] = se_config(cipher)
|
769
|
-
end
|
770
|
-
puts config.to_yaml
|
771
|
-
~~~
|
32
|
+
Please run `secret-config --help` to see the new arguments and updated operations.
|
772
33
|
|
773
34
|
## Versioning
|
774
35
|
|
775
|
-
This project adheres to [Semantic Versioning](
|
36
|
+
This project adheres to [Semantic Versioning](https://semver.org/).
|
776
37
|
|
777
38
|
## Author
|
778
39
|
|
@@ -780,7 +41,7 @@ This project adheres to [Semantic Versioning](http://semver.org/).
|
|
780
41
|
|
781
42
|
## License
|
782
43
|
|
783
|
-
Copyright
|
44
|
+
Copyright 2020 Reid Morrison
|
784
45
|
|
785
46
|
Licensed under the Apache License, Version 2.0 (the "License");
|
786
47
|
you may not use this file except in compliance with the License.
|