secret_config 0.4.5 → 0.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +424 -37
- data/lib/secret_config/cli.rb +86 -23
- data/lib/secret_config/providers/file.rb +2 -7
- data/lib/secret_config/providers/provider.rb +20 -0
- data/lib/secret_config/providers/ssm.rb +5 -1
- data/lib/secret_config/registry.rb +17 -6
- data/lib/secret_config/utils.rb +10 -2
- data/lib/secret_config/version.rb +1 -1
- data/lib/secret_config.rb +1 -0
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 515036f2cebc7abd6211448b44f80c99c0872065a7947afaf4dface54c84d808
|
4
|
+
data.tar.gz: 2a5c7a8ccfd5d7d91d459a13d8cd5fcce8b1b381e1baa3e83946d99d712cca42
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 73c14c00b3b1759b73d04415bb20724775efef6084729907d9377cb4928b513ed324b2edc67cd5953a6e2a8e71969b4add609d74b94ae2fd2467adfa5b0a48b5
|
7
|
+
data.tar.gz: 5dfb6fd8832349e7a63b8d6f02023c54cc33538d82e742b3bf24246252f1e0500fa948a9b835ca8ac054dc3c7a6d69b4090448938c782dc332640d361fa8997f
|
data/README.md
CHANGED
@@ -1,21 +1,19 @@
|
|
1
1
|
# Secret Config
|
2
|
-
[![Gem Version](https://img.shields.io/gem/v/secret_config.svg)](https://rubygems.org/gems/secret_config) [![Build Status](https://travis-ci.org/rocketjob/secret_config.svg?branch=master)](https://travis-ci.org/rocketjob/secret_config) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-
|
2
|
+
[![Gem Version](https://img.shields.io/gem/v/secret_config.svg)](https://rubygems.org/gems/secret_config) [![Build Status](https://travis-ci.org/rocketjob/secret_config.svg?branch=master)](https://travis-ci.org/rocketjob/secret_config) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) [![Gitter chat](https://img.shields.io/badge/IRC%20(gitter)-Support-brightgreen.svg)](https://gitter.im/rocketjob/support)
|
3
3
|
|
4
4
|
Centralized Configuration and Secrets Management for Ruby and Rails applications.
|
5
5
|
|
6
|
-
Securely store configuration information centrally.
|
7
|
-
|
8
|
-
## Project Status
|
9
|
-
|
10
|
-
Early development.
|
6
|
+
Securely store configuration information centrally, supporting multiple tenants of the same application.
|
11
7
|
|
12
8
|
## Features
|
13
9
|
|
14
10
|
Supports storing configuration information in:
|
15
11
|
* File
|
16
12
|
* Development and testing use.
|
13
|
+
* Environment Variables
|
14
|
+
* Environment Variables take precedence and can be used to override any setting.
|
17
15
|
* AWS System Manager Parameter Store
|
18
|
-
* Encrypt and store secrets such as passwords centrally.
|
16
|
+
* Encrypt and securely store secrets such as passwords centrally.
|
19
17
|
|
20
18
|
## Benefits
|
21
19
|
|
@@ -27,13 +25,156 @@ Benefits of moving sensitive configuration information into AWS System Manager P
|
|
27
25
|
* In a large application the number of secrets can grow dramatically.
|
28
26
|
* Removes the need to encrypt sensitive data config files.
|
29
27
|
* Including securing and managing encryption keys.
|
30
|
-
* When encryption keys change, such as during a key rotation, config files don
|
28
|
+
* When encryption keys change, such as during a key rotation, config files don't have to be changed.
|
31
29
|
* Removes security concerns with placing passwords in the clear into environment variables.
|
32
30
|
* AWS System Manager Parameter Store does not charge for parameters.
|
33
31
|
* Still recommend using a custom KMS key that charges only $1 per month.
|
34
32
|
* Amounts as of 4/2019. Confirm what AWS charges you for these services.
|
35
|
-
* AWS Secrets Manager charges for every secret being managed, which can accumulate quickly with large projects.
|
33
|
+
* AWS Secrets Manager charges for every secret being managed, which can accumulate quickly with large projects.
|
34
|
+
* Configure multiple distinct application instances to support multiple tenants.
|
35
|
+
* For example, use separate databases with unique credentials for each tenant.
|
36
|
+
* Separation of responsibilities is achieved since operations can manage production configuration.
|
37
|
+
* Developers do not need to be involved with production configuration such as host names and passwords.
|
38
|
+
* All values are encrypted by default when stored in the AWS Parameter Store.
|
39
|
+
* Prevents accidentally not encrypting sensitive data.
|
40
|
+
|
41
|
+
## Introduction
|
42
|
+
|
43
|
+
When Secret Config starts up it reads all configuration entries into memory for all keys under the configured path.
|
44
|
+
This means that once Secret Config has initialized all calls to Secret Config are extremely fast.
|
45
|
+
|
46
|
+
The in-memory copy of the registry can be refreshed at any time by calling `SecretConfig.refresh!`. It can be refreshed
|
47
|
+
via a process signal, or by calling it through an event, or via a messaging system.
|
48
|
+
|
49
|
+
It is suggested that any programmatic lookup to values stored in Secret Config are called every time a value is
|
50
|
+
being used, rather than creating a local copy of the value. This ensures that a refresh of the registry will take effect
|
51
|
+
immediately for any code reading from Secret Config.
|
52
|
+
|
53
|
+
## API
|
54
|
+
|
55
|
+
When Secret Config starts up it reads all configuration entries immediately for all keys under the configured path.
|
56
|
+
This means that once Secret Config has initialized all calls to Secret Config are extremely fast.
|
57
|
+
|
58
|
+
Secret Config supports the following programmatic interface:
|
59
|
+
|
60
|
+
### Read values
|
61
|
+
|
62
|
+
Fetch the value for the supplied key, returning nil if not found:
|
63
|
+
|
64
|
+
~~~ruby
|
65
|
+
# Key is present:
|
66
|
+
SecretConfig["logger/level"]
|
67
|
+
# => "info"
|
68
|
+
|
69
|
+
# Key is missing:
|
70
|
+
SecretConfig["logger/blah"]
|
71
|
+
# => nil
|
72
|
+
~~~
|
73
|
+
|
74
|
+
Fetch the value for the supplied key, raising `SecretConfig::MissingMandatoryKey` if not found:
|
75
|
+
|
76
|
+
~~~ruby
|
77
|
+
# Key is present:
|
78
|
+
SecretConfig.fetch("logger/level")
|
79
|
+
# => "info"
|
80
|
+
|
81
|
+
# Key is missing:
|
82
|
+
SecretConfig.fetch("logger/blah")
|
83
|
+
# => SecretConfig::MissingMandatoryKey (Missing configuration value for /development/logger/blah)
|
84
|
+
~~~
|
85
|
+
|
86
|
+
A default value can be supplied when the key is not found in the registry:
|
87
|
+
|
88
|
+
~~~ruby
|
89
|
+
SecretConfig.fetch("logger/level", default: "info")
|
90
|
+
# => "info"
|
91
|
+
~~~
|
92
|
+
|
93
|
+
Since AWS SSM Parameter store and environment variables only support string values,
|
94
|
+
it is neccessary to convert the string back to the type required by the program.
|
95
|
+
|
96
|
+
The following types are supported:
|
97
|
+
`:integer`
|
98
|
+
`:float`
|
99
|
+
`:string`
|
100
|
+
`:boolean`
|
101
|
+
`:symbol`
|
102
|
+
|
103
|
+
~~~ruby
|
104
|
+
# Without type conversion:
|
105
|
+
SecretConfig.fetch("symmetric_encryption/version")
|
106
|
+
# => "0"
|
107
|
+
|
108
|
+
# With type conversion:
|
109
|
+
SecretConfig.fetch("symmetric_encryption/version", type: :integer)
|
110
|
+
# => 0
|
111
|
+
~~~
|
112
|
+
|
113
|
+
When storing binary data, it should be encoded with strict base64 encoding. To automatically convert it back to binary
|
114
|
+
specify the encoding as `:base64`
|
115
|
+
|
116
|
+
~~~ruby
|
117
|
+
# Return a value that was stored in Base64 encoding format:
|
118
|
+
SecretConfig.fetch("symmetric_encryption/iv")
|
119
|
+
# => "FW+/wLubAYM+ZU0bWQj59Q=="
|
36
120
|
|
121
|
+
# Base64 decode a value that was stored in Base64 encoding format:
|
122
|
+
SecretConfig.fetch("symmetric_encryption/iv", encoding: :base64)
|
123
|
+
# => "\x15o\xBF\xC0\xBB\x9B\x01\x83>eM\eY\b\xF9\xF5"
|
124
|
+
~~~
|
125
|
+
|
126
|
+
### Key presence
|
127
|
+
|
128
|
+
Returns whether a key is present in the registry:
|
129
|
+
|
130
|
+
~~~ruby
|
131
|
+
SecretConfig.key?("logger/level")
|
132
|
+
# => true
|
133
|
+
~~~
|
134
|
+
|
135
|
+
### Write values
|
136
|
+
|
137
|
+
When Secret Config is configured to use the AWS SSM Parameter store, its values can be modified:
|
138
|
+
|
139
|
+
~~~ruby
|
140
|
+
SecretConfig["logger/level"] = "debug"
|
141
|
+
~~~
|
142
|
+
|
143
|
+
~~~ruby
|
144
|
+
SecretConfig.set("logger/level", "debug")
|
145
|
+
~~~
|
146
|
+
|
147
|
+
### Configuration
|
148
|
+
|
149
|
+
Returns a Hash copy of the configuration as a tree:
|
150
|
+
|
151
|
+
~~~ruby
|
152
|
+
SecretConfig.configuration
|
153
|
+
~~~
|
154
|
+
|
155
|
+
### Refresh Configuration
|
156
|
+
|
157
|
+
Tell Secret Config to refresh its in-memory copy of the configuration settings.
|
158
|
+
|
159
|
+
~~~ruby
|
160
|
+
SecretConfig.refresh!
|
161
|
+
~~~
|
162
|
+
|
163
|
+
Example, refresh the registry any time a SIGUSR2 is raised, add the following code on startup:
|
164
|
+
|
165
|
+
~~~ruby
|
166
|
+
Signal.trap('USR2') do
|
167
|
+
SecretConfig.refresh!
|
168
|
+
end
|
169
|
+
~~~
|
170
|
+
|
171
|
+
Then to make the process refresh it registry:
|
172
|
+
~~~shell
|
173
|
+
kill -SIGUSR2 1234
|
174
|
+
~~~
|
175
|
+
|
176
|
+
Where `1234` above is the process PID.
|
177
|
+
|
37
178
|
## Development and Test use
|
38
179
|
|
39
180
|
In the development environment create the file `config/application.yml` within which to store local development credentials.
|
@@ -173,7 +314,7 @@ For example, somewhere in your codebase you need a persistent http connection:
|
|
173
314
|
~~~
|
174
315
|
|
175
316
|
Then the application that uses the above library / gem just needs to add the relevant entries to their
|
176
|
-
`application.
|
317
|
+
`application.yml` file:
|
177
318
|
|
178
319
|
~~~yaml
|
179
320
|
http_client:
|
@@ -199,18 +340,39 @@ as covered above. By default it will use env var `RAILS_ENV` to define the path
|
|
199
340
|
|
200
341
|
The default settings are great for getting started in development and test, but should not be used in production.
|
201
342
|
|
202
|
-
|
203
|
-
|
343
|
+
To ensure Secret Config is configured and available for use within any of the config files, add
|
344
|
+
the following lines to the very top of `application.rb` under the line `class Application < Rails::Application`:
|
204
345
|
|
205
346
|
~~~ruby
|
206
|
-
|
207
|
-
|
208
|
-
|
347
|
+
module MyApp
|
348
|
+
class Application < Rails::Application
|
349
|
+
|
350
|
+
# Add the following lines to configure Secret Config:
|
351
|
+
if Rails.env.development? || Rails.env.test?
|
352
|
+
# Use 'config/application.yml'
|
353
|
+
config.secret_config.use :file
|
354
|
+
else
|
355
|
+
# Read configuration from AWS SSM Parameter Store
|
356
|
+
config.secret_config.use :ssm, path: "/#{Rails.env}/my_app"
|
357
|
+
end
|
358
|
+
|
359
|
+
# ....
|
360
|
+
end
|
209
361
|
end
|
210
362
|
~~~
|
211
363
|
|
212
364
|
`path` is the path from which the configuration data will be read. This path uniquely identifies the
|
213
|
-
configuration for this instance of the application.
|
365
|
+
configuration for this instance of the application. In the example above it uses the rails env and application name
|
366
|
+
by default. This can be overridden using the `SECRET_CONFIG_PATH` environment variable when needed.
|
367
|
+
|
368
|
+
By placing the secret config configuration as the very first configuration item, it allows any subsequent
|
369
|
+
configuration item to access the centralized configuration in AWS System Manager Parameter Store.
|
370
|
+
|
371
|
+
The environment variable `SECRET_CONFIG_PROVIDER` can be used to override the provider when needed.
|
372
|
+
For example:
|
373
|
+
`export SECRET_CONFIG_PROVIDER=ssm`
|
374
|
+
Or,
|
375
|
+
`export SECRET_CONFIG_PROVIDER=file`
|
214
376
|
|
215
377
|
If we need 2 completely separate instances of the application running in a single AWS account then we could use
|
216
378
|
multiple paths. For example:
|
@@ -228,11 +390,22 @@ When writing settings to the parameter store, it is recommended to use a custom
|
|
228
390
|
To supply the key to encrypt the values with, add the `key_id` parameter:
|
229
391
|
|
230
392
|
~~~ruby
|
231
|
-
|
232
|
-
|
233
|
-
|
234
|
-
|
235
|
-
|
393
|
+
module MyApp
|
394
|
+
class Application < Rails::Application
|
395
|
+
|
396
|
+
# Add the following lines to configure Secret Config:
|
397
|
+
if Rails.env.development? || Rails.env.test?
|
398
|
+
# Use 'config/application.yml'
|
399
|
+
config.secret_config.use :file
|
400
|
+
else
|
401
|
+
# Read configuration from AWS SSM Parameter Store
|
402
|
+
config.secret_config.use :ssm,
|
403
|
+
path: "/#{Rails.env}/my_app",
|
404
|
+
key_id: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
|
405
|
+
end
|
406
|
+
|
407
|
+
# ....
|
408
|
+
end
|
236
409
|
end
|
237
410
|
~~~
|
238
411
|
|
@@ -241,6 +414,31 @@ Note: The relevant KMS key must be created first prior to using it here.
|
|
241
414
|
The `key_id` is only used when writing settings to the AWS Parameter store and can be left off when that instance
|
242
415
|
will only read from the parameter store.
|
243
416
|
|
417
|
+
### Shared configuration for development and test
|
418
|
+
|
419
|
+
When running multiple engines or private "gems" inside the same code repository, the development and test
|
420
|
+
configuration file `application.yml` can be shared. Update the lines above to:
|
421
|
+
|
422
|
+
~~~ruby
|
423
|
+
module MyApp
|
424
|
+
class Application < Rails::Application
|
425
|
+
|
426
|
+
# Add the following lines:
|
427
|
+
if Rails.env.development? || Rails.env.test?
|
428
|
+
# Use 'config/application.yml'
|
429
|
+
config.secret_config.use :file
|
430
|
+
else
|
431
|
+
# Read configuration from AWS SSM Parameter Store
|
432
|
+
config.secret_config.use :ssm, path: "/#{Rails.env}/my_app"
|
433
|
+
end
|
434
|
+
|
435
|
+
# ....
|
436
|
+
end
|
437
|
+
end
|
438
|
+
~~~
|
439
|
+
|
440
|
+
Where `file_name` is the full path and filename for where `application.yml` is located.
|
441
|
+
|
244
442
|
### Authorization
|
245
443
|
|
246
444
|
The following policy needs to be added to the IAM Group under which the application will be running:
|
@@ -253,8 +451,9 @@ The following policy needs to be added to the IAM Group under which the applicat
|
|
253
451
|
"Sid": "VisualEditor0",
|
254
452
|
"Effect": "Allow",
|
255
453
|
"Action": [
|
256
|
-
"ssm:PutParameter",
|
257
454
|
"ssm:GetParametersByPath",
|
455
|
+
"ssm:PutParameter",
|
456
|
+
"ssm:DeleteParameter",
|
258
457
|
],
|
259
458
|
"Resource": "*"
|
260
459
|
}
|
@@ -279,11 +478,14 @@ Secret Config has a command line interface for exporting, importing and copying
|
|
279
478
|
secret_config [options]
|
280
479
|
-e, --export [FILE_NAME] Export configuration to a file or stdout if no file_name supplied.
|
281
480
|
-i, --import [FILE_NAME] Import configuration from a file or stdin if no file_name supplied.
|
282
|
-
-
|
481
|
+
-C, --copy SOURCE_PATH Import configuration from a file or stdin if no file_name supplied.
|
482
|
+
-D, --diff [FILE_NAME] Compare configuration from a file or stdin if no file_name supplied.
|
483
|
+
-c, --console Start interactive console.
|
283
484
|
-p, --path PATH Path to import from / export to.
|
284
485
|
-P, --provider PROVIDER Provider to use. [ssm | file]. Default: ssm
|
285
486
|
-U, --no-filter Do not filter passwords and keys.
|
286
|
-
-
|
487
|
+
-d, --prune During import delete all existing keys for which there is no key in the import file.
|
488
|
+
-k, --key_id KEY_ID AWS KMS Key id or Key Alias to use when importing configuration values. Default: AWS Default key.
|
287
489
|
-r, --region REGION AWS Region to use. Default: AWS_REGION env var.
|
288
490
|
-R, --random_size INTEGER Size to use when generating random values. Whenever $random is encountered during an import. Default: 32
|
289
491
|
-v, --version Display Symmetric Encryption version.
|
@@ -292,34 +494,219 @@ secret_config [options]
|
|
292
494
|
|
293
495
|
### CLI Examples
|
294
496
|
|
497
|
+
#### Import from a file into SSM parameters
|
498
|
+
|
499
|
+
To get started it is useful to create a YAML file with all the relevant settings and then import
|
500
|
+
it into AWS SSM Parameter store. This file is the same as `applcation.yml` except that each file
|
501
|
+
is just for one environment. I.e. It does not contain the `test` or `development` root level entries.
|
502
|
+
|
503
|
+
For example: `production.yml`
|
504
|
+
|
505
|
+
~~~yaml
|
506
|
+
mysql:
|
507
|
+
database: secret_config_production
|
508
|
+
username: secret_config
|
509
|
+
password: secret_configrules
|
510
|
+
host: mysql_server.example.net
|
511
|
+
|
512
|
+
mongo:
|
513
|
+
database: secret_config_production
|
514
|
+
primary: mongo_primary.example.net:27017
|
515
|
+
secondary: mongo_secondary.example.net:27017
|
516
|
+
|
517
|
+
secrets:
|
518
|
+
secret_key_base: somereallylongproductionstring
|
519
|
+
~~~
|
520
|
+
|
521
|
+
Import a yaml file, into a path in AWS SSM Parameter Store:
|
522
|
+
|
523
|
+
secret_config --import production.yml --path /production/my_application
|
524
|
+
|
525
|
+
Import a yaml file, into a path in AWS SSM Parameter Store, using a custom KMS key to encrypt the values:
|
526
|
+
|
527
|
+
secret_config --import production.yml --path /production/my_application --key_id "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
|
528
|
+
|
529
|
+
#### Diff
|
530
|
+
|
531
|
+
Before importing a new config file into the AWS SSM Parameter store, a diff can be performed to determine
|
532
|
+
what the differences are that will be applied when the import is run with the `--prune` option.
|
533
|
+
|
534
|
+
secret_config --diff production.yml --path /production/my_application
|
535
|
+
|
536
|
+
Key:
|
537
|
+
|
538
|
+
+ Adding a new key to the registry.
|
539
|
+
- The key will be removed from the registry during the import if --prune is specified.
|
540
|
+
* The value for that key will change during an import.
|
541
|
+
|
542
|
+
#### Export SSM parameters
|
543
|
+
|
544
|
+
In AWS SSM Parameter store it can be difficult to
|
545
|
+
Export the values from a specific path into a yaml or json file so that they are easier to read.
|
546
|
+
|
295
547
|
Export from a path in AWS SSM Parameter Store to a yaml file, where passwords are filtered:
|
296
548
|
|
297
|
-
secret_config --export
|
549
|
+
secret_config --export production.yml --path /production/my_application
|
298
550
|
|
299
551
|
Export from a path in AWS SSM Parameter Store to a yaml file, _without_ filtering out passwords:
|
300
552
|
|
301
|
-
secret_config --export
|
553
|
+
secret_config --export production.yml --path /production/my_application --no-filter
|
302
554
|
|
303
555
|
Export from a path in AWS SSM Parameter Store to a json file, where passwords are filtered:
|
304
556
|
|
305
|
-
secret_config --export
|
557
|
+
secret_config --export production.json --path /production/my_application
|
306
558
|
|
307
|
-
|
559
|
+
#### Copy values between paths in AWS SSM parameter store
|
308
560
|
|
309
|
-
|
561
|
+
It can be useful to keep a "master" copy of the values for an environment or stack in a custom path
|
562
|
+
in AWS Parameter Store. Then for each stack or environment that is spun up, copy the "master" / "common" values
|
563
|
+
into the new path. Once copied the values specific to that path can be updated accordingly.
|
310
564
|
|
311
|
-
|
565
|
+
Copy configuration from one path in AWS SSM Parameter Store to another path in AWS SSM Parameter Store:
|
312
566
|
|
313
|
-
secret_config --
|
567
|
+
secret_config --copy /production/my_application --path /tenant73/my_application
|
314
568
|
|
315
|
-
|
569
|
+
#### Generating random passwords
|
570
|
+
|
571
|
+
In the multi-tenant example above, we may want to generate a secure random password for each tenant.
|
572
|
+
In the source file or registry, set the value to `$random`, this will ensure that during the `import` or `copy`
|
573
|
+
that the destination will receive a secure random value.
|
574
|
+
|
575
|
+
By default the length of the randomized value is 32 bytes, use `--random_size` to adjust the length of
|
576
|
+
the randomized string.
|
577
|
+
|
578
|
+
## Docker
|
579
|
+
|
580
|
+
Secret Config is at its best when the application is containerized. By externalizing the configuration the same
|
581
|
+
docker container can be tested in one or more environments and then deployed directly to production without
|
582
|
+
any changes. The only difference being the path that container uses to read its configuration from.
|
583
|
+
|
584
|
+
Another important benefit is that the docker image does not contain any production or test credentials since
|
585
|
+
these are all stored in AWS SSM Parameter Store.
|
586
|
+
|
587
|
+
When a Ruby / Rails application is using Secret Config for its configuration settings, it only requires the
|
588
|
+
following environment variables when starting up the container in for example AWS ECS or AWS Fargate:
|
316
589
|
|
317
|
-
|
590
|
+
~~~shell
|
591
|
+
export SECRET_CONFIG_PATH=/production/my_application
|
592
|
+
~~~
|
593
|
+
|
594
|
+
For rails applications, typically the `RAILS_ENV` is also needed, but not required for Secret Config.
|
595
|
+
|
596
|
+
~~~shell
|
597
|
+
export RAILS_ENV=production
|
598
|
+
~~~
|
599
|
+
|
600
|
+
### Logging
|
601
|
+
|
602
|
+
When using Semantic Logger, the following code could be added to `application.rb` to facilitate configuration
|
603
|
+
of the logging output via Secret Config:
|
604
|
+
|
605
|
+
~~~ruby
|
606
|
+
# Logging
|
607
|
+
config.log_level = config.secret_config.fetch("logger/level", default: :info, type: :symbol)
|
608
|
+
config.semantic_logger.backtrace_level = config.secret_config.fetch("logger/backtrace_level", default: :error, type: :symbol)
|
609
|
+
config.semantic_logger.application = config.secret_config.fetch("logger/application", default: "my_app")
|
610
|
+
config.semantic_logger.environment = config.secret_config.fetch("logger/environment", default: Rails.env)
|
611
|
+
~~~
|
612
|
+
|
613
|
+
In any environment the log level can be changed, for example set `logger/level` to `debug`. And it can be changed
|
614
|
+
in the AWS SSM Parameter Store, or directly with the environment variable `export LOGGER_LEVEL=debug`
|
615
|
+
|
616
|
+
`logger/environment` can be used to identify which tenant the log messages are emanating from. By default it is just
|
617
|
+
the rails environment. For example set `logger/environment` to `tenant73`.
|
618
|
+
|
619
|
+
Additionally the following code can be used with containers to send log output to standard out:
|
620
|
+
|
621
|
+
~~~ruby
|
622
|
+
destination = config.secret_config.fetch("logger/destination", default: :file, type: :symbol)
|
623
|
+
if destination == :stdout
|
624
|
+
STDOUT.sync = true
|
625
|
+
config.rails_semantic_logger.add_file_appender = false
|
626
|
+
config.semantic_logger.add_appender(
|
627
|
+
io: STDOUT,
|
628
|
+
level: config.log_level,
|
629
|
+
formatter: config.secret_config.fetch("logger/formatter", default: :default, type: :symbol)
|
630
|
+
)
|
631
|
+
end
|
632
|
+
~~~
|
633
|
+
|
634
|
+
Specifically for docker containers it is necessary to turn off file logging and turn on logging to standard out
|
635
|
+
so that AWS Cloud Watch can pick up the log data.
|
636
|
+
|
637
|
+
To start with `logger/destination` of `stdout` will work with regular non-colorized output. When feeding the
|
638
|
+
log output into something that can process JSON, set `logger/formatter` to `json`.
|
639
|
+
|
640
|
+
The benefit with the above approach is that a developer can pull the exact same container image that is running
|
641
|
+
in production and configure it to run locally on their laptop. For example, set `logger/destination` to `file`.
|
318
642
|
|
319
|
-
|
320
|
-
|
321
|
-
|
322
|
-
|
643
|
+
The above code can be modified as necessary to add any Semantic Logger appender to write directly to external
|
644
|
+
centralized logging systems, instead of writing to standard out or local files.
|
645
|
+
|
646
|
+
### Email Server and Assets
|
647
|
+
|
648
|
+
An example of how to setup the email server and the assets for html emails. Add to `application.rb`:
|
649
|
+
|
650
|
+
~~~ruby
|
651
|
+
# Emails
|
652
|
+
application_url = config.secret_config.fetch("emails/asset_host")
|
653
|
+
uri = URI.parse(application_url)
|
654
|
+
|
655
|
+
config.action_mailer.default_url_options = {host: uri.host, protocol: uri.scheme}
|
656
|
+
config.action_mailer.asset_host = application_url
|
657
|
+
config.action_mailer.smtp_settings = {address: config.secret_config.fetch("emails/smtp/address", default: "localhost")}
|
658
|
+
config.action_mailer.raise_delivery_errors = config.secret_config.fetch("emails/raise_delivery_errors", default: true, type: :boolean)
|
659
|
+
~~~
|
660
|
+
|
661
|
+
### Symmetric Encryption
|
662
|
+
|
663
|
+
An example of how to setup Symmetric Encryption. Add to `application.rb`:
|
664
|
+
|
665
|
+
~~~ruby
|
666
|
+
# Encryption
|
667
|
+
config.symmetric_encryption.cipher =
|
668
|
+
SymmetricEncryption::Cipher.new(
|
669
|
+
key: config.secret_config.fetch('symmetric_encryption/key', encoding: :base64),
|
670
|
+
iv: config.secret_config.fetch('symmetric_encryption/iv', encoding: :base64),
|
671
|
+
version: config.secret_config.fetch('symmetric_encryption/version', type: :integer),
|
672
|
+
)
|
673
|
+
|
674
|
+
# Also support one prior encryption key version during key rotation
|
675
|
+
if config.secret_config.key?('symmetric_encryption/old/key')
|
676
|
+
SymmetricEncryption.secondary_ciphers = [
|
677
|
+
SymmetricEncryption::Cipher.new(
|
678
|
+
key: config.secret_config.fetch('symmetric_encryption/old/key', encoding: :base64),
|
679
|
+
iv: config.secret_config.fetch('symmetric_encryption/old/iv', encoding: :base64),
|
680
|
+
version: config.secret_config.fetch('symmetric_encryption/old/version', type: :integer),
|
681
|
+
),
|
682
|
+
]
|
683
|
+
end
|
684
|
+
~~~
|
685
|
+
|
686
|
+
Using this approach the file `config/symmetric-encryption.yml` can be removed once the keys have been moved to
|
687
|
+
the registry.
|
688
|
+
|
689
|
+
To extract existing keys from the config file so that they can be imported into the registry,
|
690
|
+
run the code below inside a console in each of the respective environments.
|
691
|
+
|
692
|
+
~~~ruby
|
693
|
+
require "yaml"
|
694
|
+
require "base64"
|
695
|
+
|
696
|
+
def se_config(cipher)
|
697
|
+
{
|
698
|
+
"key" => Base64.strict_encode64(cipher.send(:key)),
|
699
|
+
"iv" => Base64.strict_encode64(cipher.iv),
|
700
|
+
"version" => cipher.version
|
701
|
+
}
|
702
|
+
end
|
703
|
+
|
704
|
+
config = { "symmetric_encryption" => se_config(SymmetricEncryption.cipher) }
|
705
|
+
if cipher = SymmetricEncryption.secondary_ciphers.first
|
706
|
+
config["symmetric_encryption"]["old"] = se_config(cipher)
|
707
|
+
end
|
708
|
+
puts config.to_yaml
|
709
|
+
~~~
|
323
710
|
|
324
711
|
## Versioning
|
325
712
|
|
data/lib/secret_config/cli.rb
CHANGED
@@ -11,7 +11,7 @@ module SecretConfig
|
|
11
11
|
attr_reader :path, :region, :provider,
|
12
12
|
:export, :no_filter,
|
13
13
|
:import, :key_id, :random_size, :prune, :overwrite,
|
14
|
-
:copy_path,
|
14
|
+
:copy_path, :diff,
|
15
15
|
:console,
|
16
16
|
:show_version
|
17
17
|
|
@@ -35,6 +35,7 @@ module SecretConfig
|
|
35
35
|
@copy_path = nil
|
36
36
|
@show_version = false
|
37
37
|
@console = false
|
38
|
+
@diff = false
|
38
39
|
|
39
40
|
if argv.empty?
|
40
41
|
puts parser
|
@@ -51,10 +52,14 @@ module SecretConfig
|
|
51
52
|
run_console
|
52
53
|
elsif export
|
53
54
|
run_export(export, filtered: !no_filter)
|
55
|
+
elsif import && prune
|
56
|
+
run_import_and_prune(import)
|
54
57
|
elsif import
|
55
58
|
run_import(import)
|
56
59
|
elsif copy_path
|
57
60
|
run_copy(copy_path, path)
|
61
|
+
elsif diff
|
62
|
+
run_diff(diff)
|
58
63
|
else
|
59
64
|
puts parser
|
60
65
|
end
|
@@ -78,11 +83,15 @@ module SecretConfig
|
|
78
83
|
@import = file_name || STDIN
|
79
84
|
end
|
80
85
|
|
81
|
-
opts.on '-
|
86
|
+
opts.on '-C', '--copy SOURCE_PATH', 'Import configuration from a file or stdin if no file_name supplied.' do |path|
|
82
87
|
@copy_path = path
|
83
88
|
end
|
84
89
|
|
85
|
-
opts.on '-
|
90
|
+
opts.on '-D', '--diff [FILE_NAME]', 'Compare configuration from a file or stdin if no file_name supplied.' do |file_name|
|
91
|
+
@diff = file_name
|
92
|
+
end
|
93
|
+
|
94
|
+
opts.on '-c', '--console', 'Start interactive console.' do
|
86
95
|
@console = true
|
87
96
|
end
|
88
97
|
|
@@ -98,14 +107,9 @@ module SecretConfig
|
|
98
107
|
@no_filter = true
|
99
108
|
end
|
100
109
|
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
# end
|
105
|
-
#
|
106
|
-
# opts.on '-r', '--replace', 'During import replace existing keys if present' do
|
107
|
-
# @replace = true
|
108
|
-
# end
|
110
|
+
opts.on '-d', '--prune', 'During import delete all existing keys for which there is no key in the import file.' do
|
111
|
+
@prune = true
|
112
|
+
end
|
109
113
|
|
110
114
|
opts.on '-k', '--key_id KEY_ID', 'AWS KMS Key id or Key Alias to use when importing configuration values. Default: AWS Default key.' do |key_id|
|
111
115
|
@key_id = key_id
|
@@ -145,19 +149,35 @@ module SecretConfig
|
|
145
149
|
|
146
150
|
def run_export(file_name, filtered: true)
|
147
151
|
config = fetch_config(path, filtered: filtered)
|
148
|
-
|
149
|
-
format = file_format(file_name)
|
150
|
-
data = render(config, format)
|
151
|
-
write_file(file_name, data)
|
152
|
+
write_config_file(file_name, config)
|
152
153
|
|
153
154
|
puts("Exported #{path} from #{provider} to #{file_name}") if file_name.is_a?(String)
|
154
155
|
end
|
155
156
|
|
156
157
|
def run_import(file_name)
|
157
|
-
|
158
|
-
|
159
|
-
config
|
160
|
-
|
158
|
+
config = read_config_file(file_name)
|
159
|
+
|
160
|
+
set_config(config, path, current_values)
|
161
|
+
|
162
|
+
puts("Imported #{file_name} to #{provider} at #{path}") if file_name.is_a?(String)
|
163
|
+
end
|
164
|
+
|
165
|
+
def run_import_and_prune(file_name)
|
166
|
+
config = read_config_file(file_name)
|
167
|
+
delete_keys = current_values.keys - Utils.flatten(config, path).keys
|
168
|
+
|
169
|
+
unless delete_keys.empty?
|
170
|
+
puts "Going to delete the following keys:"
|
171
|
+
delete_keys.each {|key| puts " #{key}"}
|
172
|
+
sleep(5)
|
173
|
+
end
|
174
|
+
|
175
|
+
set_config(config, path, current_values)
|
176
|
+
|
177
|
+
delete_keys.each do |key|
|
178
|
+
puts "Deleting: #{key}"
|
179
|
+
provider_instance.delete(key)
|
180
|
+
end
|
161
181
|
|
162
182
|
puts("Imported #{file_name} to #{provider} at #{path}") if file_name.is_a?(String)
|
163
183
|
end
|
@@ -165,21 +185,64 @@ module SecretConfig
|
|
165
185
|
def run_copy(source_path, target_path)
|
166
186
|
config = fetch_config(source_path, filtered: false)
|
167
187
|
|
168
|
-
set_config(config, target_path)
|
188
|
+
set_config(config, target_path, current_values)
|
169
189
|
|
170
190
|
puts "Copied #{source_path} to #{target_path} using #{provider}"
|
171
191
|
end
|
172
192
|
|
193
|
+
def run_diff(file_name)
|
194
|
+
file_config = read_config_file(file_name)
|
195
|
+
file = Utils.flatten(file_config, path)
|
196
|
+
|
197
|
+
registry_config = fetch_config(path, filtered: false)
|
198
|
+
registry = Utils.flatten(registry_config, path)
|
199
|
+
|
200
|
+
(file.keys + registry.keys).sort.uniq.each do |key|
|
201
|
+
if registry.key?(key)
|
202
|
+
if file.key?(key)
|
203
|
+
if file[key].to_s != registry[key].to_s
|
204
|
+
puts "* #{key}: #{registry[key]} => #{file[key]}"
|
205
|
+
end
|
206
|
+
else
|
207
|
+
puts "- #{key}"
|
208
|
+
end
|
209
|
+
elsif file.key?(key)
|
210
|
+
puts "+ #{key}: #{file[key]}"
|
211
|
+
end
|
212
|
+
end
|
213
|
+
|
214
|
+
puts("Compared #{file_name} to #{provider} at #{path}") if file_name.is_a?(String)
|
215
|
+
end
|
216
|
+
|
173
217
|
def run_console
|
174
218
|
IRB.start
|
175
219
|
end
|
176
220
|
|
177
|
-
def
|
178
|
-
|
221
|
+
def current_values
|
222
|
+
@current_values ||= Utils.flatten(fetch_config(path, filtered: false), path)
|
223
|
+
end
|
224
|
+
|
225
|
+
def read_config_file(file_name)
|
226
|
+
format = file_format(file_name)
|
227
|
+
data = read_file(file_name)
|
228
|
+
parse(data, format)
|
229
|
+
end
|
230
|
+
|
231
|
+
def write_config_file(file_name, config)
|
232
|
+
format = file_format(file_name)
|
233
|
+
data = render(config, format)
|
234
|
+
write_file(file_name, data)
|
235
|
+
end
|
236
|
+
|
237
|
+
def set_config(config, path, current_values = {})
|
179
238
|
Utils.flatten_each(config, path) do |key, value|
|
180
239
|
next if value.nil?
|
240
|
+
next if current_values[key].to_s == value.to_s
|
181
241
|
|
182
|
-
|
242
|
+
if value.to_s.strip == '$random'
|
243
|
+
next if current_values[key]
|
244
|
+
value = random_password
|
245
|
+
end
|
183
246
|
puts "Setting: #{key}"
|
184
247
|
provider_instance.set(key, value)
|
185
248
|
end
|
@@ -4,7 +4,7 @@ require 'erb'
|
|
4
4
|
module SecretConfig
|
5
5
|
module Providers
|
6
6
|
# Read configuration from a local file
|
7
|
-
class File
|
7
|
+
class File < Provider
|
8
8
|
attr_reader :file_name
|
9
9
|
|
10
10
|
def initialize(file_name: "config/application.yml")
|
@@ -18,16 +18,11 @@ module SecretConfig
|
|
18
18
|
paths = path.sub(/\A\/*/, '').sub(/\/*\Z/, '').split("/")
|
19
19
|
settings = config.dig(*paths)
|
20
20
|
|
21
|
-
raise(ConfigurationError, "Path #{paths.join("
|
21
|
+
raise(ConfigurationError, "Path #{paths.join("/")} not found in file: #{file_name}") unless settings
|
22
22
|
|
23
23
|
Utils.flatten_each(settings, path, &block)
|
24
24
|
nil
|
25
25
|
end
|
26
|
-
|
27
|
-
def set(key, value)
|
28
|
-
raise NotImplementedError
|
29
|
-
end
|
30
|
-
|
31
26
|
end
|
32
27
|
end
|
33
28
|
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
module SecretConfig
|
2
|
+
module Providers
|
3
|
+
# Abstract Base provider
|
4
|
+
class Provider
|
5
|
+
def set(key, value)
|
6
|
+
raise NotImplementedError
|
7
|
+
end
|
8
|
+
|
9
|
+
def delete(key)
|
10
|
+
raise NotImplementedError
|
11
|
+
end
|
12
|
+
|
13
|
+
def to_h(path)
|
14
|
+
h = {}
|
15
|
+
each(path) { |key, value| h[key] = value }
|
16
|
+
h
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
@@ -7,7 +7,7 @@ end
|
|
7
7
|
module SecretConfig
|
8
8
|
module Providers
|
9
9
|
# Use the AWS System Manager Parameter Store for Centralized Configuration / Secrets Management
|
10
|
-
class Ssm
|
10
|
+
class Ssm < Provider
|
11
11
|
attr_reader :client, :key_id
|
12
12
|
|
13
13
|
def initialize(key_id: nil)
|
@@ -40,6 +40,10 @@ module SecretConfig
|
|
40
40
|
overwrite: true
|
41
41
|
)
|
42
42
|
end
|
43
|
+
|
44
|
+
def delete(key)
|
45
|
+
client.delete_parameter(name: key)
|
46
|
+
end
|
43
47
|
end
|
44
48
|
end
|
45
49
|
end
|
@@ -7,11 +7,14 @@ module SecretConfig
|
|
7
7
|
attr_reader :provider
|
8
8
|
attr_accessor :path
|
9
9
|
|
10
|
-
def initialize(path: nil, provider:
|
11
|
-
@path = path
|
10
|
+
def initialize(path: nil, provider: nil, provider_args: nil)
|
11
|
+
@path = default_path(path)
|
12
12
|
raise(UndefinedRootError, 'Root must start with /') unless @path.start_with?('/')
|
13
13
|
|
14
|
-
|
14
|
+
resolved_provider = default_provider(provider)
|
15
|
+
provider_args = nil if resolved_provider != provider
|
16
|
+
|
17
|
+
@provider = create_provider(resolved_provider, provider_args)
|
15
18
|
@cache = Concurrent::Map.new
|
16
19
|
refresh!
|
17
20
|
end
|
@@ -53,7 +56,7 @@ module SecretConfig
|
|
53
56
|
# Set the value for a key in the centralized configuration store.
|
54
57
|
def set(key:, value:, encrypt: true)
|
55
58
|
key = expand_key(key)
|
56
|
-
provider.set(key, value, encrypt:
|
59
|
+
provider.set(key, value, encrypt: encrypt)
|
57
60
|
cache[key] = value
|
58
61
|
end
|
59
62
|
|
@@ -152,13 +155,21 @@ module SecretConfig
|
|
152
155
|
args && args.size > 0 ? klass.new(**args) : klass.new
|
153
156
|
end
|
154
157
|
|
155
|
-
def default_path
|
156
|
-
path = ENV["SECRET_CONFIG_PATH"] || ENV["RAILS_ENV"]
|
158
|
+
def default_path(configured_path)
|
159
|
+
path = ENV["SECRET_CONFIG_PATH"] || configured_path || ENV["RAILS_ENV"]
|
157
160
|
path = Rails.env if path.nil? && defined?(Rails) && Rails.respond_to?(:env)
|
158
161
|
|
159
162
|
raise(UndefinedRootError, "Either set env var 'SECRET_CONFIG_PATH' or call SecretConfig.use") unless path
|
160
163
|
|
161
164
|
path.start_with?('/') ? path : "/#{path}"
|
162
165
|
end
|
166
|
+
|
167
|
+
def default_provider(provider)
|
168
|
+
provider = (ENV["SECRET_CONFIG_PROVIDER"] || provider || 'file')
|
169
|
+
|
170
|
+
return provider if provider.respond_to?(:each) && provider.respond_to?(:set)
|
171
|
+
|
172
|
+
provider.to_s.downcase.to_sym
|
173
|
+
end
|
163
174
|
end
|
164
175
|
end
|
data/lib/secret_config/utils.rb
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
module SecretConfig
|
2
2
|
module Utils
|
3
|
-
# Takes a hierarchical structure and flattens it to a single level
|
4
|
-
# If path is supplied it is prepended to every key returned
|
3
|
+
# Takes a hierarchical structure and flattens it to a single level.
|
4
|
+
# If path is supplied it is prepended to every key returned.
|
5
5
|
def self.flatten_each(hash, path = nil, &block)
|
6
6
|
hash.each_pair do |key, value|
|
7
7
|
name = path.nil? ? key : "#{path}/#{key}"
|
@@ -9,6 +9,14 @@ module SecretConfig
|
|
9
9
|
end
|
10
10
|
end
|
11
11
|
|
12
|
+
# Takes a hierarchical structure and flattens it to a single level hash.
|
13
|
+
# If path is supplied it is prepended to every key returned.
|
14
|
+
def self.flatten(hash, path = nil)
|
15
|
+
h = {}
|
16
|
+
flatten_each(hash, path) { |key, value| h[key] = value }
|
17
|
+
h
|
18
|
+
end
|
19
|
+
|
12
20
|
def self.constantize_symbol(symbol, namespace = 'SecretConfig::Providers')
|
13
21
|
klass = "#{namespace}::#{camelize(symbol.to_s)}"
|
14
22
|
begin
|
data/lib/secret_config.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: secret_config
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.5.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Reid Morrison
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2019-05-
|
11
|
+
date: 2019-05-14 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: concurrent-ruby
|
@@ -40,6 +40,7 @@ files:
|
|
40
40
|
- lib/secret_config/cli.rb
|
41
41
|
- lib/secret_config/errors.rb
|
42
42
|
- lib/secret_config/providers/file.rb
|
43
|
+
- lib/secret_config/providers/provider.rb
|
43
44
|
- lib/secret_config/providers/ssm.rb
|
44
45
|
- lib/secret_config/railtie.rb
|
45
46
|
- lib/secret_config/registry.rb
|