secret_config 0.4.5 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3c5b1779481318edb20fa0b59bc560cf55d890119fd901e7b59339f75232ae2f
-  data.tar.gz: 19bb4a25a9b9b8037f84f85e183d5be9d853be350132987cd6d2e577782c6e79
+  metadata.gz: 515036f2cebc7abd6211448b44f80c99c0872065a7947afaf4dface54c84d808
+  data.tar.gz: 2a5c7a8ccfd5d7d91d459a13d8cd5fcce8b1b381e1baa3e83946d99d712cca42
 SHA512:
-  metadata.gz: 8b7a542adb565a32e1a8a692a13f560745c7bf29f195ccbe1a681ab70526343b76fa7231a146c4b784df7acb2e429bb5b0a4b1c7f43f06740540c12b0fdee0ac
-  data.tar.gz: 751e31fce21e043d98208b299b3239b3bedd02e96e09aef6fe7f416a809f2b4236a4e26458a1b8a49ffe0460c1533f6b36bb25068d02aafc9cff47445977cfb9
+  metadata.gz: 73c14c00b3b1759b73d04415bb20724775efef6084729907d9377cb4928b513ed324b2edc67cd5953a6e2a8e71969b4add609d74b94ae2fd2467adfa5b0a48b5
+  data.tar.gz: 5dfb6fd8832349e7a63b8d6f02023c54cc33538d82e742b3bf24246252f1e0500fa948a9b835ca8ac054dc3c7a6d69b4090448938c782dc332640d361fa8997f

data/README.md

@@ -1,21 +1,19 @@
 # Secret Config
-[![Gem Version](https://img.shields.io/gem/v/secret_config.svg)](https://rubygems.org/gems/secret_config) [![Build Status](https://travis-ci.org/rocketjob/secret_config.svg?branch=master)](https://travis-ci.org/rocketjob/secret_config) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Beta-yellow.svg) [![Gitter chat](https://img.shields.io/badge/IRC%20(gitter)-Support-brightgreen.svg)](https://gitter.im/rocketjob/support)
+[![Gem Version](https://img.shields.io/gem/v/secret_config.svg)](https://rubygems.org/gems/secret_config) [![Build Status](https://travis-ci.org/rocketjob/secret_config.svg?branch=master)](https://travis-ci.org/rocketjob/secret_config) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) [![Gitter chat](https://img.shields.io/badge/IRC%20(gitter)-Support-brightgreen.svg)](https://gitter.im/rocketjob/support)
 
 Centralized Configuration and Secrets Management for Ruby and Rails applications.
 
-Securely store configuration information centrally.
-
-## Project Status
-
-Early development.
+Securely store configuration information centrally, supporting multiple tenants of the same application.
 
 ## Features
 
 Supports storing configuration information in:
 * File
     * Development and testing use.
+* Environment Variables
+    * Environment Variables take precedence and can be used to override any setting.
 * AWS System Manager Parameter Store
-    * Encrypt and store secrets such as passwords centrally. 
+    * Encrypt and securely store secrets such as passwords centrally. 
 
 ## Benefits
 
@@ -27,13 +25,156 @@ Benefits of moving sensitive configuration information into AWS System Manager P
     * In a large application the number of secrets can grow dramatically.
   * Removes the need to encrypt sensitive data config files.
     * Including securing and managing encryption keys.
-  * When encryption keys change, such as during a key rotation, config files don;t have to be changed.
+  * When encryption keys change, such as during a key rotation, config files don't have to be changed.
   * Removes security concerns with placing passwords in the clear into environment variables.
   * AWS System Manager Parameter Store does not charge for parameters.
     * Still recommend using a custom KMS key that charges only $1 per month.
     * Amounts as of 4/2019. Confirm what AWS charges you for these services.
-  * AWS Secrets Manager charges for every secret being managed, which can accumulate quickly with large projects. 
+  * AWS Secrets Manager charges for every secret being managed, which can accumulate quickly with large projects.
+  * Configure multiple distinct application instances to support multiple tenants.
+    * For example, use separate databases with unique credentials for each tenant.
+  * Separation of responsibilities is achieved since operations can manage production configuration.
+    * Developers do not need to be involved with production configuration such as host names and passwords.      
+  * All values are encrypted by default when stored in the AWS Parameter Store. 
+    * Prevents accidentally not encrypting sensitive data.
+  
+## Introduction
+
+When Secret Config starts up it reads all configuration entries into memory for all keys under the configured path.
+This means that once Secret Config has initialized all calls to Secret Config are extremely fast.
+
+The in-memory copy of the registry can be refreshed at any time by calling `SecretConfig.refresh!`. It can be refreshed
+via a process signal, or by calling it through an event, or via a messaging system.
+
+It is suggested that any programmatic lookup to values stored in Secret Config are called every time a value is
+being used, rather than creating a local copy of the value. This ensures that a refresh of the registry will take effect
+immediately for any code reading from Secret Config.
+  
+## API
+
+When Secret Config starts up it reads all configuration entries immediately for all keys under the configured path.
+This means that once Secret Config has initialized all calls to Secret Config are extremely fast.
+
+Secret Config supports the following programmatic interface:
+
+### Read values
+
+Fetch the value for the supplied key, returning nil if not found:
+
+~~~ruby
+# Key is present:
+SecretConfig["logger/level"]
+# => "info"
+  
+# Key is missing:
+SecretConfig["logger/blah"]
+# => nil
+~~~
+
+Fetch the value for the supplied key, raising `SecretConfig::MissingMandatoryKey` if not found:
+
+~~~ruby
+# Key is present:
+SecretConfig.fetch("logger/level")
+# => "info"
+  
+# Key is missing:
+SecretConfig.fetch("logger/blah")
+# => SecretConfig::MissingMandatoryKey (Missing configuration value for /development/logger/blah)
+~~~
+
+A default value can be supplied when the key is not found in the registry:
+
+~~~ruby
+SecretConfig.fetch("logger/level", default: "info")
+# => "info"
+~~~
+
+Since AWS SSM Parameter store and environment variables only support string values, 
+it is neccessary to convert the string back to the type required by the program.
+
+The following types are supported:
+    `:integer`
+    `:float`
+    `:string`
+    `:boolean`
+    `:symbol`
+
+~~~ruby
+# Without type conversion:
+SecretConfig.fetch("symmetric_encryption/version")
+# => "0"
+
+# With type conversion:
+SecretConfig.fetch("symmetric_encryption/version", type: :integer)
+# => 0
+~~~
+
+When storing binary data, it should be encoded with strict base64 encoding. To automatically convert it back to binary
+specify the encoding as `:base64`
+
+~~~ruby
+# Return a value that was stored in Base64 encoding format:
+SecretConfig.fetch("symmetric_encryption/iv")
+# => "FW+/wLubAYM+ZU0bWQj59Q=="
   
+# Base64 decode a value that was stored in Base64 encoding format:
+SecretConfig.fetch("symmetric_encryption/iv", encoding: :base64)
+# => "\x15o\xBF\xC0\xBB\x9B\x01\x83>eM\eY\b\xF9\xF5"
+~~~
+
+### Key presence
+
+Returns whether a key is present in the registry:
+
+~~~ruby
+SecretConfig.key?("logger/level")
+# => true
+~~~
+
+### Write values
+
+When Secret Config is configured to use the AWS SSM Parameter store, its values can be modified:
+
+~~~ruby
+SecretConfig["logger/level"] = "debug"
+~~~
+
+~~~ruby
+SecretConfig.set("logger/level", "debug")
+~~~
+
+### Configuration
+
+Returns a Hash copy of the configuration as a tree:
+
+~~~ruby
+SecretConfig.configuration
+~~~
+
+### Refresh Configuration
+
+Tell Secret Config to refresh its in-memory copy of the configuration settings.
+
+~~~ruby
+SecretConfig.refresh!
+~~~
+
+Example, refresh the registry any time a SIGUSR2 is raised, add the following code on startup:
+      
+~~~ruby
+Signal.trap('USR2') do
+  SecretConfig.refresh!
+end
+~~~
+
+Then to make the process refresh it registry:
+~~~shell
+kill -SIGUSR2 1234
+~~~
+
+Where `1234` above is the process PID.
+
 ## Development and Test use
 
 In the development environment create the file `config/application.yml` within which to store local development credentials.
@@ -173,7 +314,7 @@ For example, somewhere in your codebase you need a persistent http connection:
 ~~~
 
 Then the application that uses the above library / gem just needs to add the relevant entries to their
-`application.rb` file:
+`application.yml` file:
 
 ~~~yaml
 http_client:
@@ -199,18 +340,39 @@ as covered above. By default it will use env var `RAILS_ENV` to define the path
 
 The default settings are great for getting started in development and test, but should not be used in production. 
 
-Add the setting to `config/environments/production.rb` to make it fetch its settings from 
-AWS System Manager Parameter Store:
+To ensure Secret Config is configured and available for use within any of the config files, add
+the following lines to the very top of `application.rb` under the line `class Application < Rails::Application`: 
 
 ~~~ruby
-Rails.application.configure do
-  # Read configuration from AWS Parameter Store
-  config.secret_config.use :ssm, path: '/production/my_application'
+module MyApp
+  class Application < Rails::Application
+  
+    # Add the following lines to configure Secret Config:
+    if Rails.env.development? || Rails.env.test?
+      # Use 'config/application.yml'
+      config.secret_config.use :file
+    else
+      # Read configuration from AWS SSM Parameter Store
+      config.secret_config.use :ssm, path: "/#{Rails.env}/my_app"
+    end    
+    
+    # ....
+  end
 end
 ~~~
 
 `path` is the path from which the configuration data will be read. This path uniquely identifies the
-configuration for this instance of the application.
+configuration for this instance of the application. In the example above it uses the rails env and application name
+by default. This can be overridden using the `SECRET_CONFIG_PATH` environment variable when needed.
+
+By placing the secret config configuration as the very first configuration item, it allows any subsequent
+configuration item to access the centralized configuration in AWS System Manager Parameter Store.
+
+The environment variable `SECRET_CONFIG_PROVIDER` can be used to override the provider when needed.
+For example: 
+    `export SECRET_CONFIG_PROVIDER=ssm`
+Or,
+    `export SECRET_CONFIG_PROVIDER=file`
 
 If we need 2 completely separate instances of the application running in a single AWS account then we could use
 multiple paths. For example:
@@ -228,11 +390,22 @@ When writing settings to the parameter store, it is recommended to use a custom
 To supply the key to encrypt the values with, add the `key_id` parameter: 
 
 ~~~ruby
-Rails.application.configure do
-  # Read configuration from AWS Parameter Store
-  config.secret_config.use :ssm,
-                           key_id: 'alias/production/myapplication',
-                           path: '/production/my_application'
+module MyApp
+  class Application < Rails::Application
+  
+    # Add the following lines to configure Secret Config:
+    if Rails.env.development? || Rails.env.test?
+      # Use 'config/application.yml'
+      config.secret_config.use :file
+    else
+      # Read configuration from AWS SSM Parameter Store
+      config.secret_config.use :ssm, 
+        path: "/#{Rails.env}/my_app",
+        key_id: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
+    end    
+    
+    # ....
+  end
 end
 ~~~
 
@@ -241,6 +414,31 @@ Note: The relevant KMS key must be created first prior to using it here.
 The `key_id` is only used when writing settings to the AWS Parameter store and can be left off when that instance
 will only read from the parameter store.
 
+### Shared configuration for development and test
+
+When running multiple engines or private "gems" inside the same code repository, the development and test
+configuration file `application.yml` can be shared. Update the lines above to:
+
+~~~ruby
+module MyApp
+  class Application < Rails::Application
+  
+    # Add the following lines:
+    if Rails.env.development? || Rails.env.test?
+      # Use 'config/application.yml'
+      config.secret_config.use :file
+    else
+      # Read configuration from AWS SSM Parameter Store
+      config.secret_config.use :ssm, path: "/#{Rails.env}/my_app"
+    end    
+    
+    # ....
+  end
+end
+~~~
+
+Where `file_name` is the full path and filename for where `application.yml` is located.
+
 ### Authorization
 
 The following policy needs to be added to the IAM Group under which the application will be running:
@@ -253,8 +451,9 @@ The following policy needs to be added to the IAM Group under which the applicat
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": [
-                "ssm:PutParameter",
                 "ssm:GetParametersByPath",
+                "ssm:PutParameter",
+                "ssm:DeleteParameter",
             ],
             "Resource": "*"
         }
@@ -279,11 +478,14 @@ Secret Config has a command line interface for exporting, importing and copying
 secret_config [options]
     -e, --export [FILE_NAME]         Export configuration to a file or stdout if no file_name supplied.
     -i, --import [FILE_NAME]         Import configuration from a file or stdin if no file_name supplied.
-    -c, --copy SOURCE_PATH           Import configuration from a file or stdin if no file_name supplied.
+    -C, --copy SOURCE_PATH           Import configuration from a file or stdin if no file_name supplied.
+    -D, --diff [FILE_NAME]           Compare configuration from a file or stdin if no file_name supplied.
+    -c, --console                    Start interactive console.
     -p, --path PATH                  Path to import from / export to.
     -P, --provider PROVIDER          Provider to use. [ssm | file]. Default: ssm
     -U, --no-filter                  Do not filter passwords and keys.
-    -k, --key KEY_ID | KEY_ALIAS     AWS KMS Key id or Key Alias to use when importing configuration values. Default: AWS Default key.
+    -d, --prune                      During import delete all existing keys for which there is no key in the import file.
+    -k, --key_id KEY_ID              AWS KMS Key id or Key Alias to use when importing configuration values. Default: AWS Default key.
     -r, --region REGION              AWS Region to use. Default: AWS_REGION env var.
     -R, --random_size INTEGER        Size to use when generating random values. Whenever $random is encountered during an import. Default: 32
     -v, --version                    Display Symmetric Encryption version.
@@ -292,34 +494,219 @@ secret_config [options]
 
 ### CLI Examples
 
+#### Import from a file into SSM parameters
+
+To get started it is useful to create a YAML file with all the relevant settings and then import
+it into AWS SSM Parameter store. This file is the same as `applcation.yml` except that each file
+is just for one environment. I.e. It does not contain the `test` or `development` root level entries.
+
+For example: `production.yml`
+
+~~~yaml
+mysql:
+  database:   secret_config_production
+  username:   secret_config
+  password:   secret_configrules
+  host:       mysql_server.example.net
+
+mongo:
+  database:   secret_config_production
+  primary:    mongo_primary.example.net:27017
+  secondary:  mongo_secondary.example.net:27017
+
+secrets:
+  secret_key_base: somereallylongproductionstring
+~~~
+
+Import a yaml file, into a path in AWS SSM Parameter Store:
+
+    secret_config --import production.yml --path /production/my_application
+
+Import a yaml file, into a path in AWS SSM Parameter Store, using a custom KMS key to encrypt the values:
+
+    secret_config --import production.yml --path /production/my_application --key_id "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
+
+#### Diff
+
+Before importing a new config file into the AWS SSM Parameter store, a diff can be performed to determine
+what the differences are that will be applied when the import is run with the `--prune` option.
+
+    secret_config --diff production.yml --path /production/my_application
+
+Key:
+
+    + Adding a new key to the registry.
+    - The key will be removed from the registry during the import if --prune is specified.
+    * The value for that key will change during an import.
+
+#### Export SSM parameters
+
+In AWS SSM Parameter store it can be difficult to 
+Export the values from a specific path into a yaml or json file so that they are easier to read.
+
 Export from a path in AWS SSM Parameter Store to a yaml file, where passwords are filtered:
 
-    secret_config --export test.yml --path /test/my_application
+    secret_config --export production.yml --path /production/my_application
 
 Export from a path in AWS SSM Parameter Store to a yaml file, _without_ filtering out passwords:
 
-    secret_config --export test.yml --path /test/my_application --no-filter
+    secret_config --export production.yml --path /production/my_application --no-filter
 
 Export from a path in AWS SSM Parameter Store to a json file, where passwords are filtered:
 
-    secret_config --export test.json --path /test/my_application
+    secret_config --export production.json --path /production/my_application
 
-Import a yaml file, into a path in AWS SSM Parameter Store:
+#### Copy values between paths in AWS SSM parameter store
 
-    secret_config --import test.yml --path /production/my_application
+It can be useful to keep a "master" copy of the values for an environment or stack in a custom path
+in AWS Parameter Store. Then for each stack or environment that is spun up, copy the "master" / "common" values
+into the new path. Once copied the values specific to that path can be updated accordingly.
 
-Import a yaml file, into a path in AWS SSM Parameter Store, using a custom KMS key to encrypt the values:
+Copy configuration from one path in AWS SSM Parameter Store to another path in AWS SSM Parameter Store:
 
-    secret_config --import test.yml --path /production/my_application --key_id "arn:aws:kms:us-east-1:23643632463:key/UUID"
+    secret_config --copy /production/my_application --path /tenant73/my_application
 
-Copy configuration from one path in AWS SSM Parameter Store to another path in AWS SSM Parameter Store:
+#### Generating random passwords
+
+In the multi-tenant example above, we may want to generate a secure random password for each tenant.
+In the source file or registry, set the value to `$random`, this will ensure that during the `import` or `copy`
+that the destination will receive a secure random value. 
+
+By default the length of the randomized value is 32 bytes, use `--random_size` to adjust the length of 
+the randomized string. 
+
+## Docker
+
+Secret Config is at its best when the application is containerized. By externalizing the configuration the same
+docker container can be tested in one or more environments and then deployed directly to production without
+any changes. The only difference being the path that container uses to read its configuration from.
+
+Another important benefit is that the docker image does not contain any production or test credentials since
+these are all stored in AWS SSM Parameter Store.
+
+When a Ruby / Rails application is using Secret Config for its configuration settings, it only requires the 
+following environment variables when starting up the container in for example AWS ECS or AWS Fargate:
 
-    secret_config --copy /test/my_application --path /production/my_application
+~~~shell
+export SECRET_CONFIG_PATH=/production/my_application
+~~~
+
+For rails applications, typically the `RAILS_ENV` is also needed, but not required for Secret Config.
+
+~~~shell
+export RAILS_ENV=production
+~~~
+
+### Logging
+
+When using Semantic Logger, the following code could be added to `application.rb` to facilitate configuration
+of the logging output via Secret Config:
+
+~~~ruby
+# Logging
+config.log_level                       = config.secret_config.fetch("logger/level", default: :info, type: :symbol)
+config.semantic_logger.backtrace_level = config.secret_config.fetch("logger/backtrace_level", default: :error, type: :symbol)
+config.semantic_logger.application     = config.secret_config.fetch("logger/application", default: "my_app")
+config.semantic_logger.environment     = config.secret_config.fetch("logger/environment", default: Rails.env)
+~~~
+
+In any environment the log level can be changed, for example set `logger/level` to `debug`. And it can be changed
+in the AWS SSM Parameter Store, or directly with the environment variable `export LOGGER_LEVEL=debug`
+
+`logger/environment` can be used to identify which tenant the log messages are emanating from. By default it is just
+the rails environment. For example set `logger/environment` to `tenant73`.
+
+Additionally the following code can be used with containers to send log output to standard out:
+
+~~~ruby
+destination = config.secret_config.fetch("logger/destination", default: :file, type: :symbol)
+if destination == :stdout
+  STDOUT.sync                                    = true
+  config.rails_semantic_logger.add_file_appender = false
+  config.semantic_logger.add_appender(
+    io:        STDOUT,
+    level:     config.log_level,
+    formatter: config.secret_config.fetch("logger/formatter", default: :default, type: :symbol)
+  )
+end
+~~~
+
+Specifically for docker containers it is necessary to turn off file logging and turn on logging to standard out
+so that AWS Cloud Watch can pick up the log data.
+
+To start with `logger/destination` of `stdout` will work with regular non-colorized output. When feeding the 
+log output into something that can process JSON, set `logger/formatter` to `json`.
+
+The benefit with the above approach is that a developer can pull the exact same container image that is running
+in production and configure it to run locally on their laptop. For example, set `logger/destination` to `file`.
 
-During an `import` or `copy` if any of the source values consist only of `$random`, 
-they will be replaced with a secure 32 byte random value.
-This is deal for when a secure random password needs to be generated. 
-Use `--random_size` to adjust the length of the randomized string.
+The above code can be modified as necessary to add any Semantic Logger appender to write directly to external
+centralized logging systems, instead of writing to standard out or local files. 
+
+### Email Server and Assets
+
+An example of how to setup the email server and the assets for html emails. Add to `application.rb`:
+
+~~~ruby
+# Emails
+application_url = config.secret_config.fetch("emails/asset_host")
+uri             = URI.parse(application_url)
+
+config.action_mailer.default_url_options   = {host: uri.host, protocol: uri.scheme}
+config.action_mailer.asset_host            = application_url
+config.action_mailer.smtp_settings         = {address: config.secret_config.fetch("emails/smtp/address", default: "localhost")}
+config.action_mailer.raise_delivery_errors = config.secret_config.fetch("emails/raise_delivery_errors", default: true, type: :boolean)
+~~~
+
+### Symmetric Encryption
+
+An example of how to setup Symmetric Encryption. Add to `application.rb`:
+
+~~~ruby
+# Encryption
+config.symmetric_encryption.cipher =
+  SymmetricEncryption::Cipher.new(
+    key:     config.secret_config.fetch('symmetric_encryption/key', encoding: :base64),
+    iv:      config.secret_config.fetch('symmetric_encryption/iv', encoding: :base64),
+    version: config.secret_config.fetch('symmetric_encryption/version', type: :integer),
+  )
+
+# Also support one prior encryption key version during key rotation
+if config.secret_config.key?('symmetric_encryption/old/key')
+  SymmetricEncryption.secondary_ciphers = [
+    SymmetricEncryption::Cipher.new(
+      key:     config.secret_config.fetch('symmetric_encryption/old/key', encoding: :base64),
+      iv:      config.secret_config.fetch('symmetric_encryption/old/iv', encoding: :base64),
+      version: config.secret_config.fetch('symmetric_encryption/old/version', type: :integer),
+    ),
+  ]
+end
+~~~
+
+Using this approach the file `config/symmetric-encryption.yml` can be removed once the keys have been moved to
+the registry.
+
+To extract existing keys from the config file so that they can be imported into the registry, 
+run the code below inside a console in each of the respective environments.
+
+~~~ruby
+require "yaml"
+require "base64"
+
+def se_config(cipher)
+  {
+    "key"     => Base64.strict_encode64(cipher.send(:key)),
+    "iv"      => Base64.strict_encode64(cipher.iv),
+    "version" => cipher.version
+  }
+end
+
+config = { "symmetric_encryption" => se_config(SymmetricEncryption.cipher) }
+if cipher = SymmetricEncryption.secondary_ciphers.first
+  config["symmetric_encryption"]["old"] = se_config(cipher)
+end
+puts config.to_yaml
+~~~
 
 ## Versioning
 

data/lib/secret_config/cli.rb

@@ -11,7 +11,7 @@ module SecretConfig
     attr_reader :path, :region, :provider,
                 :export, :no_filter,
                 :import, :key_id, :random_size, :prune, :overwrite,
-                :copy_path,
+                :copy_path, :diff,
                 :console,
                 :show_version
 
@@ -35,6 +35,7 @@ module SecretConfig
       @copy_path    = nil
       @show_version = false
       @console      = false
+      @diff         = false
 
       if argv.empty?
         puts parser
@@ -51,10 +52,14 @@ module SecretConfig
         run_console
       elsif export
         run_export(export, filtered: !no_filter)
+      elsif import && prune
+        run_import_and_prune(import)
       elsif import
         run_import(import)
       elsif copy_path
         run_copy(copy_path, path)
+      elsif diff
+        run_diff(diff)
       else
         puts parser
       end
@@ -78,11 +83,15 @@ module SecretConfig
           @import = file_name || STDIN
         end
 
-        opts.on '-c', '--copy SOURCE_PATH', 'Import configuration from a file or stdin if no file_name supplied.' do |path|
+        opts.on '-C', '--copy SOURCE_PATH', 'Import configuration from a file or stdin if no file_name supplied.' do |path|
           @copy_path = path
         end
 
-        opts.on '-i', '--console', 'Start interactive console.' do
+        opts.on '-D', '--diff [FILE_NAME]', 'Compare configuration from a file or stdin if no file_name supplied.' do |file_name|
+          @diff = file_name
+        end
+
+        opts.on '-c', '--console', 'Start interactive console.' do
           @console = true
         end
 
@@ -98,14 +107,9 @@ module SecretConfig
           @no_filter = true
         end
 
-        # TODO:
-        # opts.on '-Q', '--prune', 'During import delete all existing keys for which their is no key in the import file' do
-        #   @prune = true
-        # end
-        #
-        # opts.on '-r', '--replace', 'During import replace existing keys if present' do
-        #   @replace = true
-        # end
+        opts.on '-d', '--prune', 'During import delete all existing keys for which there is no key in the import file.' do
+          @prune = true
+        end
 
         opts.on '-k', '--key_id KEY_ID', 'AWS KMS Key id or Key Alias to use when importing configuration values. Default: AWS Default key.' do |key_id|
           @key_id = key_id
@@ -145,19 +149,35 @@ module SecretConfig
 
     def run_export(file_name, filtered: true)
       config = fetch_config(path, filtered: filtered)
-
-      format = file_format(file_name)
-      data   = render(config, format)
-      write_file(file_name, data)
+      write_config_file(file_name, config)
 
       puts("Exported #{path} from #{provider} to #{file_name}") if file_name.is_a?(String)
     end
 
     def run_import(file_name)
-      format = file_format(file_name)
-      data   = read_file(file_name)
-      config = parse(data, format)
-      set_config(config, path)
+      config = read_config_file(file_name)
+
+      set_config(config, path, current_values)
+
+      puts("Imported #{file_name} to #{provider} at #{path}") if file_name.is_a?(String)
+    end
+
+    def run_import_and_prune(file_name)
+      config      = read_config_file(file_name)
+      delete_keys = current_values.keys - Utils.flatten(config, path).keys
+
+      unless delete_keys.empty?
+        puts "Going to delete the following keys:"
+        delete_keys.each {|key| puts "  #{key}"}
+        sleep(5)
+      end
+
+      set_config(config, path, current_values)
+
+      delete_keys.each do |key|
+        puts "Deleting: #{key}"
+        provider_instance.delete(key)
+      end
 
       puts("Imported #{file_name} to #{provider} at #{path}") if file_name.is_a?(String)
     end
@@ -165,21 +185,64 @@ module SecretConfig
     def run_copy(source_path, target_path)
       config = fetch_config(source_path, filtered: false)
 
-      set_config(config, target_path)
+      set_config(config, target_path, current_values)
 
       puts "Copied #{source_path} to #{target_path} using #{provider}"
     end
 
+    def run_diff(file_name)
+      file_config = read_config_file(file_name)
+      file        = Utils.flatten(file_config, path)
+
+      registry_config = fetch_config(path, filtered: false)
+      registry        = Utils.flatten(registry_config, path)
+
+      (file.keys + registry.keys).sort.uniq.each do |key|
+        if registry.key?(key)
+          if file.key?(key)
+            if file[key].to_s != registry[key].to_s
+              puts "* #{key}: #{registry[key]} => #{file[key]}"
+            end
+          else
+            puts "- #{key}"
+          end
+        elsif file.key?(key)
+          puts "+ #{key}: #{file[key]}"
+        end
+      end
+
+      puts("Compared #{file_name} to #{provider} at #{path}") if file_name.is_a?(String)
+    end
+
     def run_console
       IRB.start
     end
 
-    def set_config(config, path)
-      # TODO: prune, replace
+    def current_values
+      @current_values ||= Utils.flatten(fetch_config(path, filtered: false), path)
+    end
+
+    def read_config_file(file_name)
+      format = file_format(file_name)
+      data   = read_file(file_name)
+      parse(data, format)
+    end
+
+    def write_config_file(file_name, config)
+      format = file_format(file_name)
+      data   = render(config, format)
+      write_file(file_name, data)
+    end
+
+    def set_config(config, path, current_values = {})
       Utils.flatten_each(config, path) do |key, value|
         next if value.nil?
+        next if current_values[key].to_s == value.to_s
 
-        value = random_password if value.to_s.strip == '$random'
+        if value.to_s.strip == '$random'
+          next if current_values[key]
+          value = random_password
+        end
         puts "Setting: #{key}"
         provider_instance.set(key, value)
       end

data/lib/secret_config/providers/file.rb

@@ -4,7 +4,7 @@ require 'erb'
 module SecretConfig
   module Providers
     # Read configuration from a local file
-    class File
+    class File < Provider
       attr_reader :file_name
 
       def initialize(file_name: "config/application.yml")
@@ -18,16 +18,11 @@ module SecretConfig
         paths    = path.sub(/\A\/*/, '').sub(/\/*\Z/, '').split("/")
         settings = config.dig(*paths)
 
-        raise(ConfigurationError, "Path #{paths.join(".")} not found in file: #{file_name}") unless settings
+        raise(ConfigurationError, "Path #{paths.join("/")} not found in file: #{file_name}") unless settings
 
         Utils.flatten_each(settings, path, &block)
         nil
       end
-
-      def set(key, value)
-        raise NotImplementedError
-      end
-
     end
   end
 end

data/lib/secret_config/providers/provider.rb

@@ -0,0 +1,20 @@
+module SecretConfig
+  module Providers
+    # Abstract Base provider
+    class Provider
+      def set(key, value)
+        raise NotImplementedError
+      end
+
+      def delete(key)
+        raise NotImplementedError
+      end
+
+      def to_h(path)
+        h = {}
+        each(path) { |key, value| h[key] = value }
+        h
+      end
+    end
+  end
+end

data/lib/secret_config/providers/ssm.rb

@@ -7,7 +7,7 @@ end
 module SecretConfig
   module Providers
     # Use the AWS System Manager Parameter Store for Centralized Configuration / Secrets Management
-    class Ssm
+    class Ssm < Provider
       attr_reader :client, :key_id
 
       def initialize(key_id: nil)
@@ -40,6 +40,10 @@ module SecretConfig
           overwrite: true
         )
       end
+
+      def delete(key)
+        client.delete_parameter(name: key)
+      end
     end
   end
 end

data/lib/secret_config/registry.rb

@@ -7,11 +7,14 @@ module SecretConfig
     attr_reader :provider
     attr_accessor :path
 
-    def initialize(path: nil, provider: :file, provider_args: nil)
-      @path = path || default_path
+    def initialize(path: nil, provider: nil, provider_args: nil)
+      @path = default_path(path)
       raise(UndefinedRootError, 'Root must start with /') unless @path.start_with?('/')
 
-      @provider = create_provider(provider, provider_args)
+      resolved_provider = default_provider(provider)
+      provider_args     = nil if resolved_provider != provider
+
+      @provider = create_provider(resolved_provider, provider_args)
       @cache    = Concurrent::Map.new
       refresh!
     end
@@ -53,7 +56,7 @@ module SecretConfig
     # Set the value for a key in the centralized configuration store.
     def set(key:, value:, encrypt: true)
       key = expand_key(key)
-      provider.set(key, value, encrypt: true)
+      provider.set(key, value, encrypt: encrypt)
       cache[key] = value
     end
 
@@ -152,13 +155,21 @@ module SecretConfig
       args && args.size > 0 ? klass.new(**args) : klass.new
     end
 
-    def default_path
-      path = ENV["SECRET_CONFIG_PATH"] || ENV["RAILS_ENV"]
+    def default_path(configured_path)
+      path = ENV["SECRET_CONFIG_PATH"] || configured_path || ENV["RAILS_ENV"]
       path = Rails.env if path.nil? && defined?(Rails) && Rails.respond_to?(:env)
 
       raise(UndefinedRootError, "Either set env var 'SECRET_CONFIG_PATH' or call SecretConfig.use") unless path
 
       path.start_with?('/') ? path : "/#{path}"
     end
+
+    def default_provider(provider)
+      provider = (ENV["SECRET_CONFIG_PROVIDER"] || provider || 'file')
+
+      return provider if provider.respond_to?(:each) && provider.respond_to?(:set)
+
+      provider.to_s.downcase.to_sym
+    end
   end
 end

data/lib/secret_config/utils.rb

@@ -1,7 +1,7 @@
 module SecretConfig
   module Utils
-    # Takes a hierarchical structure and flattens it to a single level
-    # If path is supplied it is prepended to every key returned
+    # Takes a hierarchical structure and flattens it to a single level.
+    # If path is supplied it is prepended to every key returned.
     def self.flatten_each(hash, path = nil, &block)
       hash.each_pair do |key, value|
         name = path.nil? ? key : "#{path}/#{key}"
@@ -9,6 +9,14 @@ module SecretConfig
       end
     end
 
+    # Takes a hierarchical structure and flattens it to a single level hash.
+    # If path is supplied it is prepended to every key returned.
+    def self.flatten(hash, path = nil)
+      h = {}
+      flatten_each(hash, path) { |key, value| h[key] = value }
+      h
+    end
+
     def self.constantize_symbol(symbol, namespace = 'SecretConfig::Providers')
       klass = "#{namespace}::#{camelize(symbol.to_s)}"
       begin

data/lib/secret_config/version.rb

@@ -1,3 +1,3 @@
 module SecretConfig
-  VERSION = '0.4.5'
+  VERSION = '0.5.0'
 end

data/lib/secret_config.rb

@@ -8,6 +8,7 @@ require 'secret_config/railtie' if defined?(Rails)
 module SecretConfig
   module Providers
     autoload :File, 'secret_config/providers/file'
+    autoload :Provider, 'secret_config/providers/provider'
     autoload :Ssm, 'secret_config/providers/ssm'
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: secret_config
 version: !ruby/object:Gem::Version
-  version: 0.4.5
+  version: 0.5.0
 platform: ruby
 authors:
 - Reid Morrison
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-05-01 00:00:00.000000000 Z
+date: 2019-05-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -40,6 +40,7 @@ files:
 - lib/secret_config/cli.rb
 - lib/secret_config/errors.rb
 - lib/secret_config/providers/file.rb
+- lib/secret_config/providers/provider.rb
 - lib/secret_config/providers/ssm.rb
 - lib/secret_config/railtie.rb
 - lib/secret_config/registry.rb