cloud-config 0.1.1 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 103ce286b688715803e0542a5b739d4719ff0c002113368d2db8828dcbe07be5
-  data.tar.gz: e400a9d26f3f3e291147699f3077d4fac31d5d9d9ce82712bbe3690d75607e84
+  metadata.gz: 4f4463471bbbffc8cdfcda8361581f051804c106523c0fa86838f5e62de59a6e
+  data.tar.gz: 8c31a0844247d7e210e219099170f8bda4212fca7e2e61ab323872a8d68f8ed7
 SHA512:
-  metadata.gz: 9317560df6ac2d68e26a189994f8090386c466ba39720874911ee79f6e8869b52ed6b2d0c703d7d9b66ec291e625306ad58a85f044b558baff540798a1229d83
-  data.tar.gz: 6d634f9653b9df6d443524ab931ed0f79768db9db33a650e020ee43f9c15e32d13b94bbace9a2f3af231fcdf24bd26d690f1474420afde4f8d9f86ffaef77614
+  metadata.gz: 75a640f3415c4ae1b66e1d9260f1e02c77b5d4a692ef1de609ef3fbef5afcd6e858120a74ca15ef3a8a77259cce702bf856d60c1bac3d549cad1c32f24f2343e
+  data.tar.gz: 9e033c6fa3481c798ff064666f419478a5ff150e05cce1fe0bd6a44a235601b7a28f54fc7e37fa95bed89ddb38a2fb97b9e4587f1375a2531084cfbd55aad9e3

data/CHANGELOG.md

@@ -1,5 +1,26 @@
 ## [Unreleased]
 
-## [0.1.0] - 2023-01-15
+## [Released]
+
+## [0.2.1] - 2023-05-21
+
+Enhancements:
+
+- Allow options to be passed when fetching configuration from the provider
+
+
+## [0.2.0] - 2023-05-20
+
+Bug fixes:
+
+- Fix in memory cache expiry
+
+Enhancements:
+
+- Reset cache for single key
+- Lookup providers by configuration key
+
+
+## [0.1.0] - 2023-05-17
 
 - Initial release

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    cloud-config (0.1.0)
+    cloud-config (0.2.0)
       parallel
 
 GEM

data/README.md

@@ -2,7 +2,9 @@
 
 A library to modernise the way applications fetch configuration. Typically an application will use environment variables and settings files to store and retrieve configurations, but there are many issues with this approach. Often environment variables create a security risk and settings files hard code infrastructure dependencies into the code. Changing configuration settings will usually require redeploying large parts of the infrastructure and perhaps even need to go through the application deployment lifecycle.
 
-A modern approach stores configuration remotely, often using key/value databases. Storing configurations in a single database, separately from the codebase reduces infrastructure dependency. Configuration updates can automatically sync with any application, without requiring redeployments. Security risk is greatly reduced, since the configurations can be securely stored. Another goal with this approach is creating an environmentless codebase. The application no longer needs to know which environment it's running in, since all the configuration is handled by the infrastucture.
+A modern approach stores configuration remotely, often using key/value databases. Storing configurations in a single database, separately from the codebase reduces infrastructure dependency. Configuration updates can automatically sync with any application, without requiring redeployments. Security risk is greatly reduced, since configurations can be securely stored. Another goal with this approach is creating an environmentless codebase. The application no longer needs to know which environment it's running in, since all the configuration is handled by the infrastucture.
+
+Another common problem is password rotation. A typical application will need to be restarted or even redeployed when the configuration settings change. CloudConfig can handle this elegantly, improving application uptime and resilience.
 
 ## Installation
 
@@ -16,7 +18,7 @@ If bundler is not being used to manage dependencies, install the gem by executin
 
 ## Usage
 
-CloudConfig can be configured to fetch keys from multiple providers. Since CloudConfig will need to know which provider has the correct key, all the keys will need to be preconfigured. An example configuration may look like
+CloudConfig can be configured to fetch keys from multiple providers. Since CloudConfig will need to know which provider has the correct key, all the keys will need to be preconfigured. Duplicate keys will be overriden with the latest provider. An example configuration may look like
 
 ```ruby
 CloudConfig.configure do
@@ -27,6 +29,18 @@ CloudConfig.configure do
 end
 ```
 
+Fetch a setting with
+
+```ruby
+url = CloudConfig.get(:api_url)
+```
+
+If caching has been configured, the cache can be reset using
+
+```ruby
+url = CloudConfig.get(:api_url, reset_cache: true)
+```
+
 ### Configuration Options
 
 ```ruby
@@ -61,12 +75,17 @@ CloudConfig.configure do
   provider :in_memory do
     setting :key1, cache: 60 # 60 seconds
   end
+
+  # If the datastore has configurable options
+  provider :aws_parameter_store do
+    setting :key1, with_decryption: true
+  end
 end
 ```
 
 ### Preloading
 
-CloudConfig can preload all the keys configured for preload enabled providers. A cache client must be configured, otherwise
+CloudConfig can preload all the keys configured for preload enabled providers. A cache client must also be configured, otherwise
 preloading won't do anything.
 
 ```ruby
@@ -74,6 +93,7 @@ CloudConfig.configure do
   cache_client CloudConfig::Cache::InMemory.new
 
   provider :in_memory, preload: true do
+    setting :key1
   end
 end
 ```
@@ -84,9 +104,77 @@ Call preload to cache all the keys.
 CloudConfig.preload
 ```
 
+### Custom Providers
+
+CloudConfig comes equipped with a set of providers for fetching configuration from remote datastores. The limit set of providers will not be sufficient for most real world applications, so creating custom providers will be necessary. There is a very simple interface consisting of 3 methods necessary for creating a custom provider.
+
+```ruby
+class CustomProvider
+  # Define initialize with a params argument.
+  def initialize(params = {}); end
+
+  # Define `get` for fetching keys from the remote datastore
+  def get(key, opts = {}); end
+
+  # Define `set` for storing keys in the remote datastore
+  def set(key, value); end
+end
+```
+
+Specify the class when configuring the custom provider.
+
+```ruby
+CloudConfig.configure do
+  provider :custom_provider, provider_class: CustomProvider do
+  end
+end
+```
+
+### Custom Cache
+
+Defining a custom cache client will require an interface of 3 methods.
+
+```ruby
+class CustomCache
+  # Define `key?` for checking whether the key exists in the cache client.
+  # This check allows nil values to be cached.
+  def key?(key); end
+
+  # Define `get` for fetching keys from the cache
+  def get(key); end
+
+  # Define `set` for storing keys in the cache
+  def set(key, value); end
+end
+```
+
+Configure CloudConfig to use the cache in the usual way
+
+```ruby
+CloudConfig.configure do
+  cache_client CustomCache.new
+end
+```
+
+## Connection Pooling
+
+CloudConfig does not do any connection pooling. It is the responsibility of the application to handle connection pooling. For example, Rails handles its own connection pools, so CloudConfig will not attempt to interfere with the pool.
+
+## Examples
+
+There are some example configurations in the [examples](examples) folder. In the terminal, execute
+
+    $ examples/001_parameter_store.rb
+
+These examples make use of AWS infrastructure, so make sure the AWS enviroment variables have been correctly configured.
+
+    $ export AWS_ACCESS_KEY_ID=
+    $ export AWS_SECRET_ACCESS_KEY=
+    $ export AWS_REGION=
+
 ## Development
 
-It's recommended to use Docker when working with the library. Assuming Docker is installed, run
+It's recommended to use Docker when working with this library. Assuming Docker is installed, run
 
     $ docker-compose run config /bin/bash
 

data/examples/001_parameter_store.rb

@@ -14,12 +14,13 @@ end
 require 'benchmark'
 require 'cloud-config/providers/aws_parameter_store'
 
+# @note: Make sure the `parameter_store_example` key is set in the parameter store
 CloudConfig.configure do
   provider :aws_parameter_store do
     setting :parameter_store_example
   end
 end
 
-puts 'Fetching key parameter_store_example'
+puts 'Fetching key: parameter_store_example'
 value = CloudConfig.get(:parameter_store_example)
-puts "Fetched value #{value}"
+puts "Fetched value: #{value}"

data/examples/002_parameter_store_with_cache.rb

@@ -15,6 +15,7 @@ require 'benchmark'
 require 'cloud-config/cache/in_memory'
 require 'cloud-config/providers/aws_parameter_store'
 
+# @note: Make sure the `parameter_store_example` key is set in the parameter store
 CloudConfig.configure do
   cache_client CloudConfig::Cache::InMemory.new
 
@@ -25,16 +26,20 @@ end
 
 value = nil
 
-puts 'Fetching key parameter_store_example'
+puts 'Fetching key; parameter_store_example'
 time = Benchmark.measure { value = CloudConfig.get(:parameter_store_example) }
-puts "Fetched value #{value} in #{time.real.round(5)} seconds"
+puts "Fetched value: #{value} in #{time.real.round(5)} seconds"
+
+puts
 
-puts 'Fetching key parameter_store_example (cached)'
+puts 'Fetching key: parameter_store_example (cached)'
 time = Benchmark.measure { value = CloudConfig.get(:parameter_store_example) }
-puts "Fetched value #{value} in #{time.real.round(5)} seconds"
+puts "Fetched value: #{value} in #{time.real.round(5)} seconds"
 
 sleep 5
 
+puts
+
 puts 'Fetching key parameter_store_example'
 time = Benchmark.measure { value = CloudConfig.get(:parameter_store_example) }
 puts "Fetched value #{value} in #{time.real.round(5)} seconds"

data/examples/003_parameter_store_with_preload.rb

@@ -15,6 +15,7 @@ require 'benchmark'
 require 'cloud-config/cache/in_memory'
 require 'cloud-config/providers/aws_parameter_store'
 
+# @note: Make sure the `parameter_store_example` key is set in the parameter store
 CloudConfig.configure do
   cache_client CloudConfig::Cache::InMemory.new
 
@@ -27,12 +28,14 @@ CloudConfig.preload
 
 value = nil
 
-puts 'Fetching key parameter_store_example (cached)'
+puts 'Fetching key: parameter_store_example (cached)'
 time = Benchmark.measure { value = CloudConfig.get(:parameter_store_example) }
-puts "Fetched value #{value} in #{time.real.round(5)} seconds"
+puts "Fetched value: #{value} in #{time.real.round(5)} seconds"
 
 sleep 5
 
-puts 'Fetching key parameter_store_example'
+puts
+
+puts 'Fetching key: parameter_store_example'
 time = Benchmark.measure { value = CloudConfig.get(:parameter_store_example) }
-puts "Fetched value #{value} in #{time.real.round(5)} seconds"
+puts "Fetched value: #{value} in #{time.real.round(5)} seconds"

data/examples/004_handling_configuration_changes.rb

@@ -0,0 +1,47 @@
+#! /usr/bin/env ruby
+
+require 'bundler/inline'
+
+gemfile do
+  source 'https://rubygems.org'
+
+  gem 'faraday'
+
+  gem 'cloud-config', path: '..'
+end
+
+require 'faraday'
+require 'cloud-config/cache/in_memory'
+require 'cloud-config/providers/in_memory'
+
+CloudConfig.configure do
+  cache_client CloudConfig::Cache::InMemory.new
+
+  provider :in_memory, preload: true do
+    setting :url, cache: 60
+  end
+end
+
+CloudConfig.set(:url, 'https://httpstat.us/503')
+
+url = CloudConfig.get(:url)
+
+# Mimick a configuration change
+CloudConfig.providers_by_key[:url].provider.set(:url, 'https://httpstat.us/200')
+
+req = Faraday.new do |f|
+  f.response :raise_error
+  f.response :logger
+end
+
+begin
+  req.get(url)
+rescue Faraday::ServerError
+  url = CloudConfig.get(:url, reset_cache: true)
+
+  sleep 1
+
+  puts
+
+  retry
+end

data/examples/005_parameter_store_encrypted_key.rb

@@ -0,0 +1,40 @@
+#! /usr/bin/env ruby
+
+require 'bundler/inline'
+
+gemfile do
+  source 'https://rubygems.org'
+
+  gem 'nokogiri'
+  gem 'aws-sdk-ssm'
+
+  gem 'cloud-config', path: '..'
+end
+
+require 'benchmark'
+require 'cloud-config/providers/aws_parameter_store'
+
+# @note: Make sure the `encrypted_key` key is set in the parameter store
+CloudConfig.configure do
+  provider :aws_parameter_store do
+    setting :encrypted_key
+  end
+end
+
+puts 'Fetching key: encrypted_key'
+value = CloudConfig.get(:encrypted_key)
+puts "Fetched value: #{value}"
+
+puts
+
+CloudConfig.reset!
+
+CloudConfig.configure do
+  provider :aws_parameter_store do
+    setting :encrypted_key, with_decryption: true
+  end
+end
+
+puts 'Fetching key: encrypted_key (decrypt)'
+value = CloudConfig.get(:encrypted_key)
+puts "Fetched value: #{value}"

data/lib/cloud-config/cache/in_memory.rb

@@ -55,7 +55,7 @@ module CloudConfig
       # @param value [Object] Value of the key
       # @option options [Hash] :expire_in Time in seconds until key expires
       def set(key, value, options = {})
-        expire_in = options.fetch(:expire_in) { DEFAULT_EXPIRE }
+        expire_in = options[:expire_in] || DEFAULT_EXPIRE
 
         expire_at = Time.now + expire_in
 

data/lib/cloud-config/cache.rb

@@ -30,7 +30,7 @@ module CloudConfig
       #
       # @return [Object] Value of the key
       def with_cache(key, options = {})
-        return cache.get(key) if cache&.key?(key)
+        return cache.get(key) if !options[:reset_cache] && cache&.key?(key)
 
         value = yield
 

data/lib/cloud-config/providers/aws_parameter_store.rb

@@ -13,18 +13,31 @@ module CloudConfig
       #   @return [Aws::SSM::Client]
       attr_reader :client
 
+      # @!attribute [r] Whether parameters need to be decrypted
+      #   @return [Boolean]
+      attr_reader :with_decryption
+
       # Create a new instance of {AwsParameterStore}.
-      def initialize(_params = {})
+      #
+      # @param [Hash] opts Parameter store options
+      # @option opts [Boolean] :with_decryption Whether all keys need to be decrypted
+      def initialize(opts = {})
         @client = Aws::SSM::Client.new
+
+        @with_decryption = opts.fetch(:with_decryption, false)
       end
 
       # Fetch the value of the key
       #
       # @param key [String,Symbol] Key to fetch
+      # @param [Hash] opts for fetching the key
+      # @option opts [Boolean] :with_decryption Whether the key needs decrypting
       #
       # @return [String] Value of the key
-      def get(key)
-        client.get_parameter(name: key).parameter.value
+      def get(key, opts = {})
+        decrypt = opts.fetch(:with_decryption) { with_decryption }
+
+        client.get_parameter(name: key, with_decryption: decrypt).parameter.value
       end
 
       # Set the value of the key
@@ -34,10 +47,6 @@ module CloudConfig
       def set(key, value)
         client.put_parameter(name: key, value:)
       end
-
-      # def client
-      #   @client ||= Aws::SSM::Client.new
-      # end
     end
   end
 end

data/lib/cloud-config/providers/in_memory.rb

@@ -14,7 +14,7 @@ module CloudConfig
       attr_reader :settings
 
       # Create an instance of {InMemory}
-      def initialize(_params = {})
+      def initialize(_opts = {})
         @settings = {}
       end
 
@@ -23,7 +23,7 @@ module CloudConfig
       # @param key [String,Symbol] Key to fetch
       #
       # @return [Object] Value of the key
-      def get(key)
+      def get(key, _opts = {})
         settings[key]
       end
 

data/lib/cloud-config/providers/yaml_file.rb

@@ -15,9 +15,10 @@ module CloudConfig
 
       # Create an instance of {YamlFile}
       #
-      # @option params [Hash] :filename Name of the YAML file
-      def initialize(params = {})
-        @contents = YAML.load_file(params[:filename]) || {}
+      # @param [Hash] opts Yaml file options
+      # @option opts [String] :filename Name of the YAML file
+      def initialize(opts = {})
+        @contents = YAML.load_file(opts[:filename]) || {}
       end
 
       # Fetch the value of the key
@@ -25,7 +26,7 @@ module CloudConfig
       # @param key [String,Symbol] Key to fetch
       #
       # @return [Object] Value of the key
-      def get(key)
+      def get(key, _opts = {})
         contents[key]
       end
 

data/lib/cloud-config/providers.rb

@@ -14,6 +14,10 @@ module CloudConfig
       #   @return [Hash<Symbol,ProviderConfig>]
       attr_reader :providers
 
+      # @!attribute [r] Fetch provider configuration by setting key
+      #   @return [Hash<Symbol,ProviderConfig>]
+      attr_reader :providers_by_key
+
       # Add a provider to the list of provider configurations.
       #
       # @param provider_name [Symbol] Name of the provider
@@ -25,6 +29,18 @@ module CloudConfig
         provider_config.instance_eval(&blk) if blk
         @providers ||= {}
         @providers[provider_name] = provider_config
+
+        update_provider_keys(provider_config)
+      end
+
+      # Update {#providers_by_key} with settings from the provider config.
+      #
+      # @param provider_config [ProviderConfig] Provider config containing the keys
+      def update_provider_keys(provider_config)
+        @providers_by_key ||= {}
+        provider_config.settings.each_key do |key|
+          @providers_by_key[key] = provider_config
+        end
       end
     end
   end

data/lib/cloud-config/version.rb

@@ -2,5 +2,5 @@
 
 module CloudConfig
   # Version of CloudConfig
-  VERSION = '0.1.1'
+  VERSION = '0.2.1'
 end

data/lib/cloud-config.rb

@@ -35,14 +35,15 @@ module CloudConfig
   # Fetch the value of a key using the appropriate provider.
   #
   # @param key [String,Symbol] Key to lookup
+  # @param reset_cache [Boolean] Whether the cache for the key should be reset
   #
   # @return [Object] Value of the key
-  def get(key)
-    provider_config = providers.values.find { |provider| provider.settings.key?(key) }
+  def get(key, reset_cache: false)
+    provider_config = providers_by_key[key]
 
     raise MissingKey, 'Key not found' if provider_config.nil?
 
-    load_key(provider_config, key)
+    load_key(provider_config, key, reset_cache:)
   end
 
   # Set the value of a key with the configured provider.
@@ -50,7 +51,7 @@ module CloudConfig
   # @param key [String,Symobl] Key to update
   # @param value [Object] Value of key
   def set(key, value)
-    provider_config = providers.values.find { |provider| provider.settings.key?(key) }
+    provider_config = providers_by_key[key]
 
     raise MissingKey, 'Key not found' if provider_config.nil?
 
@@ -84,17 +85,21 @@ module CloudConfig
   #
   # @param provider_config [CloudConfig::ProviderConfig] provider configuration
   # @param key [String,Symbol] Key to fetch
+  # @param reset_cache [Boolean] Whether the cache for the key should be reset
   #
   # @return [Object] Value of the key
-  def load_key(provider_config, key)
-    with_cache(key, expire_in: provider_config.settings[key][:cache]) do
-      provider_config.provider.get(key)
+  def load_key(provider_config, key, reset_cache: false)
+    with_cache(key, reset_cache:, expire_in: provider_config.settings[key][:cache]) do
+      provider_config.provider.get(key, provider_config.settings[key])
     end
   end
 
   # Reset the {CloudConfig} configuration
   def reset!
     @cache = nil
-    @provider = nil
+    @providers = {}
+    @providers_by_key = {}
   end
 end
+
+CloudConfig.reset!

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cloud-config
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.1
 platform: ruby
 authors:
 - hypernova2002
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2023-05-17 00:00:00.000000000 Z
+date: 2023-05-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: parallel
@@ -230,6 +230,8 @@ files:
 - examples/001_parameter_store.rb
 - examples/002_parameter_store_with_cache.rb
 - examples/003_parameter_store_with_preload.rb
+- examples/004_handling_configuration_changes.rb
+- examples/005_parameter_store_encrypted_key.rb
 - lib/cloud-config.rb
 - lib/cloud-config/cache.rb
 - lib/cloud-config/cache/in_memory.rb