process_settings 0.4.0.1 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 191a509df72fd317ad7475301c1c1f9ed2b9234d
-  data.tar.gz: 32913ebed29db8091d71c1e669d75964f0b2170f
+  metadata.gz: 587893ae016c645a43a870af29b50f4da16eb916
+  data.tar.gz: 9b3b4ad7e24c9d52a5dea8d459df143c2c671aac
 SHA512:
-  metadata.gz: b09368b7c8d4d091e955d492be4231cc909719e544d170484e79f1ee27af606c42e927e87f465e7cb7587b02ecbe7750a0ff2bee4f2d91a75dd4dfd8cfc6f327
-  data.tar.gz: 7aa5af5281aec6c0c9ec14f9c47c9bdf5e276ba7d6ceeb348dc2e62eebddb5f0553eeb13bcc6271c0a3ef7c957f6faaa6fbee27515ce007eed79c0ec5cf306bc
+  metadata.gz: 969389525e75f8b8e7884d2300c2726a6eb657b291110aa6347b5f4b07e0716e5587e2e3c0651ce6720f296849a77809eab0e6478b9e23157ea30f622d4ddacc
+  data.tar.gz: 80294ade9ce5babf95249641fbdddd815abaddf2cb3e70516377376ffc2dd0e839a1f731a1f6ae8db13197e2385ec72f4e95bd9fa29625a69d130f61bcd98b85

data/README.md

@@ -1,7 +1,9 @@
-# ProcessSettings [![Build Status](https://travis-ci.org/Invoca/process_settings.svg?branch=master)](https://travis-ci.org/Invoca/process_settings) [![Coverage Status](https://coveralls.io/repos/github/Invoca/process_settings/badge.svg?branch=master)](https://coveralls.io/github/Invoca/process_settings?branch=master) [![Gem Version](https://badge.fury.io/rb/process_settings.svg)](https://badge.fury.io/rb/process_settings)
-This gem provides dynamic settings for Linux processes. These settings are stored in JSON.
-They including a targeting notation so that each settings group can be targeted based on matching context values.
-The context can be either static to the process (for example, service_name or data_center) or dynamic (for example, the domain of the current web request).
+# ProcessSettings
+This gem provides dynamic settings for Ruby processes. The settings are stored in YAML.
+Settings are managed in a git repo, in separate YAML files for each concern (for example, each micro-service). Each YAML file can be targeted based on matching context values (for example, `service_name`).
+
+
+The context can be either static to the process (for example, `service_name` or `data_center`) or dynamic (for example, the current web request `domain`).
 
 ## Installation
 To install this gem directly on your machine from rubygems, run the following:
@@ -11,27 +13,151 @@ gem install process_settings
 
 To install this gem in your bundler project, add the following to your Gemfile:
 ```ruby
-gem 'process_settings', '~> 0.3'
-```
-
-To use an unreleased version, add it to your Gemfile for Bundler:
-```ruby
-gem 'process_settings', git: 'git@github.com:Invoca/process_settings'
+gem 'process_settings', '~> 0.4'
 ```
 
 ## Usage
-### Initialization
-To use the contextual logger, all you need to do is initailize the object with your existing logger
+The `ProcessSettings::Monitor` and related classes can be freely created and used at any time.
+But typical usage is through the `ProcessSettings::Monitor.instance`.
+That should be configured at process startup before use.
+### Configuration
+Before using `ProcessSettings::Monitor.instance`, you must first configure the path to the combined process settings file on disk,
+and provide a logger.
 ```ruby
 require 'process_settings'
+
+ProcessSettings::Monitor.file_path = "/etc/process_settings/combined_process_settings.yml"
+ProcessSettings::Monitor.logger = logger
+```
+### Monitor Initialization
+The `ProcessSettings::Monitor` is a hybrid singleton. The class attribute `instance` returns
+the current instance. If not already set, this is lazy-created based on the above configuration.
+
+The monitor should be initialized with static (unchanging) context for your process:
+```
+ProcessSettings::Monitor.static_context = {
+  "service_name" => "frontend",
+  "data_center" => "AWS-US-EAST-1"
+}
+```
+The `static_context` is important because it is used to pre-filter settings for the process.
+For example, a setting that is targeted to `service_name: frontend` will match the above static context and
+be simplified to `true`. In other processes with a different `service_name`, such a targeted setting will be
+simplified to `false` and removed from memory.
+
+Note that the `static_context` as well as `dynamic_context` must use strings, not symbols, for both keys and values.
+
+### Reading Settings
+For the following section, consider this `combined_process_settings.yml` file:
+```
+---
+- filename: frontend.yml
+  settings:
+    frontend:
+      log_level: info
+- filename: frontend-microsite.yml
+  target:
+    domain: microsite.example.com
+  settings:
+    frontend:
+      log_level: debug
+- meta:
+    version: 27
+    END: true
+```
+
+To read a setting, application code should call the `[]` method on the `ProcessSettings` class. For example:
+```
+log_level = ProcessSettings['frontend', 'log_level']
+=> "info"
+```
+#### ProcessSettings[] interface
+The `ProcessSettings[]` method delegates to `ProcessSettings::Monitor#[]` on the `instance`.
+
+`[]` interface:
+
+```
+[](*path, dynamic_context: {}, required: true)
+```
+
+|argument|description|
+|--------|-------------|
+|_path_    |A series of 1 or more comma-separated strings forming a path to navigate the `settings` hash, starting at the top.|
+|`dynamic_context:` |An optional hash of dynamic settings, used to target the settings. This will automatically be deep-merged with the static context. It may not contradict the static context. |
+|`required:` |A boolean indicating if the setting is required to be present. If a setting is missing, then if `required` is truthy, a `ProcesssSettings::SettingsPathNotFound` exception will be raised. Otherwise, `nil` will be returned. Default: `true`.
+
+Example with `dynamic_context`:
+```
+log_level = ProcessSettings['frontend', 'log_level',
+                            dynamic_context: { "domain" => "microsite.example.com" }
+                           ]
+=> "debug"
 ```
 
-TODO: Fill in here how to use the Monitor's instance method to get current settings, how to register for on_change callbacks, etc.
+Example with `required: true` (default) that was not found:
+```
+http_version = ProcessSettings['frontend', 'http_version']
+
+exception raised!
+
+ProcessSettings::SettingsPathNotFound: No settings found for path ["frontend", "http_version"]
+```
+
+Here is the same example with `required: false`, applying a default value of `2`:
+```
+http_version = ProcessSettings['frontend', 'http_version', required: false] || 2
+```
 
 ### Dynamic Settings
 
-In order to load changes dynamically, `ProcessSettings` relies on INotify module of the Linux kernel. On kernels that do not have this module (MacOS for example), you will see a warning on STDERR that changes will not be loaded while the process runs.
+The `ProcessSettings::Monitor` loads settings changes dynamically whenever the file changes,
+by using the [listen](https://github.com/guard/listen) gem which in turn uses the `INotify` module of the Linux kernel, or `FSEvents` on MacOS. There is no need to restart the process or send it a signal to tell it to reload changes.
+
+There are two ways to get access the latest settings from inside the process:
+
+#### Read Latest Setting Through `ProcessSettings[]`
+
+The simplest approach--as shown above--is to read the latest settings at any time through `ProcessSettings[]` (which delegates to `ProcessSettings::Monitor.instance`):
+```
+http_version = ProcessSettings['frontend', 'http_version']
+```
+
+#### Register an `on_change` Callback
+Alternatively, if you need to execute some code when there is a change, register a callback with `ProcessSettings::Monitor#on_change`:
+```
+ProcessSettings::Monitor.instance.on_change do
+  logger.level = ProcessSettings['frontend', 'log_level']
+end
+```
+Note that all callbacks run sequentially on the shared change monitoring thread, so please be considerate!
+
+There is no provision for unregistering callbacks. Instead, replace the `instance` of the monitor with a new one.
+
+## Targeting
+Each settings YAML file has an optional `target` key at the top level, next to `settings`.
+
+If there is no `target` key, the target defaults to `true`, meaning all processes are targeted for these settings. (However, the settings may be overridden by other YAML files. See "Precedence" below.)
+
+### Hash Key-Values Are AND'd
+To `target` on context values, provide a hash of key-value pairs. All keys must match for the target to be met. For example, consider this target hash:
+```
+target:
+  service_name: frontend
+  data_center: AWS-US-EAST-1
+```
+This will be applied in any process that has `service_name == "frontend"` AND is running in `data_center == "AWS-US-EAST-1"`.
+
+### Multiple Values Are OR'd
+Values may be set to an array, in which case the key matches if _any_ of the values matches. For example, consider this target hash:
+```
+target:
+  service_name: [frontend, auth]
+  data_center: AWS-US-EAST-1
+```
+This will be applied in any process that has (`service_name == "frontend"` OR `service_name == "auth"`) AND `data_center == "AWS-US-EAST-1"`.
 
+### Precedence
+The settings YAML files are always combined in alphabetical order by file path. Later settings take precedence over the earlier ones.
 
 ## Contributions
 

data/bin/combine_process_settings

@@ -32,15 +32,16 @@ def parse_options(argv)
   options.verbose = false
   option_parser = OptionParser.new(argv) do |opt|
     opt.on('-v', '--verbose', 'Verbose mode.') { options.verbose = true }
+    opt.on('-n', '--version=VERSION', 'Set version number.') { |value| options.version = value.to_i unless value.empty? }
     opt.on('-r', '--root_folder=ROOT') { |o| options.root_folder = o }
     opt.on('-o', '--output=FILENAME', 'Output file.') { |o| options.output_filename = o }
     opt.on('-i', '--initial=FILENAME', 'Initial settings file for version inference.') { |o| options.initial_filename = o }
   end
 
-  if option_parser.parse! && options.root_folder && options.output_filename && (ENV['BUILD_NUMBER'] || options.initial_filename)
+  if option_parser.parse! && options.root_folder && options.output_filename && (options.version || options.initial_filename)
     options
   else
-    warn "usage: #{PROGRAM_NAME} -r staging|production -o combined_process_settings.yml [-i initial_combined_process_settings.yml] (required if BUILD_NUMBER not set)"
+    warn "usage: #{PROGRAM_NAME} -r staging|production -o combined_process_settings.yml [--version=VERSION] [-i initial_combined_process_settings.yml] (-i required if --version= not set)"
     option_parser.summarize(STDERR)
     exit(1)
   end
@@ -82,7 +83,7 @@ options = parse_options(ARGV.dup)
 
 combined_settings = read_and_combine_settings(Pathname.new(options.root_folder) + SETTINGS_FOLDER)
 
-version_number = ENV['BUILD_NUMBER']&.to_i || default_version_number(options.initial_filename)
+version_number = options.version || default_version_number(options.initial_filename)
 combined_settings << end_marker(version_number)
 
 yaml = combined_settings.to_yaml

data/bin/diff_process_settings

@@ -34,6 +34,6 @@ system("rm -f tmp/combined_process_settings-A.yml tmp/combined_process_settings-
 system("sed '/^- meta:$/,$d' #{input_files[0]} > tmp/combined_process_settings-A.yml")
 system("sed '/^- meta:$/,$d' #{input_files[1]} > tmp/combined_process_settings-B.yml")
 
-system("diff tmp/combined_process_settings-A.yml tmp/combined_process_settings-B.yml")
+system("diff -c tmp/combined_process_settings-A.yml tmp/combined_process_settings-B.yml | sed '1,3d'")
 
 system("rm -f tmp/combined_process_settings-A.yml tmp/combined_process_settings-B.yml")

data/bin/process_settings_for_services

@@ -107,7 +107,7 @@ if correlation_scorecard.any? { |_correllation_key, scorecard| scorecard[true].a
     system("rm -f tmp/old.yml tmp/new.yml")
     File.write("tmp/old.yml", ProcessSettings.plain_hash(best_correlation.last['__changes__']['old']).to_yaml)
     File.write("tmp/new.yml", ProcessSettings.plain_hash(best_correlation.last['__changes__']['new']).to_yaml)
-    STDOUT << `diff tmp/old.yml tmp/new.yml`
+    STDOUT << `diff -c tmp/old.yml tmp/new.yml | sed '1,3d'`
   else
     puts "No best correlation found?"
   end

data/lib/process_settings.rb

@@ -7,8 +7,8 @@ require 'process_settings/monitor'
 
 module ProcessSettings
   class << self
-    def [](value, dynamic_context = {})
-      Monitor.instance.targeted_value(value, dynamic_context)
+    def [](*path, dynamic_context: {}, required: true)
+      Monitor.instance.targeted_value(*path, dynamic_context: dynamic_context, required: required)
     end
   end
 end

data/lib/process_settings/hash_path.rb

@@ -2,13 +2,24 @@
 
 module ProcessSettings
   # Module for mixing into `Hash` or other class with `[]` that you want to be able to index
-  # with a hash path like:  hash['app.service_name' => 'frontend']
+  # with a hash path like:  hash.mine('honeypot', 'answer_odds')
   module HashPath
-    def [](key)
-      if key.is_a?(Hash)
-        HashPath.hash_at_path(self, key)
-      else
-        super
+    # returns the value found at the given path
+    # if the path is not found:
+    #   if a block is given, it is called and its value returned (may also raise an exception from the block)
+    #   else, the not_found_value: is returned
+    def mine(*path_array, not_found_value: nil)
+      path_array.is_a?(Enumerable) && path_array.size > 0 or raise ArgumentError, "path must be 1 or more keys; got #{path_array.inspect}"
+      path_array.reduce(self) do |hash, key|
+        if hash.has_key?(key)
+          hash[key]
+        else
+          if block_given?
+            break yield
+          else
+            break not_found_value
+          end
+        end
       end
     end
 
@@ -31,24 +42,6 @@ module ProcessSettings
           hash[path]
         end
       end
-
-      def set_hash_at_path(hash, path)
-        path.is_a?(Hash) or raise ArgumentError, "got unexpected non-hash value (#{hash[path]}"
-        case path.size
-        when 0
-          hash
-        when 1
-          path_key, path_value = path.first
-          if path_value.is_a?(Hash)
-            set_hash_at_path(remaining_hash, remaining_path)
-          else
-            hash[path_key] = path_value
-          end
-        else
-          raise ArgumentError, "path may have at most 1 key (got #{path.inspect})"
-        end
-        hash
-      end
     end
   end
 end

data/lib/process_settings/monitor.rb

@@ -7,6 +7,8 @@ require 'listen'
 require 'active_support'
 
 module ProcessSettings
+  class SettingsPathNotFound < StandardError; end
+
   class Monitor
     attr_reader :file_path, :min_polling_seconds, :logger
     attr_reader :static_context, :untargeted_settings, :statically_targeted_settings
@@ -26,8 +28,8 @@ module ProcessSettings
 
       path = File.dirname(@file_path)
 
-      @listener = file_change_notifier.to(path) do |modified, _added, _removed|
-        if modified.include?(@file_path)
+      @listener = file_change_notifier.to(path) do |modified, added, _removed|
+        if modified.include?(@file_path) || added.include?(@file_path)
           @logger.info("ProcessSettings::Monitor file #{@file_path} changed. Reloading.")
           load_untargeted_settings
 
@@ -54,7 +56,10 @@ module ProcessSettings
     end
 
     # Assigns a new static context. Recomputes statically_targeted_settings.
+    # Keys must be strings or integers. No symbols.
     def static_context=(context)
+      self.class.ensure_no_symbols(context)
+
       @static_context = context
 
       load_statically_targeted_settings(force_retarget: true)
@@ -62,20 +67,32 @@ module ProcessSettings
 
     # Returns the process settings value at the given `path` using the given `dynamic_context`.
     # (It is assumed that the static context was already set through static_context=.)
-    # Returns `nil` if nothing set at the given `path`.
-    def targeted_value(path, dynamic_context)
+    # If nothing set at the given `path`:
+    #   if required, raises SettingsPathNotFound
+    #   else returns nil
+    def targeted_value(*path, dynamic_context:, required: true)
       # Merging the static context in is necessary to make sure that the static context isn't shifting
       # this can be rather costly to do every time if the dynamic context is not changing
       # TODO: Warn in the case where dynamic context was attempting to change a static value
       # TODO: Cache the last used dynamic context as a potential optimization to avoid unnecessary deep merges
       full_context = dynamic_context.deep_merge(static_context)
-      statically_targeted_settings.reduce(nil) do |result, target_and_settings|
+      result = statically_targeted_settings.reduce(:not_found) do |latest_result, target_and_settings|
         # find last value from matching targets
         if target_and_settings.target.target_key_matches?(full_context)
-          unless (value = HashPath.hash_at_path(target_and_settings.process_settings, path)).nil?
-            result = value
+          if (value = target_and_settings.process_settings.json_doc.mine(*path, not_found_value: :not_found)) != :not_found
+            latest_result = value
           end
         end
+        latest_result
+      end
+
+      if result == :not_found
+        if required
+          raise SettingsPathNotFound, "no settings found for path #{path.inspect}"
+        else
+          nil
+        end
+      else
         result
       end
     end
@@ -124,6 +141,20 @@ module ProcessSettings
         @logger = new_logger
         Listen.logger = new_logger
       end
+
+      def ensure_no_symbols(value)
+        case value
+        when Symbol
+          raise ArgumentError, "symbol value #{value.inspect} found--should be String"
+        when Hash
+          value.each do |k, v|
+            k.is_a?(Symbol) and raise ArgumentError, "symbol key #{k.inspect} found--should be String"
+            ensure_no_symbols(v)
+          end
+        when Array
+          value.each { |v| ensure_no_symbols(v) }
+        end
+      end
     end
 
     # Calls all registered on_change callbacks. Rescues any exceptions they may raise.

data/lib/process_settings/replace_versioned_file.rb

@@ -5,31 +5,48 @@ require_relative 'targeted_settings'
 module ProcessSettings
   # This class will override a file with a higher version file; it accounts for minor version number use
   module ReplaceVersionedFile
+    class SourceVersionOlderError < StandardError; end
+    class FileDoesNotExistError < StandardError; end
+
     class << self
+      # Contracts
+      #   source_file_path must be present
+      #   destination_file_path must be present
+      #   source_file_path must exist on filesystem
+      #   source file version cannot be older destination version
       def replace_file_on_newer_file_version(source_file_path, destination_file_path)
         source_file_path.to_s      == '' and raise ArgumentError, "source_file_path not present"
         destination_file_path.to_s == '' and raise ArgumentError, "destination_file_path not present"
+        File.exist?(source_file_path) or raise FileDoesNotExistError, "source file '#{source_file_path}' does not exist"
+        validate_source_version_is_not_older(source_file_path, destination_file_path)
 
         if source_version_is_newer?(source_file_path, destination_file_path)
-          FileUtils.mv(source_file_path, destination_file_path)
-        elsif source_file_path != destination_file_path # make sure we're not deleting destination file
-          if File.exist?(source_file_path)
-            FileUtils.remove_file(source_file_path) # clean up, remove left over file
-          end
+          FileUtils.cp(source_file_path, destination_file_path)
         end
       end
 
       private
 
       def source_version_is_newer?(source_file_path, destination_file_path)
-        if File.exist?(source_file_path)
-          if File.exist?(destination_file_path)
-            source_version      = ProcessSettings::TargetedSettings.from_file(source_file_path, only_meta: true).version
-            destination_version = ProcessSettings::TargetedSettings.from_file(destination_file_path, only_meta: true).version
-
-            Gem::Version.new(source_version) > Gem::Version.new(destination_version)
-          else
-            true
+        if File.exist?(destination_file_path)
+          source_version      = ProcessSettings::TargetedSettings.from_file(source_file_path, only_meta: true).version
+          destination_version = ProcessSettings::TargetedSettings.from_file(destination_file_path, only_meta: true).version
+
+          source_version.to_f > destination_version.to_f
+        else
+          true
+        end
+      end
+
+      def validate_source_version_is_not_older(source_file_path, destination_file_path)
+        if File.exist?(destination_file_path)
+          source_version      = ProcessSettings::TargetedSettings.from_file(source_file_path, only_meta: true).version
+          destination_version = ProcessSettings::TargetedSettings.from_file(destination_file_path, only_meta: true).version
+
+          if source_version.to_f < destination_version.to_f
+            raise SourceVersionOlderError,
+                  "source file '#{source_file_path}' is version #{source_version}"\
+                  " and destination file '#{destination_file_path}' is version #{destination_version}"
           end
         end
       end

data/lib/process_settings/settings.rb

@@ -21,9 +21,5 @@ module ProcessSettings
     def eql?(rhs)
       self == rhs
     end
-
-    def [](key)
-      @json_doc[key]
-    end
   end
 end

data/lib/process_settings/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ProcessSettings
-  VERSION = '0.4.0.1'
+  VERSION = '0.4.1'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: process_settings
 version: !ruby/object:Gem::Version
-  version: 0.4.0.1
+  version: 0.4.1
 platform: ruby
 authors:
 - Invoca