lhc 12.1.3 → 13.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c1d43867c7519eb1697f118fb44668142aa46bcdfb13d725be6617c74f4ed47f
-  data.tar.gz: cdfa710bc5c7ae5297eeceec6858738ad414e828237038b92a3dcbaa9c088fed
+  metadata.gz: 98300a9c64d910094f32a96b4428c71aa4d1d8927e1dc9e4b939c1119e28333c
+  data.tar.gz: 7cb93bf8911d06c5fbd015b36b2ccc62c3ff211a1e70a2aa10dfd8fe447be878
 SHA512:
-  metadata.gz: 9e11b8821556d2bb0e40b91cb080379fc82a3a90453fe36dd6127b4d26a571d065637275988ba03b46d96f6217a697022d0a6aa46532ab20a5d46e1ef5dbadfd
-  data.tar.gz: 192cabec15b2d1881a14505343f2e2f4218fad00ffaac4935dd3d1edbdc6132edb6b9ae0834f2d556dfbcacca9221f473f39dba37071ec40949e407bc953813f
+  metadata.gz: 24d10469629a7bb4942bd2a77fabc74910a98179e5998fef6e502fa88e25d583274becd33897f4f509ed97e3e9a3ed2d246145ac9a0908ae3b367d4d0f00ca9d
+  data.tar.gz: c22f9044dbc3a2052c3487eb014f04e86791017da1b3d438d6c9e48e63d30bf4bf5e19663854f3b1a2ba675ad240da84e3c528a0ca25037759a3576109275657

data/Gemfile.activesupport5

@@ -1,4 +1,4 @@
 source 'https://rubygems.org/'
 
 gemspec
-gem 'activesupport', '~> 5.0.0'
+gem 'activesupport', '~> 5.2'

data/Gemfile.activesupport6

@@ -1,4 +1,4 @@
 source 'https://rubygems.org/'
 
 gemspec
-gem 'activesupport', '~> 6.0.0'
+gem 'activesupport', '~> 6.0'

data/README.md

@@ -73,6 +73,10 @@ use it like:
            * [Installation](#installation-1)
            * [Environment](#environment)
            * [What it tracks](#what-it-tracks)
+              * [Before and after request tracking](#before-and-after-request-tracking)
+              * [Response tracking](#response-tracking)
+              * [Timeout tracking](#timeout-tracking)
+              * [Caching tracking](#caching-tracking)
            * [Configure](#configure-1)
         * [Prometheus Interceptor](#prometheus-interceptor)
         * [Retry Interceptor](#retry-interceptor)
@@ -94,6 +98,8 @@ use it like:
 
 
 
+
+
 ## Basic methods
 
 Available are `get`, `post`, `put` & `delete`.
@@ -263,7 +269,7 @@ You can also use URL templates, when [configuring endpoints](#configuring-endpoi
   LHC.configure do |c|
     c.endpoint(:find_feedback, 'http://datastore/v2/feedbacks/{id}')
   end
-  
+
   LHC.get(:find_feedback, params:{ id: 123 }) # GET http://datastore/v2/feedbacks/123
 ```
 
@@ -276,7 +282,7 @@ Working and configuring timeouts is important, to ensure your app stays alive wh
 LHC forwards two timeout options directly to typhoeus:
 
 `timeout` (in seconds) - The maximum time in seconds that you allow the libcurl transfer operation to take. Normally, name lookups can take a considerable time and limiting operations to less than a few seconds risk aborting perfectly normal operations. This option may cause libcurl to use the SIGALRM signal to timeout system calls.
-`connecttimeout` (in seconds) - It should contain the maximum time in seconds that you allow the connection phase to the server to take. This only limits the connection phase, it has no impact once it has connected. Set to zero to switch to the default built-in connection timeout - 300 seconds. 
+`connecttimeout` (in seconds) - It should contain the maximum time in seconds that you allow the connection phase to the server to take. This only limits the connection phase, it has no impact once it has connected. Set to zero to switch to the default built-in connection timeout - 300 seconds.
 
 ```ruby
 LHC.get('http://local.ch', timeout: 5, connecttimeout: 1)
@@ -481,7 +487,7 @@ You can configure global placeholders, that are used when generating urls from u
     c.placeholder(:datastore, 'http://datastore')
     c.endpoint(:feedbacks, '{+datastore}/feedbacks', { params: { has_reviews: true } })
   end
-  
+
   LHC.get(:feedbacks) # http://datastore/v2/feedbacks
 ```
 
@@ -600,7 +606,6 @@ You can configure your own cache (default Rails.cache) and logger (default Rails
 
 ```ruby
   LHC::Caching.cache = ActiveSupport::Cache::MemoryStore.new
-  LHC::Caching.logger = Logger.new(STDOUT)
 ```
 
 Caching is not enabled by default, although you added it to your basic set of interceptors.
@@ -631,6 +636,18 @@ Responses served from cache are marked as served from cache:
   response.from_cache? # true
 ```
 
+You can also use a central http cache to be used by the `LHC::Caching` interceptor.
+
+If you configure a local and a central cache, LHC will perform multi-level-caching.
+LHC will try to retrieve cached information first from the central, in case of a miss from the local cache, while writing back into both.
+
+```ruby
+  LHC::Caching.central = {
+    read: 'redis://$PASSWORD@central-http-cache-replica.namespace:6379/0',
+    write: 'redis://$PASSWORD@central-http-cache-master.namespace:6379/0'
+  }
+```
+
 ##### Options
 
 ```ruby
@@ -643,7 +660,7 @@ Responses served from cache are marked as served from cache:
 
 `race_condition_ttl` - very useful in situations where a cache entry is used very frequently and is under heavy load.
 If a cache expires and due to heavy load several different processes will try to read data natively and then they all will try to write to cache.
-To avoid that case the first process to find an expired cache entry will bump the cache expiration time by the value set in `cache_race_condition_ttl`.
+To avoid that case the first process to find an expired cache entry will bump the cache expiration time by the value set in `race_condition_ttl`.
 
 `use` - Set an explicit cache to be used for this request. If this option is missing `LHC::Caching.cache` is used.
 
@@ -729,14 +746,18 @@ LHC::Monitoring.env = ENV['DEPLOYMENT_TYPE'] || Rails.env
 
 It tracks request attempts with `before_request` and `after_request` (counts).
 
-In case your workers/processes are getting killed due limited time constraints, 
+In case your workers/processes are getting killed due limited time constraints,
 you are able to detect deltas with relying on "before_request", and "after_request" counts:
 
+###### Before and after request tracking
+
 ```ruby
   "lhc.<app_name>.<env>.<host>.<http_method>.before_request", 1
   "lhc.<app_name>.<env>.<host>.<http_method>.after_request", 1
 ```
 
+###### Response tracking
+
 In case of a successful response it reports the response code with a count and the response time with a gauge value.
 
 ```ruby
@@ -747,6 +768,17 @@ In case of a successful response it reports the response code with a count and t
   "lhc.<app_name>.<env>.<host>.<http_method>.time", 43
 ```
 
+In case of a unsuccessful response it reports the response code with a count but no time:
+
+```ruby
+  LHC.get('http://local.ch')
+
+  "lhc.<app_name>.<env>.<host>.<http_method>.count", 1
+  "lhc.<app_name>.<env>.<host>.<http_method>.500", 1
+```
+
+###### Timeout tracking
+
 Timeouts are also reported:
 
 ```ruby
@@ -755,6 +787,30 @@ Timeouts are also reported:
 
 All the dots in the host are getting replaced with underscore, because dot is the default separator in graphite.
 
+###### Caching tracking
+
+When you want to track caching stats please make sure you have enabled the `LHC::Caching` and the `LHC::Monitoring` interceptor.
+
+Make sure that the `LHC::Caching` is listed before `LHC::Monitoring` interceptor when configuring interceptors:
+
+```ruby
+LHC.configure do |c|
+  c.interceptors = [LHC::Caching, LHC::Monitoring]
+end
+```
+
+If a response was served from cache it tracks:
+
+```ruby
+  "lhc.<app_name>.<env>.<host>.<http_method>.cache.hit", 1
+```
+
+If a response was not served from cache it tracks:
+
+```ruby
+  "lhc.<app_name>.<env>.<host>.<http_method>.cache.miss", 1
+```
+
 ##### Configure
 
 It is possible to set the key for Monitoring Interceptor on per request basis:
@@ -780,7 +836,7 @@ Logs basic request/response information to prometheus.
   LHC.configure do |c|
     c.interceptors = [LHC::Prometheus]
   end
-    
+
   LHC::Prometheus.client = Prometheus::Client
   LHC::Prometheus.namespace = 'web_location_app'
 ```
@@ -802,7 +858,7 @@ If you enable the retry interceptor, you can have LHC retry requests for you:
   LHC.configure do |c|
     c.interceptors = [LHC::Retry]
   end
-  
+
   response = LHC.get('http://local.ch', retry: true)
 ```
 
@@ -877,15 +933,15 @@ The throttle interceptor allows you to raise an exception if a predefined quota
   end
 ```
 ```ruby
-options = { 
+options = {
   throttle: {
-    track: true, # enables tracking of current limit/remaining requests of rate-limiting
-    break: '80%', # quota in percent after which errors are raised. Percentage symbol is optional, values will be converted to integer (e.g. '23.5' will become 23)
-    provider: 'local.ch', # name of the provider under which throttling tracking is aggregated,
-    limit: { header: 'Rate-Limit-Limit' }, # either a hard-coded integer, or a hash pointing at the response header containing the limit value
-    remaining: { header: 'Rate-Limit-Remaining' }, # a hash pointing at the response header containing the current amount of remaining requests
-    expires: { header: 'Rate-Limit-Reset' } # a hash pointing at the response header containing the timestamp when the quota will reset
-  } 
+    track: true,
+    break: '80%',
+    provider: 'local.ch',
+    limit: { header: 'Rate-Limit-Limit' },
+    remaining: { header: 'Rate-Limit-Remaining' },
+    expires: { header: 'Rate-Limit-Reset' }
+  }
 }
 
 LHC.get('http://local.ch', options)
@@ -895,6 +951,22 @@ LHC.get('http://local.ch', options)
 # raises LHC::Throttle::OutOfQuota: Reached predefined quota for local.ch
 ```
 
+**Options Description**
+* `track`: enables tracking of current limit/remaining requests of rate-limiting
+* `break`: quota in percent after which errors are raised. Percentage symbol is optional, values will be converted to integer (e.g. '23.5' will become 23)
+* `provider`: name of the provider under which throttling tracking is aggregated,
+* `limit`:
+  * a hard-coded integer
+  * a hash pointing at the response header containing the limit value
+  * a proc that receives the response as argument and returns the limit value
+* `remaining`:
+  * a hash pointing at the response header containing the current amount of remaining requests
+  * a proc that receives the response as argument and returns the current amount of remaining requests
+* `expires`:
+  * a hash pointing at the response header containing the timestamp when the quota will reset
+  * a proc that receives the response as argument and returns the timestamp when the quota will reset
+
+
 #### Zipkin
 
 ** Zipkin 0.33 breaks our current implementation of the Zipkin interceptor **

data/cider-ci.yml

@@ -1,6 +1,5 @@
 jobs:
   include:
-    - cider-ci/jobs/rspec-activesupport-4.yml
     - cider-ci/jobs/rspec-activesupport-5.yml
     - cider-ci/jobs/rspec-activesupport-6.yml
     - cider-ci/jobs/rubocop.yml

data/lhc.gemspec

@@ -21,14 +21,15 @@ Gem::Specification.new do |s|
 
   s.requirements << 'Ruby >= 2.0.0'
 
-  s.add_dependency 'activesupport', '>= 4.2'
+  s.add_dependency 'activesupport', '>= 5.2'
   s.add_dependency 'addressable'
   s.add_dependency 'typhoeus', '>= 0.11'
 
   s.add_development_dependency 'geminabox'
   s.add_development_dependency 'prometheus-client', '~> 0.7.1'
   s.add_development_dependency 'pry'
-  s.add_development_dependency 'rails', '>= 4.2'
+  s.add_development_dependency 'rails', '>= 5.2'
+  s.add_development_dependency 'redis'
   s.add_development_dependency 'rspec-rails', '>= 3.0.0'
   s.add_development_dependency 'rubocop', '~> 0.57.1'
   s.add_development_dependency 'rubocop-rspec', '~> 1.26.0'

data/lib/lhc/error.rb

@@ -64,8 +64,10 @@ class LHC::Error < StandardError
   end
 
   def to_s
-    return response if response.is_a?(String)
+    return response.to_s unless response.is_a?(LHC::Response)
     request = response.request
+    return unless request.is_a?(LHC::Request)
+
     debug = []
     debug << [request.method, request.url].map { |str| self.class.fix_invalid_encoding(str) }.join(' ')
     debug << "Options: #{request.options}"

data/lib/lhc/interceptor.rb

@@ -29,4 +29,8 @@ class LHC::Interceptor
   def self.dup
     self
   end
+
+  def all_interceptor_classes
+    @all_interceptors ||= LHC::Interceptors.new(request).all.map(&:class)
+  end
 end

data/lib/lhc/interceptors/auth.rb

@@ -75,10 +75,6 @@ class LHC::Auth < LHC::Interceptor
     @refresh_client_token_option ||= auth_options[:refresh_client_token] || refresh_client_token
   end
 
-  def all_interceptor_classes
-    @all_interceptors ||= LHC::Interceptors.new(request).all.map(&:class)
-  end
-
   def auth_options
     request.options[:auth] || {}
   end

data/lib/lhc/interceptors/caching.rb

@@ -3,69 +3,104 @@
 class LHC::Caching < LHC::Interceptor
   include ActiveSupport::Configurable
 
-  config_accessor :cache, :logger
+  config_accessor :cache, :central
 
+  # to control cache invalidation across all applications in case of
+  # breaking changes within this inteceptor
+  # that do not lead to cache invalidation otherwise
   CACHE_VERSION = '1'
 
   # Options forwarded to the cache
   FORWARDED_OPTIONS = [:expires_in, :race_condition_ttl]
 
+  class MultilevelCache
+
+    def initialize(central: nil, local: nil)
+      @central = central
+      @local = local
+    end
+
+    def fetch(key)
+      central_response = @central[:read].fetch(key) if @central && @central[:read].present?
+      if central_response
+        puts %Q{[LHC] served from central cache: "#{key}"}
+        return central_response
+      end
+      local_response = @local.fetch(key) if @local
+      if local_response
+        puts %Q{[LHC] served from local cache: "#{key}"}
+        return local_response
+      end
+    end
+
+    def write(key, content, options)
+      @central[:write].write(key, content, options) if @central && @central[:write].present?
+      @local.write(key, content, options) if @local.present?
+    end
+  end
+
   def before_request
     return unless cache?(request)
-    deprecation_warning(request.options)
-    options = options(request.options)
-    key = key(request, options[:key])
-    response_data = cache_for(options).fetch(key)
-    return unless response_data
-    logger&.info "Served from cache: #{key}"
+    return if response_data.blank?
     from_cache(request, response_data)
   end
 
   def after_response
     return unless response.success?
-    request = response.request
     return unless cache?(request)
-    options = options(request.options)
-    cache_for(options).write(
+    return if response_data.present?
+    multilevel_cache.write(
       key(request, options[:key]),
       to_cache(response),
-      cache_options(options)
+      cache_options
     )
   end
 
   private
 
-  # return the cache for the given options
-  def cache_for(options)
+  # from cache
+  def response_data
+    # stop calling multi-level cache if it already returned nil for this interceptor instance
+    return @response_data if defined? @response_data
+    @response_data ||= multilevel_cache.fetch(key(request, options[:key]))
+  end
+
+  # performs read/write (fetch/write) on all configured cache levels (e.g. local & central)
+  def multilevel_cache
+    MultilevelCache.new(
+      central: central_cache,
+      local: local_cache
+    )
+  end
+
+  # returns the local cache either configured for entire LHC
+  # or configured locally for that particular request
+  def local_cache
     options.fetch(:use, cache)
   end
 
+  def central_cache
+    return nil if central.blank? || (central[:read].blank? && central[:write].blank?)
+    {}.tap do |options|
+      options[:read] = ActiveSupport::Cache::RedisCacheStore.new(url: central[:read]) if central[:read].present?
+      options[:write] = ActiveSupport::Cache::RedisCacheStore.new(url: central[:write]) if central[:write].present?
+    end
+  end
+
   # do we even need to bother with this interceptor?
   # based on the options, this method will
   # return false if this interceptor cannot work
   def cache?(request)
     return false unless request.options[:cache]
-    options = options(request.options)
-    cache_for(options) &&
+    (local_cache || central_cache) &&
       cached_method?(request.method, options[:methods])
   end
 
-  # returns the request_options
-  # will map deprecated options to the new format
-  def options(request_options)
-    options = (request_options[:cache] == true) ? {} : request_options[:cache].dup
-    map_deprecated_options!(request_options, options)
+  def options
+    options = (request.options[:cache] == true) ? {} : request.options[:cache].dup
     options
   end
 
-  # maps `cache_key` -> `key`, `cache_expires_in` -> `expires_in` and so on
-  def map_deprecated_options!(request_options, options)
-    deprecated_keys(request_options).each do |deprecated_key|
-      new_key = deprecated_key.to_s.gsub(/^cache_/, '').to_sym
-      options[new_key] = request_options[deprecated_key]
-    end
-  end
-
   # converts json we read from the cache to an LHC::Response object
   def from_cache(request, data)
     raw = Typhoeus::Response.new(data)
@@ -104,24 +139,10 @@ class LHC::Caching < LHC::Interceptor
 
   # extracts the options that should be forwarded to
   # the cache
-  def cache_options(input = {})
-    input.each_with_object({}) do |(key, value), result|
+  def cache_options
+    options.each_with_object({}) do |(key, value), result|
       result[key] = value if key.in? FORWARDED_OPTIONS
       result
     end
   end
-
-  # grabs the deprecated keys from the request options
-  def deprecated_keys(request_options)
-    request_options.keys.select { |k| k =~ /^cache_.*/ }.sort
-  end
-
-  # emits a deprecation warning if necessary
-  def deprecation_warning(request_options)
-    unless deprecated_keys(request_options).empty?
-      ActiveSupport::Deprecation.warn(
-        "Cache options have changed! #{deprecated_keys(request_options).join(', ')} are deprecated and will be removed in future versions."
-      )
-    end
-  end
 end