barkibu-kb 1.2.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9b46793e55bdd7e53d2e98e3e792972ec67a86048074aeec1bc9ffdde8028965
-  data.tar.gz: 44d5748d6fb445709add87f4bea42c9c19516ff9bb81be8b4112b716c17469c9
+  metadata.gz: 72ba3947460c029c6dee0dc965b1066b03b645d3bb6a891b291c2383b8d581e3
+  data.tar.gz: d0b5ca779b79f1e7017dbafca93376215faac264ec55210ef1af8f981f352ec5
 SHA512:
-  metadata.gz: 38cb22fa193c859a72448aea1cecfbad17671a4deae2dbd5763881c05f3ffa7338ab80b6218de5e041463c1e0575d190bba204fcf6851a6655107b1894eb1ca6
-  data.tar.gz: f66b4353ac4cc0e3432310ecf7d54b212dfe8b1b245ca6f9f95062b49185acc5028ac1b5ab69503f9099de9cfc89f2ff39e3c62df12531e8c4f593a3bef2e034
+  metadata.gz: 0d596248f6f49451a5a4bcb9ea1648b3175b3d8b1d7cbf0665082332841d63c32867e7f408d221f2a47ef7387e1b70836c74af5bfd8e5e37ae9536ea9b19ec42
+  data.tar.gz: e832f256a27bfbe905d6e122b81245370c932dad1801d5772608778851131c9da1e761817e164df8a91e5e770af79804cd2c0cfdc90310148048bd92ef761ea7

data/CHANGELOG.md

@@ -6,7 +6,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [unreleased]
-- See diff: https://github.com/barkibu/kb-ruby/compare/v1.2.0...HEAD
+- See diff: https://github.com/barkibu/kb-ruby/compare/v1.3.0...HEAD
+
+## [1.3.0]
+- Retry transport failures once (`faraday-retry`, already in the Faraday 1.10 bundle, now an explicit dependency). `KB::RetryPolicy` decides by the underlying error, not the Faraday class: failures where the request never left (`Net::OpenTimeout`, `ECONNREFUSED`, `EHOSTUNREACH`, `ENETUNREACH`, `EADDRNOTAVAIL`, `SocketError`) retry for every verb; any other transport failure (read/write timeout, reset, EOF, TLS) retries for GET/HEAD only (and not when the call raised its own `read_timeout:`, so a 30s read isn't doubled); HTTP error responses never retry. New settings `KB.config.request.retries` (default 1, 0 disables) and `retry_interval` (default 0.1s, randomized up to 2x). Worst-case latency is now two attempts' worth of phase budgets.
+- `request.kb_client` payload gains `retries` and `retry_errors` on retried calls; the Datadog subscriber tags them as `kb.retries` / `kb.retry_errors`. The event and span cover all attempts, so a call that succeeded on retry is not an error.
+- `KB::Listable.all` (and so `PetParent.all`, `Pet.all`, `Breed.all`, `Product.all`, `Plan.all`, `Assessment.all`) now wraps `Faraday::ConnectionFailed` in `KB::Error` like every other model call, instead of re-raising it raw. Code rescuing `Faraday::ConnectionFailed` around `.all` must rescue `KB::Error` instead; none of Funnel, Global Admin or connected_health does.
 
 ## [1.2.0]
 - `KB::Client` emits one `request.kb_client` `ActiveSupport::Notifications` event per KB call (`KB::Client::REQUEST_EVENT`), wrapping cache lookup, connect, TLS, write, read and parsing. Payload: `verb`, `path`, `base_url`, `cache_hit` (GET only), `status`, plus ActiveSupport's `exception`/`exception_object` when the call raised. Every public method now goes through one private `perform` seam; no behaviour change (same cache keys, params and error classes).

data/Gemfile.lock

@@ -1,17 +1,18 @@
 PATH
   remote: .
   specs:
-    barkibu-kb (1.2.0)
+    barkibu-kb (1.3.0)
       activemodel (>= 4.0.2)
       activerecord
       activesupport (>= 3.0.0)
       dry-configurable (~> 0.9)
       faraday
       faraday-net_http (~> 1.0)
+      faraday-retry (~> 1.0)
       faraday_middleware
       i18n
-    barkibu-kb-fake (1.2.0)
-      barkibu-kb (= 1.2.0)
+    barkibu-kb-fake (1.3.0)
+      barkibu-kb (= 1.3.0)
       countries
       sinatra
       webmock

data/README.md

@@ -74,14 +74,49 @@ and write budgets stay global:
 KB::Pet.kb_client.request('birthdays', filters: { month: 9, day: 22, size: 1000 }, read_timeout: 30)
 ```
 
+#### Retries
+
+The client retries a failed call once when the failure is in the transport, never
+when KB answered with an HTTP error. Which calls retry depends on whether the
+request can have reached KB (`KB::RetryPolicy`):
+
+| Failure | Retried for |
+|---|---|
+| Never sent: connect/TLS timeout (`Net::OpenTimeout`), connection refused, host/network unreachable, DNS failure | every verb, POST included |
+| Maybe sent: read/write timeout, connection reset, EOF, TLS error mid-stream | GET and HEAD only |
+| HTTP 4xx/5xx | never |
+
+PUT and DELETE are not retried on "maybe sent" failures: `upsert` and
+`PetParent#merge!` are PUTs whose second run is not a no-op on KB's side, and a
+repeated DELETE would turn a success into a 404. A call that raised its own read
+budget (`read_timeout:` on `KB::Client#request`) is not retried on "maybe sent"
+failures either, so a 30s birthdays read can't become 60s; failures that never
+reached KB are still retried.
+
+```ruby
+# config/initializers/kb_ruby.rb
+KB.config.request.retries = 1          # default; 0 disables retries
+KB.config.request.retry_interval = 0.1 # default, seconds; each wait is 1x-2x this
+```
+
+Like the timeouts, these are read when a client builds its connection, i.e. on
+its first call, so set them in an initializer.
+
+Worst case, a call now takes two attempts' worth of phase budgets plus the
+interval, e.g. a GET that read-times-out twice takes about 2 x (1 + 3 + 5)s with
+the default timeouts.
+
 #### Instrumentation
 
 Every KB call emits one `request.kb_client` event through
 `ActiveSupport::Notifications`, wrapping the whole call: cache lookup, TCP
 connect, TLS, write, read and JSON parsing. The payload carries `verb`, `path`,
-`base_url`, `cache_hit` (GET calls only), `status` (when a response arrived) and
-ActiveSupport's `exception` / `exception_object` when the call raised. Subscribe
-to it for logging, metrics or anything else:
+`base_url`, `cache_hit` (GET calls only), `status` (when a response arrived),
+`retries` and `retry_errors` (only when the call was retried: the count, and the
+underlying error class of each failed attempt, e.g. `["Net::OpenTimeout"]`) and
+ActiveSupport's `exception` / `exception_object` when the call raised. The event
+covers the whole call including retries, so `exception` is set only when every
+attempt failed. Subscribe to it for logging, metrics or anything else:
 
 ```ruby
 ActiveSupport::Notifications.subscribe(KB::Client::REQUEST_EVENT) do |event|
@@ -110,8 +145,10 @@ tracer's own Net::HTTP spans nest under it. It inherits the app's service
 knowledge-base service either: it measures the client's whole call, not a KB
 operation. Resources are low-cardinality (`GET /v1/pets/birthdays`,
 `GET /v1/pets/?/contracts`). Tags: `peer.hostname` (the KB host used),
-`kb.method`, `kb.cache_hit` (GET calls only), `http.status_code`, plus the
-standard `error.type`/`error.message` when the call raises.
+`kb.method`, `kb.cache_hit` (GET calls only), `http.status_code`, `kb.retries`
+and `kb.retry_errors` (retried calls only), plus the standard
+`error.type`/`error.message` when the call raises. A span with `kb.retries` and
+no error is a failure the retry absorbed.
 
 Why not rely on the Net::HTTP tracer alone: faraday-net_http opens the socket
 before `Net::HTTP#request`, the method that tracer patches, so a connect timeout

data/barkibu-kb.gemspec

@@ -54,5 +54,6 @@ Gem::Specification.new do |spec|
   spec.add_runtime_dependency 'faraday'
   spec.add_runtime_dependency 'faraday-net_http', '~> 1.0'
   spec.add_runtime_dependency 'faraday_middleware'
+  spec.add_runtime_dependency 'faraday-retry', '~> 1.0'
   spec.add_runtime_dependency 'i18n'
 end

data/lib/barkibu-kb.rb

@@ -21,6 +21,9 @@ module KB
     setting :connect_timeout, default: 1
     setting :write_timeout, default: 3
     setting :read_timeout, default: 5
+    # Retries after a transport failure, per KB::RetryPolicy. 0 disables retries.
+    setting :retries, default: 1
+    setting :retry_interval, default: 0.1
   end
 end
 
@@ -29,6 +32,7 @@ require 'kb/inflections'
 require 'kb/cache'
 require 'kb/client_resolver'
 require 'kb/errors'
+require 'kb/retry_policy'
 require 'kb/client'
 
 require 'kb/concerns'

data/lib/kb/client.rb

@@ -2,6 +2,7 @@ module KB
   class Client
     # Emitted once per KB call, wrapping cache lookup and the HTTP request.
     # Payload: verb, path, base_url, cache_hit (GET only), status (when a response arrived),
+    # retries / retry_errors (only when the call was retried, see KB::RetryPolicy),
     # plus ActiveSupport's exception/exception_object when the call raised.
     REQUEST_EVENT = 'request.kb_client'.freeze
 
@@ -74,7 +75,10 @@ module KB
     end
 
     def http(event, payload, read_timeout)
-      response = connection.public_send(event[:verb], event[:path], payload, &request_options(read_timeout))
+      response = connection.public_send(event[:verb], event[:path], payload) do |req|
+        req.options.read_timeout = read_timeout if read_timeout
+        RetryPolicy.track(req, event, read_timeout)
+      end
       event[:status] = response.status
       response.body
     rescue Faraday::ClientError, Faraday::ServerError => e
@@ -101,6 +105,7 @@ module KB
 
     def connection
       @connection ||= Faraday.new(url: base_url, headers: headers, request: request_timeouts) do |conn|
+        conn.request :retry, RetryPolicy.middleware_options
         conn.response :json
         conn.response :raise_error
         if KB.config.log_level == :debugger
@@ -112,12 +117,6 @@ module KB
       end
     end
 
-    def request_options(read_timeout)
-      return nil if read_timeout.nil?
-
-      ->(req) { req.options.read_timeout = read_timeout }
-    end
-
     def request_timeouts
       {
         open_timeout: KB.config.request.connect_timeout,

data/lib/kb/instrumentation/datadog.rb

@@ -72,9 +72,21 @@ module KB
 
           span.set_tag('kb.cache_hit', payload[:cache_hit].to_s) if payload.key?(:cache_hit)
           span.set_tag('http.status_code', payload[:status].to_s) if payload[:status]
+          tag_retries(span, payload)
           span.set_error(payload[:exception_object]) if payload[:exception_object]
           span.finish
         end
+
+        private
+
+        # Only on retried calls: `kb.retries` (numeric) and the distinct underlying
+        # errors that triggered them, e.g. `Net::OpenTimeout`.
+        def tag_retries(span, payload)
+          return unless payload[:retries]
+
+          span.set_tag('kb.retries', payload[:retries])
+          span.set_tag('kb.retry_errors', payload[:retry_errors].uniq.join(','))
+        end
       end
     end
   end

data/lib/kb/models/concerns/listable.rb

@@ -11,8 +11,6 @@ module KB
         kb_client.all(filters).map do |pet_parent|
           from_api pet_parent
         end
-      rescue Faraday::ConnectionFailed => e
-        raise e
       rescue Faraday::Error => e
         raise KB::Error.from_faraday(e)
       end

data/lib/kb/retry_policy.rb

@@ -0,0 +1,88 @@
+require 'socket'
+require 'net/http'
+require 'faraday/retry'
+
+module KB
+  # Decides which failed KB calls the client retries.
+  #
+  # Two classes of transport failure, told apart by the underlying Ruby error
+  # rather than the Faraday class (adapters disagree on the Faraday class: the
+  # net_http adapter wraps Net::OpenTimeout as ConnectionFailed, the persistent
+  # one as TimeoutError):
+  #
+  # - never sent: TCP connect / TLS handshake did not complete, so KB cannot have
+  #   seen the request. Safe to retry for every verb, POST included.
+  # - maybe sent: anything else the transport raises (read/write timeout, reset,
+  #   EOF, TLS error mid-stream). KB may have processed it, so only GET/HEAD are
+  #   retried. PUT/DELETE are left out on purpose: `upsert` and `merge!` are PUTs
+  #   whose second run is not a no-op on KB's side, and a repeated DELETE would
+  #   turn a success into a 404.
+  #   A call that raised its own read budget (`read_timeout:`, e.g. 30s for
+  #   birthdays) is not retried on these either, so its worst case isn't doubled.
+  #
+  # HTTP responses (4xx/5xx) are never retried: KB answered.
+  module RetryPolicy
+    NOT_SENT_ERRORS = [
+      Net::OpenTimeout,
+      Errno::ECONNREFUSED,
+      Errno::EHOSTUNREACH,
+      Errno::ENETUNREACH,
+      Errno::EADDRNOTAVAIL,
+      SocketError # DNS resolution
+    ].freeze
+    TRANSPORT_ERRORS = [Faraday::ConnectionFailed, Faraday::TimeoutError, Faraday::SSLError].freeze
+    MAYBE_SENT_VERBS = %i[get head].freeze
+
+    module_function
+
+    def retry?(verb, error, own_read_budget: false)
+      return false unless TRANSPORT_ERRORS.any? { |klass| error.is_a?(klass) }
+
+      not_sent?(error) || (MAYBE_SENT_VERBS.include?(verb) && !own_read_budget)
+    end
+
+    def not_sent?(error)
+      cause = root_cause(error)
+      NOT_SENT_ERRORS.any? { |klass| cause.is_a?(klass) }
+    end
+
+    # The Ruby error behind a Faraday error. `wrapped_exception` is Faraday's own
+    # explicit link to it (Ruby's `cause` is only whatever was being rescued at
+    # the raise, usually the same object). Faraday's adapters wrap the Ruby error
+    # one level deep, so one level is enough.
+    def root_cause(error)
+      (error.respond_to?(:wrapped_exception) && error.wrapped_exception) || error.cause || error
+    end
+
+    # Options for faraday-retry's middleware, read from KB.config.request.
+    def middleware_options
+      {
+        max: KB.config.request.retries,
+        interval: KB.config.request.retry_interval,
+        interval_randomness: 1, # 1x-2x the interval, so a burst of callers doesn't retry in lockstep
+        exceptions: TRANSPORT_ERRORS,
+        methods: [], # always ask retry_if
+        retry_if: lambda do |env, error|
+          retry?(env[:method], error, own_read_budget: env[:request].context&.dig(:kb_own_read_budget))
+        end,
+        retry_block: ->(env, _options, _retries_left, error) { record(env, error) }
+      }
+    end
+
+    # Hands the request.kb_client event payload (and whether the call set its own
+    # read budget) to the middleware through the request context, so each retry
+    # is reported on the call's own event.
+    def track(request, event, read_timeout = nil)
+      tracking = { kb_event: event, kb_own_read_budget: !read_timeout.nil? }
+      request.options.context = (request.options.context || {}).merge(tracking)
+    end
+
+    def record(env, error)
+      event = env[:request].context&.dig(:kb_event)
+      return unless event
+
+      event[:retries] = event.fetch(:retries, 0) + 1
+      (event[:retry_errors] ||= []) << root_cause(error).class.name
+    end
+  end
+end

data/lib/kb/version.rb

@@ -1,3 +1,3 @@
 module KB
-  VERSION = '1.2.0'.freeze
+  VERSION = '1.3.0'.freeze
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: barkibu-kb
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.3.0
 platform: ruby
 authors:
 - Léo Figea
@@ -261,6 +261,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: faraday-retry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: i18n
   requirement: !ruby/object:Gem::Requirement
@@ -352,6 +366,7 @@ files:
 - lib/kb/models/referral.rb
 - lib/kb/models/search_result.rb
 - lib/kb/models/symptom.rb
+- lib/kb/retry_policy.rb
 - lib/kb/type/array_of_conditions_type.rb
 - lib/kb/type/array_of_strings_type.rb
 - lib/kb/type/array_of_symptoms_type.rb