tonic-rails 0.6.0-x86_64-linux-gnu → 0.7.0-x86_64-linux-gnu

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 69a7ded16d01d0e9fc54dd2a24eecc85b36aa49362c4984dd5f4f234429e24a4
-  data.tar.gz: 267553cdb615846fd3e3c39c0e3417154cce6e7cbdb470b42b4a7229717908d1
+  metadata.gz: 83bded7e4b04dc72602f73fa065ac32bde6f97631fe4450525432ef6f93df506
+  data.tar.gz: 13bb870aa4bfb1487fa9827745aceac8c1a0a211053a9b5adafebe494975307e
 SHA512:
-  metadata.gz: 50ad4688167a303a40c8b64bca81de9301c997c9bc29e2d355cd773bcbec1696f029c357ec343fc96fa1eca9b1e936ba40e8e602a015dc49a804f6428090e425
-  data.tar.gz: d14c5ab406b10f432169da4917c44647cbca451c1e6d1b607b9ee1fab8d1038edfe998f2f7b597edf94404323a0654f9d6e98b8c18d682c747c7cf0b4bf042e1
+  metadata.gz: 539c6220bfa47fb98b506c826112ad5728bde68693c37a3479dd2727baff3b14fd07cb689c4c72419d05228a4c3042ea4ce636b44f7c1cdbad06013a2084f8ba
+  data.tar.gz: 9ac35b6b569b9f4083ea342c5e55fe321dda57b89ec4d797ec41624ae2477e47957f71cd4569f9c42fb592d5b1860037ba0fdf2378a75fbba2cf3ed894e1c3da

data/CHANGELOG.md

@@ -7,6 +7,26 @@ minor versions may contain breaking changes.
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-10-11
+
+### Added
+
+- Client: opt-in retries, per client or per call (`retries: true` or a
+  Hash of `max_attempts`, `codes`, `initial_backoff`, `max_backoff` and
+  `multiplier`), following gRPC's retry design. Only calls the server
+  never started on are retried: a failed connection, or a status sent
+  without response headers. Waits are random with exponential caps, stop
+  before the deadline and follow `grpc-retry-pushback-ms`. Each retry is
+  logged, and `request_rpc.tonic_rails` reports `:attempts`. Closes #39.
+
+### Changed
+
+- The server answers calls it turns away before a handler runs (shutting
+  down, dispatch queue full, malformed requests) with a Trailers-Only
+  response: the status in the headers and no body. Clients that follow
+  gRPC's retry rules can tell those calls never started and retry them
+  safely.
+
 ## [0.6.0] - 2026-10-11
 
 ### Added
@@ -166,7 +186,8 @@ First public release.
 - Precompiled native gems for Linux (x86_64, aarch64; glibc and musl) and
   macOS (x86_64, arm64), Ruby 3.3 to 4.0.
 
-[Unreleased]: https://github.com/CodingAnarchy/tonic-rails/compare/v0.6.0...HEAD
+[Unreleased]: https://github.com/CodingAnarchy/tonic-rails/compare/v0.7.0...HEAD
+[0.7.0]: https://github.com/CodingAnarchy/tonic-rails/compare/v0.6.0...v0.7.0
 [0.6.0]: https://github.com/CodingAnarchy/tonic-rails/compare/v0.5.0...v0.6.0
 [0.5.0]: https://github.com/CodingAnarchy/tonic-rails/compare/v0.4.0...v0.5.0
 [0.4.0]: https://github.com/CodingAnarchy/tonic-rails/compare/v0.3.0...v0.4.0

data/README.md

@@ -507,6 +507,68 @@ If a handler lets the error propagate, its caller receives the same
 status, message, details and trailing metadata. Rescue it and raise your
 own error when those are internal.
 
+### Retries
+
+Retries are off unless you turn them on, for a client or a single call:
+
+```ruby
+ITEMS = Tonic::Rails::Client.new("items.internal:50051", deadline: 2, retries: true)
+
+items.get_item({ id: 1 }, retries: false)                    # not this call
+items.get_item({ id: 1 }, retries: { max_attempts: 5 })      # or differently
+```
+
+`retries: true` tries a call up to 3 times when it fails with
+`UNAVAILABLE`. A Hash changes any of these:
+
+| Field | Default | |
+|---|---|---|
+| `max_attempts` | 3 | attempts in all, from 2 to 5 |
+| `codes` | `[:unavailable]` | status codes worth retrying |
+| `initial_backoff` | 0.1 | seconds; the cap on the first wait |
+| `max_backoff` | 1 | seconds; the cap on any wait |
+| `multiplier` | 2 | how much the cap grows per attempt |
+
+The rules come from gRPC's own retry design, so they avoid the usual
+mistakes of a hand-written retry loop:
+
+- **A call is retried only if the server never started on it.** That
+  means the connection failed (refused, reset, a pod restarting), or the
+  server answered with only a status and no response headers. Servers do
+  that when they turn a call away before any handler runs: a
+  tonic-rails server that's shutting down or whose queue is full, or a
+  proxy with no healthy backend. Once response headers arrive, the call
+  is never repeated, even if it then fails with a code you listed.
+- **Waits are random.** Each one is anywhere from zero up to a cap that
+  starts at `initial_backoff` and grows by `multiplier` per attempt, up to
+  `max_backoff`. Clients that failed together don't retry together.
+- **The deadline covers every attempt.** If the next attempt couldn't
+  start before the deadline, the client stops and raises the last
+  failure.
+- **The server can steer.** A `grpc-retry-pushback-ms` trailer sets the
+  wait, and a negative one stops retries.
+- A server stream is retried only before its first message.
+
+Which calls are safe to retry? With the default `[:unavailable]`, nearly
+all of them. By gRPC convention, `UNAVAILABLE` means the call can be tried
+again, and from a tonic-rails server it's retried only when no handler
+ran. For a tonic-rails server, `:resource_exhausted` is worth adding too:
+a full dispatch queue returns it before any handler runs.
+
+Take more care with codes a handler might raise after doing some work,
+like `:aborted`. A tonic-rails server sends headers before it runs a
+handler, so those calls still won't be retried. Other servers may answer
+with only a status after a handler ran, so list those codes only for
+methods that are safe to run twice.
+
+Each retry is logged as a warning, through the client's `logger:` (by
+default `Tonic::Rails.configuration.logger`, which is the Rails logger in
+a Rails app):
+
+```
+tonic-rails: /demo.v1.ItemService/GetItem failed with UNAVAILABLE: ... (attempt 1 of 3); retrying in 64ms
+```
+
 ### Metadata
 
 `metadata:` on a call, and on the client for every call, works like
@@ -549,6 +611,8 @@ the client is created, not on the first call.
 | `metadata` | `{}` | sent with every call |
 | `max_message_size` | 4 MiB | largest message sent or received |
 | `connect_timeout` | 5 | seconds to wait for a connection |
+| `retries` | `false` | retry calls the server never started on; see [Retries](#retries) |
+| `logger` | `Tonic::Rails.configuration.logger` | where retries are logged |
 
 ### Fork safety
 
@@ -830,7 +894,8 @@ When a call fails, `event.payload[:exception_object]` is the
 unexpected, that error's `cause` is the original exception.
 
 Calls made with `Tonic::Rails::Client` publish `request_rpc.tonic_rails`,
-with the same payload plus `:address`, the server called. For a
+with the same payload plus `:address`, the server called, and
+`:attempts`, how many times it was tried. For a
 server-streaming call, the event covers the whole stream, including the
 time your block spends on each message.
 

data/lib/tonic/rails/client/retry_policy.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+module Tonic
+  module Rails
+    class Client
+      # When and how often a call is retried, following gRPC's retry design
+      # (A6).  Pass one, a Hash of its fields or `true` for the defaults as
+      # `retries:` to Client.new or to a single call.
+      #
+      #   Tonic::Rails::Client.new("items.internal:50051", retries: true)
+      #   Tonic::Rails::Client.new("items.internal:50051", retries: { max_attempts: 4 })
+      #   items.get_item({ id: 1 }, retries: false)
+      #
+      # Only calls the server hasn't started on are retried: the connection
+      # failed, or the server answered with a status and no response
+      # headers.  Each wait is random, up to a cap that grows from
+      # `initial_backoff` by `multiplier` per attempt to `max_backoff`.
+      # Retrying stops when the next attempt couldn't start before the
+      # call's deadline.
+      RetryPolicy = Data.define(:max_attempts, :codes, :initial_backoff, :max_backoff, :multiplier) do
+        # Defaults for the fields left out.
+        def self.defaults
+          { max_attempts: 3, codes: [:unavailable], initial_backoff: 0.1, max_backoff: 1, multiplier: 2 }
+        end
+
+        # @param option [RetryPolicy, Hash, true, false, nil]
+        # @return [RetryPolicy, nil] nil for no retries
+        def self.from(option)
+          case option
+          when nil, false then nil
+          when true then new(**defaults)
+          when Hash then new(**defaults, **option)
+          when self then option
+          else raise ArgumentError, "retries must be true, false, a Hash or a #{name}, got #{option.class}"
+          end
+        end
+
+        # @param max_attempts [Integer] attempts in all, from 2 to 5.  gRPC
+        #   caps it at 5, so a misconfigured client can't multiply an
+        #   outage's traffic by more than that.
+        # @param codes [Array<Symbol, Integer>] status codes worth retrying
+        # @param initial_backoff [Numeric] seconds; the cap on the first wait
+        # @param max_backoff [Numeric] seconds; the cap on any wait
+        # @param multiplier [Numeric] how much the cap grows per attempt
+        def initialize(max_attempts:, codes:, initial_backoff:, max_backoff:, multiplier:)
+          unless max_attempts.is_a?(Integer) && max_attempts.between?(2, 5)
+            raise ArgumentError, "max_attempts must be an Integer from 2 to 5"
+          end
+
+          codes = Array(codes).map { |code| RpcError.normalize_code(code) }.uniq.freeze
+          raise ArgumentError, "codes must name at least one status code other than OK" if codes.empty? || codes.include?(0)
+
+          [[:initial_backoff, initial_backoff], [:max_backoff, max_backoff]].each do |field, value|
+            raise ArgumentError, "#{field} must be a positive number of seconds" unless value.is_a?(Numeric) && value.positive?
+          end
+          raise ArgumentError, "multiplier must be at least 1" unless multiplier.is_a?(Numeric) && multiplier >= 1
+
+          super
+        end
+
+        # @param code [Integer]
+        def retry?(code)
+          codes.include?(code)
+        end
+
+        # Seconds to wait after failed attempt number `attempt` (from 1).
+        def backoff(attempt)
+          rand * [initial_backoff * (multiplier**(attempt - 1)), max_backoff].min
+        end
+      end
+    end
+  end
+end

data/lib/tonic/rails/client.rb

@@ -14,6 +14,9 @@ module Tonic
     #   item = items.get_item({ id: 1 }, deadline: 0.5, metadata: { "x-request-id" => request_id })
     #   items.list_items(category: "books").each { |item| ... }
     #
+    # Retries are opt-in: `retries: true` (or a Hash; see RetryPolicy) on
+    # the client or a call retries calls the server never started on.
+    #
     # Stubs are built from the protobuf descriptors that
     # `tonic_rails:compile_protos` (protoc --ruby_out) generates; no
     # `_services_pb.rb` is needed.  A failed call raises
@@ -29,12 +32,20 @@ module Tonic
 
       DEFAULT_MAX_MESSAGE_SIZE = 4 * 1024 * 1024
 
+      # Trailer a server sends to ask for a retry after so many
+      # milliseconds, or (with a negative value) for none.
+      RETRY_PUSHBACK = "grpc-retry-pushback-ms"
+      private_constant :RETRY_PUSHBACK
+
       # @return [String] "host:port"
       attr_reader :address
 
       # @return [Numeric, nil] default deadline in seconds
       attr_reader :deadline
 
+      # @return [RetryPolicy, nil] how calls are retried, unless they say otherwise
+      attr_reader :retries
+
       # @param address [String] "host:port"
       # @param tls [Boolean] false for plaintext, which must be explicit
       # @param tls_ca_path [String, nil] CA certificates (PEM) to trust instead
@@ -48,9 +59,13 @@ module Tonic
       # @param metadata [Hash] sent with every call (see Tonic::Rails::Metadata)
       # @param max_message_size [Integer] largest message sent or received, in bytes
       # @param connect_timeout [Numeric] seconds to wait for a connection
+      # @param retries [RetryPolicy, Hash, Boolean] retry calls the server
+      #   never started on (see RetryPolicy); off by default
+      # @param logger [Logger, nil] where retries are logged; defaults to
+      #   Tonic::Rails.configuration.logger
       def initialize(address, tls: true, tls_ca_path: nil, tls_cert_path: nil, tls_key_path: nil,
                      tls_server_name: nil, deadline: nil, metadata: {}, max_message_size: DEFAULT_MAX_MESSAGE_SIZE,
-                     connect_timeout: 5)
+                     connect_timeout: 5, retries: false, logger: nil)
         Tonic::Rails.native_extension!
         validate_address!(address)
         validate_tls!(tls, tls_cert_path, tls_key_path)
@@ -62,6 +77,8 @@ module Tonic
 
         @address = address
         @deadline = deadline
+        @retries = RetryPolicy.from(retries)
+        @logger = logger
         @metadata = Metadata.encode(metadata)
         @stubs = {}
         tls_options = [read_pem(tls_ca_path, "CA"), read_pem(tls_cert_path, "certificate"),
@@ -106,14 +123,16 @@ module Tonic
 
       # Make a unary call.  Stubs call this; use them instead.
       # @api private
-      def unary(path, input, output, request, deadline: nil, metadata: {}, return_call: false)
-        message = request_message(path, input, request)
+      def unary(path, input, output, request, deadline: nil, metadata: {}, return_call: false, retries: nil)
+        bytes = input.encode(request_message(path, input, request))
         metadata = call_metadata(metadata)
-        timeout_ms = remaining_ms(deadline_at(deadline))
+        deadline_at = deadline_at(deadline)
+        policy = retry_policy(retries)
 
-        Instrumentation.instrument_request(path, address, metadata) do
-          code, error_message, response, headers, trailers, details =
-            @native.unary(path, input.encode(message), metadata, timeout_ms)
+        Instrumentation.instrument_request(path, address, metadata) do |payload|
+          code, error_message, response, headers, trailers, details = with_retries(path, policy, deadline_at, payload) do
+            @native.unary(path, bytes, metadata, remaining_ms(deadline_at))
+          end
           headers = decode_metadata(headers)
           trailers = decode_metadata(trailers)
           check_status!(code, error_message, trailers, details)
@@ -125,28 +144,25 @@ module Tonic
 
       # Start a server-streaming call.  Stubs call this; use them instead.
       # @api private
-      def server_streaming(path, input, output, request, deadline: nil, metadata: {})
-        message = request_message(path, input, request)
+      def server_streaming(path, input, output, request, deadline: nil, metadata: {}, retries: nil)
+        bytes = input.encode(request_message(path, input, request))
         metadata = call_metadata(metadata)
         # Fixed now, so the stream keeps the deadline of the RPC this
         # thread is handling even if it's read later.
         deadline_at = deadline_at(deadline)
+        policy = retry_policy(retries)
 
         ServerStream.new(path) do |stream, block|
-          timeout_ms = remaining_ms(deadline_at)
-          Instrumentation.instrument_request(path, address, metadata) do
-            native = @native.server_streaming(path, input.encode(message), metadata, timeout_ms)
-            begin
-              while (bytes = native.next)
-                block.call(decode_response(output, bytes, path))
-              end
-              code, error_message, _, headers, trailers, details = native.result
-              trailers = decode_metadata(trailers)
-              stream.finish(decode_metadata(headers), trailers)
-              check_status!(code, error_message, trailers, details)
-            ensure
-              native.close
+          Instrumentation.instrument_request(path, address, metadata) do |payload|
+            # A message arrives only after the response headers, which
+            # commit the call, so an attempt that yielded anything is never
+            # retried.
+            code, error_message, _, headers, trailers, details = with_retries(path, policy, deadline_at, payload) do
+              read_stream(@native.server_streaming(path, bytes, metadata, remaining_ms(deadline_at)), output, path, &block)
             end
+            trailers = decode_metadata(trailers)
+            stream.finish(decode_metadata(headers), trailers)
+            check_status!(code, error_message, trailers, details)
           end
         end
       end
@@ -222,6 +238,75 @@ module Tonic
         (remaining * 1000).ceil
       end
 
+      def retry_policy(retries)
+        retries.nil? ? @retries : RetryPolicy.from(retries)
+      end
+
+      # Yield each message of a native stream, then return its result.
+      def read_stream(native, output, path)
+        while (bytes = native.next)
+          yield decode_response(output, bytes, path)
+        end
+        native.result
+      ensure
+        native.close
+      end
+
+      # Run attempts of a call (the block returns a native result) until one
+      # succeeds or shouldn't be retried; return its result.
+      def with_retries(path, policy, deadline_at, payload)
+        attempt = 1
+        loop do
+          payload[:attempts] = attempt if payload
+          result = yield
+          delay = retry_delay(policy, attempt, result, deadline_at)
+          if delay.nil?
+            log_gave_up(path, attempt, result) if attempt > 1 && !result[0].zero?
+            return result
+          end
+
+          log_retry(path, attempt, policy.max_attempts, result, delay)
+          sleep(delay)
+          attempt += 1
+        end
+      end
+
+      # Seconds to wait before retrying, or nil not to retry.  The server
+      # can ask for a delay, or for no retry, with grpc-retry-pushback-ms.
+      def retry_delay(policy, attempt, result, deadline_at)
+        code, _, _, _, trailers, _, committed = result
+        return if policy.nil? || code.zero? || committed || attempt >= policy.max_attempts || !policy.retry?(code)
+
+        pushback = trailers.assoc(RETRY_PUSHBACK)&.last
+        delay = if pushback.nil? then policy.backoff(attempt)
+        elsif pushback.match?(/\A\d+\z/) then pushback.to_i / 1000.0
+        end
+        return if delay.nil?
+        return if deadline_at && Process.clock_gettime(Process::CLOCK_MONOTONIC) + delay >= deadline_at
+
+        delay
+      end
+
+      def logger
+        @logger || Tonic::Rails.configuration.logger
+      end
+
+      def log_retry(path, attempt, max_attempts, result, delay)
+        logger&.warn(format("tonic-rails: %<path>s failed with %<status>s (attempt %<attempt>d of %<max>d); " \
+                            "retrying in %<ms>dms", path: path, status: status_text(result), attempt: attempt,
+                                                    max: max_attempts, ms: (delay * 1000).round))
+      end
+
+      def log_gave_up(path, attempts, result)
+        logger&.warn("tonic-rails: #{path} failed with #{status_text(result)} after #{attempts} attempts")
+      end
+
+      def status_text(result)
+        code, message = result
+        text = RpcError::CODE_NAMES.fetch(code, code).to_s.upcase
+        message.empty? ? text : "#{text}: #{message}"
+      end
+
       def check_status!(code, message, trailers, details)
         return if code.zero?
 
@@ -282,16 +367,18 @@ module Tonic
             input = method.input_type.msgclass
             output = method.output_type.msgclass
             if method.server_streaming
-              define_method(name) do |request = nil, deadline: nil, metadata: {}, **fields|
+              define_method(name) do |request = nil, deadline: nil, metadata: {}, retries: nil, **fields|
                 request = rpc_request(name, request, fields)
-                @client.server_streaming(path, input, output, request, deadline: deadline, metadata: metadata)
+                @client.server_streaming(path, input, output, request, deadline: deadline, metadata: metadata,
+                                                                       retries: retries)
               end
             else
               # `get_item(id: 1)` works as well as `get_item({ id: 1 })`.
-              define_method(name) do |request = nil, deadline: nil, metadata: {}, return_call: false, **fields|
+              define_method(name) do |request = nil, deadline: nil, metadata: {}, return_call: false, retries: nil,
+                                      **fields|
                 request = rpc_request(name, request, fields)
                 @client.unary(path, input, output, request, deadline: deadline, metadata: metadata,
-                                                            return_call: return_call)
+                                                            return_call: return_call, retries: retries)
               end
             end
           end
@@ -369,3 +456,5 @@ module Tonic
     end
   end
 end
+
+require "tonic/rails/client/retry_policy"

data/lib/tonic/rails/instrumentation.rb

@@ -25,8 +25,8 @@ module Tonic
       EVENT = "process_rpc.tonic_rails"
 
       # Outgoing calls made with Tonic::Rails::Client publish
-      # "request_rpc.tonic_rails", with the same payload plus :address,
-      # the server called.
+      # "request_rpc.tonic_rails", with the same payload plus :address, the
+      # server called, and :attempts, how many times it was tried.
       REQUEST_EVENT = "request_rpc.tonic_rails"
 
       module_function
@@ -41,16 +41,17 @@ module Tonic
         publish(REQUEST_EVENT, path, metadata, address: address, &)
       end
 
+      # Yields the payload, or nil when nothing is listening.
       # @api private
       def publish(event, path, metadata, **extra)
         # Without subscribers, skip building the payload: this runs on
         # every RPC.
-        return yield unless ActiveSupport::Notifications.notifier.listening?(event)
+        return yield(nil) unless ActiveSupport::Notifications.notifier.listening?(event)
 
         service, rpc = path.delete_prefix("/").split("/", 2)
         payload = { path: path, service: service, rpc: rpc, metadata: metadata, code: nil, status: nil, **extra }
         ActiveSupport::Notifications.instrument(event, payload) do
-          result = yield
+          result = yield(payload)
           payload[:code] = 0
           payload[:status] = :ok
           result

data/lib/tonic/rails/version.rb

@@ -2,6 +2,6 @@
 
 module Tonic
   module Rails
-    VERSION = "0.6.0"
+    VERSION = "0.7.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: tonic-rails
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: x86_64-linux-gnu
 authors:
 - Matt Tanous
@@ -68,6 +68,7 @@ files:
 - lib/tonic/rails/4.0/tonic_rails_native.so
 - lib/tonic/rails/base.rb
 - lib/tonic/rails/client.rb
+- lib/tonic/rails/client/retry_policy.rb
 - lib/tonic/rails/cluster.rb
 - lib/tonic/rails/configuration.rb
 - lib/tonic/rails/health.rb