redis-client 0.18.0 → 0.24.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0903a6d92200235e71b687f16de1ec058ca88979a42cc14d2a783ba5d5605061'
-  data.tar.gz: 1e772676e34700120320659108e1be493cd4d2067a286056cb8ac87f1c3b51e7
+  metadata.gz: 7c3cb528ecb15130b80ce97d00a5e0e45c31edfc6147068516e1a52295a94045
+  data.tar.gz: b4517fbd99d0f3997e510b6d0d56c345c1fceb1ad44fcac59cceb28faf4326b5
 SHA512:
-  metadata.gz: 66e2d650fe13befeb44b06514e0523d571dfa749f0f787bf7c95126b16d7e54a6b7cb3aa7ed99375f761a302892c2de50c14ad2c011bd42266f4a441a6c9b24b
-  data.tar.gz: 14b28ce893f07569688acb16a404c2f98bb8cdcd3eaca6d68264a2f1c790cb52c483cd252cc243e8f3f8b4f6e858daf34204d78c7689dcfb2146fc960d45bf3a
+  metadata.gz: dbc7fcacc084fcd9d765337ad48a2a8ee413eecfec48f1a166fbd58510299cf92b3e058ce08620055deadeca86464c2c25fe6cddcccab532ab3d2cd741f9cef0
+  data.tar.gz: '09164c4ea9f7021d58df97690f78e7940821a44eb7e12cc4c25324bd17e0e784f447443accedbe2299e6d8c0a0a6f15391491465c4c32baaa56286501b77b693'

data/CHANGELOG.md

@@ -1,5 +1,62 @@
 # Unreleased
 
+# 0.24.0
+
+- Allow `sentinel_password` to be provided as a `Proc`.
+- Ensure `Config#inspect` and `Config#to_s` do not display stored passwords.
+
+# 0.23.2
+
+- Fix retry logic not to attempt to retry on an open circuit breaker. Fix #227.
+
+# 0.23.1
+
+- Fix a potential crash in `hiredis-client` when using subcriptions (`next_event`). See #221.
+
+# 0.23.0
+
+- Allow `password` to be a callable. Makes it easy to implement short lived password authentication strategies.
+- Fix a thread safety issue in `hiredis-client` when using the `pubsub` client concurrently.
+
+# 0.22.2
+
+- Fix the sentinel client to properly extend timeout for blocking commands.
+- Fix IPv6 support in `RedisClient::Config#server_url`.
+
+# 0.22.1
+
+- Fix `ProtocolError: Unknown sigil type` errors when using SSL connection. See #190.
+
+# 0.22.0
+
+- Made various performance optimizations to the Ruby driver. See #184.
+- Always assume UTF-8 encoding instead of relying on `Encoding.default_external`.
+- Add `exception` flag in `pipelined` allowing failed commands to be returned in the result array when set to `false`. See #187.
+
+# 0.21.1
+
+- Handle unresolved Sentinel master/replica error when displaying server URL in exceptions. See #182.
+
+# 0.21.0
+
+- Include redis server URL in most error messages. See #178.
+- Close Redis Sentinel connection after resolving role. See #176.
+
+# 0.20.0
+
+- Accept `unix://` schemes as well as simple paths in the `url:` config parameter. #170.
+- Make basic usage Ractor compatible.
+
+# 0.19.1
+
+- Fixed a bug in `hiredis-client` that could cause a crash if interrupted by `Timeout.timeout` or other `Thread#raise` based mecanism.
+- Fixed a GC bug that could cause crashes in `hiredis-client`.
+
+# 0.19.0
+
+- Revalidate connection in `RedisClient#connected?`
+- Eagerly fail if `db:` isn't an Integer. #151.
+
 # 0.18.0
 
 - Expose more connection details such as `host`, `db`, etc on `RedisClient`.
@@ -23,6 +80,7 @@
 - Discard sockets rather than explictly close them when a fork is detected. #126.
 - Allow to configure sentinel client via url. #117.
 - Fix sentinel to preverse the auth/password when refreshing the sentinel list. #107.
+- Added `RedisClient#measure_round_trip_delay` method. #113.
 
 # 0.14.1
 

data/README.md

@@ -1,9 +1,10 @@
 # RedisClient
 
-`redis-client` is a simple, low-level, client for Redis 6+.
+`redis-client` is a simple, low-level, client for [Redis](https://redis.io/) 6+, [Valkey](https://valkey.io/) 7+, [KeyDB](https://docs.keydb.dev/),
+and several other databases that implement the same `RESP3` protocol.
 
 Contrary to the `redis` gem, `redis-client` doesn't try to map all Redis commands to Ruby constructs,
-it merely is a thin wrapper on top of the RESP3 protocol.
+it merely is a thin wrapper on top of the `RESP3` protocol.
 
 ## Installation
 
@@ -62,7 +63,9 @@ redis.call("GET", "mykey")
 
 ### Configuration
 
-- `url`: A Redis connection URL, e.g. `redis://example.com:6379/5`, a `rediss://` scheme enable SSL, and the path is interpreted as a database number.
+- `url`: A Redis connection URL, e.g. `redis://example.com:6379/5` - a `rediss://` scheme enables SSL, and the path is interpreted as a database number.
+  To connect to UNIX domain sockets, the `url` can also just be a path, and the database specified as query parameter: `/run/redis/foo.sock?db=5`, or optionally
+  have a `unix://` scheme: `unix:///run/redis/foo.sock?db=5`
   Note that all other configurations take precedence, e.g. `RedisClient.config(url: "redis://localhost:3000", port: 6380)` will connect on port `6380`.
 - `host`: The server hostname or IP address. Defaults to `"localhost"`.
 - `port`: The server port. Defaults to `6379`.
@@ -75,7 +78,7 @@ redis.call("GET", "mykey")
 - `db`: The database to select after connecting, defaults to `0`.
 - `id` ID for the client connection, assigns name to current connection by sending `CLIENT SETNAME`.
 - `username` Username to authenticate against server, defaults to `"default"`.
-- `password` Password to authenticate against server.
+- `password` Password to authenticate against server. Can either be a String or a callable that recieve `username` as argument and return a passowrd as a String.
 - `timeout`: The general timeout in seconds, default to `1.0`.
 - `connect_timeout`: The connection timeout, takes precedence over the general timeout when connecting to the server.
 - `read_timeout`: The read timeout, takes precedence over the general timeout when reading responses from the server.
@@ -87,7 +90,7 @@ redis.call("GET", "mykey")
 
 ### Sentinel support
 
-The client is able to perform automatic failover by using [Redis Sentinel](https://redis.io/docs/manual/sentinel/).
+The client is able to perform automatic failover by using [Redis Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/).
 
 To connect using Sentinel, use:
 
@@ -147,7 +150,7 @@ SENTINELS = [{ host: '127.0.0.1', port: 26380 },
 redis_config = RedisClient.sentinel(name: 'mymaster', sentinels: SENTINELS, role: :master, password: 'mysecret')
 ```
 
-So you have to provide Sentinel credential and Redis explictly even they are the same
+So you have to provide Sentinel credential and Redis explicitly even they are the same
 
 ```ruby
 # Use 'mysecret' to authenticate against the mymaster instance and sentinel
@@ -330,6 +333,27 @@ end
 # => ["OK", 1]
 ```
 
+#### Exception management
+
+The `exception` flag in the `#pipelined` method of `RedisClient` is a feature that modifies the pipeline execution
+behavior. When set to `false`, it doesn't raise an exception when a command error occurs. Instead, it allows the
+pipeline to execute all commands, and any failed command will be available in the returned array. (Defaults to `true`)
+
+```ruby
+results = redis.pipelined(exception: false) do |pipeline|
+  pipeline.call("SET", "foo", "bar") # => nil
+  pipeline.call("DOESNOTEXIST", 12) # => nil
+  pipeline.call("INCR", "baz") # => nil
+end
+# results => ["OK", #<RedisClient::CommandError: ERR unknown command 'DOESNOTEXIST', with args beginning with: '12'>, 2]
+
+results.each do |result|
+  if result.is_a?(RedisClient::CommandError)
+    # Do something with the failed result
+  end
+end
+```
+
 ### Transactions
 
 You can use [`MULTI/EXEC` to run a number of commands in an atomic fashion](https://redis.io/topics/transactions).
@@ -501,7 +525,7 @@ recover for a while.
 
 [Circuit breakers are a pattern that does exactly that](https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern).
 
-Configuation options:
+Configuration options:
 
   - `error_threshold`. The amount of errors to encounter within `error_threshold_timeout` amount of time before opening the circuit, that is to start rejecting requests instantly.
   - `error_threshold_timeout`. The amount of time in seconds that `error_threshold` errors must occur to open the circuit. Defaults to `error_timeout` seconds if not set.

data/lib/redis_client/circuit_breaker.rb

@@ -63,7 +63,7 @@ class RedisClient
     private
 
     def refresh_state
-      now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      now = RedisClient.now
       @lock.synchronize do
         if @errors.last < (now - @error_timeout)
           if @success_threshold > 0
@@ -78,7 +78,7 @@ class RedisClient
     end
 
     def record_error
-      now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      now = RedisClient.now
       expiry = now - @error_timeout
       @lock.synchronize do
         if @state == :closed

data/lib/redis_client/config.rb

@@ -12,8 +12,8 @@ class RedisClient
     DEFAULT_DB = 0
 
     module Common
-      attr_reader :db, :password, :id, :ssl, :ssl_params, :command_builder, :inherit_socket,
-        :connect_timeout, :read_timeout, :write_timeout, :driver, :connection_prelude, :protocol,
+      attr_reader :db, :id, :ssl, :ssl_params, :command_builder, :inherit_socket,
+        :connect_timeout, :read_timeout, :write_timeout, :driver, :protocol,
         :middlewares_stack, :custom, :circuit_breaker
 
       alias_method :ssl?, :ssl
@@ -40,8 +40,13 @@ class RedisClient
         circuit_breaker: nil
       )
         @username = username
-        @password = password
-        @db = db || DEFAULT_DB
+        @password = password && !password.respond_to?(:call) ? ->(_) { password } : password
+        @db = begin
+          Integer(db || DEFAULT_DB)
+        rescue ArgumentError
+          raise ArgumentError, "db: must be an Integer, got: #{db.inspect}"
+        end
+
         @id = id
         @ssl = ssl || false
 
@@ -65,7 +70,6 @@ class RedisClient
 
         reconnect_attempts = Array.new(reconnect_attempts, 0).freeze if reconnect_attempts.is_a?(Integer)
         @reconnect_attempts = reconnect_attempts
-        @connection_prelude = build_connection_prelude
 
         circuit_breaker = CircuitBreaker.new(**circuit_breaker) if circuit_breaker.is_a?(Hash)
         if @circuit_breaker = circuit_breaker
@@ -82,10 +86,47 @@ class RedisClient
         @middlewares_stack = middlewares_stack
       end
 
+      def connection_prelude
+        prelude = []
+        pass = password
+        if protocol == 3
+          prelude << if pass
+            ["HELLO", "3", "AUTH", username, pass]
+          else
+            ["HELLO", "3"]
+          end
+        elsif pass
+          prelude << if @username && !@username.empty?
+            ["AUTH", @username, pass]
+          else
+            ["AUTH", pass]
+          end
+        end
+
+        if @db && @db != 0
+          prelude << ["SELECT", @db.to_s]
+        end
+
+        # Deep freeze all the strings and commands
+        prelude.map! do |commands|
+          commands = commands.map { |str| str.frozen? ? str : str.dup.freeze }
+          commands.freeze
+        end
+        prelude.freeze
+      end
+
+      def password
+        @password&.call(username)
+      end
+
       def username
         @username || DEFAULT_USERNAME
       end
 
+      def resolved?
+        true
+      end
+
       def sentinel?
         false
       end
@@ -119,34 +160,23 @@ class RedisClient
 
       def server_url
         if path
-          "#{path}/#{db}"
+          url = "unix://#{path}"
+          if db != 0
+            url = "#{url}?db=#{db}"
+          end
         else
-          "redis#{'s' if ssl?}://#{host}:#{port}/#{db}"
-        end
-      end
-
-      private
-
-      def build_connection_prelude
-        prelude = []
-        if protocol == 3
-          prelude << if @password
-            ["HELLO", "3", "AUTH", @username || DEFAULT_USERNAME, @password]
+          # add brackets to IPv6 address
+          redis_host = if host.count(":") >= 2
+            "[#{host}]"
           else
-            ["HELLO", "3"]
+            host
           end
-        elsif @password
-          prelude << if @username && !@username.empty?
-            ["AUTH", @username, @password]
-          else
-            ["AUTH", @password]
+          url = "redis#{'s' if ssl?}://#{redis_host}:#{port}"
+          if db != 0
+            url = "#{url}/#{db}"
           end
         end
-
-        if @db && @db != 0
-          prelude << ["SELECT", @db.to_s]
-        end
-        prelude.freeze
+        url
       end
     end
 
@@ -171,15 +201,20 @@ class RedisClient
         }.compact.merge(kwargs)
         host ||= url_config.host
         port ||= url_config.port
+        path ||= url_config.path
         username ||= url_config.username
         password ||= url_config.password
       end
 
       super(username: username, password: password, **kwargs)
 
-      @host = host || DEFAULT_HOST
-      @port = Integer(port || DEFAULT_PORT)
-      @path = path
+      if @path = path
+        @host = nil
+        @port = nil
+      else
+        @host = host || DEFAULT_HOST
+        @port = Integer(port || DEFAULT_PORT)
+      end
     end
   end
 end

data/lib/redis_client/connection_mixin.rb

@@ -28,18 +28,19 @@ class RedisClient
     def call(command, timeout)
       @pending_reads += 1
       write(command)
-      result = read(timeout)
+      result = read(connection_timeout(timeout))
       @pending_reads -= 1
       if result.is_a?(Error)
         result._set_command(command)
+        result._set_config(config)
         raise result
       else
         result
       end
     end
 
-    def call_pipelined(commands, timeouts)
-      exception = nil
+    def call_pipelined(commands, timeouts, exception: true)
+      first_exception = nil
 
       size = commands.size
       results = Array.new(commands.size)
@@ -48,20 +49,38 @@ class RedisClient
 
       size.times do |index|
         timeout = timeouts && timeouts[index]
-        result = read(timeout)
+        result = read(connection_timeout(timeout))
         @pending_reads -= 1
-        if result.is_a?(Error)
+
+        # A multi/exec command can return an array of results.
+        # An error from a multi/exec command is handled in Multi#_coerce!.
+        if result.is_a?(Array)
+          result.each do |res|
+            res._set_config(config) if res.is_a?(Error)
+          end
+        elsif result.is_a?(Error)
           result._set_command(commands[index])
-          exception ||= result
+          result._set_config(config)
+          first_exception ||= result
         end
+
         results[index] = result
       end
 
-      if exception
-        raise exception
+      if first_exception && exception
+        raise first_exception
       else
         results
       end
     end
+
+    def connection_timeout(timeout)
+      return timeout unless timeout && timeout > 0
+
+      # Can't use the command timeout argument as the connection timeout
+      # otherwise it would be very racy. So we add the regular read_timeout on top
+      # to account for the network delay.
+      timeout + config.read_timeout
+    end
   end
 end

data/lib/redis_client/decorator.rb

@@ -47,8 +47,8 @@ class RedisClient
       end
       ruby2_keywords :with if respond_to?(:ruby2_keywords, true)
 
-      def pipelined
-        @client.pipelined { |p| yield @_pipeline_class.new(p) }
+      def pipelined(exception: true)
+        @client.pipelined(exception: exception) { |p| yield @_pipeline_class.new(p) }
       end
 
       def multi(**kwargs)

data/lib/redis_client/ruby_connection/buffered_io.rb

@@ -10,14 +10,80 @@ class RedisClient
 
       attr_accessor :read_timeout, :write_timeout
 
-      def initialize(io, read_timeout:, write_timeout:, chunk_size: 4096)
-        @io = io
-        @buffer = "".b
-        @offset = 0
-        @chunk_size = chunk_size
-        @read_timeout = read_timeout
-        @write_timeout = write_timeout
-        @blocking_reads = false
+      if String.method_defined?(:byteindex) # Ruby 3.2+
+        ENCODING = Encoding::UTF_8
+
+        def initialize(io, read_timeout:, write_timeout:, chunk_size: 4096)
+          @io = io
+          @buffer = +""
+          @offset = 0
+          @chunk_size = chunk_size
+          @read_timeout = read_timeout
+          @write_timeout = write_timeout
+          @blocking_reads = false
+        end
+
+        def gets_chomp
+          fill_buffer(false) if @offset >= @buffer.bytesize
+          until eol_index = @buffer.byteindex(EOL, @offset)
+            fill_buffer(false)
+          end
+
+          line = @buffer.byteslice(@offset, eol_index - @offset)
+          @offset = eol_index + EOL_SIZE
+          line
+        end
+
+        def read_chomp(bytes)
+          ensure_remaining(bytes + EOL_SIZE)
+          str = @buffer.byteslice(@offset, bytes)
+          @offset += bytes + EOL_SIZE
+          str
+        end
+
+        private def ensure_line
+          fill_buffer(false) if @offset >= @buffer.bytesize
+          until @buffer.byteindex(EOL, @offset)
+            fill_buffer(false)
+          end
+        end
+      else
+        ENCODING = Encoding::BINARY
+
+        def initialize(io, read_timeout:, write_timeout:, chunk_size: 4096)
+          @io = io
+          @buffer = "".b
+          @offset = 0
+          @chunk_size = chunk_size
+          @read_timeout = read_timeout
+          @write_timeout = write_timeout
+          @blocking_reads = false
+        end
+
+        def gets_chomp
+          fill_buffer(false) if @offset >= @buffer.bytesize
+          until eol_index = @buffer.index(EOL, @offset)
+            fill_buffer(false)
+          end
+
+          line = @buffer.byteslice(@offset, eol_index - @offset)
+          @offset = eol_index + EOL_SIZE
+          line
+        end
+
+        def read_chomp(bytes)
+          ensure_remaining(bytes + EOL_SIZE)
+          str = @buffer.byteslice(@offset, bytes)
+          @offset += bytes + EOL_SIZE
+          str.force_encoding(Encoding::UTF_8)
+        end
+
+        private def ensure_line
+          fill_buffer(false) if @offset >= @buffer.bytesize
+          until @buffer.index(EOL, @offset)
+            fill_buffer(false)
+          end
+        end
       end
 
       def close
@@ -82,28 +148,35 @@ class RedisClient
       end
 
       def getbyte
-        ensure_remaining(1)
-        byte = @buffer.getbyte(@offset)
+        unless byte = @buffer.getbyte(@offset)
+          ensure_remaining(1)
+          byte = @buffer.getbyte(@offset)
+        end
         @offset += 1
         byte
       end
 
-      def gets_chomp
-        fill_buffer(false) if @offset >= @buffer.bytesize
-        until eol_index = @buffer.index(EOL, @offset)
-          fill_buffer(false)
-        end
+      def gets_integer
+        int = 0
+        offset = @offset
+        while true
+          chr = @buffer.getbyte(offset)
 
-        line = @buffer.byteslice(@offset, eol_index - @offset)
-        @offset = eol_index + EOL_SIZE
-        line
-      end
+          if chr
+            if chr == 13 # "\r".ord
+              @offset = offset + 2
+              break
+            else
+              int = (int * 10) + chr - 48
+            end
+            offset += 1
+          else
+            ensure_line
+            return gets_integer
+          end
+        end
 
-      def read_chomp(bytes)
-        ensure_remaining(bytes + EOL_SIZE)
-        str = @buffer.byteslice(@offset, bytes)
-        @offset += bytes + EOL_SIZE
-        str
+        int
       end
 
       private
@@ -117,7 +190,9 @@ class RedisClient
 
       def fill_buffer(strict, size = @chunk_size)
         remaining = size
-        empty_buffer = @offset >= @buffer.bytesize
+        buffer_size = @buffer.bytesize
+        start = @offset - buffer_size
+        empty_buffer = start >= 0
 
         loop do
           bytes = if empty_buffer
@@ -125,26 +200,40 @@ class RedisClient
           else
             @io.read_nonblock([remaining, @chunk_size].max, exception: false)
           end
+
           case bytes
-          when String
-            if empty_buffer
-              @offset = 0
-              empty_buffer = false
-            else
-              @buffer << bytes
-            end
-            remaining -= bytes.bytesize
-            return if !strict || remaining <= 0
           when :wait_readable
+            # Ref: https://github.com/redis-rb/redis-client/issues/190
+            # SSLSocket always clear the provided buffer, even when it didn't
+            # read anything. So we need to reset the offset accordingly.
+            if empty_buffer && @buffer.empty?
+              @offset -= buffer_size
+            end
+
             unless @io.to_io.wait_readable(@read_timeout)
               raise ReadTimeoutError, "Waited #{@read_timeout} seconds" unless @blocking_reads
             end
           when :wait_writable
+            # Ref: https://github.com/redis-rb/redis-client/issues/190
+            # SSLSocket always clear the provided buffer, even when it didn't
+            # read anything. So we need to reset the offset accordingly.
+            if empty_buffer && @buffer.empty?
+              @offset -= buffer_size
+            end
+
             @io.to_io.wait_writable(@write_timeout) or raise(WriteTimeoutError, "Waited #{@write_timeout} seconds")
           when nil
             raise EOFError
           else
-            raise "Unexpected `read_nonblock` return: #{bytes.inspect}"
+            if empty_buffer
+              @offset = start
+              empty_buffer = false
+              @buffer.force_encoding(ENCODING) unless @buffer.encoding == ENCODING
+            else
+              @buffer << bytes.force_encoding(ENCODING)
+            end
+            remaining -= bytes.bytesize
+            return if !strict || remaining <= 0
           end
         end
       end