grape-idempotency 1.1.1 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: be4536a211550d89d0c48cd4c83c7704cb5242bedd7028fe10037e1ceac11f76
-  data.tar.gz: 5fed5897f5416f70e99771aa7a773433420829acb60c9682fec55d943ac5080c
+  metadata.gz: 482197066de124a0866f6e8a96a3506b0301f892f8cab0ec04ea550a88ba8046
+  data.tar.gz: b0eb8cd9a3f5fd8979180ca234501e0be9781d33f079dc1085ef2021713960e9
 SHA512:
-  metadata.gz: 797f3ba565676d3eb97c5b0d38af871ec0c017b3f1a0705b08cd80c7f8b6fa30d32d3c921bb46aa2910827e06bb21500d3bf83e63c017177feb33e13bb5a0ab3
-  data.tar.gz: 071c4135447ceb9389cd2c5d8c2bb92477465318c523efc25eca43967e0c4533366a5d576cac3c52ee097c59ccd1164628124d73a5aea1072f6bfed7da3ae78b
+  metadata.gz: 6bb352d73edaf86330aa4585df7ad5124fec16cb10314cf5eb2b338be83b6e8754592615a7ab833e989c3e11d65df15030378e698d51b435d2613078eaf2c779
+  data.tar.gz: 40d0a3e83170df6bed8b8daece96dec1241ae953fab54c56490e79dc18ca439a342b26a6cbc9214cfca3f14c059dd2bbd9e2e30bc9a694353693307acf9f2dcb

data/CHANGELOG.md

@@ -4,6 +4,34 @@ All changes to `grape-idempotency` will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.2.0] - 2026-10-15
+
+### Feature
+
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Replay the response headers of the original request (e.g. `Content-Type`, `Location` or custom headers), except `Set-Cookie`, hop-by-hop headers, `Content-Length`, `Date`, `Age`, `Via`, `Server-Timing`, `X-Runtime`, `Original-Request` and the configured idempotency key and request id headers - [@jcagarcia](https://github.com/jcagarcia).
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Add the `replay_headers_denylist` configuration option to exclude more response headers from being stored and replayed - [@jcagarcia](https://github.com/jcagarcia).
+* Add `encryption_key` and `encryptor` configuration options for encrypting the requests and responses stored in `Redis` - [@jcagarcia](https://github.com/jcagarcia).
+
+### Fix
+
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Store and replay binary and non UTF-8 response bodies, which raised an error while storing the response and replaced the original response - [@jcagarcia](https://github.com/jcagarcia).
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Replay the formatted response body byte for byte. APIs not using the JSON format returned a different body when replaying (e.g. `{:id=>1}` became `{"id"=>1}`) - [@jcagarcia](https://github.com/jcagarcia).
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Keep the headers passed to `error!` inside an `idempotent` block in the first response, and replay them - [@jcagarcia](https://github.com/jcagarcia).
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Return the `Original-Request` and idempotency key headers in responses generated by `rescue_from` handlers with Grape < 2.4 - [@jcagarcia](https://github.com/jcagarcia).
+
+### Changed
+
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Store the final response (status, headers and formatted body) built by Grape, for both `idempotent` blocks and `rescue_from` handlers. Responses stored by previous versions are still replayed - [@jcagarcia](https://github.com/jcagarcia).
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): When replaying a response, the code placed after the `idempotent` block in the endpoint (and the `after` callbacks) is no longer run. Previously it ran on every replay - [@jcagarcia](https://github.com/jcagarcia).
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): `error!` called inside an `idempotent` block is rendered by Grape's error formatter, as it is without the gem. Previously the message was rendered by the regular formatter (e.g. `error!('Not allowed', 403)` returned `"Not allowed"` instead of `{"error":"Not allowed"}` with `format :json`) - [@jcagarcia](https://github.com/jcagarcia).
+* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Responses whose body is a stream or a file are not stored. The idempotency key is released and a warning is logged, so these endpoints are not idempotent - [@jcagarcia](https://github.com/jcagarcia).
+* Store a SHA-256 digest of the request params instead of the params. Requests stored by previous versions are still read. While deploying, instances running a previous version return the conflict response for retries of requests stored by this version - [@jcagarcia](https://github.com/jcagarcia).
+* Do not log idempotency keys, stored requests or response bodies. Fix the inverted values of the "Same params?" and "Same path?" debug log - [@jcagarcia](https://github.com/jcagarcia).
+* Log a warning the first time an idempotent endpoint is performed when the stored data is not encrypted - [@jcagarcia](https://github.com/jcagarcia).
+* Document the data stored in `Redis` and for how long - [@jcagarcia](https://github.com/jcagarcia).
+* Improve the gem summary and description, and add source code, changelog, bug tracker and documentation links to the gem metadata - [@jcagarcia](https://github.com/jcagarcia).
+
+
 ## [1.1.1] - 2026-10-15
 
 ### Fix

data/README.md

@@ -15,6 +15,7 @@ Topics covered in this README:
 - [How it works](#how-it-works-)
   - [Making idempotency key header mandatory](#making-idempotency-key-header-mandatory-)
   - [Redis Storage Connectivity Issue](#redis-storage-connectivity-issue)
+  - [Stored data](#stored-data-)
 - [Configuration](#configuration-)
 - [Changelog](#changelog)
 - [Contributing](#contributing)
@@ -88,9 +89,15 @@ This key allows your consumer to make the same request again in case of a connec
 
 To execute an idempotent request, simply request your user to include an extra `Idempotency-Key: <key>` header as part of his request.
 
-This gem operates by storing the initial request's status code and response body, regardless of whether the request succeeded or failed, using a specific idempotency key. Subsequent requests with the same key will consistently yield the same result, even if there were 500 errors.
+This gem operates by storing the initial request's status code, response headers and response body, regardless of whether the request succeeded or failed, using a specific idempotency key. Subsequent requests with the same key will consistently yield the same result, even if there were 500 errors.
 
-Keys are automatically removed from the system if they are at least 24 hours old, and a new request is generated when a key is reused after the original has been removed. The idempotency layer compares incoming parameters to those of the original request and returns a `422 - Unprocessable Entity` status code if they don't match, preventing accidental misuse.
+The stored response is the final one built by Grape (the formatted body, also for responses generated by `error!` or by `rescue_from` handlers), and it is returned byte for byte. When a response is replayed, your endpoint is not run, including any code placed after the `idempotent` block.
+
+Some response headers are never stored nor replayed: `Set-Cookie`, hop-by-hop headers (e.g. `Connection` or `Transfer-Encoding`), `Content-Length`, `Date`, `Age`, `Via`, `Server-Timing`, `X-Runtime`, `Original-Request` and the configured idempotency key and request id headers. You can exclude more headers with the [replay_headers_denylist](#replay_headers_denylist) configuration option.
+
+Responses whose body is a stream or a file (e.g. using `stream` or `sendfile`) are not stored: the idempotency key is released and a warning is logged, so those endpoints are not idempotent.
+
+Keys are automatically removed from the system once they are older than the configured [expires_in](#expires_in) (60 hours by default), and a new request is generated when a key is reused after the original has been removed. The idempotency layer compares incoming parameters to those of the original request and returns a `422 - Unprocessable Entity` status code if they don't match, preventing accidental misuse.
 If a request is received while another one with the same idempotency key is still being processed the idempotency layer returns a `409 - Conflict` status
 
 Results are only saved if an API endpoint begins its execution. If incoming parameters fail validation or if the request conflicts with another one executing concurrently, no idempotent result is stored because no API endpoint has initiated execution. In such cases, retrying these requests is safe.
@@ -160,13 +167,30 @@ If you want to avoid this functionality, and you want the gem handles the potent
 
 🚨 WARNING: If a `Redis` exception appears AFTER performing the wrapped code, nothing will be re-raised. The process will continue working and the response will be returned to the consumer of your API. However, a `409 Conflict` response can be returned to your consumer if it retried the same call with the same idempotency key. This is because the gem was not able to associate the response of the original request to the original idempotency key because those connectivity issues.
 
+### Stored data 🔐
+
+For each request with an idempotency key, the gem stores in `Redis`, under the `grape:idempotency:<idempotency-key>` key:
+
+* The request path.
+* A SHA-256 digest of the request params, used to detect when an idempotency key is reused with different params. The params themselves are NOT stored.
+* The response status, headers (except the ones listed in [How it works](#how-it-works-)) and body, which are returned again when the request is retried.
+* The original request id.
+
+Stored requests are removed after [expires_in](#expires_in) seconds (60 hours by default).
+
+Since `1.2.0` the request params are not stored. While deploying `1.2.0`, instances still running an older version of the gem return the [conflict_error_response](#conflict_error_response) for retries of requests stored by `1.2.0`, as they compare the params of the request with the stored ones.
+
+The response is stored as returned by your endpoint, so it may include sensitive data. Configure an [encryption_key](#encryption_key) or an [encryptor](#encryptor) to encrypt everything the gem stores. Until you do, the gem logs a warning the first time an idempotent endpoint is performed.
+
+The gem never logs idempotency keys, request params or response bodies.
+
 ## Configuration 🪚
 
 In addition to the storage aspect, you have the option to supply additional configuration details to tailor the gem to the specific requirements of your project.
 
 ### expires_in
 
-As we have mentioned in the [How it works](#how-it-works-) section, keys are automatically removed from the system if they are at least 24 hours old. However, a 24-hour timeframe may not be suitable for your project. To accommodate your specific needs, you can adjust this duration by using the `expires_in` parameter for configuration:
+As we have mentioned in the [How it works](#how-it-works-) section, keys are automatically removed from the system once they are older than `expires_in` (60 hours by default). However, this timeframe may not be suitable for your project. To accommodate your specific needs, you can adjust this duration by using the `expires_in` parameter for configuration:
 
 ```ruby
 Grape::Idempotency.configure do |c|
@@ -209,7 +233,7 @@ In the case above, you request your consumers to use the `X-Trace-Id: <trace-id>
 
 ### logger, logger_level and logger_prefix
 
-By default, the logger used by the gem is configured like `Logger.new(STDOUT)` and `INFO` level. As this gem does not log any message with `INFO` level, only `ERROR` messages will be logged.
+By default, the logger used by the gem is configured like `Logger.new(STDOUT)` and `INFO` level. As this gem does not log any message with `INFO` level, only `WARN` and `ERROR` messages will be logged.
 
 
 If you want to provide your own logger, you want to change the level to `DEBUG` or you want to provide your own prefix, you can configure the gem like:
@@ -228,7 +252,7 @@ An example of the logged information when changing the level of the log to `DEBU
 ```shell
 I, [2023-11-23T22:41:39.148163 #1]  DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
 I, [2023-11-23T22:41:39.148176 #1]  DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
-I, [2023-11-23T22:41:39.148192 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key" => "fd77c9d6-b7da-4966-aac8-40ee258f24aa"
+I, [2023-11-23T22:41:39.148192 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key".
 I, [2023-11-23T22:41:39.148210 #1]  DEBUG -- : [my-own-prefix] Previous request information has NOT been found for the provided idempotency key.
 I, [2023-11-23T22:41:39.148248 #1]  DEBUG -- : [my-own-prefix] Request stored as processing.
 I, [2023-11-23T22:41:39.148261 #1]  DEBUG -- : [my-own-prefix] Performing the provided block.
@@ -237,8 +261,8 @@ I, [2023-11-23T22:41:39.148287 #1]  DEBUG -- : [my-own-prefix] Storing response.
 I, [2023-11-23T22:41:39.148317 #1]  DEBUG -- : [my-own-prefix] Response stored.
 I, [2023-11-23T22:41:39.148473 #1]  DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
 I, [2023-11-23T22:41:39.148486 #1]  DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
-I, [2023-11-23T22:41:39.148502 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key" => "fd77c9d6-b7da-4966-aac8-40ee258f24aa"
-I, [2023-11-23T22:41:39.148523 #1]  DEBUG -- : [my-own-prefix] Request has been found for the provided idempotency key => {"path"=>"/payments", "params"=>{"locale"=>"undefined", "{\"amount\":10000}"=>nil}, "status"=>500, "original_request"=>"wadus", "response"=>"{\"error\":\"Internal Server Error\"}"}
+I, [2023-11-23T22:41:39.148502 #1]  DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key".
+I, [2023-11-23T22:41:39.148523 #1]  DEBUG -- : [my-own-prefix] Request has been found for the provided idempotency key.
 I, [2023-11-23T22:41:39.148537 #1]  DEBUG -- : [my-own-prefix] Returning the response from the original request.
 ```
 
@@ -257,6 +281,59 @@ end
 
 However, this approach carries a certain level of risk. In the case that `Redis` experiences an outage, the idempotent functionality will be lost, the endpoint will behave as no idempotent, and this issue may go unnoticed.
 
+### replay_headers_denylist
+
+By default, all the response headers are stored and replayed except the ones listed in the [How it works](#how-it-works-) section. If your API returns other headers that must not be replayed, you can add them using the `replay_headers_denylist` configuration option. Header names are case-insensitive, and the headers excluded by default are always excluded:
+
+```ruby
+Grape::Idempotency.configure do |c|
+  c.storage = @storage
+  c.replay_headers_denylist = ["X-Session-Token", "X-Rate-Limit-Remaining"]
+end
+```
+
+### encryption_key
+
+Encrypts everything the gem stores in `Redis` (see [Stored data](#stored-data-)) using `ActiveSupport::MessageEncryptor` with `aes-256-gcm`. The encryption key is derived from the provided value, so use a long random secret and share the same value across all the instances of your API, otherwise a retried request reaching another instance would not be able to read the stored response.
+
+```ruby
+Grape::Idempotency.configure do |c|
+  c.storage = @storage
+  c.encryption_key = ENV.fetch("IDEMPOTENCY_ENCRYPTION_KEY") # e.g. generated with `SecureRandom.hex(32)`
+end
+```
+
+Requests stored before configuring the encryption are still read, so you can enable it without affecting the retries in progress. While deploying it, instances running a version older than `1.2.0` are not able to read the encrypted requests, so upgrade the gem before configuring the encryption.
+
+If a stored request cannot be decrypted (e.g. after changing the key, or after removing the encryption), the gem returns the [conflict_error_response](#conflict_error_response) instead of performing the request again. Those requests are removed once [expires_in](#expires_in) has elapsed.
+
+### encryptor
+
+If you want to use your own encryption (e.g. a KMS, or your existing `ActiveSupport::MessageEncryptor`), provide an object responding to `encrypt(string)` and `decrypt(string)`, both returning a `String`. `decrypt` must raise an error (or return `nil`) when the value cannot be decrypted.
+
+```ruby
+class IdempotencyEncryptor
+  def initialize(message_encryptor)
+    @message_encryptor = message_encryptor
+  end
+
+  def encrypt(value)
+    @message_encryptor.encrypt_and_sign(value)
+  end
+
+  def decrypt(value)
+    @message_encryptor.decrypt_and_verify(value)
+  end
+end
+
+Grape::Idempotency.configure do |c|
+  c.storage = @storage
+  c.encryptor = IdempotencyEncryptor.new(my_message_encryptor)
+end
+```
+
+Configure either an `encryption_key` or an `encryptor`, not both.
+
 ### conflict_error_response
 
 When providing a `Idempotency-Key: <key>` header, this gem compares incoming parameters to those of the original request (if exists) and returns a `409 - Conflict` status code if they don't match, preventing accidental misuse. The response body returned by the gem looks like:

data/grape-idempotency.gemspec

@@ -8,10 +8,21 @@ Gem::Specification.new do |spec|
   spec.version     = Grape::Idempotency::VERSION
   spec.authors     = ['Juan Carlos García']
   spec.email       = ['jugade92@gmail.com']
-  spec.description = 'Add idempotency support to your Grape APIs for safely retrying requests without accidentally performing the same operation twice. When creating or updating an object, use an idempotency key. Then, if a connection error occurs, you can safely repeat the request without risk of creating a second object or performing the update twice.'
-  spec.summary     = 'Gem for supporting idempotency in your Grape APIs'
+  spec.description = 'Idempotency-Key header support for Grape REST APIs, based on the IETF Idempotency-Key HTTP header draft. ' \
+                     'Wrap a POST or PATCH endpoint in an idempotent block and clients can safely retry requests after timeouts or ' \
+                     'network errors without creating duplicate payments, orders or records. The first response (status, headers and ' \
+                     'body) is stored in Redis and replayed for retries with the same key. Concurrent duplicate requests get a 409 Conflict, ' \
+                     'and reusing a key with a different payload gets a 422 Unprocessable Entity. Works with Grape 1.x to 4.x and Ruby 2.6+.'
+  spec.summary     = 'Idempotency-Key support for Grape APIs: safe retries and duplicate request prevention backed by Redis'
   spec.homepage    = 'https://github.com/jcagarcia/grape-idempotency'
   spec.license     = 'MIT'
+  spec.metadata    = {
+    'homepage_uri' => spec.homepage,
+    'source_code_uri' => 'https://github.com/jcagarcia/grape-idempotency',
+    'changelog_uri' => 'https://github.com/jcagarcia/grape-idempotency/blob/main/CHANGELOG.md',
+    'bug_tracker_uri' => 'https://github.com/jcagarcia/grape-idempotency/issues',
+    'documentation_uri' => 'https://github.com/jcagarcia/grape-idempotency#readme'
+  }
 
   files = Dir["lib/**/*.rb"]
   rootfiles = ["CHANGELOG.md", "grape-idempotency.gemspec", "Rakefile", "README.md"]

data/lib/grape/idempotency/encryptor.rb

@@ -0,0 +1,38 @@
+require 'openssl'
+
+module Grape
+  module Idempotency
+    # Encryptor built by the gem from the configured `encryption_key`.
+    class Encryptor
+      CIPHER = "aes-256-gcm".freeze
+      KEY_SALT = "grape-idempotency".freeze
+      KEY_ITERATIONS = 65_536
+
+      # Stored records are already JSON strings, so they are encrypted as they are
+      module RawSerializer
+        def self.dump(value)
+          value
+        end
+
+        def self.load(value)
+          value
+        end
+      end
+
+      def initialize(encryption_key)
+        require 'active_support/message_encryptor'
+
+        key = OpenSSL::PKCS5.pbkdf2_hmac(encryption_key, KEY_SALT, KEY_ITERATIONS, 32, OpenSSL::Digest::SHA256.new)
+        @message_encryptor = ActiveSupport::MessageEncryptor.new(key, cipher: CIPHER, serializer: RawSerializer)
+      end
+
+      def encrypt(value)
+        @message_encryptor.encrypt_and_sign(value)
+      end
+
+      def decrypt(value)
+        @message_encryptor.decrypt_and_verify(value)
+      end
+    end
+  end
+end

data/lib/grape/idempotency/middleware/error.rb

@@ -23,14 +23,10 @@ module Grape
         response = error_response(response) if thrown_error?(response)
 
         if response.is_a?(Rack::Response)
-          update_idempotency_error_with(error, response)
           response
         else
           # default_rescue_handler is private since grape 2.1
-          response = run_rescue_handler(method(:default_rescue_handler), Grape::Exceptions::InvalidResponse.new, endpoint)
-          # Store it under the original error, the one the pending idempotent request was registered with
-          update_idempotency_error_with(error, response)
-          response
+          run_rescue_handler(method(:default_rescue_handler), Grape::Exceptions::InvalidResponse.new, endpoint)
         end
       end
 
@@ -42,19 +38,6 @@ module Grape
 
         response.is_a?(Hash) && response.key?(:message) && response.key?(:status) && response.key?(:headers)
       end
-
-      def update_idempotency_error_with(error, response)
-        stored_error_key = env && env[Grape::Idempotency::ERROR_KEY_ENV]
-        return unless stored_error_key
-
-        begin
-          body = JSON.parse(response.body.join)
-        rescue JSON::ParserError
-          body = response.body.join
-        end
-
-        Grape::Idempotency.update_error_with_rescue_from_result(error, response.status, body, error_key: stored_error_key)
-      end
     end
   end
 end

data/lib/grape/idempotency/middleware/response_capture.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Grape
+  module Idempotency
+    module Middleware
+      # Prepended to Grape::Middleware::Error, the outermost Grape middleware of every endpoint.
+      # It receives the final response (formatted body, status and headers) of the success path,
+      # of `error!` and of rescue_from handlers, so it can be stored or replayed as it is.
+      module ResponseCapture
+        def call!(env)
+          Grape::Idempotency.finalize_response(env, super)
+        end
+      end
+    end
+  end
+end
+
+Grape::Middleware::Error.prepend(Grape::Idempotency::Middleware::ResponseCapture)

data/lib/grape/idempotency/version.rb

@@ -2,6 +2,6 @@
 
 module Grape
   module Idempotency
-    VERSION = '1.1.1'
+    VERSION = '1.2.0'
   end
 end

data/lib/grape/idempotency.rb

@@ -2,18 +2,34 @@ require 'grape'
 require 'redis'
 require 'logger'
 require 'securerandom'
+require 'digest'
+require 'json'
 require 'grape/idempotency/version'
+require 'grape/idempotency/encryptor'
 require 'grape/idempotency/middleware/error'
+require 'grape/idempotency/middleware/response_capture'
 
 module Grape
   module Idempotency
     autoload :Helpers, 'grape/idempotency/helpers'
     Grape::Endpoint.send :include, Grape::Idempotency::Helpers
 
-    ERROR_KEY_ENV = "grape.idempotency.error_key".freeze
+    ENCRYPTED_RECORD_PREFIX = "encrypted:v1:".freeze
+
+    class UnreadableRecordError < StandardError; end
 
     class << self
       ORIGINAL_REQUEST_HEADER = "Original-Request".freeze
+      STORAGE_SCHEMA_VERSION = 2
+      BODY_ENCODING_UTF8 = "utf8".freeze
+      BODY_ENCODING_BASE64 = "base64".freeze
+      PENDING_ENV_KEY = "grape.idempotency.pending".freeze
+      REPLAY_ENV_KEY = "grape.idempotency.replay".freeze
+      # Response headers that are never stored nor replayed
+      EXCLUDED_REPLAY_HEADERS = %w[
+        set-cookie connection keep-alive proxy-connection transfer-encoding te trailer upgrade
+        content-length date age via server-timing x-runtime original-request
+      ].freeze
 
       def configure(&block)
         yield(configuration)
@@ -23,30 +39,46 @@ module Grape
         clean_configuration = Configuration.new
         clean_configuration.storage = storage
         @configuration = clean_configuration
+        @unencrypted_storage_warned = false
       end
 
       def idempotent(grape, required: false, &block)
         validate_config!
+        warn_unencrypted_storage unless encryptor
         log(:debug, "Performing endpoint \"#{grape.request.path}\" with idempotency.")
         log(:debug, "Idempotency key is #{!required ? 'NOT' : ''} mandatory for this endpoint.")
 
         idempotency_key = get_idempotency_key(grape.request.headers)
-        log(:debug, "Idempotency key received in request header \"#{configuration.idempotency_key_header.downcase}\" => \"#{idempotency_key}\"") if idempotency_key
+        log(:debug, "Idempotency key received in request header \"#{configuration.idempotency_key_header.downcase}\".") if idempotency_key
 
         grape.error!(configuration.mandatory_header_response, 400) if required && !idempotency_key
         return block.call if !idempotency_key
 
-        stored_request = get_from_storage(idempotency_key)
-        log(:debug, "Request has been found for the provided idempotency key => #{stored_request}") if stored_request
-        if stored_request && (stored_request["params"] != grape.request.params || stored_request["path"] != grape.request.path)
-          log(:debug, "Request has conflicts. Same params? => #{stored_request["params"] != grape.request.params}. Same path? => #{stored_request["path"] != grape.request.path}")
+        request_params_digest = params_digest(grape.request.params)
+        begin
+          stored_request = get_from_storage(idempotency_key)
+        rescue UnreadableRecordError
+          log(:error, "The request stored for the provided idempotency key could NOT be read. Check the encryption configuration.")
+          log(:debug, "Returning conflict error response.")
+          grape.error!(configuration.conflict_error_response, 422)
+        end
+        log(:debug, "Request has been found for the provided idempotency key.") if stored_request
+        same_params = stored_request && stored_params_digest(stored_request) == request_params_digest
+        same_path = stored_request && stored_request["path"] == grape.request.path
+        if stored_request && (!same_params || !same_path)
+          log(:debug, "Request has conflicts. Same params? => #{same_params}. Same path? => #{same_path}")
           log(:debug, "Returning conflict error response.")
           grape.error!(configuration.conflict_error_response, 422)
         elsif stored_request && stored_request["processing"] == true
           log(:debug, "Returning processing error response.")
           grape.error!(configuration.processing_response, 409)
-        elsif stored_request
+        elsif stored_request && stored_request["v"].to_i >= STORAGE_SCHEMA_VERSION
           log(:debug, "Returning the response from the original request.")
+          # Stop the endpoint here. The stored response replaces Grape's response in #finalize_response.
+          grape.env[REPLAY_ENV_KEY] = { idempotency_key: idempotency_key, stored_request: stored_request }
+          grape.error!('', stored_request["status"])
+        elsif stored_request
+          log(:debug, "Returning the response from the original request (stored by a previous version of the gem).")
           grape.status stored_request["status"]
           grape.header(ORIGINAL_REQUEST_HEADER, stored_request["original_request"])
           grape.header(configuration.idempotency_key_header, idempotency_key)
@@ -56,7 +88,7 @@ module Grape
         log(:debug, "Previous request information has NOT been found for the provided idempotency key.")
 
         original_request_id = get_request_id(grape.request.headers)
-        success = store_processing_request(idempotency_key, grape.request.path, grape.request.params, original_request_id)
+        success = store_processing_request(idempotency_key, grape.request.path, request_params_digest, original_request_id)
         if !success
           log(:error, "Request NOT stored as processing. Concurrent requests for the same idempotency key appeared.")
           grape.error!(configuration.processing_response, 409)
@@ -64,67 +96,60 @@ module Grape
           log(:debug, "Request stored as processing.")
         end
 
+        # The final response (formatted body, status and headers) is stored by #finalize_response,
+        # once Grape has built it, both for the success path and for the rescue_from path.
+        grape.env[PENDING_ENV_KEY] = {
+          idempotency_key: idempotency_key,
+          path: grape.request.path,
+          params_digest: request_params_digest,
+          original_request: original_request_id
+        }
+        grape.header(ORIGINAL_REQUEST_HEADER, original_request_id)
+        grape.header(configuration.idempotency_key_header, idempotency_key)
+
         log(:debug, "Performing the provided block.")
+        block.call
+      end
 
-        response = catch(:error) do
-          block.call
-        end
+      # Called with the final Rack response of an endpoint (see Grape::Idempotency::ResponseCapture).
+      # Stores it for a pending idempotent request, or replaces it with the stored one when replaying.
+      # It never raises: on any failure the original response is returned.
+      def finalize_response(env, response)
+        replay = env.delete(REPLAY_ENV_KEY)
+        pending = env.delete(PENDING_ENV_KEY)
+        return response unless replay || pending
 
-        log(:debug, "Block has been performed.")
+        parts = rack_response_parts(response)
 
-        if is_an_error?(response)
-          log(:debug, "An error response was returned by the performed block. => #{response}")
-          response = response.is_a?(Hash) ? response[:message] : response.message
-        end
+        if replay
+          return response unless parts
 
-        grape.header(ORIGINAL_REQUEST_HEADER, original_request_id)
-        grape.body response
-      rescue Redis::BaseError => e
-        raise
-      rescue => e
-        log(:debug, "An unexpected error was raised when performing the block.")
-        if idempotency_key && !stored_request && !response
-          validate_config!
-          log(:debug, "Storing error response.")
-          original_request_id = get_request_id(grape.request.headers)
-          stored_key = store_error_request(idempotency_key, grape.request.path, grape.request.params, grape.status, original_request_id, e)
-          if stored_key
-            log(:debug, "Error response stored.")
-            grape.env[ERROR_KEY_ENV] = error_key(stored_key)
-            grape.header(ORIGINAL_REQUEST_HEADER, original_request_id)
-            grape.header(configuration.idempotency_key_header, stored_key)
-          end
+          log(:debug, "Replaying the stored response.")
+          return replay_response(replay[:stored_request], replay[:idempotency_key], parts[1])
         end
-        log(:debug, "Re-raising the error.")
-        raise
-      ensure
-        if !stored_request && response
-          validate_config!
-          log(:debug, "Storing response.")
-          stored_key = store_request_response(idempotency_key, grape.request.path, grape.request.params, grape.status, original_request_id, response)
-          if stored_key
-            log(:debug, "Response stored.")
-            grape.header(configuration.idempotency_key_header, stored_key)
-          end
-        end
-      end
 
-      def update_error_with_rescue_from_result(error, status, response, error_key: nil)
-        validate_config!
+        if parts.nil? || !storable_body?(parts[2])
+          log(:warn, "The response body of \"#{pending[:path]}\" is a stream or a file. It is NOT stored and its idempotency key is released.")
+          release_idempotency_key(pending[:idempotency_key])
+          return response
+        end
 
-        stored_error = error_key ? get_error_request_by_key(error_key) : get_error_request_for(error)
-        return unless stored_error
+        status, headers, body = parts
+        log(:debug, "Storing response.")
+        store_request_response(pending, status, headers, body.join)
+        log(:debug, "Response stored.")
 
-        request_with_unmanaged_error = stored_error[:request]
-        idempotency_key = stored_error[:idempotency_key]
-        path = request_with_unmanaged_error["path"]
-        params = request_with_unmanaged_error["params"]
-        original_request_id = request_with_unmanaged_error["original_request"]
+        set_response_header(headers, ORIGINAL_REQUEST_HEADER, pending[:original_request])
+        set_response_header(headers, configuration.idempotency_key_header, pending[:idempotency_key])
+        response
+      rescue => e
+        log(:error, "The response could NOT be stored or replayed => #{e.class}")
+        response
+      end
 
-        store_request_response(idempotency_key, path, params, status, original_request_id, response)
-        storage.del(stored_error[:error_key])
-      rescue Redis::BaseError => e
-        log(:error, "Storage error => #{e.message} - #{e}")
+      # Kept for backwards compatibility. Responses generated by rescue_from handlers are now
+      # stored by #finalize_response.
+      def update_error_with_rescue_from_result(_error, _status, _response, error_key: nil)
         nil
       end
 
@@ -132,6 +157,21 @@ module Grape
 
       def validate_config!
         raise Configuration::Error.new("A Redis instance must be configured as cache storage") unless valid_storage?
+        validate_encryption_config!
+      end
+
+      def validate_encryption_config!
+        if configuration.encryption_key && configuration.encryptor
+          raise Configuration::Error.new("Configure either an encryption_key or an encryptor, not both")
+        end
+
+        if configuration.encryption_key && !(configuration.encryption_key.is_a?(String) && !configuration.encryption_key.empty?)
+          raise Configuration::Error.new("The encryption_key must be a non-empty String")
+        end
+
+        if configuration.encryptor && !(configuration.encryptor.respond_to?(:encrypt) && configuration.encryptor.respond_to?(:decrypt))
+          raise Configuration::Error.new("The encryptor must respond to encrypt and decrypt")
+        end
       end
 
       def valid_storage?
@@ -161,120 +201,179 @@ module Grape
         value = storage.get(key(idempotency_key))
         return unless value
 
-        JSON.parse(value)
+        deserialize(value)
       rescue Redis::BaseError => e
         log(:error, "Storage error => #{e.message} - #{e}")
         return if configuration.manage_redis_exceptions
         raise
       end
 
-      def store_processing_request(idempotency_key, path, params, request_id)
+      def store_processing_request(idempotency_key, path, request_params_digest, request_id)
         body = {
           path: path,
-          params: params,
+          params_digest: request_params_digest,
           original_request: request_id,
           processing: true
         }
 
-        storage.set(key(idempotency_key), body.to_json, ex: configuration.expires_in, nx: true)
+        storage.set(key(idempotency_key), serialize(body), ex: configuration.expires_in, nx: true)
       rescue Redis::BaseError => e
         return true if configuration.manage_redis_exceptions
         raise
       end
 
-      def store_request_response(idempotency_key, path, params, status, request_id, response)
-        body = {
-          path: path,
-          params: params,
-          status: status,
-          original_request: request_id,
-          response: response
+      def store_request_response(pending, status, headers, body)
+        utf8_body = body.dup.force_encoding(Encoding::UTF_8)
+        body_encoding = utf8_body.valid_encoding? ? BODY_ENCODING_UTF8 : BODY_ENCODING_BASE64
+        replayable_headers = replayable_headers(headers)
+
+        stored_response = {
+          v: STORAGE_SCHEMA_VERSION,
+          path: pending[:path],
+          params_digest: pending[:params_digest],
+          status: status.to_i,
+          original_request: pending[:original_request],
+          headers: replayable_headers,
+          body_encoding: body_encoding,
+          body: body_encoding == BODY_ENCODING_UTF8 ? utf8_body : [body].pack('m0')
         }
+        # Read by versions of the gem that do not know the schema above (e.g. during a rolling deploy)
+        stored_response[:response] = legacy_response(replayable_headers, utf8_body) if body_encoding == BODY_ENCODING_UTF8
 
-        storage.set(key(idempotency_key), body.to_json, ex: configuration.expires_in, nx: false)
+        storage.set(key(pending[:idempotency_key]), serialize(stored_response), ex: configuration.expires_in, nx: false)
 
-        idempotency_key
+        pending[:idempotency_key]
       rescue Redis::BaseError => e
         log(:error, "Storage error => #{e.message} - #{e}")
-        idempotency_key
+        pending[:idempotency_key]
       end
 
-      def store_error_request(idempotency_key, path, params, status, request_id, error)
-        body = {
-          path: path,
-          params: params,
-          status: status,
-          original_request: request_id,
-          error: {
-            class_name: error.class.to_s,
-            message: error.message
-          }
-        }.to_json
-
-        storage.set(error_key(idempotency_key), body, ex: 30, nx: false)
-
-        idempotency_key
+      def release_idempotency_key(idempotency_key)
+        storage.del(key(idempotency_key))
       rescue Redis::BaseError => e
         log(:error, "Storage error => #{e.message} - #{e}")
-        idempotency_key
+        nil
       end
 
-      def get_error_request_by_key(error_key)
-        raw_request_with_error = storage.get(error_key)
-        return unless raw_request_with_error
+      def replay_response(stored_request, idempotency_key, headers)
+        headers.keys.each do |name|
+          headers.delete(name) if %w[content-type content-length].include?(name.to_s.downcase)
+        end
+        (stored_request["headers"] || {}).each { |name, value| set_response_header(headers, name, value) }
+        set_response_header(headers, ORIGINAL_REQUEST_HEADER, stored_request["original_request"])
+        set_response_header(headers, configuration.idempotency_key_header, idempotency_key)
+
+        body = stored_request["body"].to_s
+        body = body.unpack1('m0') if stored_request["body_encoding"] == BODY_ENCODING_BASE64
 
-        {
-          error_key: error_key,
-          request: JSON.parse(raw_request_with_error),
-          idempotency_key: error_key.delete_prefix(error_key_prefix)
-        }
-      rescue Redis::BaseError, JSON::ParserError, TypeError => e
-        log(:error, "Error retrieving stored error request => #{e.message} - #{e}")
-        nil
+        [stored_request["status"].to_i, headers, [body]]
       end
 
-      def get_error_request_for(error)
-        error_keys = storage.keys("#{error_key_prefix}*")
-        return if error_keys.empty?
-
-        error_keys.map do |key|
-          raw_request_with_error = storage.get(key)
-          next unless raw_request_with_error
-
-          request_with_error = JSON.parse(raw_request_with_error)
-          error_class_name = request_with_error["error"]["class_name"]
-          error_message = request_with_error["error"]["message"]
-
-          if error_class_name == error.class.to_s && error_message == error.message
-            {
-              error_key: key,
-              request: request_with_error,
-              idempotency_key: key.delete_prefix(error_key_prefix)
-            }
-          end
-        end.compact.first
-      rescue Redis::BaseError, JSON::ParserError, TypeError => e
-        log(:error, "Error retrieving stored error request => #{e.message} - #{e}")
-        nil
+      def rack_response_parts(response)
+        if response.is_a?(Rack::Response)
+          [response.status, response.headers, response.body]
+        elsif response.is_a?(Array) && response.size == 3
+          response
+        end
       end
 
-      def is_an_error?(response)
-        # grape >= 3.3 throws Grape::Exceptions::ErrorResponse instead of a Hash
-        return true if defined?(Grape::Exceptions::ErrorResponse) && response.is_a?(Grape::Exceptions::ErrorResponse)
+      # Streams and files are not buffered, so they cannot be stored
+      def storable_body?(body)
+        body.is_a?(Array) && body.all? { |chunk| chunk.is_a?(String) }
+      end
 
-        response.is_a?(Hash) && response.has_key?(:message) && response.has_key?(:headers) && response.has_key?(:status)
+      def replayable_headers(headers)
+        excluded = excluded_headers
+        headers.each_with_object({}) do |(name, value), replayable|
+          replayable[name.to_s] = value unless excluded.include?(name.to_s.downcase)
+        end
       end
 
-      def key(idempotency_key)
-        "#{gem_prefix}#{idempotency_key}"
+      def excluded_headers
+        EXCLUDED_REPLAY_HEADERS +
+          [configuration.idempotency_key_header, configuration.request_id_header].map(&:downcase) +
+          Array(configuration.replay_headers_denylist).map { |name| name.to_s.downcase }
+      end
+
+      def legacy_response(headers, body)
+        content_type = headers.find { |name, _| name.downcase == 'content-type' }&.last.to_s
+        return body unless content_type =~ /\bjson\b/i
+
+        JSON.parse(body)
+      rescue JSON::ParserError
+        body
       end
 
-      def error_key(idempotency_key)
-        "#{error_key_prefix}#{idempotency_key}"
+      def set_response_header(headers, name, value)
+        headers.keys.each { |existing| headers.delete(existing) if existing.to_s.casecmp?(name) }
+        # Rack 3 requires lowercase header names
+        headers[defined?(Rack::Headers) ? name.downcase : name] = value
       end
 
-      def error_key_prefix
-        "#{gem_prefix}error:"
+      def serialize(record)
+        json = record.to_json
+        return json unless encryptor
+
+        "#{ENCRYPTED_RECORD_PREFIX}#{encryptor.encrypt(json)}"
+      end
+
+      def deserialize(value)
+        if value.start_with?(ENCRYPTED_RECORD_PREFIX)
+          raise UnreadableRecordError, "Stored request is encrypted but no encryption is configured" unless encryptor
+
+          value = encryptor.decrypt(value.delete_prefix(ENCRYPTED_RECORD_PREFIX))
+        end
+
+        # Records stored before configuring the encryption are not encrypted and are still accepted
+        JSON.parse(value)
+      rescue UnreadableRecordError
+        raise
+      rescue StandardError => e
+        raise UnreadableRecordError, "Stored request could not be read (#{e.class})"
+      end
+
+      def encryptor
+        return configuration.encryptor if configuration.encryptor
+        return unless configuration.encryption_key
+
+        if @encryptor.nil? || @encryptor_key != configuration.encryption_key
+          @encryptor = Encryptor.new(configuration.encryption_key)
+          @encryptor_key = configuration.encryption_key
+        end
+        @encryptor
+      end
+
+      def warn_unencrypted_storage
+        return if @unencrypted_storage_warned
+
+        @unencrypted_storage_warned = true
+        log(:warn, "Stored requests and responses are NOT encrypted. Configure an encryption_key or an encryptor to encrypt them.")
+      end
+
+      def params_digest(params)
+        digest(JSON.generate(canonical(JSON.parse(params.to_json))))
+      end
+
+      def stored_params_digest(stored_request)
+        # Requests stored by versions < 1.2.0 contain the params instead of their digest
+        stored_request["params_digest"] || params_digest(stored_request["params"])
+      end
+
+      # Sorts hash keys so the digest does not depend on the params order
+      def canonical(value)
+        case value
+        when Hash then value.keys.sort.map { |k| [k, canonical(value[k])] }
+        when Array then value.map { |v| canonical(v) }
+        else value
+        end
+      end
+
+      def digest(value)
+        Digest::SHA256.hexdigest(value)
+      end
+
+      def key(idempotency_key)
+        "#{gem_prefix}#{idempotency_key}"
       end
 
       def gem_prefix
@@ -290,7 +389,7 @@ module Grape
       end
 
       def logger
-        return @logger if @logger
+        return @logger if @logger && @logger.equal?(configuration.logger)
 
         @logger = configuration.logger
         @logger.level = configuration.logger_level
@@ -305,7 +404,7 @@ module Grape
     class Configuration
       attr_accessor :storage, :logger, :logger_level, :logger_prefix, :expires_in, :idempotency_key_header,
         :request_id_header, :conflict_error_response, :processing_response, :mandatory_header_response,
-        :manage_redis_exceptions
+        :manage_redis_exceptions, :replay_headers_denylist, :encryption_key, :encryptor
 
       class Error < StandardError; end
 
@@ -318,6 +417,9 @@ module Grape
         @idempotency_key_header = "idempotency-key"
         @request_id_header = "x-request-id"
         @manage_redis_exceptions = false
+        @replay_headers_denylist = []
+        @encryption_key = nil
+        @encryptor = nil
         @conflict_error_response = {
           "title" => "Idempotency-Key is already used",
           "detail" => "This operation is idempotent and it requires correct usage of Idempotency Key. Idempotency Key MUST not be reused across different payloads of this operation."
@@ -333,4 +435,4 @@ module Grape
       end
     end
   end
-end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: grape-idempotency
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.2.0
 platform: ruby
 authors:
 - Juan Carlos García
@@ -94,11 +94,13 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: Add idempotency support to your Grape APIs for safely retrying requests
-  without accidentally performing the same operation twice. When creating or updating
-  an object, use an idempotency key. Then, if a connection error occurs, you can safely
-  repeat the request without risk of creating a second object or performing the update
-  twice.
+description: Idempotency-Key header support for Grape REST APIs, based on the IETF
+  Idempotency-Key HTTP header draft. Wrap a POST or PATCH endpoint in an idempotent
+  block and clients can safely retry requests after timeouts or network errors without
+  creating duplicate payments, orders or records. The first response (status, headers
+  and body) is stored in Redis and replayed for retries with the same key. Concurrent
+  duplicate requests get a 409 Conflict, and reusing a key with a different payload
+  gets a 422 Unprocessable Entity. Works with Grape 1.x to 4.x and Ruby 2.6+.
 email:
 - jugade92@gmail.com
 executables: []
@@ -111,13 +113,20 @@ files:
 - grape-idempotency.gemspec
 - lib/grape-idempotency.rb
 - lib/grape/idempotency.rb
+- lib/grape/idempotency/encryptor.rb
 - lib/grape/idempotency/helpers.rb
 - lib/grape/idempotency/middleware/error.rb
+- lib/grape/idempotency/middleware/response_capture.rb
 - lib/grape/idempotency/version.rb
 homepage: https://github.com/jcagarcia/grape-idempotency
 licenses:
 - MIT
-metadata: {}
+metadata:
+  homepage_uri: https://github.com/jcagarcia/grape-idempotency
+  source_code_uri: https://github.com/jcagarcia/grape-idempotency
+  changelog_uri: https://github.com/jcagarcia/grape-idempotency/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/jcagarcia/grape-idempotency/issues
+  documentation_uri: https://github.com/jcagarcia/grape-idempotency#readme
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -136,5 +145,6 @@ requirements: []
 rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
-summary: Gem for supporting idempotency in your Grape APIs
+summary: 'Idempotency-Key support for Grape APIs: safe retries and duplicate request
+  prevention backed by Redis'
 test_files: []