grape-idempotency 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +44 -5
- data/README.md +101 -8
- data/grape-idempotency.gemspec +13 -2
- data/lib/grape/idempotency/encryptor.rb +38 -0
- data/lib/grape/idempotency/middleware/error.rb +19 -16
- data/lib/grape/idempotency/middleware/response_capture.rb +18 -0
- data/lib/grape/idempotency/version.rb +1 -1
- data/lib/grape/idempotency.rb +243 -118
- metadata +22 -12
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 482197066de124a0866f6e8a96a3506b0301f892f8cab0ec04ea550a88ba8046
|
|
4
|
+
data.tar.gz: b0eb8cd9a3f5fd8979180ca234501e0be9781d33f079dc1085ef2021713960e9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6bb352d73edaf86330aa4585df7ad5124fec16cb10314cf5eb2b338be83b6e8754592615a7ab833e989c3e11d65df15030378e698d51b435d2613078eaf2c779
|
|
7
|
+
data.tar.gz: 40d0a3e83170df6bed8b8daece96dec1241ae953fab54c56490e79dc18ca439a342b26a6cbc9214cfca3f14c059dd2bbd9e2e30bc9a694353693307acf9f2dcb
|
data/CHANGELOG.md
CHANGED
|
@@ -4,20 +4,59 @@ All changes to `grape-idempotency` will be documented in this file.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
-
## [1.
|
|
7
|
+
## [1.2.0] - 2026-10-15
|
|
8
|
+
|
|
9
|
+
### Feature
|
|
10
|
+
|
|
11
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Replay the response headers of the original request (e.g. `Content-Type`, `Location` or custom headers), except `Set-Cookie`, hop-by-hop headers, `Content-Length`, `Date`, `Age`, `Via`, `Server-Timing`, `X-Runtime`, `Original-Request` and the configured idempotency key and request id headers - [@jcagarcia](https://github.com/jcagarcia).
|
|
12
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Add the `replay_headers_denylist` configuration option to exclude more response headers from being stored and replayed - [@jcagarcia](https://github.com/jcagarcia).
|
|
13
|
+
* Add `encryption_key` and `encryptor` configuration options for encrypting the requests and responses stored in `Redis` - [@jcagarcia](https://github.com/jcagarcia).
|
|
8
14
|
|
|
9
15
|
### Fix
|
|
10
16
|
|
|
11
|
-
*
|
|
17
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Store and replay binary and non UTF-8 response bodies, which raised an error while storing the response and replaced the original response - [@jcagarcia](https://github.com/jcagarcia).
|
|
18
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Replay the formatted response body byte for byte. APIs not using the JSON format returned a different body when replaying (e.g. `{:id=>1}` became `{"id"=>1}`) - [@jcagarcia](https://github.com/jcagarcia).
|
|
19
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Keep the headers passed to `error!` inside an `idempotent` block in the first response, and replay them - [@jcagarcia](https://github.com/jcagarcia).
|
|
20
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Return the `Original-Request` and idempotency key headers in responses generated by `rescue_from` handlers with Grape < 2.4 - [@jcagarcia](https://github.com/jcagarcia).
|
|
12
21
|
|
|
13
22
|
### Changed
|
|
14
23
|
|
|
15
|
-
*
|
|
24
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Store the final response (status, headers and formatted body) built by Grape, for both `idempotent` blocks and `rescue_from` handlers. Responses stored by previous versions are still replayed - [@jcagarcia](https://github.com/jcagarcia).
|
|
25
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): When replaying a response, the code placed after the `idempotent` block in the endpoint (and the `after` callbacks) is no longer run. Previously it ran on every replay - [@jcagarcia](https://github.com/jcagarcia).
|
|
26
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): `error!` called inside an `idempotent` block is rendered by Grape's error formatter, as it is without the gem. Previously the message was rendered by the regular formatter (e.g. `error!('Not allowed', 403)` returned `"Not allowed"` instead of `{"error":"Not allowed"}` with `format :json`) - [@jcagarcia](https://github.com/jcagarcia).
|
|
27
|
+
* [#27](https://github.com/jcagarcia/grape-idempotency/issues/27): Responses whose body is a stream or a file are not stored. The idempotency key is released and a warning is logged, so these endpoints are not idempotent - [@jcagarcia](https://github.com/jcagarcia).
|
|
28
|
+
* Store a SHA-256 digest of the request params instead of the params. Requests stored by previous versions are still read. While deploying, instances running a previous version return the conflict response for retries of requests stored by this version - [@jcagarcia](https://github.com/jcagarcia).
|
|
29
|
+
* Do not log idempotency keys, stored requests or response bodies. Fix the inverted values of the "Same params?" and "Same path?" debug log - [@jcagarcia](https://github.com/jcagarcia).
|
|
30
|
+
* Log a warning the first time an idempotent endpoint is performed when the stored data is not encrypted - [@jcagarcia](https://github.com/jcagarcia).
|
|
31
|
+
* Document the data stored in `Redis` and for how long - [@jcagarcia](https://github.com/jcagarcia).
|
|
32
|
+
* Improve the gem summary and description, and add source code, changelog, bug tracker and documentation links to the gem metadata - [@jcagarcia](https://github.com/jcagarcia).
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
## [1.1.1] - 2026-10-15
|
|
36
|
+
|
|
37
|
+
### Fix
|
|
38
|
+
|
|
39
|
+
* Catch errors thrown with `error!` inside `rescue_from` handlers, which raised `UncaughtThrowError` with Grape >= 2.1 - [@jcagarcia](https://github.com/jcagarcia).
|
|
40
|
+
* Return Grape's `500 Invalid response` when a `rescue_from` handler does not return a valid response, instead of raising `NoMethodError: undefined method 'default_rescue_handler'` with Grape >= 2.1 - [@jcagarcia](https://github.com/jcagarcia).
|
|
41
|
+
* Store that `500 Invalid response` for the idempotency key, so retrying the request replays it instead of returning `409 Conflict` until the key expires - [@jcagarcia](https://github.com/jcagarcia).
|
|
42
|
+
* Support `rescue_from ..., with: :helper_method` with Grape >= 3.0, which resolves symbol handlers on the endpoint - [@jcagarcia](https://github.com/jcagarcia).
|
|
43
|
+
* Only treat a Hash returned by a `rescue_from` handler as an error response when it contains `:message`, `:status` and `:headers`, matching Grape - [@jcagarcia](https://github.com/jcagarcia).
|
|
44
|
+
* Handle `Grape::Exceptions::ErrorResponse` thrown by `error!` with Grape >= 3.3, both inside `idempotent` blocks and inside `rescue_from` handlers - [@jcagarcia](https://github.com/jcagarcia).
|
|
45
|
+
* [#22](https://github.com/jcagarcia/grape-idempotency/pull/22): Do not turn `rescue_from` responses into 500 errors when a stored error request expires or is invalid while being looked up. - [@salrepe](https://github.com/salrepe).
|
|
46
|
+
* [#22](https://github.com/jcagarcia/grape-idempotency/pull/22): Only access the storage from the error middleware for requests that stored an error inside `idempotent`, reading that exact key instead of scanning all error keys. - [@salrepe](https://github.com/salrepe).
|
|
47
|
+
* [#22](https://github.com/jcagarcia/grape-idempotency/pull/22): Do not store error requests when no idempotency key was provided. - [@salrepe](https://github.com/salrepe).
|
|
48
|
+
|
|
49
|
+
### Changed
|
|
50
|
+
|
|
51
|
+
* [#24](https://github.com/jcagarcia/grape-idempotency/issues/24): Test Ruby (2.6 - 4.0) and Grape (1.5 - 4.0) in CI: the combinations where behaviour changes on every push and pull request, and every supported combination plus the latest Grape release weekly or on demand - [@jcagarcia](https://github.com/jcagarcia).
|
|
52
|
+
* Document Ruby and Grape compatibility in the README - [@jcagarcia](https://github.com/jcagarcia).
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
## [1.1.0] - 2023-12-12
|
|
16
56
|
|
|
17
57
|
### Feature
|
|
18
58
|
|
|
19
|
-
* [#20](https://github.com/jcagarcia/grape-idempotency/pull/20):
|
|
20
|
-
* Your contribution here.
|
|
59
|
+
* [#20](https://github.com/jcagarcia/grape-idempotency/pull/20): Allow consumers to configure the gem for handling `Redis` exceptions - [@jcagarcia](https://github.com/jcagarcia).
|
|
21
60
|
|
|
22
61
|
## [1.0.0] - 2023-11-23
|
|
23
62
|
|
data/README.md
CHANGED
|
@@ -10,10 +10,12 @@ Implementation based on the Active Internet-Draft [draft-ietf-httpapi-idempotenc
|
|
|
10
10
|
Topics covered in this README:
|
|
11
11
|
|
|
12
12
|
- [Installation](#installation-)
|
|
13
|
+
- [Compatibility](#compatibility-)
|
|
13
14
|
- [Basic Usage](#basic-usage-)
|
|
14
15
|
- [How it works](#how-it-works-)
|
|
15
16
|
- [Making idempotency key header mandatory](#making-idempotency-key-header-mandatory-)
|
|
16
17
|
- [Redis Storage Connectivity Issue](#redis-storage-connectivity-issue)
|
|
18
|
+
- [Stored data](#stored-data-)
|
|
17
19
|
- [Configuration](#configuration-)
|
|
18
20
|
- [Changelog](#changelog)
|
|
19
21
|
- [Contributing](#contributing)
|
|
@@ -30,6 +32,20 @@ Ruby 2.6 or newer is required.
|
|
|
30
32
|
gem install grape-idempotency
|
|
31
33
|
```
|
|
32
34
|
|
|
35
|
+
## Compatibility 🤝
|
|
36
|
+
|
|
37
|
+
- Ruby 2.6 or newer.
|
|
38
|
+
- Grape 1.x and 2.x or newer are supported from a single codebase.
|
|
39
|
+
- Grape 1.0 - 1.4 are allowed by the gemspec but are not tested.
|
|
40
|
+
- Grape 1.x does not work with Rack 3, so if you are using Grape 1.x add `gem 'rack', '< 3'` to your `Gemfile`.
|
|
41
|
+
- Grape 1.5 does not load with ActiveSupport 7.1.x (7.0 and 7.2+ work). On Ruby 2.7 / 3.0, where 7.1 is the newest installable ActiveSupport, add `gem 'activesupport', '< 7.1'` to your `Gemfile`.
|
|
42
|
+
|
|
43
|
+
The tested Ruby and Grape combinations live in the [CI workflow](.github/workflows/ruby.yml): pushes and pull requests run the combinations where Ruby or Grape behaviour changes, and a weekly (or manually triggered) run tests every Grape minor line from 1.5 onwards against every Ruby version it supports, plus the latest Grape release on the newest Ruby. To run the specs against a specific Grape line locally, set `GRAPE_VERSION` (this rewrites your local `Gemfile.lock`, do not commit it):
|
|
44
|
+
|
|
45
|
+
```shell
|
|
46
|
+
GRAPE_VERSION="~> 1.8.0" bundle update grape && bundle exec rspec
|
|
47
|
+
```
|
|
48
|
+
|
|
33
49
|
## Basic Usage 📖
|
|
34
50
|
|
|
35
51
|
Configure the `Grape::Idempotency` class with a `Redis` instance.
|
|
@@ -73,9 +89,15 @@ This key allows your consumer to make the same request again in case of a connec
|
|
|
73
89
|
|
|
74
90
|
To execute an idempotent request, simply request your user to include an extra `Idempotency-Key: <key>` header as part of his request.
|
|
75
91
|
|
|
76
|
-
This gem operates by storing the initial request's status code and response body, regardless of whether the request succeeded or failed, using a specific idempotency key. Subsequent requests with the same key will consistently yield the same result, even if there were 500 errors.
|
|
92
|
+
This gem operates by storing the initial request's status code, response headers and response body, regardless of whether the request succeeded or failed, using a specific idempotency key. Subsequent requests with the same key will consistently yield the same result, even if there were 500 errors.
|
|
93
|
+
|
|
94
|
+
The stored response is the final one built by Grape (the formatted body, also for responses generated by `error!` or by `rescue_from` handlers), and it is returned byte for byte. When a response is replayed, your endpoint is not run, including any code placed after the `idempotent` block.
|
|
77
95
|
|
|
78
|
-
|
|
96
|
+
Some response headers are never stored nor replayed: `Set-Cookie`, hop-by-hop headers (e.g. `Connection` or `Transfer-Encoding`), `Content-Length`, `Date`, `Age`, `Via`, `Server-Timing`, `X-Runtime`, `Original-Request` and the configured idempotency key and request id headers. You can exclude more headers with the [replay_headers_denylist](#replay_headers_denylist) configuration option.
|
|
97
|
+
|
|
98
|
+
Responses whose body is a stream or a file (e.g. using `stream` or `sendfile`) are not stored: the idempotency key is released and a warning is logged, so those endpoints are not idempotent.
|
|
99
|
+
|
|
100
|
+
Keys are automatically removed from the system once they are older than the configured [expires_in](#expires_in) (60 hours by default), and a new request is generated when a key is reused after the original has been removed. The idempotency layer compares incoming parameters to those of the original request and returns a `422 - Unprocessable Entity` status code if they don't match, preventing accidental misuse.
|
|
79
101
|
If a request is received while another one with the same idempotency key is still being processed the idempotency layer returns a `409 - Conflict` status
|
|
80
102
|
|
|
81
103
|
Results are only saved if an API endpoint begins its execution. If incoming parameters fail validation or if the request conflicts with another one executing concurrently, no idempotent result is stored because no API endpoint has initiated execution. In such cases, retrying these requests is safe.
|
|
@@ -145,13 +167,30 @@ If you want to avoid this functionality, and you want the gem handles the potent
|
|
|
145
167
|
|
|
146
168
|
🚨 WARNING: If a `Redis` exception appears AFTER performing the wrapped code, nothing will be re-raised. The process will continue working and the response will be returned to the consumer of your API. However, a `409 Conflict` response can be returned to your consumer if it retried the same call with the same idempotency key. This is because the gem was not able to associate the response of the original request to the original idempotency key because those connectivity issues.
|
|
147
169
|
|
|
170
|
+
### Stored data 🔐
|
|
171
|
+
|
|
172
|
+
For each request with an idempotency key, the gem stores in `Redis`, under the `grape:idempotency:<idempotency-key>` key:
|
|
173
|
+
|
|
174
|
+
* The request path.
|
|
175
|
+
* A SHA-256 digest of the request params, used to detect when an idempotency key is reused with different params. The params themselves are NOT stored.
|
|
176
|
+
* The response status, headers (except the ones listed in [How it works](#how-it-works-)) and body, which are returned again when the request is retried.
|
|
177
|
+
* The original request id.
|
|
178
|
+
|
|
179
|
+
Stored requests are removed after [expires_in](#expires_in) seconds (60 hours by default).
|
|
180
|
+
|
|
181
|
+
Since `1.2.0` the request params are not stored. While deploying `1.2.0`, instances still running an older version of the gem return the [conflict_error_response](#conflict_error_response) for retries of requests stored by `1.2.0`, as they compare the params of the request with the stored ones.
|
|
182
|
+
|
|
183
|
+
The response is stored as returned by your endpoint, so it may include sensitive data. Configure an [encryption_key](#encryption_key) or an [encryptor](#encryptor) to encrypt everything the gem stores. Until you do, the gem logs a warning the first time an idempotent endpoint is performed.
|
|
184
|
+
|
|
185
|
+
The gem never logs idempotency keys, request params or response bodies.
|
|
186
|
+
|
|
148
187
|
## Configuration 🪚
|
|
149
188
|
|
|
150
189
|
In addition to the storage aspect, you have the option to supply additional configuration details to tailor the gem to the specific requirements of your project.
|
|
151
190
|
|
|
152
191
|
### expires_in
|
|
153
192
|
|
|
154
|
-
As we have mentioned in the [How it works](#how-it-works-) section, keys are automatically removed from the system
|
|
193
|
+
As we have mentioned in the [How it works](#how-it-works-) section, keys are automatically removed from the system once they are older than `expires_in` (60 hours by default). However, this timeframe may not be suitable for your project. To accommodate your specific needs, you can adjust this duration by using the `expires_in` parameter for configuration:
|
|
155
194
|
|
|
156
195
|
```ruby
|
|
157
196
|
Grape::Idempotency.configure do |c|
|
|
@@ -194,7 +233,7 @@ In the case above, you request your consumers to use the `X-Trace-Id: <trace-id>
|
|
|
194
233
|
|
|
195
234
|
### logger, logger_level and logger_prefix
|
|
196
235
|
|
|
197
|
-
By default, the logger used by the gem is configured like `Logger.new(STDOUT)` and `INFO` level. As this gem does not log any message with `INFO` level, only `ERROR` messages will be logged.
|
|
236
|
+
By default, the logger used by the gem is configured like `Logger.new(STDOUT)` and `INFO` level. As this gem does not log any message with `INFO` level, only `WARN` and `ERROR` messages will be logged.
|
|
198
237
|
|
|
199
238
|
|
|
200
239
|
If you want to provide your own logger, you want to change the level to `DEBUG` or you want to provide your own prefix, you can configure the gem like:
|
|
@@ -213,7 +252,7 @@ An example of the logged information when changing the level of the log to `DEBU
|
|
|
213
252
|
```shell
|
|
214
253
|
I, [2023-11-23T22:41:39.148163 #1] DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
|
|
215
254
|
I, [2023-11-23T22:41:39.148176 #1] DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
|
|
216
|
-
I, [2023-11-23T22:41:39.148192 #1] DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key"
|
|
255
|
+
I, [2023-11-23T22:41:39.148192 #1] DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key".
|
|
217
256
|
I, [2023-11-23T22:41:39.148210 #1] DEBUG -- : [my-own-prefix] Previous request information has NOT been found for the provided idempotency key.
|
|
218
257
|
I, [2023-11-23T22:41:39.148248 #1] DEBUG -- : [my-own-prefix] Request stored as processing.
|
|
219
258
|
I, [2023-11-23T22:41:39.148261 #1] DEBUG -- : [my-own-prefix] Performing the provided block.
|
|
@@ -222,8 +261,8 @@ I, [2023-11-23T22:41:39.148287 #1] DEBUG -- : [my-own-prefix] Storing response.
|
|
|
222
261
|
I, [2023-11-23T22:41:39.148317 #1] DEBUG -- : [my-own-prefix] Response stored.
|
|
223
262
|
I, [2023-11-23T22:41:39.148473 #1] DEBUG -- : [my-own-prefix] Performing endpoint "/payments" with idempotency.
|
|
224
263
|
I, [2023-11-23T22:41:39.148486 #1] DEBUG -- : [my-own-prefix] Idempotency key is NOT mandatory for this endpoint.
|
|
225
|
-
I, [2023-11-23T22:41:39.148502 #1] DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key"
|
|
226
|
-
I, [2023-11-23T22:41:39.148523 #1] DEBUG -- : [my-own-prefix] Request has been found for the provided idempotency key
|
|
264
|
+
I, [2023-11-23T22:41:39.148502 #1] DEBUG -- : [my-own-prefix] Idempotency key received in request header "x-custom-idempotency-key".
|
|
265
|
+
I, [2023-11-23T22:41:39.148523 #1] DEBUG -- : [my-own-prefix] Request has been found for the provided idempotency key.
|
|
227
266
|
I, [2023-11-23T22:41:39.148537 #1] DEBUG -- : [my-own-prefix] Returning the response from the original request.
|
|
228
267
|
```
|
|
229
268
|
|
|
@@ -242,6 +281,59 @@ end
|
|
|
242
281
|
|
|
243
282
|
However, this approach carries a certain level of risk. In the case that `Redis` experiences an outage, the idempotent functionality will be lost, the endpoint will behave as no idempotent, and this issue may go unnoticed.
|
|
244
283
|
|
|
284
|
+
### replay_headers_denylist
|
|
285
|
+
|
|
286
|
+
By default, all the response headers are stored and replayed except the ones listed in the [How it works](#how-it-works-) section. If your API returns other headers that must not be replayed, you can add them using the `replay_headers_denylist` configuration option. Header names are case-insensitive, and the headers excluded by default are always excluded:
|
|
287
|
+
|
|
288
|
+
```ruby
|
|
289
|
+
Grape::Idempotency.configure do |c|
|
|
290
|
+
c.storage = @storage
|
|
291
|
+
c.replay_headers_denylist = ["X-Session-Token", "X-Rate-Limit-Remaining"]
|
|
292
|
+
end
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
### encryption_key
|
|
296
|
+
|
|
297
|
+
Encrypts everything the gem stores in `Redis` (see [Stored data](#stored-data-)) using `ActiveSupport::MessageEncryptor` with `aes-256-gcm`. The encryption key is derived from the provided value, so use a long random secret and share the same value across all the instances of your API, otherwise a retried request reaching another instance would not be able to read the stored response.
|
|
298
|
+
|
|
299
|
+
```ruby
|
|
300
|
+
Grape::Idempotency.configure do |c|
|
|
301
|
+
c.storage = @storage
|
|
302
|
+
c.encryption_key = ENV.fetch("IDEMPOTENCY_ENCRYPTION_KEY") # e.g. generated with `SecureRandom.hex(32)`
|
|
303
|
+
end
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
Requests stored before configuring the encryption are still read, so you can enable it without affecting the retries in progress. While deploying it, instances running a version older than `1.2.0` are not able to read the encrypted requests, so upgrade the gem before configuring the encryption.
|
|
307
|
+
|
|
308
|
+
If a stored request cannot be decrypted (e.g. after changing the key, or after removing the encryption), the gem returns the [conflict_error_response](#conflict_error_response) instead of performing the request again. Those requests are removed once [expires_in](#expires_in) has elapsed.
|
|
309
|
+
|
|
310
|
+
### encryptor
|
|
311
|
+
|
|
312
|
+
If you want to use your own encryption (e.g. a KMS, or your existing `ActiveSupport::MessageEncryptor`), provide an object responding to `encrypt(string)` and `decrypt(string)`, both returning a `String`. `decrypt` must raise an error (or return `nil`) when the value cannot be decrypted.
|
|
313
|
+
|
|
314
|
+
```ruby
|
|
315
|
+
class IdempotencyEncryptor
|
|
316
|
+
def initialize(message_encryptor)
|
|
317
|
+
@message_encryptor = message_encryptor
|
|
318
|
+
end
|
|
319
|
+
|
|
320
|
+
def encrypt(value)
|
|
321
|
+
@message_encryptor.encrypt_and_sign(value)
|
|
322
|
+
end
|
|
323
|
+
|
|
324
|
+
def decrypt(value)
|
|
325
|
+
@message_encryptor.decrypt_and_verify(value)
|
|
326
|
+
end
|
|
327
|
+
end
|
|
328
|
+
|
|
329
|
+
Grape::Idempotency.configure do |c|
|
|
330
|
+
c.storage = @storage
|
|
331
|
+
c.encryptor = IdempotencyEncryptor.new(my_message_encryptor)
|
|
332
|
+
end
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
Configure either an `encryption_key` or an `encryptor`, not both.
|
|
336
|
+
|
|
245
337
|
### conflict_error_response
|
|
246
338
|
|
|
247
339
|
When providing a `Idempotency-Key: <key>` header, this gem compares incoming parameters to those of the original request (if exists) and returns a `409 - Conflict` status code if they don't match, preventing accidental misuse. The response body returned by the gem looks like:
|
|
@@ -368,4 +460,5 @@ Open issues on the GitHub issue tracker with clear information.
|
|
|
368
460
|
### Contributors
|
|
369
461
|
|
|
370
462
|
* Juan Carlos García - Creator - https://github.com/jcagarcia
|
|
371
|
-
* Carlos Cabanero - Contributor - https://github.com/Flip120
|
|
463
|
+
* Carlos Cabanero - Contributor - https://github.com/Flip120
|
|
464
|
+
* Salvador Revert Pedron - Contributor - https://github.com/salrepe
|
data/grape-idempotency.gemspec
CHANGED
|
@@ -8,10 +8,21 @@ Gem::Specification.new do |spec|
|
|
|
8
8
|
spec.version = Grape::Idempotency::VERSION
|
|
9
9
|
spec.authors = ['Juan Carlos García']
|
|
10
10
|
spec.email = ['jugade92@gmail.com']
|
|
11
|
-
spec.description = '
|
|
12
|
-
|
|
11
|
+
spec.description = 'Idempotency-Key header support for Grape REST APIs, based on the IETF Idempotency-Key HTTP header draft. ' \
|
|
12
|
+
'Wrap a POST or PATCH endpoint in an idempotent block and clients can safely retry requests after timeouts or ' \
|
|
13
|
+
'network errors without creating duplicate payments, orders or records. The first response (status, headers and ' \
|
|
14
|
+
'body) is stored in Redis and replayed for retries with the same key. Concurrent duplicate requests get a 409 Conflict, ' \
|
|
15
|
+
'and reusing a key with a different payload gets a 422 Unprocessable Entity. Works with Grape 1.x to 4.x and Ruby 2.6+.'
|
|
16
|
+
spec.summary = 'Idempotency-Key support for Grape APIs: safe retries and duplicate request prevention backed by Redis'
|
|
13
17
|
spec.homepage = 'https://github.com/jcagarcia/grape-idempotency'
|
|
14
18
|
spec.license = 'MIT'
|
|
19
|
+
spec.metadata = {
|
|
20
|
+
'homepage_uri' => spec.homepage,
|
|
21
|
+
'source_code_uri' => 'https://github.com/jcagarcia/grape-idempotency',
|
|
22
|
+
'changelog_uri' => 'https://github.com/jcagarcia/grape-idempotency/blob/main/CHANGELOG.md',
|
|
23
|
+
'bug_tracker_uri' => 'https://github.com/jcagarcia/grape-idempotency/issues',
|
|
24
|
+
'documentation_uri' => 'https://github.com/jcagarcia/grape-idempotency#readme'
|
|
25
|
+
}
|
|
15
26
|
|
|
16
27
|
files = Dir["lib/**/*.rb"]
|
|
17
28
|
rootfiles = ["CHANGELOG.md", "grape-idempotency.gemspec", "Rakefile", "README.md"]
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
require 'openssl'
|
|
2
|
+
|
|
3
|
+
module Grape
|
|
4
|
+
module Idempotency
|
|
5
|
+
# Encryptor built by the gem from the configured `encryption_key`.
|
|
6
|
+
class Encryptor
|
|
7
|
+
CIPHER = "aes-256-gcm".freeze
|
|
8
|
+
KEY_SALT = "grape-idempotency".freeze
|
|
9
|
+
KEY_ITERATIONS = 65_536
|
|
10
|
+
|
|
11
|
+
# Stored records are already JSON strings, so they are encrypted as they are
|
|
12
|
+
module RawSerializer
|
|
13
|
+
def self.dump(value)
|
|
14
|
+
value
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def self.load(value)
|
|
18
|
+
value
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def initialize(encryption_key)
|
|
23
|
+
require 'active_support/message_encryptor'
|
|
24
|
+
|
|
25
|
+
key = OpenSSL::PKCS5.pbkdf2_hmac(encryption_key, KEY_SALT, KEY_ITERATIONS, 32, OpenSSL::Digest::SHA256.new)
|
|
26
|
+
@message_encryptor = ActiveSupport::MessageEncryptor.new(key, cipher: CIPHER, serializer: RawSerializer)
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def encrypt(value)
|
|
30
|
+
@message_encryptor.encrypt_and_sign(value)
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def decrypt(value)
|
|
34
|
+
@message_encryptor.decrypt_and_verify(value)
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
@@ -5,36 +5,39 @@ module Grape
|
|
|
5
5
|
class Error < Base
|
|
6
6
|
def run_rescue_handler(handler, error, endpoint=nil)
|
|
7
7
|
if handler.instance_of?(Symbol)
|
|
8
|
-
|
|
8
|
+
# grape >= 3.0 resolves symbol handlers (`rescue_from ..., with: :helper`) on the endpoint
|
|
9
|
+
handler_owner = respond_to?(handler) || endpoint.nil? ? self : endpoint
|
|
10
|
+
raise NoMethodError, "undefined method '#{handler}'" unless handler_owner.respond_to?(handler)
|
|
9
11
|
|
|
10
|
-
handler = public_method(handler)
|
|
12
|
+
handler = handler_owner.public_method(handler)
|
|
11
13
|
end
|
|
12
14
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
15
|
+
response = catch(:error) do
|
|
16
|
+
if endpoint
|
|
17
|
+
handler.arity.zero? ? endpoint.instance_exec(&handler) : endpoint.instance_exec(error, &handler)
|
|
18
|
+
else
|
|
19
|
+
handler.arity.zero? ? instance_exec(&handler) : instance_exec(error, &handler)
|
|
20
|
+
end
|
|
17
21
|
end
|
|
18
22
|
|
|
23
|
+
response = error_response(response) if thrown_error?(response)
|
|
24
|
+
|
|
19
25
|
if response.is_a?(Rack::Response)
|
|
20
|
-
update_idempotency_error_with(error, response)
|
|
21
26
|
response
|
|
22
27
|
else
|
|
23
|
-
|
|
28
|
+
# default_rescue_handler is private since grape 2.1
|
|
29
|
+
run_rescue_handler(method(:default_rescue_handler), Grape::Exceptions::InvalidResponse.new, endpoint)
|
|
24
30
|
end
|
|
25
31
|
end
|
|
26
32
|
|
|
27
33
|
private
|
|
28
34
|
|
|
29
|
-
def
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
rescue JSON::ParserError
|
|
33
|
-
body = response.body.join
|
|
34
|
-
end
|
|
35
|
+
def thrown_error?(response)
|
|
36
|
+
# grape >= 3.3 throws Grape::Exceptions::ErrorResponse instead of a Hash
|
|
37
|
+
return true if defined?(Grape::Exceptions::ErrorResponse) && response.is_a?(Grape::Exceptions::ErrorResponse)
|
|
35
38
|
|
|
36
|
-
|
|
39
|
+
response.is_a?(Hash) && response.key?(:message) && response.key?(:status) && response.key?(:headers)
|
|
37
40
|
end
|
|
38
41
|
end
|
|
39
42
|
end
|
|
40
|
-
end
|
|
43
|
+
end
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Grape
|
|
4
|
+
module Idempotency
|
|
5
|
+
module Middleware
|
|
6
|
+
# Prepended to Grape::Middleware::Error, the outermost Grape middleware of every endpoint.
|
|
7
|
+
# It receives the final response (formatted body, status and headers) of the success path,
|
|
8
|
+
# of `error!` and of rescue_from handlers, so it can be stored or replayed as it is.
|
|
9
|
+
module ResponseCapture
|
|
10
|
+
def call!(env)
|
|
11
|
+
Grape::Idempotency.finalize_response(env, super)
|
|
12
|
+
end
|
|
13
|
+
end
|
|
14
|
+
end
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
Grape::Middleware::Error.prepend(Grape::Idempotency::Middleware::ResponseCapture)
|
data/lib/grape/idempotency.rb
CHANGED
|
@@ -2,16 +2,34 @@ require 'grape'
|
|
|
2
2
|
require 'redis'
|
|
3
3
|
require 'logger'
|
|
4
4
|
require 'securerandom'
|
|
5
|
+
require 'digest'
|
|
6
|
+
require 'json'
|
|
5
7
|
require 'grape/idempotency/version'
|
|
8
|
+
require 'grape/idempotency/encryptor'
|
|
6
9
|
require 'grape/idempotency/middleware/error'
|
|
10
|
+
require 'grape/idempotency/middleware/response_capture'
|
|
7
11
|
|
|
8
12
|
module Grape
|
|
9
13
|
module Idempotency
|
|
10
14
|
autoload :Helpers, 'grape/idempotency/helpers'
|
|
11
15
|
Grape::Endpoint.send :include, Grape::Idempotency::Helpers
|
|
12
16
|
|
|
17
|
+
ENCRYPTED_RECORD_PREFIX = "encrypted:v1:".freeze
|
|
18
|
+
|
|
19
|
+
class UnreadableRecordError < StandardError; end
|
|
20
|
+
|
|
13
21
|
class << self
|
|
14
22
|
ORIGINAL_REQUEST_HEADER = "Original-Request".freeze
|
|
23
|
+
STORAGE_SCHEMA_VERSION = 2
|
|
24
|
+
BODY_ENCODING_UTF8 = "utf8".freeze
|
|
25
|
+
BODY_ENCODING_BASE64 = "base64".freeze
|
|
26
|
+
PENDING_ENV_KEY = "grape.idempotency.pending".freeze
|
|
27
|
+
REPLAY_ENV_KEY = "grape.idempotency.replay".freeze
|
|
28
|
+
# Response headers that are never stored nor replayed
|
|
29
|
+
EXCLUDED_REPLAY_HEADERS = %w[
|
|
30
|
+
set-cookie connection keep-alive proxy-connection transfer-encoding te trailer upgrade
|
|
31
|
+
content-length date age via server-timing x-runtime original-request
|
|
32
|
+
].freeze
|
|
15
33
|
|
|
16
34
|
def configure(&block)
|
|
17
35
|
yield(configuration)
|
|
@@ -21,30 +39,46 @@ module Grape
|
|
|
21
39
|
clean_configuration = Configuration.new
|
|
22
40
|
clean_configuration.storage = storage
|
|
23
41
|
@configuration = clean_configuration
|
|
42
|
+
@unencrypted_storage_warned = false
|
|
24
43
|
end
|
|
25
44
|
|
|
26
45
|
def idempotent(grape, required: false, &block)
|
|
27
46
|
validate_config!
|
|
47
|
+
warn_unencrypted_storage unless encryptor
|
|
28
48
|
log(:debug, "Performing endpoint \"#{grape.request.path}\" with idempotency.")
|
|
29
49
|
log(:debug, "Idempotency key is #{!required ? 'NOT' : ''} mandatory for this endpoint.")
|
|
30
50
|
|
|
31
51
|
idempotency_key = get_idempotency_key(grape.request.headers)
|
|
32
|
-
log(:debug, "Idempotency key received in request header \"#{configuration.idempotency_key_header.downcase}\"
|
|
52
|
+
log(:debug, "Idempotency key received in request header \"#{configuration.idempotency_key_header.downcase}\".") if idempotency_key
|
|
33
53
|
|
|
34
54
|
grape.error!(configuration.mandatory_header_response, 400) if required && !idempotency_key
|
|
35
55
|
return block.call if !idempotency_key
|
|
36
56
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
57
|
+
request_params_digest = params_digest(grape.request.params)
|
|
58
|
+
begin
|
|
59
|
+
stored_request = get_from_storage(idempotency_key)
|
|
60
|
+
rescue UnreadableRecordError
|
|
61
|
+
log(:error, "The request stored for the provided idempotency key could NOT be read. Check the encryption configuration.")
|
|
62
|
+
log(:debug, "Returning conflict error response.")
|
|
63
|
+
grape.error!(configuration.conflict_error_response, 422)
|
|
64
|
+
end
|
|
65
|
+
log(:debug, "Request has been found for the provided idempotency key.") if stored_request
|
|
66
|
+
same_params = stored_request && stored_params_digest(stored_request) == request_params_digest
|
|
67
|
+
same_path = stored_request && stored_request["path"] == grape.request.path
|
|
68
|
+
if stored_request && (!same_params || !same_path)
|
|
69
|
+
log(:debug, "Request has conflicts. Same params? => #{same_params}. Same path? => #{same_path}")
|
|
41
70
|
log(:debug, "Returning conflict error response.")
|
|
42
71
|
grape.error!(configuration.conflict_error_response, 422)
|
|
43
72
|
elsif stored_request && stored_request["processing"] == true
|
|
44
73
|
log(:debug, "Returning processing error response.")
|
|
45
74
|
grape.error!(configuration.processing_response, 409)
|
|
46
|
-
elsif stored_request
|
|
75
|
+
elsif stored_request && stored_request["v"].to_i >= STORAGE_SCHEMA_VERSION
|
|
47
76
|
log(:debug, "Returning the response from the original request.")
|
|
77
|
+
# Stop the endpoint here. The stored response replaces Grape's response in #finalize_response.
|
|
78
|
+
grape.env[REPLAY_ENV_KEY] = { idempotency_key: idempotency_key, stored_request: stored_request }
|
|
79
|
+
grape.error!('', stored_request["status"])
|
|
80
|
+
elsif stored_request
|
|
81
|
+
log(:debug, "Returning the response from the original request (stored by a previous version of the gem).")
|
|
48
82
|
grape.status stored_request["status"]
|
|
49
83
|
grape.header(ORIGINAL_REQUEST_HEADER, stored_request["original_request"])
|
|
50
84
|
grape.header(configuration.idempotency_key_header, idempotency_key)
|
|
@@ -54,7 +88,7 @@ module Grape
|
|
|
54
88
|
log(:debug, "Previous request information has NOT been found for the provided idempotency key.")
|
|
55
89
|
|
|
56
90
|
original_request_id = get_request_id(grape.request.headers)
|
|
57
|
-
success = store_processing_request(idempotency_key, grape.request.path,
|
|
91
|
+
success = store_processing_request(idempotency_key, grape.request.path, request_params_digest, original_request_id)
|
|
58
92
|
if !success
|
|
59
93
|
log(:error, "Request NOT stored as processing. Concurrent requests for the same idempotency key appeared.")
|
|
60
94
|
grape.error!(configuration.processing_response, 409)
|
|
@@ -62,66 +96,60 @@ module Grape
|
|
|
62
96
|
log(:debug, "Request stored as processing.")
|
|
63
97
|
end
|
|
64
98
|
|
|
99
|
+
# The final response (formatted body, status and headers) is stored by #finalize_response,
|
|
100
|
+
# once Grape has built it, both for the success path and for the rescue_from path.
|
|
101
|
+
grape.env[PENDING_ENV_KEY] = {
|
|
102
|
+
idempotency_key: idempotency_key,
|
|
103
|
+
path: grape.request.path,
|
|
104
|
+
params_digest: request_params_digest,
|
|
105
|
+
original_request: original_request_id
|
|
106
|
+
}
|
|
107
|
+
grape.header(ORIGINAL_REQUEST_HEADER, original_request_id)
|
|
108
|
+
grape.header(configuration.idempotency_key_header, idempotency_key)
|
|
109
|
+
|
|
65
110
|
log(:debug, "Performing the provided block.")
|
|
111
|
+
block.call
|
|
112
|
+
end
|
|
66
113
|
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
114
|
+
# Called with the final Rack response of an endpoint (see Grape::Idempotency::ResponseCapture).
|
|
115
|
+
# Stores it for a pending idempotent request, or replaces it with the stored one when replaying.
|
|
116
|
+
# It never raises: on any failure the original response is returned.
|
|
117
|
+
def finalize_response(env, response)
|
|
118
|
+
replay = env.delete(REPLAY_ENV_KEY)
|
|
119
|
+
pending = env.delete(PENDING_ENV_KEY)
|
|
120
|
+
return response unless replay || pending
|
|
70
121
|
|
|
71
|
-
|
|
122
|
+
parts = rack_response_parts(response)
|
|
72
123
|
|
|
73
|
-
if
|
|
74
|
-
|
|
75
|
-
response = response[:message]
|
|
76
|
-
end
|
|
124
|
+
if replay
|
|
125
|
+
return response unless parts
|
|
77
126
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
rescue Redis::BaseError => e
|
|
81
|
-
raise
|
|
82
|
-
rescue => e
|
|
83
|
-
log(:debug, "An unexpected error was raised when performing the block.")
|
|
84
|
-
if !stored_request && !response
|
|
85
|
-
validate_config!
|
|
86
|
-
log(:debug, "Storing error response.")
|
|
87
|
-
original_request_id = get_request_id(grape.request.headers)
|
|
88
|
-
stored_key = store_error_request(idempotency_key, grape.request.path, grape.request.params, grape.status, original_request_id, e)
|
|
89
|
-
if stored_key
|
|
90
|
-
log(:debug, "Error response stored.")
|
|
91
|
-
grape.header(ORIGINAL_REQUEST_HEADER, original_request_id)
|
|
92
|
-
grape.header(configuration.idempotency_key_header, stored_key)
|
|
93
|
-
end
|
|
127
|
+
log(:debug, "Replaying the stored response.")
|
|
128
|
+
return replay_response(replay[:stored_request], replay[:idempotency_key], parts[1])
|
|
94
129
|
end
|
|
95
|
-
log(:debug, "Re-raising the error.")
|
|
96
|
-
raise
|
|
97
|
-
ensure
|
|
98
|
-
if !stored_request && response
|
|
99
|
-
validate_config!
|
|
100
|
-
log(:debug, "Storing response.")
|
|
101
|
-
stored_key = store_request_response(idempotency_key, grape.request.path, grape.request.params, grape.status, original_request_id, response)
|
|
102
|
-
if stored_key
|
|
103
|
-
log(:debug, "Response stored.")
|
|
104
|
-
grape.header(configuration.idempotency_key_header, stored_key)
|
|
105
|
-
end
|
|
106
|
-
end
|
|
107
|
-
end
|
|
108
130
|
|
|
109
|
-
|
|
110
|
-
|
|
131
|
+
if parts.nil? || !storable_body?(parts[2])
|
|
132
|
+
log(:warn, "The response body of \"#{pending[:path]}\" is a stream or a file. It is NOT stored and its idempotency key is released.")
|
|
133
|
+
release_idempotency_key(pending[:idempotency_key])
|
|
134
|
+
return response
|
|
135
|
+
end
|
|
111
136
|
|
|
112
|
-
|
|
113
|
-
|
|
137
|
+
status, headers, body = parts
|
|
138
|
+
log(:debug, "Storing response.")
|
|
139
|
+
store_request_response(pending, status, headers, body.join)
|
|
140
|
+
log(:debug, "Response stored.")
|
|
114
141
|
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
142
|
+
set_response_header(headers, ORIGINAL_REQUEST_HEADER, pending[:original_request])
|
|
143
|
+
set_response_header(headers, configuration.idempotency_key_header, pending[:idempotency_key])
|
|
144
|
+
response
|
|
145
|
+
rescue => e
|
|
146
|
+
log(:error, "The response could NOT be stored or replayed => #{e.class}")
|
|
147
|
+
response
|
|
148
|
+
end
|
|
120
149
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
log(:error, "Storage error => #{e.message} - #{e}")
|
|
150
|
+
# Kept for backwards compatibility. Responses generated by rescue_from handlers are now
|
|
151
|
+
# stored by #finalize_response.
|
|
152
|
+
def update_error_with_rescue_from_result(_error, _status, _response, error_key: nil)
|
|
125
153
|
nil
|
|
126
154
|
end
|
|
127
155
|
|
|
@@ -129,6 +157,21 @@ module Grape
|
|
|
129
157
|
|
|
130
158
|
def validate_config!
|
|
131
159
|
raise Configuration::Error.new("A Redis instance must be configured as cache storage") unless valid_storage?
|
|
160
|
+
validate_encryption_config!
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
def validate_encryption_config!
|
|
164
|
+
if configuration.encryption_key && configuration.encryptor
|
|
165
|
+
raise Configuration::Error.new("Configure either an encryption_key or an encryptor, not both")
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
if configuration.encryption_key && !(configuration.encryption_key.is_a?(String) && !configuration.encryption_key.empty?)
|
|
169
|
+
raise Configuration::Error.new("The encryption_key must be a non-empty String")
|
|
170
|
+
end
|
|
171
|
+
|
|
172
|
+
if configuration.encryptor && !(configuration.encryptor.respond_to?(:encrypt) && configuration.encryptor.respond_to?(:decrypt))
|
|
173
|
+
raise Configuration::Error.new("The encryptor must respond to encrypt and decrypt")
|
|
174
|
+
end
|
|
132
175
|
end
|
|
133
176
|
|
|
134
177
|
def valid_storage?
|
|
@@ -158,100 +201,179 @@ module Grape
|
|
|
158
201
|
value = storage.get(key(idempotency_key))
|
|
159
202
|
return unless value
|
|
160
203
|
|
|
161
|
-
|
|
204
|
+
deserialize(value)
|
|
162
205
|
rescue Redis::BaseError => e
|
|
163
206
|
log(:error, "Storage error => #{e.message} - #{e}")
|
|
164
207
|
return if configuration.manage_redis_exceptions
|
|
165
208
|
raise
|
|
166
209
|
end
|
|
167
210
|
|
|
168
|
-
def store_processing_request(idempotency_key, path,
|
|
211
|
+
def store_processing_request(idempotency_key, path, request_params_digest, request_id)
|
|
169
212
|
body = {
|
|
170
213
|
path: path,
|
|
171
|
-
|
|
214
|
+
params_digest: request_params_digest,
|
|
172
215
|
original_request: request_id,
|
|
173
216
|
processing: true
|
|
174
217
|
}
|
|
175
218
|
|
|
176
|
-
storage.set(key(idempotency_key), body
|
|
219
|
+
storage.set(key(idempotency_key), serialize(body), ex: configuration.expires_in, nx: true)
|
|
177
220
|
rescue Redis::BaseError => e
|
|
178
221
|
return true if configuration.manage_redis_exceptions
|
|
179
222
|
raise
|
|
180
223
|
end
|
|
181
224
|
|
|
182
|
-
def store_request_response(
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
225
|
+
def store_request_response(pending, status, headers, body)
|
|
226
|
+
utf8_body = body.dup.force_encoding(Encoding::UTF_8)
|
|
227
|
+
body_encoding = utf8_body.valid_encoding? ? BODY_ENCODING_UTF8 : BODY_ENCODING_BASE64
|
|
228
|
+
replayable_headers = replayable_headers(headers)
|
|
229
|
+
|
|
230
|
+
stored_response = {
|
|
231
|
+
v: STORAGE_SCHEMA_VERSION,
|
|
232
|
+
path: pending[:path],
|
|
233
|
+
params_digest: pending[:params_digest],
|
|
234
|
+
status: status.to_i,
|
|
235
|
+
original_request: pending[:original_request],
|
|
236
|
+
headers: replayable_headers,
|
|
237
|
+
body_encoding: body_encoding,
|
|
238
|
+
body: body_encoding == BODY_ENCODING_UTF8 ? utf8_body : [body].pack('m0')
|
|
189
239
|
}
|
|
240
|
+
# Read by versions of the gem that do not know the schema above (e.g. during a rolling deploy)
|
|
241
|
+
stored_response[:response] = legacy_response(replayable_headers, utf8_body) if body_encoding == BODY_ENCODING_UTF8
|
|
190
242
|
|
|
191
|
-
storage.set(key(idempotency_key),
|
|
243
|
+
storage.set(key(pending[:idempotency_key]), serialize(stored_response), ex: configuration.expires_in, nx: false)
|
|
192
244
|
|
|
193
|
-
idempotency_key
|
|
245
|
+
pending[:idempotency_key]
|
|
194
246
|
rescue Redis::BaseError => e
|
|
195
247
|
log(:error, "Storage error => #{e.message} - #{e}")
|
|
196
|
-
idempotency_key
|
|
248
|
+
pending[:idempotency_key]
|
|
197
249
|
end
|
|
198
250
|
|
|
199
|
-
def
|
|
200
|
-
|
|
201
|
-
path: path,
|
|
202
|
-
params: params,
|
|
203
|
-
status: status,
|
|
204
|
-
original_request: request_id,
|
|
205
|
-
error: {
|
|
206
|
-
class_name: error.class.to_s,
|
|
207
|
-
message: error.message
|
|
208
|
-
}
|
|
209
|
-
}.to_json
|
|
210
|
-
|
|
211
|
-
storage.set(error_key(idempotency_key), body, ex: 30, nx: false)
|
|
212
|
-
|
|
213
|
-
idempotency_key
|
|
251
|
+
def release_idempotency_key(idempotency_key)
|
|
252
|
+
storage.del(key(idempotency_key))
|
|
214
253
|
rescue Redis::BaseError => e
|
|
215
254
|
log(:error, "Storage error => #{e.message} - #{e}")
|
|
216
|
-
|
|
255
|
+
nil
|
|
217
256
|
end
|
|
218
257
|
|
|
219
|
-
def
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
request: request_with_error,
|
|
232
|
-
idempotency_key: key.gsub(error_key_prefix, '')
|
|
233
|
-
}
|
|
234
|
-
end
|
|
235
|
-
end.first
|
|
236
|
-
rescue Redis::BaseError => e
|
|
237
|
-
log(:error, "Storage error => #{e.message} - #{e}")
|
|
238
|
-
nil
|
|
258
|
+
def replay_response(stored_request, idempotency_key, headers)
|
|
259
|
+
headers.keys.each do |name|
|
|
260
|
+
headers.delete(name) if %w[content-type content-length].include?(name.to_s.downcase)
|
|
261
|
+
end
|
|
262
|
+
(stored_request["headers"] || {}).each { |name, value| set_response_header(headers, name, value) }
|
|
263
|
+
set_response_header(headers, ORIGINAL_REQUEST_HEADER, stored_request["original_request"])
|
|
264
|
+
set_response_header(headers, configuration.idempotency_key_header, idempotency_key)
|
|
265
|
+
|
|
266
|
+
body = stored_request["body"].to_s
|
|
267
|
+
body = body.unpack1('m0') if stored_request["body_encoding"] == BODY_ENCODING_BASE64
|
|
268
|
+
|
|
269
|
+
[stored_request["status"].to_i, headers, [body]]
|
|
239
270
|
end
|
|
240
271
|
|
|
241
|
-
def
|
|
242
|
-
response.is_a?(
|
|
272
|
+
def rack_response_parts(response)
|
|
273
|
+
if response.is_a?(Rack::Response)
|
|
274
|
+
[response.status, response.headers, response.body]
|
|
275
|
+
elsif response.is_a?(Array) && response.size == 3
|
|
276
|
+
response
|
|
277
|
+
end
|
|
243
278
|
end
|
|
244
279
|
|
|
245
|
-
|
|
246
|
-
|
|
280
|
+
# Streams and files are not buffered, so they cannot be stored
|
|
281
|
+
def storable_body?(body)
|
|
282
|
+
body.is_a?(Array) && body.all? { |chunk| chunk.is_a?(String) }
|
|
283
|
+
end
|
|
284
|
+
|
|
285
|
+
def replayable_headers(headers)
|
|
286
|
+
excluded = excluded_headers
|
|
287
|
+
headers.each_with_object({}) do |(name, value), replayable|
|
|
288
|
+
replayable[name.to_s] = value unless excluded.include?(name.to_s.downcase)
|
|
289
|
+
end
|
|
290
|
+
end
|
|
291
|
+
|
|
292
|
+
def excluded_headers
|
|
293
|
+
EXCLUDED_REPLAY_HEADERS +
|
|
294
|
+
[configuration.idempotency_key_header, configuration.request_id_header].map(&:downcase) +
|
|
295
|
+
Array(configuration.replay_headers_denylist).map { |name| name.to_s.downcase }
|
|
296
|
+
end
|
|
297
|
+
|
|
298
|
+
def legacy_response(headers, body)
|
|
299
|
+
content_type = headers.find { |name, _| name.downcase == 'content-type' }&.last.to_s
|
|
300
|
+
return body unless content_type =~ /\bjson\b/i
|
|
301
|
+
|
|
302
|
+
JSON.parse(body)
|
|
303
|
+
rescue JSON::ParserError
|
|
304
|
+
body
|
|
305
|
+
end
|
|
306
|
+
|
|
307
|
+
def set_response_header(headers, name, value)
|
|
308
|
+
headers.keys.each { |existing| headers.delete(existing) if existing.to_s.casecmp?(name) }
|
|
309
|
+
# Rack 3 requires lowercase header names
|
|
310
|
+
headers[defined?(Rack::Headers) ? name.downcase : name] = value
|
|
311
|
+
end
|
|
312
|
+
|
|
313
|
+
def serialize(record)
|
|
314
|
+
json = record.to_json
|
|
315
|
+
return json unless encryptor
|
|
316
|
+
|
|
317
|
+
"#{ENCRYPTED_RECORD_PREFIX}#{encryptor.encrypt(json)}"
|
|
247
318
|
end
|
|
248
319
|
|
|
249
|
-
def
|
|
250
|
-
|
|
320
|
+
def deserialize(value)
|
|
321
|
+
if value.start_with?(ENCRYPTED_RECORD_PREFIX)
|
|
322
|
+
raise UnreadableRecordError, "Stored request is encrypted but no encryption is configured" unless encryptor
|
|
323
|
+
|
|
324
|
+
value = encryptor.decrypt(value.delete_prefix(ENCRYPTED_RECORD_PREFIX))
|
|
325
|
+
end
|
|
326
|
+
|
|
327
|
+
# Records stored before configuring the encryption are not encrypted and are still accepted
|
|
328
|
+
JSON.parse(value)
|
|
329
|
+
rescue UnreadableRecordError
|
|
330
|
+
raise
|
|
331
|
+
rescue StandardError => e
|
|
332
|
+
raise UnreadableRecordError, "Stored request could not be read (#{e.class})"
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
def encryptor
|
|
336
|
+
return configuration.encryptor if configuration.encryptor
|
|
337
|
+
return unless configuration.encryption_key
|
|
338
|
+
|
|
339
|
+
if @encryptor.nil? || @encryptor_key != configuration.encryption_key
|
|
340
|
+
@encryptor = Encryptor.new(configuration.encryption_key)
|
|
341
|
+
@encryptor_key = configuration.encryption_key
|
|
342
|
+
end
|
|
343
|
+
@encryptor
|
|
344
|
+
end
|
|
345
|
+
|
|
346
|
+
def warn_unencrypted_storage
|
|
347
|
+
return if @unencrypted_storage_warned
|
|
348
|
+
|
|
349
|
+
@unencrypted_storage_warned = true
|
|
350
|
+
log(:warn, "Stored requests and responses are NOT encrypted. Configure an encryption_key or an encryptor to encrypt them.")
|
|
251
351
|
end
|
|
252
352
|
|
|
253
|
-
def
|
|
254
|
-
|
|
353
|
+
def params_digest(params)
|
|
354
|
+
digest(JSON.generate(canonical(JSON.parse(params.to_json))))
|
|
355
|
+
end
|
|
356
|
+
|
|
357
|
+
def stored_params_digest(stored_request)
|
|
358
|
+
# Requests stored by versions < 1.2.0 contain the params instead of their digest
|
|
359
|
+
stored_request["params_digest"] || params_digest(stored_request["params"])
|
|
360
|
+
end
|
|
361
|
+
|
|
362
|
+
# Sorts hash keys so the digest does not depend on the params order
|
|
363
|
+
def canonical(value)
|
|
364
|
+
case value
|
|
365
|
+
when Hash then value.keys.sort.map { |k| [k, canonical(value[k])] }
|
|
366
|
+
when Array then value.map { |v| canonical(v) }
|
|
367
|
+
else value
|
|
368
|
+
end
|
|
369
|
+
end
|
|
370
|
+
|
|
371
|
+
def digest(value)
|
|
372
|
+
Digest::SHA256.hexdigest(value)
|
|
373
|
+
end
|
|
374
|
+
|
|
375
|
+
def key(idempotency_key)
|
|
376
|
+
"#{gem_prefix}#{idempotency_key}"
|
|
255
377
|
end
|
|
256
378
|
|
|
257
379
|
def gem_prefix
|
|
@@ -267,7 +389,7 @@ module Grape
|
|
|
267
389
|
end
|
|
268
390
|
|
|
269
391
|
def logger
|
|
270
|
-
return @logger if @logger
|
|
392
|
+
return @logger if @logger && @logger.equal?(configuration.logger)
|
|
271
393
|
|
|
272
394
|
@logger = configuration.logger
|
|
273
395
|
@logger.level = configuration.logger_level
|
|
@@ -282,7 +404,7 @@ module Grape
|
|
|
282
404
|
class Configuration
|
|
283
405
|
attr_accessor :storage, :logger, :logger_level, :logger_prefix, :expires_in, :idempotency_key_header,
|
|
284
406
|
:request_id_header, :conflict_error_response, :processing_response, :mandatory_header_response,
|
|
285
|
-
:manage_redis_exceptions
|
|
407
|
+
:manage_redis_exceptions, :replay_headers_denylist, :encryption_key, :encryptor
|
|
286
408
|
|
|
287
409
|
class Error < StandardError; end
|
|
288
410
|
|
|
@@ -295,6 +417,9 @@ module Grape
|
|
|
295
417
|
@idempotency_key_header = "idempotency-key"
|
|
296
418
|
@request_id_header = "x-request-id"
|
|
297
419
|
@manage_redis_exceptions = false
|
|
420
|
+
@replay_headers_denylist = []
|
|
421
|
+
@encryption_key = nil
|
|
422
|
+
@encryptor = nil
|
|
298
423
|
@conflict_error_response = {
|
|
299
424
|
"title" => "Idempotency-Key is already used",
|
|
300
425
|
"detail" => "This operation is idempotent and it requires correct usage of Idempotency Key. Idempotency Key MUST not be reused across different payloads of this operation."
|
|
@@ -310,4 +435,4 @@ module Grape
|
|
|
310
435
|
end
|
|
311
436
|
end
|
|
312
437
|
end
|
|
313
|
-
end
|
|
438
|
+
end
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: grape-idempotency
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Juan Carlos García
|
|
8
|
-
autorequire:
|
|
8
|
+
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date:
|
|
11
|
+
date: 2026-10-15 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: grape
|
|
@@ -94,11 +94,13 @@ dependencies:
|
|
|
94
94
|
- - ">="
|
|
95
95
|
- !ruby/object:Gem::Version
|
|
96
96
|
version: '0'
|
|
97
|
-
description:
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
97
|
+
description: Idempotency-Key header support for Grape REST APIs, based on the IETF
|
|
98
|
+
Idempotency-Key HTTP header draft. Wrap a POST or PATCH endpoint in an idempotent
|
|
99
|
+
block and clients can safely retry requests after timeouts or network errors without
|
|
100
|
+
creating duplicate payments, orders or records. The first response (status, headers
|
|
101
|
+
and body) is stored in Redis and replayed for retries with the same key. Concurrent
|
|
102
|
+
duplicate requests get a 409 Conflict, and reusing a key with a different payload
|
|
103
|
+
gets a 422 Unprocessable Entity. Works with Grape 1.x to 4.x and Ruby 2.6+.
|
|
102
104
|
email:
|
|
103
105
|
- jugade92@gmail.com
|
|
104
106
|
executables: []
|
|
@@ -111,14 +113,21 @@ files:
|
|
|
111
113
|
- grape-idempotency.gemspec
|
|
112
114
|
- lib/grape-idempotency.rb
|
|
113
115
|
- lib/grape/idempotency.rb
|
|
116
|
+
- lib/grape/idempotency/encryptor.rb
|
|
114
117
|
- lib/grape/idempotency/helpers.rb
|
|
115
118
|
- lib/grape/idempotency/middleware/error.rb
|
|
119
|
+
- lib/grape/idempotency/middleware/response_capture.rb
|
|
116
120
|
- lib/grape/idempotency/version.rb
|
|
117
121
|
homepage: https://github.com/jcagarcia/grape-idempotency
|
|
118
122
|
licenses:
|
|
119
123
|
- MIT
|
|
120
|
-
metadata:
|
|
121
|
-
|
|
124
|
+
metadata:
|
|
125
|
+
homepage_uri: https://github.com/jcagarcia/grape-idempotency
|
|
126
|
+
source_code_uri: https://github.com/jcagarcia/grape-idempotency
|
|
127
|
+
changelog_uri: https://github.com/jcagarcia/grape-idempotency/blob/main/CHANGELOG.md
|
|
128
|
+
bug_tracker_uri: https://github.com/jcagarcia/grape-idempotency/issues
|
|
129
|
+
documentation_uri: https://github.com/jcagarcia/grape-idempotency#readme
|
|
130
|
+
post_install_message:
|
|
122
131
|
rdoc_options: []
|
|
123
132
|
require_paths:
|
|
124
133
|
- lib
|
|
@@ -134,7 +143,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
134
143
|
version: '0'
|
|
135
144
|
requirements: []
|
|
136
145
|
rubygems_version: 3.4.10
|
|
137
|
-
signing_key:
|
|
146
|
+
signing_key:
|
|
138
147
|
specification_version: 4
|
|
139
|
-
summary:
|
|
148
|
+
summary: 'Idempotency-Key support for Grape APIs: safe retries and duplicate request
|
|
149
|
+
prevention backed by Redis'
|
|
140
150
|
test_files: []
|