gitlab-labkit 3.0.1 → 4.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/lib/labkit/rate_limit/README.md +95 -41
- data/lib/labkit/rate_limit/evaluator.rb +54 -40
- data/lib/labkit/rate_limit/limiter.rb +1 -1
- data/lib/labkit/rate_limit/metrics.rb +4 -3
- data/lib/labkit/rate_limit/result.rb +118 -19
- data/lib/labkit/rate_limit/rule.rb +8 -7
- data/lib/labkit/rate_limit.rb +2 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c872947164d25ef7f75203c6195fe04bbc8c9002639d59fadf10672d47201862
|
|
4
|
+
data.tar.gz: d0308016fd538669e466d2e499aa283d47e82bb4d53bd530c07bce74162b16b4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e6cb4000c77a3ebe764b61e0d536db95dbe3c79378a1d95744747e7eff2509cb435561eba2963ae83107eea37364f5515e3a64bd9491c799c2b9508c9293c793
|
|
7
|
+
data.tar.gz: 64cc36f40cb6140b09fc54d873a102098a7fe05b2e967960317c782dc53b224c513fe676e6435dae48b6b0fc33e9e95dd8f32f10230900dea809bbef8bdd03e0
|
|
@@ -77,8 +77,8 @@ RACK_LIMITER = Labkit::RateLimit::Limiter.new(
|
|
|
77
77
|
- `name` must match `/\A[a-z0-9_]+\z/`. It is used as the first segment of
|
|
78
78
|
every Redis counter key for this limiter, so renaming a `Limiter` abandons
|
|
79
79
|
any in-flight counters.
|
|
80
|
-
- `rules` is an ordered array of `Rule` objects.
|
|
81
|
-
|
|
80
|
+
- `rules` is an ordered array of `Rule` objects. Every rule whose `match` hash
|
|
81
|
+
is satisfied is evaluated and counted (see [Evaluation flow](#evaluation-flow)).
|
|
82
82
|
- `redis` and `logger` are optional; they fall back to the global
|
|
83
83
|
`Labkit::RateLimit.config` values.
|
|
84
84
|
|
|
@@ -99,7 +99,7 @@ result = RACK_LIMITER.check(
|
|
|
99
99
|
endpoint: request.path
|
|
100
100
|
)
|
|
101
101
|
|
|
102
|
-
if result.
|
|
102
|
+
if result.action == :block
|
|
103
103
|
response.headers.merge!(result.to_response_headers)
|
|
104
104
|
render plain: "Too Many Requests", status: 429
|
|
105
105
|
return
|
|
@@ -115,9 +115,11 @@ the same counter.
|
|
|
115
115
|
`Limiter#peek(identifier)` returns the same `Result` shape but does not
|
|
116
116
|
mutate Redis. It is useful when one code path should account for the request
|
|
117
117
|
(`check`) and another should gate a side-effect on whether the caller is
|
|
118
|
-
already over-limit. `peek`
|
|
119
|
-
|
|
120
|
-
|
|
118
|
+
already over-limit. `peek` reads `:log` rules too: `check` can report a
|
|
119
|
+
`:log` rule's evaluation, so excluding them would make `peek` answer a
|
|
120
|
+
different question than `check`. A matched `:skip` rule terminates `peek`
|
|
121
|
+
the same way it terminates `check`: matched, `:allow`, no Redis read, no
|
|
122
|
+
`info`.
|
|
121
123
|
|
|
122
124
|
## Identifier
|
|
123
125
|
|
|
@@ -146,7 +148,7 @@ A `Rule` is a `Data.define` value object with the following fields:
|
|
|
146
148
|
| `match` | Hash of identifier key/value predicates that must **all** be satisfied for the rule to apply. Empty hash matches anything. See [Matchers](#matchers). |
|
|
147
149
|
| `limit` | Integer request threshold per `period`. May be a callable resolved on every check. |
|
|
148
150
|
| `period` | Window length in seconds. May be a callable resolved on every check. |
|
|
149
|
-
| `action` | What the
|
|
151
|
+
| `action` | What the rule does when it matches. One of `:limit`, `:log`, `:skip`. Default `:limit`. See [Actions](#actions). |
|
|
150
152
|
| `characteristics` | Array of identifier keys whose values are folded into the Redis counter key. Each unique combination gets its own counter. |
|
|
151
153
|
|
|
152
154
|
Making `limit` or `period` callable is the supported pattern for
|
|
@@ -180,10 +182,37 @@ Glob and prefix matchers are intentionally out of scope.
|
|
|
180
182
|
|
|
181
183
|
### Evaluation flow
|
|
182
184
|
|
|
183
|
-
`check` walks the rule list in order
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
185
|
+
`check` walks the rule list in order and evaluates **every** rule that matches:
|
|
186
|
+
each one increments its own counter, and the request debits `cost` from all of
|
|
187
|
+
them. Matching alone does not stop the walk. Only two things terminate it:
|
|
188
|
+
|
|
189
|
+
- a matched `:skip` rule (bypass, nothing counted), and
|
|
190
|
+
- a `:limit` rule that is **over its limit** — the request is rejected, so the
|
|
191
|
+
remaining rules cannot change the outcome. Rules declared after it are neither
|
|
192
|
+
counted nor evaluated, meaning a blocked request debits every rule up to and
|
|
193
|
+
including the one that blocked, and none after it.
|
|
194
|
+
|
|
195
|
+
A `:limit` rule that is under its limit does not terminate, so it behaves
|
|
196
|
+
exactly like `:log` until the moment it blocks.
|
|
197
|
+
|
|
198
|
+
The returned `Result` collects one `Result::Evaluation` per counted rule
|
|
199
|
+
(`result.evaluations`), and its readers (`rule`, `info`, `exceeded?`,
|
|
200
|
+
`to_response_headers`) report the **most constraining** one: a blocking
|
|
201
|
+
evaluation first, then exceeded ones, then the fewest requests remaining,
|
|
202
|
+
ties broken by declaration order (the ranking is `Result::Evaluation#<=>`).
|
|
203
|
+
That is what keeps `to_response_headers` honest — reporting any other matched
|
|
204
|
+
rule would advertise headroom that a different rule is about to refuse:
|
|
205
|
+
|
|
206
|
+
```
|
|
207
|
+
per_ip limit 1000 count 20 remaining 980
|
|
208
|
+
per_user limit 100 count 99 remaining 1 <- reported
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
An exceeded rule has `remaining` 0 and therefore outranks any rule still under
|
|
212
|
+
its limit. For a `:log` rule that means shadow traffic surfaces to the caller
|
|
213
|
+
as `exceeded? == true` (with `action` still `:allow`) — visible, but unable to
|
|
214
|
+
block. A `:log`-only path that matches therefore returns `matched? == true`
|
|
215
|
+
and emits no `rule="unmatched"` metric.
|
|
187
216
|
|
|
188
217
|
```mermaid
|
|
189
218
|
flowchart TD
|
|
@@ -194,31 +223,53 @@ flowchart TD
|
|
|
194
223
|
Skip -->|"yes (no Redis op)"| SkipEmit[Emit calls_total<br/>action=skip]
|
|
195
224
|
SkipEmit --> SkipReturn([Return matched=true<br/>action=:allow])
|
|
196
225
|
Skip -->|no| Eval["INCR Redis counter<br/>(see Redis sequence below)"]
|
|
197
|
-
Eval --> Build[Build
|
|
198
|
-
Build -->
|
|
199
|
-
|
|
200
|
-
Act
|
|
201
|
-
Act
|
|
202
|
-
|
|
226
|
+
Eval --> Build[Build Evaluation<br/>resolve limit/period]
|
|
227
|
+
Build --> Add[Add evaluation to Result]
|
|
228
|
+
Add --> Emit[Emit calls_total + limit/period gauges]
|
|
229
|
+
Emit --> Act{"result.block?<br/>(:limit rule over limit)"}
|
|
230
|
+
Act -->|yes| Return([Return Result<br/>action=:block])
|
|
231
|
+
Act -->|"no (:log, or :limit under limit)"| Iter
|
|
232
|
+
Iter -->|no more rules| Any{any rule<br/>evaluated?}
|
|
233
|
+
Any -->|yes| ReturnFold([Return Result reporting<br/>most-constraining evaluation])
|
|
234
|
+
Any -->|no| Unmatched[Emit calls_total<br/>rule=unmatched, action=allow]
|
|
203
235
|
Unmatched --> ReturnUnmatched([Return matched=false<br/>action=:allow])
|
|
204
236
|
Eval -. StandardError .-> Error[Emit errors_total<br/>log warn]
|
|
205
237
|
Error --> ReturnErr([Return error=true<br/>action=:allow])
|
|
206
238
|
```
|
|
207
239
|
|
|
240
|
+
Error handling is whole-check, not per-rule: a Redis failure part-way through
|
|
241
|
+
discards the verdicts of the rules already evaluated, even though their counters
|
|
242
|
+
were incremented. Those requests are counted but produce no verdict — the
|
|
243
|
+
fail-open trade-off is that a request is never blocked on a partial evaluation.
|
|
244
|
+
|
|
245
|
+
Note that `calls_total` is emitted **per matched rule**, so summing it by
|
|
246
|
+
`rate_limiter` counts rule evaluations, not requests.
|
|
247
|
+
|
|
208
248
|
### Actions
|
|
209
249
|
|
|
210
|
-
The rule's `action`
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
250
|
+
The rule's `action` describes what the rule does; the result's `action`
|
|
251
|
+
describes the outcome — what the caller should do — and is only ever `:allow`
|
|
252
|
+
or `:block`. The counter is always incremented when a rule matches, except for
|
|
253
|
+
`:skip` rules, which never touch Redis:
|
|
254
|
+
|
|
255
|
+
| rule action | what it does | exceeded? | result action | terminating? |
|
|
256
|
+
|-------------|-----------------------------------------------|-----------|---------------|---------------|
|
|
257
|
+
| `:limit` | count against the limit | no | `:allow` | no — continue |
|
|
258
|
+
| `:limit` | count against the limit | yes | `:block` | yes — stop |
|
|
259
|
+
| `:log` | count against the limit (observability only) | no | `:allow` | no — continue |
|
|
260
|
+
| `:log` | count against the limit (observability only) | yes | `:allow` | no — continue |
|
|
261
|
+
| `:skip` | don't count (bypass) | n/a | `:allow` | yes — stop |
|
|
262
|
+
|
|
263
|
+
- `:limit` — when exceeded, `Result#action` is `:block` and evaluation
|
|
264
|
+
terminates. Caller should reject the request (e.g. with HTTP 429). When under
|
|
265
|
+
the limit, evaluation continues to the next rule.
|
|
266
|
+
- `:log` — **never terminating and never blocking**. The rule counts the
|
|
267
|
+
request and records metrics, but evaluation always continues to the next
|
|
268
|
+
rule. This is the mechanism for shadow rules during rollout: stack a `:log`
|
|
269
|
+
rule and a `:limit` rule together and the `:log` rule cannot disable the
|
|
270
|
+
`:limit` rule. When exceeded, a `:log` rule can still be the rule reported in
|
|
271
|
+
the `Result` (`exceeded? == true`, `action == :allow`) if nothing more
|
|
272
|
+
constraining matched.
|
|
222
273
|
- `:skip` — bypass. A matching rule terminates evaluation with
|
|
223
274
|
`Result#action` `:allow` **without any Redis operation**: nothing is
|
|
224
275
|
counted, so `limit`, `period`, `characteristics`, and `count_distinct` are
|
|
@@ -277,21 +328,25 @@ no reason to read it again before mutating.
|
|
|
277
328
|
|
|
278
329
|
`peek` does not use a script: it pipelines `GET` + `TTL` (or `SCARD` + `TTL`)
|
|
279
330
|
and never issues `EXPIRE`, so it cannot start or extend a window. A missing
|
|
280
|
-
key (`GET → nil`, `TTL → -2`) is reported as `count = 0`, and `
|
|
331
|
+
key (`GET → nil`, `TTL → -2`) is reported as `count = 0`, and `build_evaluation`
|
|
281
332
|
falls back to the rule's period for `reset_at` since there is no Redis-side
|
|
282
333
|
window to read.
|
|
283
334
|
|
|
284
335
|
## Result
|
|
285
336
|
|
|
286
|
-
`Result` carries the decision back to the caller
|
|
337
|
+
`Result` carries the decision back to the caller. It collects one
|
|
338
|
+
`Result::Evaluation` per counted rule; `rule`, `exceeded?`, and `info` report
|
|
339
|
+
the most constraining one (see [Evaluation flow](#evaluation-flow)):
|
|
287
340
|
|
|
288
341
|
```ruby
|
|
289
342
|
result.matched? # => true if some rule matched
|
|
290
|
-
result.exceeded? # => true if the
|
|
291
|
-
result.action # => :block | :
|
|
292
|
-
result.
|
|
343
|
+
result.exceeded? # => true if the reported rule's counter > limit
|
|
344
|
+
result.action # => :block | :allow — what the caller should do
|
|
345
|
+
result.block? # => result.action == :block
|
|
346
|
+
result.rule # => the reported Rule, or nil
|
|
293
347
|
result.error? # => true if Redis failed (see Fail-open)
|
|
294
348
|
result.info # => Result::Info or nil
|
|
349
|
+
result.evaluations # => every counted Result::Evaluation, in rule order
|
|
295
350
|
result.to_response_headers
|
|
296
351
|
# => { "RateLimit-Limit" => "...", "RateLimit-Remaining" => "...", "RateLimit-Reset" => "<unix-ts>" }
|
|
297
352
|
```
|
|
@@ -313,8 +368,8 @@ safe to merge unconditionally.
|
|
|
313
368
|
|
|
314
369
|
The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
|
|
315
370
|
(Redis connection failure, timeout, OOM in user-supplied callables, …) is
|
|
316
|
-
logged at WARN with `message: "rate_limit_error"` and returned as
|
|
317
|
-
`Result(matched
|
|
371
|
+
logged at WARN with `message: "rate_limit_error"` and returned as an error
|
|
372
|
+
`Result` (`matched?` false, `error?` true, `action` `:allow`). The
|
|
318
373
|
`gitlab_labkit_rate_limiter_errors_total` counter is incremented. The caller
|
|
319
374
|
should treat the request as allowed.
|
|
320
375
|
|
|
@@ -325,15 +380,14 @@ should treat the request as allowed.
|
|
|
325
380
|
|
|
326
381
|
| metric | type | labels | meaning |
|
|
327
382
|
|-------------------------------------------------|---------|-------------------------------------|----------------------------------------------------------------------|
|
|
328
|
-
| `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | One increment per
|
|
383
|
+
| `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | One increment per counted rule (plus one per matched `:skip` rule). `action` is the rule-level outcome: `"allow"` (under limit), `"limit"` (blocking `:limit` rule), `"log"` (exceeded `:log` rule), `"skip"`. `rule="unmatched", action="allow"` when no rule matched. |
|
|
329
384
|
| `gitlab_labkit_rate_limiter_errors_total` | counter | `rate_limiter` | Fail-open events (any `StandardError` in the labkit path). |
|
|
330
385
|
| `gitlab_labkit_rate_limiter_limit` | gauge | `rate_limiter`, `rule` | Resolved limit at the last check (useful when `limit:` is callable). |
|
|
331
386
|
| `gitlab_labkit_rate_limiter_period_seconds` | gauge | `rate_limiter`, `rule` | Resolved period at the last check. |
|
|
332
387
|
|
|
333
|
-
Because
|
|
334
|
-
**multiple** `calls_total` increments: one per
|
|
335
|
-
|
|
336
|
-
rule fired).
|
|
388
|
+
Because every matching rule is counted, a single `check` call can emit
|
|
389
|
+
**multiple** `calls_total` increments: one per counted rule (or a single
|
|
390
|
+
`rule="unmatched"` increment if nothing matched).
|
|
337
391
|
|
|
338
392
|
## Dev/test vs production guards
|
|
339
393
|
|
|
@@ -74,7 +74,7 @@ module Labkit
|
|
|
74
74
|
# timeout, OOM) not only Redis protocol errors.
|
|
75
75
|
report_error_metrics
|
|
76
76
|
log_error(e, identifier)
|
|
77
|
-
Result.
|
|
77
|
+
Result.error
|
|
78
78
|
end
|
|
79
79
|
|
|
80
80
|
# Read-without-increment counterpart to {#check}. Same matching and Result
|
|
@@ -85,30 +85,48 @@ module Labkit
|
|
|
85
85
|
rescue StandardError => e
|
|
86
86
|
report_error_metrics
|
|
87
87
|
log_error(e, identifier)
|
|
88
|
-
Result.
|
|
88
|
+
Result.error
|
|
89
89
|
end
|
|
90
90
|
|
|
91
91
|
private
|
|
92
92
|
|
|
93
|
-
#
|
|
94
|
-
#
|
|
93
|
+
# Every rule that matches is evaluated and counted; matching does not
|
|
94
|
+
# short-circuit the loop. Exactly two cases terminate it early:
|
|
95
95
|
#
|
|
96
|
-
# :skip
|
|
97
|
-
#
|
|
98
|
-
#
|
|
99
|
-
#
|
|
96
|
+
# - :skip terminates on match without touching Redis. No counter is
|
|
97
|
+
# incremented, so the branch sits before the count_distinct check
|
|
98
|
+
# (identifier completeness is irrelevant to a rule that builds no key).
|
|
99
|
+
# calls_total still increments so the bypass stays observable.
|
|
100
|
+
# - a :limit rule over its limit terminates, because the request is
|
|
101
|
+
# rejected and later rules cannot change that. Rules declared after it
|
|
102
|
+
# are neither counted nor evaluated, so a blocked request debits every
|
|
103
|
+
# rule up to and including the one that blocked, and none after it.
|
|
104
|
+
#
|
|
105
|
+
# Every other matched rule - :log, and :limit while under its limit - is
|
|
106
|
+
# counted and collected into the returned Result, which reports the
|
|
107
|
+
# most-constraining evaluation (ranking lives in Result::Evaluation#<=>).
|
|
108
|
+
# cost is therefore debited from every matching rule, not just the first.
|
|
100
109
|
#
|
|
101
110
|
# SET-mode rules (rule.count_distinct set) that match but whose identifier
|
|
102
111
|
# is missing the count_distinct key fail open + log + bump errors_total, and
|
|
103
112
|
# the loop continues to the next rule (the rule is treated as not applicable
|
|
104
113
|
# rather than aborting the whole evaluation).
|
|
114
|
+
#
|
|
115
|
+
# Error handling stays whole-check (see #check): a raise part-way through
|
|
116
|
+
# discards the results of the rules already evaluated, even though their
|
|
117
|
+
# counters were incremented. No blocking verdict is lost that way - a
|
|
118
|
+
# :limit rule over its limit returns before any later rule can raise -
|
|
119
|
+
# but a rule declared after the failing one loses its chance to block.
|
|
120
|
+
# The request is counted and allowed.
|
|
105
121
|
def check_rules(identifier, cost, rule_context)
|
|
122
|
+
result = Result.new
|
|
123
|
+
|
|
106
124
|
@rules.each do |rule|
|
|
107
125
|
next unless rule_matches?(rule, identifier)
|
|
108
126
|
|
|
109
127
|
if rule.action == :skip
|
|
110
128
|
report_skipped_metrics(rule)
|
|
111
|
-
return
|
|
129
|
+
return result.skip!(rule)
|
|
112
130
|
end
|
|
113
131
|
|
|
114
132
|
if rule.count_distinct && missing_count_distinct_value?(rule, identifier)
|
|
@@ -117,44 +135,42 @@ module Labkit
|
|
|
117
135
|
next
|
|
118
136
|
end
|
|
119
137
|
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
138
|
+
evaluation = evaluate_rule(rule, identifier, cost, rule_context)
|
|
139
|
+
result.add_evaluation(evaluation)
|
|
140
|
+
report_matched_metrics(evaluation)
|
|
141
|
+
return result if result.block?
|
|
123
142
|
end
|
|
124
143
|
|
|
125
|
-
report_unmatched_metrics
|
|
126
|
-
|
|
144
|
+
report_unmatched_metrics unless result.matched?
|
|
145
|
+
result
|
|
127
146
|
end
|
|
128
147
|
|
|
129
|
-
# Mirror of check_rules without metrics
|
|
130
|
-
#
|
|
148
|
+
# Mirror of check_rules without metrics or writes. :log rules are read
|
|
149
|
+
# here too: check can report a :log rule's evaluation, so excluding them
|
|
150
|
+
# would make peek answer a different question than check.
|
|
131
151
|
#
|
|
132
152
|
# peek does not need the count_distinct identifier key - it reads SCARD on
|
|
133
153
|
# the rule-keyed compound key, which contains the cardinality across all
|
|
134
154
|
# members. So missing-key fail-open does not apply here.
|
|
135
155
|
def peek_rules(identifier, rule_context)
|
|
156
|
+
result = Result.new
|
|
157
|
+
|
|
136
158
|
@rules.each do |rule|
|
|
137
|
-
next if rule.action == :log
|
|
138
159
|
next unless rule_matches?(rule, identifier)
|
|
139
160
|
|
|
140
|
-
return
|
|
161
|
+
return result.skip!(rule) if rule.action == :skip
|
|
141
162
|
|
|
142
|
-
|
|
163
|
+
result.add_evaluation(peek_rule(rule, identifier, rule_context))
|
|
164
|
+
return result if result.block?
|
|
143
165
|
end
|
|
144
166
|
|
|
145
|
-
|
|
167
|
+
result
|
|
146
168
|
end
|
|
147
169
|
|
|
148
170
|
def rule_matches?(rule, identifier)
|
|
149
171
|
rule.match.all? { |key, matcher| matcher.match?(identifier[key]) }
|
|
150
172
|
end
|
|
151
173
|
|
|
152
|
-
# A matched :skip rule allows without evaluating: no counter exists, so
|
|
153
|
-
# info is nil (to_response_headers is {} for skip results).
|
|
154
|
-
def skip_result(rule)
|
|
155
|
-
Result.new(matched: true, action: :allow, rule: rule)
|
|
156
|
-
end
|
|
157
|
-
|
|
158
174
|
def missing_count_distinct_value?(rule, identifier)
|
|
159
175
|
value = identifier[rule.count_distinct]
|
|
160
176
|
value.nil? || value.to_s.empty?
|
|
@@ -174,7 +190,7 @@ module Labkit
|
|
|
174
190
|
incr_with_ttl(redis_key, resolved_period, cost)
|
|
175
191
|
end
|
|
176
192
|
|
|
177
|
-
|
|
193
|
+
build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
|
|
178
194
|
end
|
|
179
195
|
|
|
180
196
|
def peek_rule(rule, identifier, rule_context)
|
|
@@ -183,12 +199,10 @@ module Labkit
|
|
|
183
199
|
resolved_period = Integer(resolve_value(rule.period, rule_context))
|
|
184
200
|
|
|
185
201
|
count, ttl = rule.count_distinct ? scard_with_ttl(redis_key) : read_with_ttl(redis_key)
|
|
186
|
-
|
|
202
|
+
build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
|
|
187
203
|
end
|
|
188
204
|
|
|
189
|
-
def
|
|
190
|
-
exceeded = count > resolved_limit
|
|
191
|
-
action = exceeded ? rule.action : :allow
|
|
205
|
+
def build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
|
|
192
206
|
info = Result::Info.new(
|
|
193
207
|
resolved_limit: resolved_limit, resolved_period: resolved_period,
|
|
194
208
|
count: count,
|
|
@@ -196,7 +210,7 @@ module Labkit
|
|
|
196
210
|
reset_at: Time.now.utc + (ttl >= 0 ? ttl : resolved_period)
|
|
197
211
|
)
|
|
198
212
|
|
|
199
|
-
Result.new(
|
|
213
|
+
Result::Evaluation.new(rule: rule, exceeded: count > resolved_limit, info: info)
|
|
200
214
|
end
|
|
201
215
|
|
|
202
216
|
def build_redis_key(rule, identifier)
|
|
@@ -264,7 +278,7 @@ module Labkit
|
|
|
264
278
|
|
|
265
279
|
# Pipelined GET + TTL. No EXPIRE: peek must not extend the window.
|
|
266
280
|
# A missing key (GET => nil, TTL => -2) is reported as count=0; the
|
|
267
|
-
#
|
|
281
|
+
# build_evaluation fallback then derives reset_at from the rule period
|
|
268
282
|
# since there is no Redis-side window to read.
|
|
269
283
|
#
|
|
270
284
|
# Float parsing accepts both INCR-stored ("5") and INCRBYFLOAT-stored
|
|
@@ -325,19 +339,19 @@ module Labkit
|
|
|
325
339
|
)
|
|
326
340
|
end
|
|
327
341
|
|
|
328
|
-
def report_matched_metrics(
|
|
342
|
+
def report_matched_metrics(evaluation)
|
|
329
343
|
Metrics.calls_total.increment(
|
|
330
344
|
rate_limiter: @name,
|
|
331
|
-
rule:
|
|
332
|
-
action:
|
|
345
|
+
rule: evaluation.rule.name,
|
|
346
|
+
action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s
|
|
333
347
|
)
|
|
334
348
|
Metrics.limit_gauge.set(
|
|
335
|
-
{ rate_limiter: @name, rule:
|
|
336
|
-
|
|
349
|
+
{ rate_limiter: @name, rule: evaluation.rule.name },
|
|
350
|
+
evaluation.info.resolved_limit
|
|
337
351
|
)
|
|
338
352
|
Metrics.period_gauge.set(
|
|
339
|
-
{ rate_limiter: @name, rule:
|
|
340
|
-
|
|
353
|
+
{ rate_limiter: @name, rule: evaluation.rule.name },
|
|
354
|
+
evaluation.info.resolved_period
|
|
341
355
|
)
|
|
342
356
|
end
|
|
343
357
|
|
|
@@ -15,7 +15,7 @@ module Labkit
|
|
|
15
15
|
# rules: [Labkit::RateLimit::Rule.new(name: "api_user", limit: 100, period: 60, characteristics: [:user])]
|
|
16
16
|
# )
|
|
17
17
|
# result = limiter.check({ user: 42, ip: "1.2.3.4" })
|
|
18
|
-
# render_429 if result.
|
|
18
|
+
# render_429 if result.action == :block
|
|
19
19
|
class Limiter
|
|
20
20
|
NAME_PATTERN = /\A[a-z0-9_]+\z/
|
|
21
21
|
|
|
@@ -5,9 +5,10 @@ module Labkit
|
|
|
5
5
|
module Metrics
|
|
6
6
|
module_function
|
|
7
7
|
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
# rule="unmatched", action="allow"
|
|
8
|
+
# Emitted once per *matched rule*, not once per check: every rule that
|
|
9
|
+
# matches is evaluated, so summing this by rate_limiter counts rule
|
|
10
|
+
# evaluations rather than requests. rule="unmatched", action="allow" is
|
|
11
|
+
# emitted only when no rule matched at all.
|
|
11
12
|
def calls_total
|
|
12
13
|
Labkit::Metrics::Client.counter(
|
|
13
14
|
:gitlab_labkit_rate_limiter_calls_total,
|
|
@@ -3,34 +3,100 @@
|
|
|
3
3
|
module Labkit
|
|
4
4
|
module RateLimit
|
|
5
5
|
# Result is the return value of Limiter#check.
|
|
6
|
-
#
|
|
7
|
-
#
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
#
|
|
11
|
-
#
|
|
12
|
-
#
|
|
13
|
-
#
|
|
14
|
-
#
|
|
15
|
-
#
|
|
16
|
-
#
|
|
17
|
-
#
|
|
18
|
-
#
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
6
|
+
#
|
|
7
|
+
# It accumulates one Result::Evaluation per matched-and-counted rule; the
|
|
8
|
+
# reader methods report the single most-constraining evaluation (see
|
|
9
|
+
# #most_constraining). A matched :skip rule or a fail-open error replaces
|
|
10
|
+
# the evaluations as the source of the reported outcome.
|
|
11
|
+
#
|
|
12
|
+
# Result is a mutable accumulator, not a value object: two Results built
|
|
13
|
+
# from the same evaluations are not #==.
|
|
14
|
+
#
|
|
15
|
+
# matched? - true if at least one rule's match conditions were satisfied
|
|
16
|
+
# exceeded? - true if the reported evaluation's counter exceeded its limit
|
|
17
|
+
# action - the outcome: what the caller should do
|
|
18
|
+
# :block = a :limit rule matched and is over its limit
|
|
19
|
+
# :allow = everything else, including an exceeded :log rule
|
|
20
|
+
# (visible via exceeded?), a matched :skip rule, no rule
|
|
21
|
+
# matched, and error (fail-open).
|
|
22
|
+
# The rule's configured action is available via rule.action.
|
|
23
|
+
# rule - the reported Rule: the :skip rule when one matched, otherwise
|
|
24
|
+
# the most-constraining evaluated Rule (nil when matched? is
|
|
25
|
+
# false). Other rules may also have matched and been counted;
|
|
26
|
+
# see #evaluations.
|
|
27
|
+
# error? - true if Redis was unavailable; result fails open (exceeded? is false)
|
|
28
|
+
# info - Result::Info with per-window counters for the reported rule;
|
|
29
|
+
# nil when matched? is false, error?, or the matched rule is
|
|
30
|
+
# :skip (no counter exists)
|
|
31
|
+
# evaluations - every counted evaluation, in rule declaration order
|
|
32
|
+
class Result
|
|
33
|
+
attr_reader :evaluations
|
|
34
|
+
|
|
35
|
+
def self.error
|
|
36
|
+
new(error: true)
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def initialize(error: false)
|
|
40
|
+
@evaluations = []
|
|
41
|
+
@skip_rule = nil
|
|
42
|
+
@error = error
|
|
43
|
+
@most_constraining = nil
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def add_evaluation(evaluation)
|
|
47
|
+
@most_constraining = nil
|
|
48
|
+
@evaluations << evaluation
|
|
49
|
+
self
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# Records a matched :skip rule: the request is allowed and the skip rule
|
|
53
|
+
# is the one reported, regardless of any evaluations already collected.
|
|
54
|
+
def skip!(rule)
|
|
55
|
+
@skip_rule = rule
|
|
56
|
+
self
|
|
22
57
|
end
|
|
23
58
|
|
|
24
59
|
def matched?
|
|
25
|
-
|
|
60
|
+
skipped? || @evaluations.any?
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def skipped?
|
|
64
|
+
!!@skip_rule
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def block?
|
|
68
|
+
!skipped? && @evaluations.any?(&:block?)
|
|
26
69
|
end
|
|
27
70
|
|
|
28
71
|
def exceeded?
|
|
29
|
-
|
|
72
|
+
return false if skipped?
|
|
73
|
+
|
|
74
|
+
!most_constraining.nil? && most_constraining.exceeded?
|
|
30
75
|
end
|
|
31
76
|
|
|
32
77
|
def error?
|
|
33
|
-
error
|
|
78
|
+
@error
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def action
|
|
82
|
+
block? ? :block : :allow
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def rule
|
|
86
|
+
@skip_rule || most_constraining&.rule
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def info
|
|
90
|
+
return nil if skipped?
|
|
91
|
+
|
|
92
|
+
most_constraining&.info
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
# The evaluation with the strongest claim on the outcome; ties keep the
|
|
96
|
+
# earliest-declared rule (min returns the first of tied elements).
|
|
97
|
+
# Memoized; add_evaluation invalidates.
|
|
98
|
+
def most_constraining
|
|
99
|
+
@most_constraining ||= @evaluations.min
|
|
34
100
|
end
|
|
35
101
|
|
|
36
102
|
# Returns RFC-compliant rate limit response headers, or {} when no rule matched or an error occurred.
|
|
@@ -48,6 +114,39 @@ module Labkit
|
|
|
48
114
|
end
|
|
49
115
|
end
|
|
50
116
|
|
|
117
|
+
# The outcome of counting one matched rule.
|
|
118
|
+
# rule - the evaluated Rule
|
|
119
|
+
# exceeded - whether the post-increment count exceeded the resolved limit
|
|
120
|
+
# info - Result::Info with the per-window counters
|
|
121
|
+
#
|
|
122
|
+
# Evaluations order by constraint: a blocking evaluation (:limit rule over
|
|
123
|
+
# its limit) ranks strictly first - it decides the request no matter what
|
|
124
|
+
# any other rule reports - then exceeded ones, then fewest remaining.
|
|
125
|
+
# remaining floors at 0, so an exceeded :log rule ties with one sitting
|
|
126
|
+
# exactly on its limit; ranking exceeded ahead keeps a breach from being
|
|
127
|
+
# hidden by a rule that merely reached its limit.
|
|
128
|
+
Result::Evaluation = Data.define(:rule, :exceeded, :info) do
|
|
129
|
+
include Comparable
|
|
130
|
+
|
|
131
|
+
def exceeded?
|
|
132
|
+
exceeded
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
def block?
|
|
136
|
+
exceeded? && rule.action == :limit
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
def <=>(other)
|
|
140
|
+
constraint_rank <=> other.constraint_rank
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
protected
|
|
144
|
+
|
|
145
|
+
def constraint_rank
|
|
146
|
+
[block? ? 0 : 1, exceeded? ? 0 : 1, info.remaining]
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
|
|
51
150
|
# Per-window counter data attached to a matched Result.
|
|
52
151
|
# resolved_limit - the evaluated limit Integer for this rule
|
|
53
152
|
# resolved_period - the evaluated period Integer (seconds) for this rule
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
module Labkit
|
|
4
4
|
module RateLimit
|
|
5
|
-
KNOWN_ACTIONS = %i[
|
|
5
|
+
KNOWN_ACTIONS = %i[limit log skip].freeze
|
|
6
6
|
RULE_NAME_PATTERN = /\A[a-z0-9_]+\z/
|
|
7
7
|
RULE_NAME_MAX_LENGTH = 64
|
|
8
8
|
|
|
@@ -12,11 +12,12 @@ module Labkit
|
|
|
12
12
|
# the rule to apply; empty hash matches any identifier
|
|
13
13
|
# limit - request threshold; may be a callable (resolved per check)
|
|
14
14
|
# period - window in seconds; may be a callable (resolved per check)
|
|
15
|
-
# action - :
|
|
16
|
-
#
|
|
17
|
-
#
|
|
18
|
-
#
|
|
19
|
-
#
|
|
15
|
+
# action - :limit (enforce; the result blocks when the rule is over
|
|
16
|
+
# its limit, which also terminates evaluation), :log (count
|
|
17
|
+
# and log only, never blocks and never terminates), or
|
|
18
|
+
# :skip (bypass: permit and terminate evaluation on match
|
|
19
|
+
# without counting; performs no Redis operation, so limit,
|
|
20
|
+
# period, characteristics, and count_distinct are inert)
|
|
20
21
|
# characteristics - identifier keys used to build the compound Redis counter key
|
|
21
22
|
# count_distinct - optional Symbol naming an identifier key. When set, the rule
|
|
22
23
|
# counts the number of distinct values seen for that key within
|
|
@@ -43,7 +44,7 @@ module Labkit
|
|
|
43
44
|
sym
|
|
44
45
|
end
|
|
45
46
|
|
|
46
|
-
def initialize(name:, limit:, period:, characteristics:, match: {}, action: :
|
|
47
|
+
def initialize(name:, limit:, period:, characteristics:, match: {}, action: :limit, count_distinct: nil)
|
|
47
48
|
raise ArgumentError, "name must be a String or Symbol, got #{name.class}" unless name.is_a?(String) || name.is_a?(Symbol)
|
|
48
49
|
|
|
49
50
|
name_str = name.to_s
|
data/lib/labkit/rate_limit.rb
CHANGED
|
@@ -40,7 +40,8 @@ module Labkit
|
|
|
40
40
|
#
|
|
41
41
|
# @param name [String] call site name
|
|
42
42
|
# @param identifier [Identifier, Hash] caller attributes
|
|
43
|
-
# @param rules [Array<Rule>] ordered list of rules
|
|
43
|
+
# @param rules [Array<Rule>] ordered list of rules; every matching rule is
|
|
44
|
+
# counted, and the Result reports the most constraining one
|
|
44
45
|
# @param redis [Object, nil] Redis client; falls back to config.redis
|
|
45
46
|
# @param logger [Logger, nil] logger; falls back to config.logger
|
|
46
47
|
# @param cost [Numeric] amount to add to the counter; see Limiter#check
|