gitlab-labkit 4.4.1 → 4.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitlab-ci-asdf-versions.yml +1 -1
- data/.tool-versions +1 -1
- data/lib/labkit/rate_limit/README.md +175 -42
- data/lib/labkit/rate_limit/evaluator.rb +256 -36
- data/lib/labkit/rate_limit/limiter.rb +23 -2
- data/lib/labkit/rate_limit/metrics.rb +62 -4
- data/lib/labkit/rate_limit/result.rb +18 -3
- data/lib/labkit/rate_limit/rule.rb +39 -7
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 22a38c0fe1a2f28aa538b80348911947c6de64c16d7d870d24835332796faa99
|
|
4
|
+
data.tar.gz: 6259d696d178f3008aefdf91de71731c81d460cef08259555829f15d8abd462c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: dd404f717fe9cee20cb544ad23cd5aa3195683ac0efe8bf8363988fdb8dcacc4fcaa817ac6f4cb3fcdb4f2e3d193c6d93cc13deea0344067e44fac7f3c593fb3
|
|
7
|
+
data.tar.gz: cc8d6bb5fe6fde8c17f969d4a708bd41901de5f9383f4821e3acf13a2eb35c240a80d18d3853ac5f29b689d7841df254f540233174646204358e698b9e0f643e
|
data/.tool-versions
CHANGED
|
@@ -121,6 +121,29 @@ different question than `check`. A matched `:skip` rule terminates `peek`
|
|
|
121
121
|
the same way it terminates `check`: matched, `:allow`, no Redis read, no
|
|
122
122
|
`info`.
|
|
123
123
|
|
|
124
|
+
### Clearing state
|
|
125
|
+
|
|
126
|
+
`Limiter#clear(identifier)` deletes this limiter's counters for one
|
|
127
|
+
identifier, along with any ban a rule carrying `ban_for` has written, and returns the
|
|
128
|
+
number of keys removed.
|
|
129
|
+
|
|
130
|
+
```ruby
|
|
131
|
+
RACK_LIMITER.clear({ ip: "1.2.3.4" }) # => 2
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Counters otherwise only disappear when their window expires; this is the only
|
|
135
|
+
way to end one early. It exists for call sites where a later success should
|
|
136
|
+
wipe earlier failures, such as an authentication ban cleared by a valid
|
|
137
|
+
login.
|
|
138
|
+
|
|
139
|
+
Every rule in the limiter is cleared, matched or not, because a caller
|
|
140
|
+
clearing state after a success knows the identifier rather than which rules
|
|
141
|
+
happened to match on the way in. Each rule is deleted in its own call, since
|
|
142
|
+
rules do not share a Redis Cluster slot. Clearing an identifier with no state
|
|
143
|
+
is a no-op returning `0`. A Redis failure fails open like `check`, leaving the
|
|
144
|
+
rest of the state to expire on its own, and the return value counts whatever
|
|
145
|
+
was removed before the failure.
|
|
146
|
+
|
|
124
147
|
## Identifier
|
|
125
148
|
|
|
126
149
|
`Identifier` is a small value object wrapping a hash of caller attributes.
|
|
@@ -150,6 +173,8 @@ A `Rule` is a `Data.define` value object with the following fields:
|
|
|
150
173
|
| `period` | Window length in seconds. May be a callable resolved on every check. |
|
|
151
174
|
| `action` | What the rule does when it matches. One of `:limit`, `:log`, `:skip`. Default `:limit`. See [Actions](#actions). |
|
|
152
175
|
| `characteristics` | Array of identifier keys whose values are folded into the Redis counter key. Each unique combination gets its own counter. |
|
|
176
|
+
| `count_distinct` | Optional identifier key. When set, the rule counts distinct values seen for that key within the window, backed by a Redis SET rather than a counter. Must not overlap `characteristics`. |
|
|
177
|
+
| `ban_for` | Optional ban duration, whole seconds, minimum 1. Changes the accounting; `action` still decides who is blocked. Rejected on `:skip`. May be a callable resolved on every check. |
|
|
153
178
|
|
|
154
179
|
Making `limit` or `period` callable is the supported pattern for
|
|
155
180
|
runtime-tunable thresholds (e.g. feature flags or database-backed settings):
|
|
@@ -219,7 +244,7 @@ An exceeded rule has `remaining` 0 and therefore outranks any rule still under
|
|
|
219
244
|
its limit. For a `:log` rule that means shadow traffic surfaces to the caller
|
|
220
245
|
as `exceeded? == true` (with `action` still `:allow`) — visible, but unable to
|
|
221
246
|
block. A `:log`-only path that matches therefore returns `matched? == true`
|
|
222
|
-
and
|
|
247
|
+
and its check is counted as `checks_total{matched="true"}`.
|
|
223
248
|
|
|
224
249
|
```mermaid
|
|
225
250
|
flowchart TD
|
|
@@ -227,20 +252,22 @@ flowchart TD
|
|
|
227
252
|
Iter -->|yes| Match{rule.match<br/>all satisfied?}
|
|
228
253
|
Match -->|no| Iter
|
|
229
254
|
Match -->|yes| Skip{rule.action<br/>== :skip?}
|
|
230
|
-
Skip -->|"yes (no Redis op)"| SkipEmit[Emit
|
|
255
|
+
Skip -->|"yes (no Redis op)"| SkipEmit[Emit rule_evaluations_total<br/>action=skip, result=skip]
|
|
231
256
|
SkipEmit --> SkipReturn([Return matched=true<br/>action=:allow])
|
|
232
|
-
Skip -->|no|
|
|
257
|
+
Skip -->|no| Ban{"ban_for rule with<br/>a ban in force?"}
|
|
258
|
+
Ban -->|"yes (no increment)"| Build
|
|
259
|
+
Ban -->|no| Eval["INCR Redis counter<br/>(see Redis sequence below)"]
|
|
233
260
|
Eval --> Build[Build Evaluation<br/>resolve limit/period]
|
|
234
261
|
Build --> Add[Add evaluation to Result]
|
|
235
|
-
Add --> Emit[Emit
|
|
236
|
-
Emit --> Act{"result.block?<br/>(:limit rule over limit)"}
|
|
262
|
+
Add --> Emit[Emit rule_evaluations_total<br/>+ limit/period gauges]
|
|
263
|
+
Emit --> Act{"result.block?<br/>(:limit rule over its limit,<br/>or its ban in force)"}
|
|
237
264
|
Act -->|yes| Return([Return Result<br/>action=:block])
|
|
238
|
-
Act -->|"no (:log, or
|
|
239
|
-
Iter -->|no more rules|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
Eval -. StandardError .-> Error[Emit errors_total<br/>log warn]
|
|
265
|
+
Act -->|"no (:log, or under limit)"| Iter
|
|
266
|
+
Iter -->|no more rules| Return2([Return Result reporting<br/>most-constraining evaluation,<br/>or matched=false if none])
|
|
267
|
+
Return --> Check[Emit checks_total<br/>action, matched, error]
|
|
268
|
+
SkipReturn --> Check
|
|
269
|
+
Return2 --> Check
|
|
270
|
+
Eval -. StandardError .-> Error[Emit errors_total +<br/>checks_total error=true<br/>log warn]
|
|
244
271
|
Error --> ReturnErr([Return error=true<br/>action=:allow])
|
|
245
272
|
```
|
|
246
273
|
|
|
@@ -249,23 +276,51 @@ discards the verdicts of the rules already evaluated, even though their counters
|
|
|
249
276
|
were incremented. Those requests are counted but produce no verdict — the
|
|
250
277
|
fail-open trade-off is that a request is never blocked on a partial evaluation.
|
|
251
278
|
|
|
252
|
-
|
|
253
|
-
`
|
|
279
|
+
`rule_evaluations_total` is emitted **per evaluated rule** (including matched
|
|
280
|
+
`:skip` rules); `checks_total` is emitted **exactly once per check**, whatever
|
|
281
|
+
path the evaluation takes — so summing `checks_total` by `rate_limiter` counts
|
|
282
|
+
requests through the limiter, and "no rule matched" is `matched="false"` on the
|
|
283
|
+
check rather than a placeholder rule.
|
|
284
|
+
|
|
285
|
+
Only matched rules are evaluated: a rule whose `match:` conditions are not
|
|
286
|
+
satisfied emits nothing to `rule_evaluations_total` (match-testing is not an
|
|
287
|
+
evaluation), so the counter has no "didn't match" population and needs no
|
|
288
|
+
label for it. A matched `count_distinct` rule skipped by the missing-key
|
|
289
|
+
fail-open also emits nothing here — it was never evaluated; that check is
|
|
290
|
+
visible via `checks_total{error="true"}` and the
|
|
291
|
+
`rate_limit_missing_count_distinct` log, which carries the rule name.
|
|
292
|
+
|
|
293
|
+
During the transition the deprecated `calls_total` counter is additionally
|
|
294
|
+
emitted at every point the diagram emits `rule_evaluations_total` (with its
|
|
295
|
+
historical per-rule semantics, including the `rule="unmatched"` placeholder
|
|
296
|
+
after the loop). It is not shown above to keep the diagram legible; see the
|
|
297
|
+
Metrics table below.
|
|
254
298
|
|
|
255
299
|
### Actions
|
|
256
300
|
|
|
257
301
|
The rule's `action` describes what the rule does; the result's `action`
|
|
258
302
|
describes the outcome — what the caller should do — and is only ever `:allow`
|
|
259
|
-
or `:block`.
|
|
260
|
-
`:skip` rules
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
|
265
|
-
|
|
266
|
-
| `:
|
|
267
|
-
| `:
|
|
268
|
-
| `:
|
|
303
|
+
or `:block`. A matching rule increments its counter, with two exceptions:
|
|
304
|
+
`:skip` rules never touch Redis, and a rule carrying `ban_for` whose ban is
|
|
305
|
+
already in force short-circuits before incrementing. The `result` label on
|
|
306
|
+
`rule_evaluations_total` records what each evaluation decided:
|
|
307
|
+
|
|
308
|
+
| rule action | what it does | exceeded? | result action | `result` label | terminating? |
|
|
309
|
+
|-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|
|
|
310
|
+
| `:limit` | count against the limit | no | `:allow` | `allow` | no — continue |
|
|
311
|
+
| `:limit` | count against the limit | yes | `:block` | `block` | yes — stop |
|
|
312
|
+
| `:log` | count against the limit (observability only) | no | `:allow` | `allow` | no — continue |
|
|
313
|
+
| `:log` | count against the limit (observability only) | yes | `:allow` | `log` | no — continue |
|
|
314
|
+
| `:skip` | don't count (bypass) | n/a | `:allow` | `skip` | yes — stop |
|
|
315
|
+
|
|
316
|
+
A rule may also carry `ban_for`, which changes the accounting rather than the
|
|
317
|
+
action. On crossing the limit it writes a ban that outlives the counting
|
|
318
|
+
window, and it stops counting while that ban holds:
|
|
319
|
+
|
|
320
|
+
| rule action | with `ban_for` | exceeded? | result action | `result` label | terminating? |
|
|
321
|
+
|-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|
|
|
322
|
+
| `:limit` | count, or skip counting while banned | yes | `:block` | `banned` | yes — stop |
|
|
323
|
+
| `:log` | same accounting, records what it would do | yes | `:allow` | `banned` | no — continue |
|
|
269
324
|
|
|
270
325
|
- `:limit` — when exceeded, `Result#action` is `:block` and evaluation
|
|
271
326
|
terminates. Caller should reject the request (e.g. with HTTP 429). When under
|
|
@@ -281,17 +336,53 @@ or `:block`. The counter is always incremented when a rule matches, except for
|
|
|
281
336
|
`Result#action` `:allow` **without any Redis operation**: nothing is
|
|
282
337
|
counted, so `limit`, `period`, `characteristics`, and `count_distinct` are
|
|
283
338
|
inert and the result carries no `info` (`to_response_headers` is `{}`).
|
|
284
|
-
The match is still observable via `
|
|
285
|
-
for bypasses.
|
|
339
|
+
The match is still observable via `rule_evaluations_total{action="skip"}`.
|
|
340
|
+
Use this for bypasses.
|
|
341
|
+
- `ban_for` — enforcement that outlives the window, on either counting action.
|
|
342
|
+
On crossing the limit the rule writes a second key that keeps blocking for
|
|
343
|
+
`ban_for` seconds, long after the counter window has expired. While that ban
|
|
344
|
+
is in force the rule reports `exceeded? == true` and does not count, so a
|
|
345
|
+
caller cannot extend its own ban, and `reset_at` reports when the ban lifts
|
|
346
|
+
rather than when the window rolls. Use this where repeated failures should
|
|
347
|
+
cost more than a window, such as authentication attempts. State can be
|
|
348
|
+
discarded early with [`#clear`](#clearing-state).
|
|
349
|
+
|
|
350
|
+
With `action: :limit` the ban blocks. With `action: :log` the rule does all
|
|
351
|
+
of the same accounting, including writing the ban and suppressing counting
|
|
352
|
+
while it holds, and only skips the blocking, so a shadow measures what
|
|
353
|
+
enforcement would have produced rather than approximating it.
|
|
354
|
+
|
|
355
|
+
A `ban_for` shorter than `period` is legal but surprising: the ban can lapse
|
|
356
|
+
while the counter is still over the limit, so `peek` reports not-exceeded
|
|
357
|
+
while the next `check` re-bans immediately. Bans are normally longer than the
|
|
358
|
+
window they are counted in.
|
|
359
|
+
|
|
360
|
+
The duration is whole seconds and must be at least 1, since the ban is
|
|
361
|
+
written with `SET EX`. A callable is checked again each time it resolves, and
|
|
362
|
+
a value under a second raises rather than reaching Redis. Being an error, it
|
|
363
|
+
fails open like any other, so a rule whose `ban_for` cannot resolve stops
|
|
364
|
+
blocking entirely: watch `errors_total` after changing one.
|
|
365
|
+
|
|
366
|
+
Take care combining `ban_for` with `cost:`. One expensive call can cross the
|
|
367
|
+
limit on its own, and with a ban attached that costs the caller the whole ban
|
|
368
|
+
rather than a single rejection. Checking with `peek` first has the same edge,
|
|
369
|
+
since the call it deliberately lets through is enough to start a ban. Bans
|
|
370
|
+
suit counting discrete failures, such as bad credentials, better than they
|
|
371
|
+
suit variable-cost resource limits.
|
|
286
372
|
|
|
287
373
|
### Redis keys
|
|
288
374
|
|
|
289
375
|
Each matched check writes a key shaped:
|
|
290
376
|
|
|
291
377
|
```
|
|
292
|
-
labkit:rl
|
|
378
|
+
labkit:rl:{<limiter_name>:<rule_name>:<char>:<value>[:<char>:<value>...]}
|
|
293
379
|
```
|
|
294
380
|
|
|
381
|
+
The braces are a Redis [hash tag](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#hash-tags):
|
|
382
|
+
on a Redis Cluster only the braced part decides the slot, so every key for one
|
|
383
|
+
rule and identifier lands on the same shard. `BAN_SCRIPT` touches two keys in
|
|
384
|
+
one call, and a cluster rejects a script whose keys span slots.
|
|
385
|
+
|
|
295
386
|
Characteristic values longer than 200 bytes are replaced with a SHA-256
|
|
296
387
|
hexdigest to bound key length. Missing or empty characteristic values are
|
|
297
388
|
encoded as `_unknown_`. The TTL is set on the first write of each window and
|
|
@@ -299,9 +390,24 @@ is not extended on subsequent increments, so the window is a true fixed
|
|
|
299
390
|
window starting at the first request, not a sliding window.
|
|
300
391
|
|
|
301
392
|
A check is a single `EVALSHA` of `INCR_SCRIPT` (or `SADD_SCRIPT` for
|
|
302
|
-
`count_distinct` rules). Doing the whole
|
|
303
|
-
there is no window between the increment
|
|
304
|
-
could be left without a TTL.
|
|
393
|
+
`count_distinct` rules, or `BAN_SCRIPT` for rules carrying `ban_for`). Doing the whole
|
|
394
|
+
read-modify-write inside Lua means there is no window between the increment
|
|
395
|
+
and the `EXPIRE` in which a key could be left without a TTL.
|
|
396
|
+
|
|
397
|
+
A rule carrying `ban_for` writes a second key alongside its counter, sharing the
|
|
398
|
+
same hash tag and carrying its own `ban_for` TTL:
|
|
399
|
+
|
|
400
|
+
```
|
|
401
|
+
labkit:rl:{<limiter_name>:<rule_name>:<char>:<value>[:<char>:<value>...]}:ban
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
The suffix sits outside the tag, so it cannot change the slot and cannot
|
|
405
|
+
collide with a characteristic value, and both keys are found by the same
|
|
406
|
+
`labkit:rl:{<limiter>:<rule>*` glob. The ban key outlives its counter by
|
|
407
|
+
design: that is the whole point of `ban_for`.
|
|
408
|
+
`BAN_SCRIPT` checks the ban before incrementing, so the ban check,
|
|
409
|
+
the increment and the ban write are one atomic operation and a concurrent
|
|
410
|
+
check cannot miss the threshold crossing.
|
|
305
411
|
|
|
306
412
|
```mermaid
|
|
307
413
|
sequenceDiagram
|
|
@@ -375,26 +481,53 @@ safe to merge unconditionally.
|
|
|
375
481
|
|
|
376
482
|
The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
|
|
377
483
|
(Redis connection failure, timeout, OOM in user-supplied callables, …) is
|
|
378
|
-
logged at WARN with `
|
|
379
|
-
`Result` (`matched?` false, `error?` true, `action` `:allow`). The
|
|
380
|
-
`gitlab_labkit_rate_limiter_errors_total` counter is incremented. The caller
|
|
484
|
+
logged at WARN with `error_type: "rate_limit_error"` and returned as an error
|
|
485
|
+
`Result` (`matched?` false, `error?` true, `action` `:allow`). The caller
|
|
381
486
|
should treat the request as allowed.
|
|
382
487
|
|
|
488
|
+
A failed-open `check` is still counted: it emits
|
|
489
|
+
`checks_total{action="allow", matched="false", error="true"}`, so the fraction
|
|
490
|
+
of checks that encountered an error is `checks_total{error="true"}` over
|
|
491
|
+
`checks_total`. `error="true"` also covers a check where a matched
|
|
492
|
+
`count_distinct` rule was skipped because the identifier was missing its
|
|
493
|
+
`count_distinct` key — that check completes (`action` and `matched` describe
|
|
494
|
+
its outcome as usual), so `error="true"` is not exclusively fail-open traffic.
|
|
495
|
+
|
|
496
|
+
The deprecated `gitlab_labkit_rate_limiter_errors_total` counter is still
|
|
497
|
+
incremented on every fail-open (whole-check and per-rule `count_distinct`),
|
|
498
|
+
and remains the only error metric for `peek`, which emits no `checks_total`
|
|
499
|
+
(peek must not inflate the per-check counter). The follow-up MR that removes
|
|
500
|
+
`errors_total` must first decide where `peek` errors go — a dedicated peek
|
|
501
|
+
metric, or logs only.
|
|
502
|
+
|
|
503
|
+
Metric emission itself is best-effort: a failure in the metrics stack never
|
|
504
|
+
alters the verdict or breaks fail-open, and is logged at WARN with
|
|
505
|
+
`error_type: "rate_limit_metrics_error"` (once per process, to avoid
|
|
506
|
+
flooding).
|
|
507
|
+
|
|
383
508
|
## Metrics
|
|
384
509
|
|
|
385
510
|
`Labkit::RateLimit::Metrics` emits the following Prometheus metrics through
|
|
386
511
|
`Labkit::Metrics::Client`:
|
|
387
512
|
|
|
388
|
-
| metric
|
|
389
|
-
|
|
390
|
-
| `
|
|
391
|
-
| `
|
|
392
|
-
| `
|
|
393
|
-
| `
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
`
|
|
513
|
+
| metric | type | labels | meaning |
|
|
514
|
+
|-----------------------------------------------------|---------|----------------------------------------------|----------------------------------------------------------------------|
|
|
515
|
+
| `gitlab_labkit_rate_limiter_checks_total` | counter | `rate_limiter`, `action`, `matched`, `error` | Exactly one increment per `check` call, including fail-open. `action` is what the caller should do (`"allow"` or `"block"`); `matched` and `error` are `"true"`/`"false"`. |
|
|
516
|
+
| `gitlab_labkit_rate_limiter_rule_evaluations_total` | counter | `rate_limiter`, `rule`, `action`, `result` | One increment per evaluated rule (plus one per matched `:skip` rule). `action` is the configured rule action (`"limit"`, `"log"`, `"skip"`); `result` is what the evaluation decided (`"allow"`, `"block"`, `"log"`, `"skip"`, `"banned"` — see the Actions table). |
|
|
517
|
+
| `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | **Deprecated** — superseded by `checks_total` + `rule_evaluations_total`. Historical per-rule counter: one increment per counted rule (plus one per matched `:skip` rule), `rule="unmatched", action="allow"` when no rule matched. Emitted unchanged during the transition. |
|
|
518
|
+
| `gitlab_labkit_rate_limiter_errors_total` | counter | `rate_limiter` | **Deprecated** — use `checks_total{error="true"}`. Fail-open events (any `StandardError` in the labkit path); still the only error metric for `peek`. |
|
|
519
|
+
| `gitlab_labkit_rate_limiter_limit` | gauge | `rate_limiter`, `rule` | Resolved limit at the last check (useful when `limit:` is callable). |
|
|
520
|
+
| `gitlab_labkit_rate_limiter_period_seconds` | gauge | `rate_limiter`, `rule` | Resolved period at the last check. |
|
|
521
|
+
|
|
522
|
+
`sum by (rate_limiter) (rate(gitlab_labkit_rate_limiter_checks_total[5m]))` is
|
|
523
|
+
the request rate through a limiter — no exclusions or dedup needed. A single
|
|
524
|
+
`check` call emits **one** `checks_total` increment and as many
|
|
525
|
+
`rule_evaluations_total` increments as rules it evaluated (possibly zero).
|
|
526
|
+
|
|
527
|
+
**Transition:** the deprecated `calls_total` and `errors_total` counters keep
|
|
528
|
+
emitting exactly as before this split, so existing dashboards and alerts stay
|
|
529
|
+
correct while consumers migrate to the new counters. Both are removed in a
|
|
530
|
+
follow-up major release once nothing consumes them.
|
|
398
531
|
|
|
399
532
|
## Dev/test vs production guards
|
|
400
533
|
|
|
@@ -9,6 +9,7 @@ module Labkit
|
|
|
9
9
|
# @api private
|
|
10
10
|
class Evaluator
|
|
11
11
|
REDIS_KEY_PREFIX = "labkit:rl"
|
|
12
|
+
BAN_KEY_SUFFIX = ":ban"
|
|
12
13
|
CHAR_VALUE_MAX_LENGTH = 200
|
|
13
14
|
MISSING_VALUE_SENTINEL = "_unknown_"
|
|
14
15
|
|
|
@@ -66,6 +67,50 @@ module Labkit
|
|
|
66
67
|
return {count, ttl_after}
|
|
67
68
|
LUA
|
|
68
69
|
|
|
70
|
+
# Atomic ban check + INCR + conditional ban write, for rules carrying
|
|
71
|
+
# ban_for.
|
|
72
|
+
#
|
|
73
|
+
# An active ban short-circuits before the increment, so a banned caller
|
|
74
|
+
# cannot extend its own window, matching Rack::Attack::Allow2Ban. This
|
|
75
|
+
# applies to :log rules too: a shadow that kept counting through its own
|
|
76
|
+
# ban would report more than enforcement would produce. The
|
|
77
|
+
# counter and the ban must be read and written together or a concurrent
|
|
78
|
+
# check can miss the threshold crossing.
|
|
79
|
+
#
|
|
80
|
+
# Returns {count, counter TTL, ban TTL}, where a ban TTL below zero means
|
|
81
|
+
# not banned.
|
|
82
|
+
BAN_SCRIPT = Labkit::Redis::Script.new(<<~LUA)
|
|
83
|
+
local rule_counter_key = KEYS[1]
|
|
84
|
+
local ban_key = KEYS[2]
|
|
85
|
+
|
|
86
|
+
local ttl = ARGV[1]
|
|
87
|
+
local cost = tonumber(ARGV[2])
|
|
88
|
+
local limit = tonumber(ARGV[3])
|
|
89
|
+
local ban_for = tonumber(ARGV[4])
|
|
90
|
+
|
|
91
|
+
-- TTL answers both questions at once: >= 0 means the ban key exists
|
|
92
|
+
-- and tells us how long is left, so no separate EXISTS is needed.
|
|
93
|
+
local ban_ttl = redis.call('TTL', ban_key)
|
|
94
|
+
if ban_ttl >= 0 then
|
|
95
|
+
local existing = redis.call('GET', rule_counter_key)
|
|
96
|
+
return {existing or '0', redis.call('TTL', rule_counter_key), ban_ttl}
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
local count = redis.call('INCRBYFLOAT', rule_counter_key, cost)
|
|
100
|
+
local ttl_after = redis.call('TTL', rule_counter_key)
|
|
101
|
+
if ttl_after < 0 then
|
|
102
|
+
redis.call('EXPIRE', rule_counter_key, ttl)
|
|
103
|
+
ttl_after = tonumber(ttl)
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
if tonumber(count) > limit then
|
|
107
|
+
redis.call('SET', ban_key, '1', 'EX', ban_for)
|
|
108
|
+
return {count, ttl_after, ban_for}
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
return {count, ttl_after, -1}
|
|
112
|
+
LUA
|
|
113
|
+
|
|
69
114
|
def initialize(name:, rules:, redis:, logger:)
|
|
70
115
|
@name = name
|
|
71
116
|
@rules = rules
|
|
@@ -75,13 +120,17 @@ module Labkit
|
|
|
75
120
|
|
|
76
121
|
def check(identifier, cost: 1, rule_context: nil)
|
|
77
122
|
cursor = RuleCursor.new
|
|
78
|
-
check_rules(identifier, cost, rule_context, cursor)
|
|
123
|
+
result = check_rules(identifier, cost, rule_context, cursor)
|
|
79
124
|
rescue StandardError => e
|
|
80
125
|
# Intentionally broad: fail-open applies to any unexpected error (network,
|
|
81
126
|
# timeout, OOM) not only Redis protocol errors.
|
|
82
127
|
report_error_metrics
|
|
83
128
|
log_error(e, identifier, cursor.rule)
|
|
84
|
-
Result.error
|
|
129
|
+
result = Result.error
|
|
130
|
+
ensure
|
|
131
|
+
# StandardError-safe emission, so it cannot mask a propagating error.
|
|
132
|
+
# result is nil when a non-StandardError unwinds: emit nothing then.
|
|
133
|
+
report_check_metrics(result) if result
|
|
85
134
|
end
|
|
86
135
|
|
|
87
136
|
# Read-without-increment counterpart to {#check}. Same matching and Result
|
|
@@ -96,6 +145,41 @@ module Labkit
|
|
|
96
145
|
Result.error
|
|
97
146
|
end
|
|
98
147
|
|
|
148
|
+
# Deletes this limiter's counters, and any bans, for one identifier.
|
|
149
|
+
#
|
|
150
|
+
# Every rule is cleared, matched or not: a caller clearing state after a
|
|
151
|
+
# success knows the identifier, not which rules happened to match on the
|
|
152
|
+
# way in. Rules whose keys do not exist are a no-op.
|
|
153
|
+
#
|
|
154
|
+
# Fails open like {#check}: an unreachable Redis leaves the state to
|
|
155
|
+
# expire on its own rather than raising into the caller.
|
|
156
|
+
def clear(identifier)
|
|
157
|
+
# Counted as we go, so a connection lost mid-loop still reports the keys
|
|
158
|
+
# that did go rather than claiming nothing was cleared.
|
|
159
|
+
removed = 0
|
|
160
|
+
|
|
161
|
+
key_groups = @rules.filter_map do |rule|
|
|
162
|
+
next if rule.action == :skip
|
|
163
|
+
|
|
164
|
+
[build_redis_key(rule, identifier), build_redis_key(rule, identifier, BAN_KEY_SUFFIX)]
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
return 0 if key_groups.empty?
|
|
168
|
+
|
|
169
|
+
# One DEL per rule. The hash tag holds the rule name, so a rule's own
|
|
170
|
+
# keys share a slot but two rules do not, and Redis Cluster rejects a
|
|
171
|
+
# DEL spanning slots.
|
|
172
|
+
@redis.with do |conn|
|
|
173
|
+
key_groups.each { |group| removed += conn.del(*group) }
|
|
174
|
+
end
|
|
175
|
+
|
|
176
|
+
removed
|
|
177
|
+
rescue StandardError => e
|
|
178
|
+
report_error_metrics
|
|
179
|
+
log_error(e, identifier, nil)
|
|
180
|
+
removed
|
|
181
|
+
end
|
|
182
|
+
|
|
99
183
|
private
|
|
100
184
|
|
|
101
185
|
# Every rule that matches is evaluated and counted; matching does not
|
|
@@ -104,21 +188,29 @@ module Labkit
|
|
|
104
188
|
# - :skip terminates on match without touching Redis. No counter is
|
|
105
189
|
# incremented, so the branch sits before the count_distinct check
|
|
106
190
|
# (identifier completeness is irrelevant to a rule that builds no key).
|
|
107
|
-
#
|
|
191
|
+
# rule_evaluations_total still increments so the bypass stays observable.
|
|
108
192
|
# - a :limit rule over its limit terminates, because the request is
|
|
109
193
|
# rejected and later rules cannot change that. Rules declared after it
|
|
110
194
|
# are neither counted nor evaluated, so a blocked request debits every
|
|
111
195
|
# rule up to and including the one that blocked, and none after it.
|
|
196
|
+
# A :limit rule carrying ban_for counts as over its limit for as long
|
|
197
|
+
# as its ban holds, so it terminates for the whole ban. The same rule
|
|
198
|
+
# with action :log does identical accounting and terminates nothing,
|
|
199
|
+
# which is what makes it a faithful shadow.
|
|
112
200
|
#
|
|
113
201
|
# Every other matched rule - :log, and :limit while under its limit - is
|
|
114
202
|
# counted and collected into the returned Result, which reports the
|
|
115
203
|
# most-constraining evaluation (ranking lives in Result::Evaluation#<=>).
|
|
116
204
|
# cost is therefore debited from every matching rule, not just the first.
|
|
117
205
|
#
|
|
206
|
+
# Metrics: each evaluated rule emits rule_evaluations_total (plus the
|
|
207
|
+
# deprecated per-rule calls_total); the per-check checks_total is
|
|
208
|
+
# emitted once in #check. Full contract in the README's Metrics section.
|
|
209
|
+
#
|
|
118
210
|
# SET-mode rules (rule.count_distinct set) that match but whose identifier
|
|
119
|
-
# is missing the count_distinct key fail open + log + bump errors_total
|
|
120
|
-
# the loop continues to the next rule (the rule is
|
|
121
|
-
# rather than aborting the whole evaluation).
|
|
211
|
+
# is missing the count_distinct key fail open + log + bump errors_total +
|
|
212
|
+
# flag the Result, and the loop continues to the next rule (the rule is
|
|
213
|
+
# treated as not applicable rather than aborting the whole evaluation).
|
|
122
214
|
#
|
|
123
215
|
# Error handling stays whole-check (see #check): a raise part-way through
|
|
124
216
|
# discards the results of the rules already evaluated, even though their
|
|
@@ -143,12 +235,13 @@ module Labkit
|
|
|
143
235
|
if rule.count_distinct && missing_count_distinct_value?(rule, identifier)
|
|
144
236
|
log_missing_count_distinct(rule, identifier)
|
|
145
237
|
report_error_metrics
|
|
238
|
+
result.degraded!
|
|
146
239
|
next
|
|
147
240
|
end
|
|
148
241
|
|
|
149
242
|
evaluation = evaluate_rule(rule, identifier, cost, rule_context)
|
|
150
243
|
result.add_evaluation(evaluation)
|
|
151
|
-
|
|
244
|
+
report_evaluation_metrics(evaluation)
|
|
152
245
|
return result if result.block?
|
|
153
246
|
end
|
|
154
247
|
|
|
@@ -157,6 +250,12 @@ module Labkit
|
|
|
157
250
|
|
|
158
251
|
report_unmatched_metrics unless result.matched?
|
|
159
252
|
result
|
|
253
|
+
rescue StandardError => e # binds e for the ensure
|
|
254
|
+
raise
|
|
255
|
+
ensure
|
|
256
|
+
# Keep the rule attributed while an exception unwinds; clear it on
|
|
257
|
+
# every normal exit (early returns included).
|
|
258
|
+
cursor.rule = nil unless e
|
|
160
259
|
end
|
|
161
260
|
|
|
162
261
|
# Mirror of check_rules without metrics or writes. :log rules are read
|
|
@@ -180,9 +279,11 @@ module Labkit
|
|
|
180
279
|
return result if result.block?
|
|
181
280
|
end
|
|
182
281
|
|
|
183
|
-
cursor.rule = nil
|
|
184
|
-
|
|
185
282
|
result
|
|
283
|
+
rescue StandardError => e # binds e for the ensure
|
|
284
|
+
raise
|
|
285
|
+
ensure
|
|
286
|
+
cursor.rule = nil unless e
|
|
186
287
|
end
|
|
187
288
|
|
|
188
289
|
def rule_matches?(rule, identifier)
|
|
@@ -195,6 +296,8 @@ module Labkit
|
|
|
195
296
|
end
|
|
196
297
|
|
|
197
298
|
def evaluate_rule(rule, identifier, cost, rule_context)
|
|
299
|
+
return evaluate_ban_rule(rule, identifier, cost, rule_context) if rule.ban_for
|
|
300
|
+
|
|
198
301
|
redis_key = build_redis_key(rule, identifier)
|
|
199
302
|
resolved_limit = Integer(resolve_value(rule.limit, rule_context))
|
|
200
303
|
resolved_period = Integer(resolve_value(rule.period, rule_context))
|
|
@@ -212,6 +315,8 @@ module Labkit
|
|
|
212
315
|
end
|
|
213
316
|
|
|
214
317
|
def peek_rule(rule, identifier, rule_context)
|
|
318
|
+
return peek_ban_rule(rule, identifier, rule_context) if rule.ban_for
|
|
319
|
+
|
|
215
320
|
redis_key = build_redis_key(rule, identifier)
|
|
216
321
|
resolved_limit = Integer(resolve_value(rule.limit, rule_context))
|
|
217
322
|
resolved_period = Integer(resolve_value(rule.period, rule_context))
|
|
@@ -220,6 +325,63 @@ module Labkit
|
|
|
220
325
|
build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
|
|
221
326
|
end
|
|
222
327
|
|
|
328
|
+
def evaluate_ban_rule(rule, identifier, cost, rule_context)
|
|
329
|
+
resolved_limit = Integer(resolve_value(rule.limit, rule_context))
|
|
330
|
+
resolved_period = Integer(resolve_value(rule.period, rule_context))
|
|
331
|
+
resolved_ban_for = resolve_ban_for(rule, rule_context)
|
|
332
|
+
|
|
333
|
+
count, ttl, ban_ttl = ban_incr_with_ttl(
|
|
334
|
+
build_redis_key(rule, identifier),
|
|
335
|
+
build_redis_key(rule, identifier, BAN_KEY_SUFFIX),
|
|
336
|
+
resolved_period, cost, resolved_limit, resolved_ban_for
|
|
337
|
+
)
|
|
338
|
+
|
|
339
|
+
build_ban_evaluation(rule, resolved_limit, resolved_period, count, ttl, ban_ttl)
|
|
340
|
+
end
|
|
341
|
+
|
|
342
|
+
# A callable ban_for is only checked at construction, so it can still hand
|
|
343
|
+
# back something Redis would reject on the SET that writes the ban. Failing
|
|
344
|
+
# here names ban_for rather than surfacing a line number from the script.
|
|
345
|
+
def resolve_ban_for(rule, rule_context)
|
|
346
|
+
raw = resolve_value(rule.ban_for, rule_context)
|
|
347
|
+
seconds = Integer(raw, exception: false)
|
|
348
|
+
raise ArgumentError, "ban_for resolved to #{raw.inspect}, need at least 1 second" if seconds.nil? || seconds < 1
|
|
349
|
+
|
|
350
|
+
seconds
|
|
351
|
+
end
|
|
352
|
+
|
|
353
|
+
def peek_ban_rule(rule, identifier, rule_context)
|
|
354
|
+
resolved_limit = Integer(resolve_value(rule.limit, rule_context))
|
|
355
|
+
resolved_period = Integer(resolve_value(rule.period, rule_context))
|
|
356
|
+
|
|
357
|
+
count, ttl, ban_ttl = read_ban_with_ttl(
|
|
358
|
+
build_redis_key(rule, identifier),
|
|
359
|
+
build_redis_key(rule, identifier, BAN_KEY_SUFFIX)
|
|
360
|
+
)
|
|
361
|
+
|
|
362
|
+
build_ban_evaluation(rule, resolved_limit, resolved_period, count, ttl, ban_ttl)
|
|
363
|
+
end
|
|
364
|
+
|
|
365
|
+
# A ban blocks whether it was written by this call or an earlier one, so
|
|
366
|
+
# +exceeded+ tracks the ban rather than the count: once the counter window
|
|
367
|
+
# has expired the count can sit below the limit while the ban still holds.
|
|
368
|
+
# reset_at reports when the caller may retry, which is the ban expiry.
|
|
369
|
+
def build_ban_evaluation(rule, resolved_limit, resolved_period, count, ttl, ban_ttl)
|
|
370
|
+
banned = ban_ttl >= 0
|
|
371
|
+
window_remaining = ttl >= 0 ? ttl : resolved_period
|
|
372
|
+
|
|
373
|
+
info = Result::Info.new(
|
|
374
|
+
resolved_limit: resolved_limit, resolved_period: resolved_period,
|
|
375
|
+
count: count,
|
|
376
|
+
# A ban outlives its counter, so count reads 0 once the window has
|
|
377
|
+
# gone. Nothing is remaining while the ban still blocks.
|
|
378
|
+
remaining: banned ? 0 : [resolved_limit - count, 0].max,
|
|
379
|
+
reset_at: Time.now.utc + (banned ? ban_ttl : window_remaining)
|
|
380
|
+
)
|
|
381
|
+
|
|
382
|
+
Result::Evaluation.new(rule: rule, exceeded: banned, info: info)
|
|
383
|
+
end
|
|
384
|
+
|
|
223
385
|
def build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
|
|
224
386
|
info = Result::Info.new(
|
|
225
387
|
resolved_limit: resolved_limit, resolved_period: resolved_period,
|
|
@@ -231,12 +393,19 @@ module Labkit
|
|
|
231
393
|
Result::Evaluation.new(rule: rule, exceeded: count > resolved_limit, info: info)
|
|
232
394
|
end
|
|
233
395
|
|
|
234
|
-
|
|
235
|
-
|
|
396
|
+
# The limiter, rule and characteristics sit inside a Redis hash tag, so a
|
|
397
|
+
# counter and its ban always hash to the same cluster slot. BAN_SCRIPT
|
|
398
|
+
# touches both in one call, and Redis Cluster rejects a script whose keys
|
|
399
|
+
# span slots. +suffix+ is appended outside the tag, so it cannot change
|
|
400
|
+
# the slot and cannot collide with a characteristic value.
|
|
401
|
+
def build_redis_key(rule, identifier, suffix = nil)
|
|
402
|
+
key = "#{REDIS_KEY_PREFIX}:{#{@name}:#{rule.name}"
|
|
236
403
|
rule.characteristics.each do |char|
|
|
237
404
|
value = resolve_char_value(char, identifier)
|
|
238
|
-
key
|
|
405
|
+
key << ":#{char}:#{encode_char_value(value)}"
|
|
239
406
|
end
|
|
407
|
+
key << "}"
|
|
408
|
+
key << suffix if suffix
|
|
240
409
|
key
|
|
241
410
|
end
|
|
242
411
|
|
|
@@ -294,6 +463,30 @@ module Labkit
|
|
|
294
463
|
end
|
|
295
464
|
end
|
|
296
465
|
|
|
466
|
+
# Atomic ban check + increment + conditional ban write. See BAN_SCRIPT.
|
|
467
|
+
# The returned ban TTL is negative when no ban is in force.
|
|
468
|
+
def ban_incr_with_ttl(counter_key, ban_key, period, cost, limit, ban_for)
|
|
469
|
+
@redis.with do |conn|
|
|
470
|
+
raw_count, ttl, ban_ttl = BAN_SCRIPT.eval(
|
|
471
|
+
conn, keys: [counter_key, ban_key], argv: [period, cost, limit, ban_for]
|
|
472
|
+
)
|
|
473
|
+
[Float(raw_count), Integer(ttl), Integer(ban_ttl)]
|
|
474
|
+
end
|
|
475
|
+
end
|
|
476
|
+
|
|
477
|
+
# Pipelined read of both keys for #peek. No EXPIRE and no SET: peeking a
|
|
478
|
+
# banned identifier must not extend either the window or the ban.
|
|
479
|
+
def read_ban_with_ttl(counter_key, ban_key)
|
|
480
|
+
@redis.with do |conn|
|
|
481
|
+
raw_count, ttl, ban_ttl = conn.pipelined do |pipe|
|
|
482
|
+
pipe.get(counter_key)
|
|
483
|
+
pipe.ttl(counter_key)
|
|
484
|
+
pipe.ttl(ban_key)
|
|
485
|
+
end
|
|
486
|
+
[raw_count.nil? ? 0.0 : Float(raw_count), Integer(ttl), Integer(ban_ttl)]
|
|
487
|
+
end
|
|
488
|
+
end
|
|
489
|
+
|
|
297
490
|
# Pipelined GET + TTL. No EXPIRE: peek must not extend the window.
|
|
298
491
|
# A missing key (GET => nil, TTL => -2) is reported as count=0; the
|
|
299
492
|
# build_evaluation fallback then derives reset_at from the rule period
|
|
@@ -341,6 +534,7 @@ module Labkit
|
|
|
341
534
|
# loop reached one, or after it finished - so the field is logged as null
|
|
342
535
|
# rather than omitted, the same way identifier is. A named rule is the rule
|
|
343
536
|
# whose match or evaluation raised.
|
|
537
|
+
# Never raises: a logging failure must not break fail-open.
|
|
344
538
|
def log_error(error, identifier, rule = nil)
|
|
345
539
|
@logger.warn(
|
|
346
540
|
name: @name,
|
|
@@ -350,6 +544,8 @@ module Labkit
|
|
|
350
544
|
Labkit::Fields::ERROR_MESSAGE => error.message,
|
|
351
545
|
identifier: identifier&.to_h
|
|
352
546
|
)
|
|
547
|
+
rescue StandardError
|
|
548
|
+
nil
|
|
353
549
|
end
|
|
354
550
|
|
|
355
551
|
def log_missing_count_distinct(rule, identifier)
|
|
@@ -362,43 +558,67 @@ module Labkit
|
|
|
362
558
|
)
|
|
363
559
|
end
|
|
364
560
|
|
|
365
|
-
def
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
{ rate_limiter: @name, rule: evaluation.rule.name },
|
|
373
|
-
evaluation.info.resolved_limit
|
|
561
|
+
def report_evaluation_metrics(evaluation)
|
|
562
|
+
rule_labels = { rate_limiter: @name, rule: evaluation.rule.name }
|
|
563
|
+
|
|
564
|
+
Metrics.safe_increment(
|
|
565
|
+
:rule_evaluations_total,
|
|
566
|
+
rule_labels.merge(action: evaluation.rule.action.to_s, result: evaluation_result(evaluation)),
|
|
567
|
+
logger: @logger
|
|
374
568
|
)
|
|
375
|
-
Metrics.
|
|
376
|
-
|
|
377
|
-
|
|
569
|
+
# Deprecated dual emission - remove together with Metrics.calls_total.
|
|
570
|
+
Metrics.safe_increment(
|
|
571
|
+
:calls_total,
|
|
572
|
+
rule_labels.merge(action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s),
|
|
573
|
+
logger: @logger
|
|
378
574
|
)
|
|
575
|
+
Metrics.safe_set(:limit_gauge, rule_labels, evaluation.info.resolved_limit, logger: @logger)
|
|
576
|
+
Metrics.safe_set(:period_gauge, rule_labels, evaluation.info.resolved_period, logger: @logger)
|
|
577
|
+
end
|
|
578
|
+
|
|
579
|
+
# An exceeded :log rule reports "log" rather than the "allow" the caller
|
|
580
|
+
# sees, so shadow rules over their limit stay visible.
|
|
581
|
+
# A ban is not a distinct action, so without this a blocking ban would be
|
|
582
|
+
# indistinguishable from an ordinary block. On a ban_for rule exceeded?
|
|
583
|
+
# means the ban is in force, whether this call wrote it or an earlier one
|
|
584
|
+
# did, which is exactly what is worth counting separately.
|
|
585
|
+
def evaluation_result(evaluation)
|
|
586
|
+
return "allow" unless evaluation.exceeded?
|
|
587
|
+
return "banned" if evaluation.rule.ban_for
|
|
588
|
+
|
|
589
|
+
evaluation.rule.action == :limit ? "block" : "log"
|
|
379
590
|
end
|
|
380
591
|
|
|
381
|
-
# calls_total carries action="skip" (the rule action, not the :allow the
|
|
382
|
-
# caller sees) so bypass traffic stays distinguishable from counted
|
|
383
|
-
# allows. No limit/period gauges: a skip rule has no limit to report.
|
|
384
592
|
def report_skipped_metrics(rule)
|
|
385
|
-
Metrics.
|
|
386
|
-
|
|
387
|
-
rule: rule.name,
|
|
388
|
-
|
|
593
|
+
Metrics.safe_increment(
|
|
594
|
+
:rule_evaluations_total,
|
|
595
|
+
{ rate_limiter: @name, rule: rule.name, action: "skip", result: "skip" },
|
|
596
|
+
logger: @logger
|
|
389
597
|
)
|
|
598
|
+
# Deprecated dual emission - remove together with Metrics.calls_total.
|
|
599
|
+
Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: rule.name, action: "skip" }, logger: @logger)
|
|
390
600
|
end
|
|
391
601
|
|
|
602
|
+
# Deprecated dual emission - remove together with Metrics.calls_total.
|
|
392
603
|
def report_unmatched_metrics
|
|
393
|
-
Metrics.calls_total
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
604
|
+
Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: "unmatched", action: "allow" }, logger: @logger)
|
|
605
|
+
end
|
|
606
|
+
|
|
607
|
+
def report_check_metrics(result)
|
|
608
|
+
Metrics.safe_increment(
|
|
609
|
+
:checks_total,
|
|
610
|
+
{
|
|
611
|
+
rate_limiter: @name,
|
|
612
|
+
action: result.action.to_s,
|
|
613
|
+
matched: result.matched?.to_s,
|
|
614
|
+
error: (result.error? || result.degraded?).to_s
|
|
615
|
+
},
|
|
616
|
+
logger: @logger
|
|
397
617
|
)
|
|
398
618
|
end
|
|
399
619
|
|
|
400
620
|
def report_error_metrics
|
|
401
|
-
Metrics.errors_total
|
|
621
|
+
Metrics.safe_increment(:errors_total, { rate_limiter: @name }, logger: @logger)
|
|
402
622
|
end
|
|
403
623
|
end
|
|
404
624
|
end
|
|
@@ -45,7 +45,7 @@ module Labkit
|
|
|
45
45
|
# the rule definition and the call site colocated.
|
|
46
46
|
# @return [Result]
|
|
47
47
|
def check(identifier, cost: 1, rule_context: nil)
|
|
48
|
-
id =
|
|
48
|
+
id = to_identifier(identifier)
|
|
49
49
|
@evaluator.check(id, cost: cost, rule_context: rule_context)
|
|
50
50
|
end
|
|
51
51
|
|
|
@@ -65,12 +65,33 @@ module Labkit
|
|
|
65
65
|
# @param rule_context [Hash, nil] see {#check}
|
|
66
66
|
# @return [Result]
|
|
67
67
|
def peek(identifier, rule_context: nil)
|
|
68
|
-
id =
|
|
68
|
+
id = to_identifier(identifier)
|
|
69
69
|
@evaluator.peek(id, rule_context: rule_context)
|
|
70
70
|
end
|
|
71
71
|
|
|
72
|
+
# Discards this limiter's state for one identifier: every rule's counter,
|
|
73
|
+
# and any ban written by a rule carrying ban_for.
|
|
74
|
+
#
|
|
75
|
+
# For call sites where a later success should wipe earlier failures, such
|
|
76
|
+
# as an authentication ban cleared by a valid login. Counters otherwise
|
|
77
|
+
# only expire with their window; this is the only way to end one early.
|
|
78
|
+
#
|
|
79
|
+
# Scoped to this limiter and identifier, not to a single rule: a caller
|
|
80
|
+
# clearing after a success knows who succeeded, not which rules matched.
|
|
81
|
+
#
|
|
82
|
+
# @param identifier [Identifier, Hash] caller attributes
|
|
83
|
+
# @return [Integer] number of Redis keys removed; on a Redis error, those
|
|
84
|
+
# removed before it failed
|
|
85
|
+
def clear(identifier)
|
|
86
|
+
@evaluator.clear(to_identifier(identifier))
|
|
87
|
+
end
|
|
88
|
+
|
|
72
89
|
private
|
|
73
90
|
|
|
91
|
+
def to_identifier(identifier)
|
|
92
|
+
identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
|
|
93
|
+
end
|
|
94
|
+
|
|
74
95
|
def validate_name!(name)
|
|
75
96
|
raise ArgumentError, "name must be a non-empty String" unless name.is_a?(String) && !name.empty?
|
|
76
97
|
return name if NAME_PATTERN.match?(name)
|
|
@@ -1,14 +1,60 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "concurrent-ruby"
|
|
4
|
+
|
|
3
5
|
module Labkit
|
|
4
6
|
module RateLimit
|
|
5
7
|
module Metrics
|
|
8
|
+
# Process-wide once-latch for log_failure; specs reset via make_false.
|
|
9
|
+
FAILURE_LOGGED = Concurrent::AtomicBoolean.new(false)
|
|
10
|
+
|
|
6
11
|
module_function
|
|
7
12
|
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
|
|
11
|
-
|
|
13
|
+
# Metric emission must never affect the caller's outcome. Resolving the
|
|
14
|
+
# metric by name keeps a raising getter inside the rescue.
|
|
15
|
+
def safe_increment(counter, labels, logger: nil)
|
|
16
|
+
public_send(counter).increment(**labels) # rubocop:disable GitlabSecurity/PublicSend
|
|
17
|
+
rescue StandardError => e
|
|
18
|
+
log_failure(e, counter, labels, logger)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# Gauge counterpart of safe_increment.
|
|
22
|
+
def safe_set(gauge, labels, value, logger: nil)
|
|
23
|
+
public_send(gauge).set(labels, value) # rubocop:disable GitlabSecurity/PublicSend
|
|
24
|
+
rescue StandardError => e
|
|
25
|
+
log_failure(e, gauge, labels, logger)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# make_true returns true only for the flipping caller, so exactly one
|
|
29
|
+
# warn per process. Never raises - a raise here would defeat the
|
|
30
|
+
# callers' rescues.
|
|
31
|
+
def log_failure(error, metric, labels, logger)
|
|
32
|
+
return unless logger && FAILURE_LOGGED.make_true
|
|
33
|
+
|
|
34
|
+
logger.warn(
|
|
35
|
+
name: labels[:rate_limiter],
|
|
36
|
+
metric: metric.to_s,
|
|
37
|
+
Labkit::Fields::ERROR_TYPE => "rate_limit_metrics_error",
|
|
38
|
+
Labkit::Fields::CLASS_NAME => error.class.to_s,
|
|
39
|
+
Labkit::Fields::ERROR_MESSAGE => error.message
|
|
40
|
+
)
|
|
41
|
+
rescue StandardError
|
|
42
|
+
nil
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
# Emitted exactly once per #check call, including calls that fail open;
|
|
46
|
+
# summing by rate_limiter gives the request rate through the limiter.
|
|
47
|
+
def checks_total
|
|
48
|
+
Labkit::Metrics::Client.counter(
|
|
49
|
+
:gitlab_labkit_rate_limiter_checks_total,
|
|
50
|
+
'Total number of rate limit checks',
|
|
51
|
+
{ rate_limiter: nil, action: nil, matched: nil, error: nil }
|
|
52
|
+
)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# Deprecated: superseded by checks_total and rule_evaluations_total.
|
|
56
|
+
# Emitted unchanged (once per matched rule, rule="unmatched" when none
|
|
57
|
+
# matched) until consumers migrate; will get removed in a follow-up release.
|
|
12
58
|
def calls_total
|
|
13
59
|
Labkit::Metrics::Client.counter(
|
|
14
60
|
:gitlab_labkit_rate_limiter_calls_total,
|
|
@@ -17,6 +63,18 @@ module Labkit
|
|
|
17
63
|
)
|
|
18
64
|
end
|
|
19
65
|
|
|
66
|
+
# Emitted once per *evaluated* rule. Only matched rules are evaluated:
|
|
67
|
+
# Unmatched traffic is checks_total{matched="false"}
|
|
68
|
+
def rule_evaluations_total
|
|
69
|
+
Labkit::Metrics::Client.counter(
|
|
70
|
+
:gitlab_labkit_rate_limiter_rule_evaluations_total,
|
|
71
|
+
'Total number of rate limit rule evaluations',
|
|
72
|
+
{ rate_limiter: nil, rule: nil, action: nil, result: nil }
|
|
73
|
+
)
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
# Deprecated: superseded by checks_total{error="true"}. Still emitted
|
|
77
|
+
# because it remains the only error metric for #peek.
|
|
20
78
|
def errors_total
|
|
21
79
|
Labkit::Metrics::Client.counter(
|
|
22
80
|
:gitlab_labkit_rate_limiter_errors_total,
|
|
@@ -25,6 +25,8 @@ module Labkit
|
|
|
25
25
|
# false). Other rules may also have matched and been counted;
|
|
26
26
|
# see #evaluations.
|
|
27
27
|
# error? - true if Redis was unavailable; result fails open (exceeded? is false)
|
|
28
|
+
# degraded? - true if a matched count_distinct rule was skipped by the
|
|
29
|
+
# missing-key fail-open path; the check still completed
|
|
28
30
|
# info - Result::Info with per-window counters for the reported rule;
|
|
29
31
|
# nil when matched? is false, error?, or the matched rule is
|
|
30
32
|
# :skip (no counter exists)
|
|
@@ -40,6 +42,7 @@ module Labkit
|
|
|
40
42
|
@evaluations = []
|
|
41
43
|
@skip_rule = nil
|
|
42
44
|
@error = error
|
|
45
|
+
@degraded = false
|
|
43
46
|
@most_constraining = nil
|
|
44
47
|
end
|
|
45
48
|
|
|
@@ -78,6 +81,17 @@ module Labkit
|
|
|
78
81
|
@error
|
|
79
82
|
end
|
|
80
83
|
|
|
84
|
+
# Unlike error?, the check keeps going: the flag marks the outcome as
|
|
85
|
+
# degraded without changing it.
|
|
86
|
+
def degraded!
|
|
87
|
+
@degraded = true
|
|
88
|
+
self
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def degraded?
|
|
92
|
+
@degraded
|
|
93
|
+
end
|
|
94
|
+
|
|
81
95
|
def action
|
|
82
96
|
block? ? :block : :allow
|
|
83
97
|
end
|
|
@@ -119,9 +133,10 @@ module Labkit
|
|
|
119
133
|
# exceeded - whether the post-increment count exceeded the resolved limit
|
|
120
134
|
# info - Result::Info with the per-window counters
|
|
121
135
|
#
|
|
122
|
-
# Evaluations order by constraint: a blocking evaluation (:limit rule over
|
|
123
|
-
# its limit) ranks strictly first - it
|
|
124
|
-
# any other rule reports - then exceeded
|
|
136
|
+
# Evaluations order by constraint: a blocking evaluation (a :limit rule over
|
|
137
|
+
# its limit, or one whose ban is still in force) ranks strictly first - it
|
|
138
|
+
# decides the request no matter what any other rule reports - then exceeded
|
|
139
|
+
# ones, then fewest remaining.
|
|
125
140
|
# remaining floors at 0, so an exceeded :log rule ties with one sitting
|
|
126
141
|
# exactly on its limit; ranking exceeded ahead keeps a breach from being
|
|
127
142
|
# hidden by a rule that merely reached its limit.
|
|
@@ -14,10 +14,17 @@ module Labkit
|
|
|
14
14
|
# period - window in seconds; may be a callable (resolved per check)
|
|
15
15
|
# action - :limit (enforce; the result blocks when the rule is over
|
|
16
16
|
# its limit, which also terminates evaluation), :log (count
|
|
17
|
-
# and log only, never blocks and never terminates),
|
|
18
|
-
# :skip (bypass: permit and terminate evaluation on
|
|
19
|
-
# without counting; performs no Redis operation, so
|
|
20
|
-
# period, characteristics, and count_distinct are
|
|
17
|
+
# and log only, never blocks and never terminates),
|
|
18
|
+
# or :skip (bypass: permit and terminate evaluation on
|
|
19
|
+
# match without counting; performs no Redis operation, so
|
|
20
|
+
# limit, period, characteristics, and count_distinct are
|
|
21
|
+
# inert)
|
|
22
|
+
# ban_for - optional ban duration in seconds. On crossing the limit
|
|
23
|
+
# the rule writes a separate ban key that outlives the
|
|
24
|
+
# counter window, and stops counting while it holds.
|
|
25
|
+
# +action+ still decides who is blocked: :limit enforces
|
|
26
|
+
# the ban, :log records what it would have done. Rejected
|
|
27
|
+
# on :skip. May be a callable, resolved per check.
|
|
21
28
|
# characteristics - identifier keys used to build the compound Redis counter key
|
|
22
29
|
# count_distinct - optional Symbol naming an identifier key. When set, the rule
|
|
23
30
|
# counts the number of distinct values seen for that key within
|
|
@@ -28,7 +35,27 @@ module Labkit
|
|
|
28
35
|
# +name+ must be a lowercase alphanumeric-and-underscore string of at most 64
|
|
29
36
|
# characters. It is used as the middle segment of every Redis counter key for
|
|
30
37
|
# this rule, so changing a rule's name mid-window abandons its in-flight counters.
|
|
31
|
-
Rule = Data.define(:name, :match, :limit, :period, :action, :characteristics, :count_distinct) do
|
|
38
|
+
Rule = Data.define(:name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for) do
|
|
39
|
+
# ban_for changes the accounting; action still decides who gets blocked.
|
|
40
|
+
# A :skip rule counts nothing, so it has no limit to cross and no ban to
|
|
41
|
+
# write.
|
|
42
|
+
def self.normalize_ban_for(value, action_sym, count_distinct_sym)
|
|
43
|
+
return nil if value.nil?
|
|
44
|
+
|
|
45
|
+
raise ArgumentError, "ban_for is not valid on a :skip rule" if action_sym == :skip
|
|
46
|
+
|
|
47
|
+
# The ban path counts with INCRBYFLOAT, so a SET-backed rule would
|
|
48
|
+
# silently count calls instead of distinct values.
|
|
49
|
+
raise ArgumentError, "ban_for cannot be combined with count_distinct #{count_distinct_sym.inspect}" if count_distinct_sym
|
|
50
|
+
|
|
51
|
+
return value if value.respond_to?(:call)
|
|
52
|
+
# Whole seconds: the ban is written with SET EX, and anything under a
|
|
53
|
+
# second truncates to 0, which Redis rejects.
|
|
54
|
+
return value if value.is_a?(Numeric) && value >= 1
|
|
55
|
+
|
|
56
|
+
raise ArgumentError, "ban_for must be a Numeric of at least 1 second or a callable, got #{value.inspect}"
|
|
57
|
+
end
|
|
58
|
+
|
|
32
59
|
def self.normalize_count_distinct(value, characteristics_arr)
|
|
33
60
|
sym =
|
|
34
61
|
case value
|
|
@@ -44,7 +71,10 @@ module Labkit
|
|
|
44
71
|
sym
|
|
45
72
|
end
|
|
46
73
|
|
|
47
|
-
def initialize(
|
|
74
|
+
def initialize(
|
|
75
|
+
name:, limit:, period:, characteristics:,
|
|
76
|
+
match: {}, action: :limit, count_distinct: nil, ban_for: nil
|
|
77
|
+
)
|
|
48
78
|
raise ArgumentError, "name must be a String or Symbol, got #{name.class}" unless name.is_a?(String) || name.is_a?(Symbol)
|
|
49
79
|
|
|
50
80
|
name_str = name.to_s
|
|
@@ -59,6 +89,7 @@ module Labkit
|
|
|
59
89
|
end
|
|
60
90
|
|
|
61
91
|
characteristics_arr = Array(characteristics).map(&:to_sym).freeze
|
|
92
|
+
count_distinct_sym = self.class.normalize_count_distinct(count_distinct, characteristics_arr)
|
|
62
93
|
|
|
63
94
|
super(
|
|
64
95
|
name: name_str.freeze,
|
|
@@ -67,7 +98,8 @@ module Labkit
|
|
|
67
98
|
period: period,
|
|
68
99
|
action: action_sym,
|
|
69
100
|
characteristics: characteristics_arr,
|
|
70
|
-
count_distinct:
|
|
101
|
+
count_distinct: count_distinct_sym,
|
|
102
|
+
ban_for: self.class.normalize_ban_for(ban_for, action_sym, count_distinct_sym)
|
|
71
103
|
)
|
|
72
104
|
end
|
|
73
105
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: gitlab-labkit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 4.
|
|
4
|
+
version: 4.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrew Newdigate
|
|
@@ -716,7 +716,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
716
716
|
- !ruby/object:Gem::Version
|
|
717
717
|
version: '0'
|
|
718
718
|
requirements: []
|
|
719
|
-
rubygems_version: 4.0.
|
|
719
|
+
rubygems_version: 4.0.16
|
|
720
720
|
specification_version: 4
|
|
721
721
|
summary: Instrumentation for GitLab
|
|
722
722
|
test_files: []
|