gitlab-labkit 4.4.1 → 4.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1e7ab6ac58c584a113159e7f42678f508078c83cdeb9a21c86a7f60a5c936b21
4
- data.tar.gz: b4c3abaa02eafb546284dc68a08ba4443da4c1b13aa824c98f51e0dac3e749da
3
+ metadata.gz: 22a38c0fe1a2f28aa538b80348911947c6de64c16d7d870d24835332796faa99
4
+ data.tar.gz: 6259d696d178f3008aefdf91de71731c81d460cef08259555829f15d8abd462c
5
5
  SHA512:
6
- metadata.gz: 1b032aa2876124669bf1eac1bc09cf67e79b4df4105e95fbc7d416132dd8d5bf1088c49e4c88b304e9a796e119edea60e4c2fae49a70aba050889a928b66b206
7
- data.tar.gz: 91066213316a173023257e30507fdb8c7c0e4e0b360b91dd1efcafebbacd291c82f62e9470798aa86488ae991bb8f81b1c395856ee2e03b537e3ae5a244d78c3
6
+ metadata.gz: dd404f717fe9cee20cb544ad23cd5aa3195683ac0efe8bf8363988fdb8dcacc4fcaa817ac6f4cb3fcdb4f2e3d193c6d93cc13deea0344067e44fac7f3c593fb3
7
+ data.tar.gz: cc8d6bb5fe6fde8c17f969d4a708bd41901de5f9383f4821e3acf13a2eb35c240a80d18d3853ac5f29b689d7841df254f540233174646204358e698b9e0f643e
@@ -1,5 +1,5 @@
1
1
  # DO NOT MANUALLY EDIT; Run ./scripts/update-asdf-version-variables.sh to update this
2
2
  variables:
3
- GL_ASDF_RUBY_VERSION: "4.0.5"
3
+ GL_ASDF_RUBY_VERSION: "4.0.6"
4
4
  GL_ASDF_SHELLCHECK_VERSION: "0.11"
5
5
  GL_ASDF_SHFMT_VERSION: "3.13"
data/.tool-versions CHANGED
@@ -1,3 +1,3 @@
1
- ruby 4.0.5
1
+ ruby 4.0.6
2
2
  shfmt 3.13
3
3
  shellcheck 0.11
@@ -121,6 +121,29 @@ different question than `check`. A matched `:skip` rule terminates `peek`
121
121
  the same way it terminates `check`: matched, `:allow`, no Redis read, no
122
122
  `info`.
123
123
 
124
+ ### Clearing state
125
+
126
+ `Limiter#clear(identifier)` deletes this limiter's counters for one
127
+ identifier, along with any ban a rule carrying `ban_for` has written, and returns the
128
+ number of keys removed.
129
+
130
+ ```ruby
131
+ RACK_LIMITER.clear({ ip: "1.2.3.4" }) # => 2
132
+ ```
133
+
134
+ Counters otherwise only disappear when their window expires; this is the only
135
+ way to end one early. It exists for call sites where a later success should
136
+ wipe earlier failures, such as an authentication ban cleared by a valid
137
+ login.
138
+
139
+ Every rule in the limiter is cleared, matched or not, because a caller
140
+ clearing state after a success knows the identifier rather than which rules
141
+ happened to match on the way in. Each rule is deleted in its own call, since
142
+ rules do not share a Redis Cluster slot. Clearing an identifier with no state
143
+ is a no-op returning `0`. A Redis failure fails open like `check`, leaving the
144
+ rest of the state to expire on its own, and the return value counts whatever
145
+ was removed before the failure.
146
+
124
147
  ## Identifier
125
148
 
126
149
  `Identifier` is a small value object wrapping a hash of caller attributes.
@@ -150,6 +173,8 @@ A `Rule` is a `Data.define` value object with the following fields:
150
173
  | `period` | Window length in seconds. May be a callable resolved on every check. |
151
174
  | `action` | What the rule does when it matches. One of `:limit`, `:log`, `:skip`. Default `:limit`. See [Actions](#actions). |
152
175
  | `characteristics` | Array of identifier keys whose values are folded into the Redis counter key. Each unique combination gets its own counter. |
176
+ | `count_distinct` | Optional identifier key. When set, the rule counts distinct values seen for that key within the window, backed by a Redis SET rather than a counter. Must not overlap `characteristics`. |
177
+ | `ban_for` | Optional ban duration, whole seconds, minimum 1. Changes the accounting; `action` still decides who is blocked. Rejected on `:skip`. May be a callable resolved on every check. |
153
178
 
154
179
  Making `limit` or `period` callable is the supported pattern for
155
180
  runtime-tunable thresholds (e.g. feature flags or database-backed settings):
@@ -219,7 +244,7 @@ An exceeded rule has `remaining` 0 and therefore outranks any rule still under
219
244
  its limit. For a `:log` rule that means shadow traffic surfaces to the caller
220
245
  as `exceeded? == true` (with `action` still `:allow`) — visible, but unable to
221
246
  block. A `:log`-only path that matches therefore returns `matched? == true`
222
- and emits no `rule="unmatched"` metric.
247
+ and its check is counted as `checks_total{matched="true"}`.
223
248
 
224
249
  ```mermaid
225
250
  flowchart TD
@@ -227,20 +252,22 @@ flowchart TD
227
252
  Iter -->|yes| Match{rule.match<br/>all satisfied?}
228
253
  Match -->|no| Iter
229
254
  Match -->|yes| Skip{rule.action<br/>== :skip?}
230
- Skip -->|"yes (no Redis op)"| SkipEmit[Emit calls_total<br/>action=skip]
255
+ Skip -->|"yes (no Redis op)"| SkipEmit[Emit rule_evaluations_total<br/>action=skip, result=skip]
231
256
  SkipEmit --> SkipReturn([Return matched=true<br/>action=:allow])
232
- Skip -->|no| Eval["INCR Redis counter<br/>(see Redis sequence below)"]
257
+ Skip -->|no| Ban{"ban_for rule with<br/>a ban in force?"}
258
+ Ban -->|"yes (no increment)"| Build
259
+ Ban -->|no| Eval["INCR Redis counter<br/>(see Redis sequence below)"]
233
260
  Eval --> Build[Build Evaluation<br/>resolve limit/period]
234
261
  Build --> Add[Add evaluation to Result]
235
- Add --> Emit[Emit calls_total + limit/period gauges]
236
- Emit --> Act{"result.block?<br/>(:limit rule over limit)"}
262
+ Add --> Emit[Emit rule_evaluations_total<br/>+ limit/period gauges]
263
+ Emit --> Act{"result.block?<br/>(:limit rule over its limit,<br/>or its ban in force)"}
237
264
  Act -->|yes| Return([Return Result<br/>action=:block])
238
- Act -->|"no (:log, or :limit under limit)"| Iter
239
- Iter -->|no more rules| Any{any rule<br/>evaluated?}
240
- Any -->|yes| ReturnFold([Return Result reporting<br/>most-constraining evaluation])
241
- Any -->|no| Unmatched[Emit calls_total<br/>rule=unmatched, action=allow]
242
- Unmatched --> ReturnUnmatched([Return matched=false<br/>action=:allow])
243
- Eval -. StandardError .-> Error[Emit errors_total<br/>log warn]
265
+ Act -->|"no (:log, or under limit)"| Iter
266
+ Iter -->|no more rules| Return2([Return Result reporting<br/>most-constraining evaluation,<br/>or matched=false if none])
267
+ Return --> Check[Emit checks_total<br/>action, matched, error]
268
+ SkipReturn --> Check
269
+ Return2 --> Check
270
+ Eval -. StandardError .-> Error[Emit errors_total +<br/>checks_total error=true<br/>log warn]
244
271
  Error --> ReturnErr([Return error=true<br/>action=:allow])
245
272
  ```
246
273
 
@@ -249,23 +276,51 @@ discards the verdicts of the rules already evaluated, even though their counters
249
276
  were incremented. Those requests are counted but produce no verdict — the
250
277
  fail-open trade-off is that a request is never blocked on a partial evaluation.
251
278
 
252
- Note that `calls_total` is emitted **per matched rule**, so summing it by
253
- `rate_limiter` counts rule evaluations, not requests.
279
+ `rule_evaluations_total` is emitted **per evaluated rule** (including matched
280
+ `:skip` rules); `checks_total` is emitted **exactly once per check**, whatever
281
+ path the evaluation takes — so summing `checks_total` by `rate_limiter` counts
282
+ requests through the limiter, and "no rule matched" is `matched="false"` on the
283
+ check rather than a placeholder rule.
284
+
285
+ Only matched rules are evaluated: a rule whose `match:` conditions are not
286
+ satisfied emits nothing to `rule_evaluations_total` (match-testing is not an
287
+ evaluation), so the counter has no "didn't match" population and needs no
288
+ label for it. A matched `count_distinct` rule skipped by the missing-key
289
+ fail-open also emits nothing here — it was never evaluated; that check is
290
+ visible via `checks_total{error="true"}` and the
291
+ `rate_limit_missing_count_distinct` log, which carries the rule name.
292
+
293
+ During the transition the deprecated `calls_total` counter is additionally
294
+ emitted at every point the diagram emits `rule_evaluations_total` (with its
295
+ historical per-rule semantics, including the `rule="unmatched"` placeholder
296
+ after the loop). It is not shown above to keep the diagram legible; see the
297
+ Metrics table below.
254
298
 
255
299
  ### Actions
256
300
 
257
301
  The rule's `action` describes what the rule does; the result's `action`
258
302
  describes the outcome — what the caller should do — and is only ever `:allow`
259
- or `:block`. The counter is always incremented when a rule matches, except for
260
- `:skip` rules, which never touch Redis:
261
-
262
- | rule action | what it does | exceeded? | result action | terminating? |
263
- |-------------|-----------------------------------------------|-----------|---------------|---------------|
264
- | `:limit` | count against the limit | no | `:allow` | no continue |
265
- | `:limit` | count against the limit | yes | `:block` | yes — stop |
266
- | `:log` | count against the limit (observability only) | no | `:allow` | no — continue |
267
- | `:log` | count against the limit (observability only) | yes | `:allow` | nocontinue |
268
- | `:skip` | don't count (bypass) | n/a | `:allow` | yesstop |
303
+ or `:block`. A matching rule increments its counter, with two exceptions:
304
+ `:skip` rules never touch Redis, and a rule carrying `ban_for` whose ban is
305
+ already in force short-circuits before incrementing. The `result` label on
306
+ `rule_evaluations_total` records what each evaluation decided:
307
+
308
+ | rule action | what it does | exceeded? | result action | `result` label | terminating? |
309
+ |-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|
310
+ | `:limit` | count against the limit | no | `:allow` | `allow` | no — continue |
311
+ | `:limit` | count against the limit | yes | `:block` | `block` | yes stop |
312
+ | `:log` | count against the limit (observability only) | no | `:allow` | `allow` | no continue |
313
+ | `:log` | count against the limit (observability only) | yes | `:allow` | `log` | no — continue |
314
+ | `:skip` | don't count (bypass) | n/a | `:allow` | `skip` | yes — stop |
315
+
316
+ A rule may also carry `ban_for`, which changes the accounting rather than the
317
+ action. On crossing the limit it writes a ban that outlives the counting
318
+ window, and it stops counting while that ban holds:
319
+
320
+ | rule action | with `ban_for` | exceeded? | result action | `result` label | terminating? |
321
+ |-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|
322
+ | `:limit` | count, or skip counting while banned | yes | `:block` | `banned` | yes — stop |
323
+ | `:log` | same accounting, records what it would do | yes | `:allow` | `banned` | no — continue |
269
324
 
270
325
  - `:limit` — when exceeded, `Result#action` is `:block` and evaluation
271
326
  terminates. Caller should reject the request (e.g. with HTTP 429). When under
@@ -281,17 +336,53 @@ or `:block`. The counter is always incremented when a rule matches, except for
281
336
  `Result#action` `:allow` **without any Redis operation**: nothing is
282
337
  counted, so `limit`, `period`, `characteristics`, and `count_distinct` are
283
338
  inert and the result carries no `info` (`to_response_headers` is `{}`).
284
- The match is still observable via `calls_total{action="skip"}`. Use this
285
- for bypasses.
339
+ The match is still observable via `rule_evaluations_total{action="skip"}`.
340
+ Use this for bypasses.
341
+ - `ban_for` — enforcement that outlives the window, on either counting action.
342
+ On crossing the limit the rule writes a second key that keeps blocking for
343
+ `ban_for` seconds, long after the counter window has expired. While that ban
344
+ is in force the rule reports `exceeded? == true` and does not count, so a
345
+ caller cannot extend its own ban, and `reset_at` reports when the ban lifts
346
+ rather than when the window rolls. Use this where repeated failures should
347
+ cost more than a window, such as authentication attempts. State can be
348
+ discarded early with [`#clear`](#clearing-state).
349
+
350
+ With `action: :limit` the ban blocks. With `action: :log` the rule does all
351
+ of the same accounting, including writing the ban and suppressing counting
352
+ while it holds, and only skips the blocking, so a shadow measures what
353
+ enforcement would have produced rather than approximating it.
354
+
355
+ A `ban_for` shorter than `period` is legal but surprising: the ban can lapse
356
+ while the counter is still over the limit, so `peek` reports not-exceeded
357
+ while the next `check` re-bans immediately. Bans are normally longer than the
358
+ window they are counted in.
359
+
360
+ The duration is whole seconds and must be at least 1, since the ban is
361
+ written with `SET EX`. A callable is checked again each time it resolves, and
362
+ a value under a second raises rather than reaching Redis. Being an error, it
363
+ fails open like any other, so a rule whose `ban_for` cannot resolve stops
364
+ blocking entirely: watch `errors_total` after changing one.
365
+
366
+ Take care combining `ban_for` with `cost:`. One expensive call can cross the
367
+ limit on its own, and with a ban attached that costs the caller the whole ban
368
+ rather than a single rejection. Checking with `peek` first has the same edge,
369
+ since the call it deliberately lets through is enough to start a ban. Bans
370
+ suit counting discrete failures, such as bad credentials, better than they
371
+ suit variable-cost resource limits.
286
372
 
287
373
  ### Redis keys
288
374
 
289
375
  Each matched check writes a key shaped:
290
376
 
291
377
  ```
292
- labkit:rl:<limiter_name>:<rule_name>:<char>:<value>[:<char>:<value>...]
378
+ labkit:rl:{<limiter_name>:<rule_name>:<char>:<value>[:<char>:<value>...]}
293
379
  ```
294
380
 
381
+ The braces are a Redis [hash tag](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/#hash-tags):
382
+ on a Redis Cluster only the braced part decides the slot, so every key for one
383
+ rule and identifier lands on the same shard. `BAN_SCRIPT` touches two keys in
384
+ one call, and a cluster rejects a script whose keys span slots.
385
+
295
386
  Characteristic values longer than 200 bytes are replaced with a SHA-256
296
387
  hexdigest to bound key length. Missing or empty characteristic values are
297
388
  encoded as `_unknown_`. The TTL is set on the first write of each window and
@@ -299,9 +390,24 @@ is not extended on subsequent increments, so the window is a true fixed
299
390
  window starting at the first request, not a sliding window.
300
391
 
301
392
  A check is a single `EVALSHA` of `INCR_SCRIPT` (or `SADD_SCRIPT` for
302
- `count_distinct` rules). Doing the whole read-modify-write inside Lua means
303
- there is no window between the increment and the `EXPIRE` in which a key
304
- could be left without a TTL.
393
+ `count_distinct` rules, or `BAN_SCRIPT` for rules carrying `ban_for`). Doing the whole
394
+ read-modify-write inside Lua means there is no window between the increment
395
+ and the `EXPIRE` in which a key could be left without a TTL.
396
+
397
+ A rule carrying `ban_for` writes a second key alongside its counter, sharing the
398
+ same hash tag and carrying its own `ban_for` TTL:
399
+
400
+ ```
401
+ labkit:rl:{<limiter_name>:<rule_name>:<char>:<value>[:<char>:<value>...]}:ban
402
+ ```
403
+
404
+ The suffix sits outside the tag, so it cannot change the slot and cannot
405
+ collide with a characteristic value, and both keys are found by the same
406
+ `labkit:rl:{<limiter>:<rule>*` glob. The ban key outlives its counter by
407
+ design: that is the whole point of `ban_for`.
408
+ `BAN_SCRIPT` checks the ban before incrementing, so the ban check,
409
+ the increment and the ban write are one atomic operation and a concurrent
410
+ check cannot miss the threshold crossing.
305
411
 
306
412
  ```mermaid
307
413
  sequenceDiagram
@@ -375,26 +481,53 @@ safe to merge unconditionally.
375
481
 
376
482
  The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
377
483
  (Redis connection failure, timeout, OOM in user-supplied callables, …) is
378
- logged at WARN with `message: "rate_limit_error"` and returned as an error
379
- `Result` (`matched?` false, `error?` true, `action` `:allow`). The
380
- `gitlab_labkit_rate_limiter_errors_total` counter is incremented. The caller
484
+ logged at WARN with `error_type: "rate_limit_error"` and returned as an error
485
+ `Result` (`matched?` false, `error?` true, `action` `:allow`). The caller
381
486
  should treat the request as allowed.
382
487
 
488
+ A failed-open `check` is still counted: it emits
489
+ `checks_total{action="allow", matched="false", error="true"}`, so the fraction
490
+ of checks that encountered an error is `checks_total{error="true"}` over
491
+ `checks_total`. `error="true"` also covers a check where a matched
492
+ `count_distinct` rule was skipped because the identifier was missing its
493
+ `count_distinct` key — that check completes (`action` and `matched` describe
494
+ its outcome as usual), so `error="true"` is not exclusively fail-open traffic.
495
+
496
+ The deprecated `gitlab_labkit_rate_limiter_errors_total` counter is still
497
+ incremented on every fail-open (whole-check and per-rule `count_distinct`),
498
+ and remains the only error metric for `peek`, which emits no `checks_total`
499
+ (peek must not inflate the per-check counter). The follow-up MR that removes
500
+ `errors_total` must first decide where `peek` errors go — a dedicated peek
501
+ metric, or logs only.
502
+
503
+ Metric emission itself is best-effort: a failure in the metrics stack never
504
+ alters the verdict or breaks fail-open, and is logged at WARN with
505
+ `error_type: "rate_limit_metrics_error"` (once per process, to avoid
506
+ flooding).
507
+
383
508
  ## Metrics
384
509
 
385
510
  `Labkit::RateLimit::Metrics` emits the following Prometheus metrics through
386
511
  `Labkit::Metrics::Client`:
387
512
 
388
- | metric | type | labels | meaning |
389
- |-------------------------------------------------|---------|-------------------------------------|----------------------------------------------------------------------|
390
- | `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | One increment per counted rule (plus one per matched `:skip` rule). `action` is the rule-level outcome: `"allow"` (under limit), `"limit"` (blocking `:limit` rule), `"log"` (exceeded `:log` rule), `"skip"`. `rule="unmatched", action="allow"` when no rule matched. |
391
- | `gitlab_labkit_rate_limiter_errors_total` | counter | `rate_limiter` | Fail-open events (any `StandardError` in the labkit path). |
392
- | `gitlab_labkit_rate_limiter_limit` | gauge | `rate_limiter`, `rule` | Resolved limit at the last check (useful when `limit:` is callable). |
393
- | `gitlab_labkit_rate_limiter_period_seconds` | gauge | `rate_limiter`, `rule` | Resolved period at the last check. |
394
-
395
- Because every matching rule is counted, a single `check` call can emit
396
- **multiple** `calls_total` increments: one per counted rule (or a single
397
- `rule="unmatched"` increment if nothing matched).
513
+ | metric | type | labels | meaning |
514
+ |-----------------------------------------------------|---------|----------------------------------------------|----------------------------------------------------------------------|
515
+ | `gitlab_labkit_rate_limiter_checks_total` | counter | `rate_limiter`, `action`, `matched`, `error` | Exactly one increment per `check` call, including fail-open. `action` is what the caller should do (`"allow"` or `"block"`); `matched` and `error` are `"true"`/`"false"`. |
516
+ | `gitlab_labkit_rate_limiter_rule_evaluations_total` | counter | `rate_limiter`, `rule`, `action`, `result` | One increment per evaluated rule (plus one per matched `:skip` rule). `action` is the configured rule action (`"limit"`, `"log"`, `"skip"`); `result` is what the evaluation decided (`"allow"`, `"block"`, `"log"`, `"skip"`, `"banned"` — see the Actions table). |
517
+ | `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | **Deprecated** superseded by `checks_total` + `rule_evaluations_total`. Historical per-rule counter: one increment per counted rule (plus one per matched `:skip` rule), `rule="unmatched", action="allow"` when no rule matched. Emitted unchanged during the transition. |
518
+ | `gitlab_labkit_rate_limiter_errors_total` | counter | `rate_limiter` | **Deprecated** — use `checks_total{error="true"}`. Fail-open events (any `StandardError` in the labkit path); still the only error metric for `peek`. |
519
+ | `gitlab_labkit_rate_limiter_limit` | gauge | `rate_limiter`, `rule` | Resolved limit at the last check (useful when `limit:` is callable). |
520
+ | `gitlab_labkit_rate_limiter_period_seconds` | gauge | `rate_limiter`, `rule` | Resolved period at the last check. |
521
+
522
+ `sum by (rate_limiter) (rate(gitlab_labkit_rate_limiter_checks_total[5m]))` is
523
+ the request rate through a limiter — no exclusions or dedup needed. A single
524
+ `check` call emits **one** `checks_total` increment and as many
525
+ `rule_evaluations_total` increments as rules it evaluated (possibly zero).
526
+
527
+ **Transition:** the deprecated `calls_total` and `errors_total` counters keep
528
+ emitting exactly as before this split, so existing dashboards and alerts stay
529
+ correct while consumers migrate to the new counters. Both are removed in a
530
+ follow-up major release once nothing consumes them.
398
531
 
399
532
  ## Dev/test vs production guards
400
533
 
@@ -9,6 +9,7 @@ module Labkit
9
9
  # @api private
10
10
  class Evaluator
11
11
  REDIS_KEY_PREFIX = "labkit:rl"
12
+ BAN_KEY_SUFFIX = ":ban"
12
13
  CHAR_VALUE_MAX_LENGTH = 200
13
14
  MISSING_VALUE_SENTINEL = "_unknown_"
14
15
 
@@ -66,6 +67,50 @@ module Labkit
66
67
  return {count, ttl_after}
67
68
  LUA
68
69
 
70
+ # Atomic ban check + INCR + conditional ban write, for rules carrying
71
+ # ban_for.
72
+ #
73
+ # An active ban short-circuits before the increment, so a banned caller
74
+ # cannot extend its own window, matching Rack::Attack::Allow2Ban. This
75
+ # applies to :log rules too: a shadow that kept counting through its own
76
+ # ban would report more than enforcement would produce. The
77
+ # counter and the ban must be read and written together or a concurrent
78
+ # check can miss the threshold crossing.
79
+ #
80
+ # Returns {count, counter TTL, ban TTL}, where a ban TTL below zero means
81
+ # not banned.
82
+ BAN_SCRIPT = Labkit::Redis::Script.new(<<~LUA)
83
+ local rule_counter_key = KEYS[1]
84
+ local ban_key = KEYS[2]
85
+
86
+ local ttl = ARGV[1]
87
+ local cost = tonumber(ARGV[2])
88
+ local limit = tonumber(ARGV[3])
89
+ local ban_for = tonumber(ARGV[4])
90
+
91
+ -- TTL answers both questions at once: >= 0 means the ban key exists
92
+ -- and tells us how long is left, so no separate EXISTS is needed.
93
+ local ban_ttl = redis.call('TTL', ban_key)
94
+ if ban_ttl >= 0 then
95
+ local existing = redis.call('GET', rule_counter_key)
96
+ return {existing or '0', redis.call('TTL', rule_counter_key), ban_ttl}
97
+ end
98
+
99
+ local count = redis.call('INCRBYFLOAT', rule_counter_key, cost)
100
+ local ttl_after = redis.call('TTL', rule_counter_key)
101
+ if ttl_after < 0 then
102
+ redis.call('EXPIRE', rule_counter_key, ttl)
103
+ ttl_after = tonumber(ttl)
104
+ end
105
+
106
+ if tonumber(count) > limit then
107
+ redis.call('SET', ban_key, '1', 'EX', ban_for)
108
+ return {count, ttl_after, ban_for}
109
+ end
110
+
111
+ return {count, ttl_after, -1}
112
+ LUA
113
+
69
114
  def initialize(name:, rules:, redis:, logger:)
70
115
  @name = name
71
116
  @rules = rules
@@ -75,13 +120,17 @@ module Labkit
75
120
 
76
121
  def check(identifier, cost: 1, rule_context: nil)
77
122
  cursor = RuleCursor.new
78
- check_rules(identifier, cost, rule_context, cursor)
123
+ result = check_rules(identifier, cost, rule_context, cursor)
79
124
  rescue StandardError => e
80
125
  # Intentionally broad: fail-open applies to any unexpected error (network,
81
126
  # timeout, OOM) not only Redis protocol errors.
82
127
  report_error_metrics
83
128
  log_error(e, identifier, cursor.rule)
84
- Result.error
129
+ result = Result.error
130
+ ensure
131
+ # StandardError-safe emission, so it cannot mask a propagating error.
132
+ # result is nil when a non-StandardError unwinds: emit nothing then.
133
+ report_check_metrics(result) if result
85
134
  end
86
135
 
87
136
  # Read-without-increment counterpart to {#check}. Same matching and Result
@@ -96,6 +145,41 @@ module Labkit
96
145
  Result.error
97
146
  end
98
147
 
148
+ # Deletes this limiter's counters, and any bans, for one identifier.
149
+ #
150
+ # Every rule is cleared, matched or not: a caller clearing state after a
151
+ # success knows the identifier, not which rules happened to match on the
152
+ # way in. Rules whose keys do not exist are a no-op.
153
+ #
154
+ # Fails open like {#check}: an unreachable Redis leaves the state to
155
+ # expire on its own rather than raising into the caller.
156
+ def clear(identifier)
157
+ # Counted as we go, so a connection lost mid-loop still reports the keys
158
+ # that did go rather than claiming nothing was cleared.
159
+ removed = 0
160
+
161
+ key_groups = @rules.filter_map do |rule|
162
+ next if rule.action == :skip
163
+
164
+ [build_redis_key(rule, identifier), build_redis_key(rule, identifier, BAN_KEY_SUFFIX)]
165
+ end
166
+
167
+ return 0 if key_groups.empty?
168
+
169
+ # One DEL per rule. The hash tag holds the rule name, so a rule's own
170
+ # keys share a slot but two rules do not, and Redis Cluster rejects a
171
+ # DEL spanning slots.
172
+ @redis.with do |conn|
173
+ key_groups.each { |group| removed += conn.del(*group) }
174
+ end
175
+
176
+ removed
177
+ rescue StandardError => e
178
+ report_error_metrics
179
+ log_error(e, identifier, nil)
180
+ removed
181
+ end
182
+
99
183
  private
100
184
 
101
185
  # Every rule that matches is evaluated and counted; matching does not
@@ -104,21 +188,29 @@ module Labkit
104
188
  # - :skip terminates on match without touching Redis. No counter is
105
189
  # incremented, so the branch sits before the count_distinct check
106
190
  # (identifier completeness is irrelevant to a rule that builds no key).
107
- # calls_total still increments so the bypass stays observable.
191
+ # rule_evaluations_total still increments so the bypass stays observable.
108
192
  # - a :limit rule over its limit terminates, because the request is
109
193
  # rejected and later rules cannot change that. Rules declared after it
110
194
  # are neither counted nor evaluated, so a blocked request debits every
111
195
  # rule up to and including the one that blocked, and none after it.
196
+ # A :limit rule carrying ban_for counts as over its limit for as long
197
+ # as its ban holds, so it terminates for the whole ban. The same rule
198
+ # with action :log does identical accounting and terminates nothing,
199
+ # which is what makes it a faithful shadow.
112
200
  #
113
201
  # Every other matched rule - :log, and :limit while under its limit - is
114
202
  # counted and collected into the returned Result, which reports the
115
203
  # most-constraining evaluation (ranking lives in Result::Evaluation#<=>).
116
204
  # cost is therefore debited from every matching rule, not just the first.
117
205
  #
206
+ # Metrics: each evaluated rule emits rule_evaluations_total (plus the
207
+ # deprecated per-rule calls_total); the per-check checks_total is
208
+ # emitted once in #check. Full contract in the README's Metrics section.
209
+ #
118
210
  # SET-mode rules (rule.count_distinct set) that match but whose identifier
119
- # is missing the count_distinct key fail open + log + bump errors_total, and
120
- # the loop continues to the next rule (the rule is treated as not applicable
121
- # rather than aborting the whole evaluation).
211
+ # is missing the count_distinct key fail open + log + bump errors_total +
212
+ # flag the Result, and the loop continues to the next rule (the rule is
213
+ # treated as not applicable rather than aborting the whole evaluation).
122
214
  #
123
215
  # Error handling stays whole-check (see #check): a raise part-way through
124
216
  # discards the results of the rules already evaluated, even though their
@@ -143,12 +235,13 @@ module Labkit
143
235
  if rule.count_distinct && missing_count_distinct_value?(rule, identifier)
144
236
  log_missing_count_distinct(rule, identifier)
145
237
  report_error_metrics
238
+ result.degraded!
146
239
  next
147
240
  end
148
241
 
149
242
  evaluation = evaluate_rule(rule, identifier, cost, rule_context)
150
243
  result.add_evaluation(evaluation)
151
- report_matched_metrics(evaluation)
244
+ report_evaluation_metrics(evaluation)
152
245
  return result if result.block?
153
246
  end
154
247
 
@@ -157,6 +250,12 @@ module Labkit
157
250
 
158
251
  report_unmatched_metrics unless result.matched?
159
252
  result
253
+ rescue StandardError => e # binds e for the ensure
254
+ raise
255
+ ensure
256
+ # Keep the rule attributed while an exception unwinds; clear it on
257
+ # every normal exit (early returns included).
258
+ cursor.rule = nil unless e
160
259
  end
161
260
 
162
261
  # Mirror of check_rules without metrics or writes. :log rules are read
@@ -180,9 +279,11 @@ module Labkit
180
279
  return result if result.block?
181
280
  end
182
281
 
183
- cursor.rule = nil
184
-
185
282
  result
283
+ rescue StandardError => e # binds e for the ensure
284
+ raise
285
+ ensure
286
+ cursor.rule = nil unless e
186
287
  end
187
288
 
188
289
  def rule_matches?(rule, identifier)
@@ -195,6 +296,8 @@ module Labkit
195
296
  end
196
297
 
197
298
  def evaluate_rule(rule, identifier, cost, rule_context)
299
+ return evaluate_ban_rule(rule, identifier, cost, rule_context) if rule.ban_for
300
+
198
301
  redis_key = build_redis_key(rule, identifier)
199
302
  resolved_limit = Integer(resolve_value(rule.limit, rule_context))
200
303
  resolved_period = Integer(resolve_value(rule.period, rule_context))
@@ -212,6 +315,8 @@ module Labkit
212
315
  end
213
316
 
214
317
  def peek_rule(rule, identifier, rule_context)
318
+ return peek_ban_rule(rule, identifier, rule_context) if rule.ban_for
319
+
215
320
  redis_key = build_redis_key(rule, identifier)
216
321
  resolved_limit = Integer(resolve_value(rule.limit, rule_context))
217
322
  resolved_period = Integer(resolve_value(rule.period, rule_context))
@@ -220,6 +325,63 @@ module Labkit
220
325
  build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
221
326
  end
222
327
 
328
+ def evaluate_ban_rule(rule, identifier, cost, rule_context)
329
+ resolved_limit = Integer(resolve_value(rule.limit, rule_context))
330
+ resolved_period = Integer(resolve_value(rule.period, rule_context))
331
+ resolved_ban_for = resolve_ban_for(rule, rule_context)
332
+
333
+ count, ttl, ban_ttl = ban_incr_with_ttl(
334
+ build_redis_key(rule, identifier),
335
+ build_redis_key(rule, identifier, BAN_KEY_SUFFIX),
336
+ resolved_period, cost, resolved_limit, resolved_ban_for
337
+ )
338
+
339
+ build_ban_evaluation(rule, resolved_limit, resolved_period, count, ttl, ban_ttl)
340
+ end
341
+
342
+ # A callable ban_for is only checked at construction, so it can still hand
343
+ # back something Redis would reject on the SET that writes the ban. Failing
344
+ # here names ban_for rather than surfacing a line number from the script.
345
+ def resolve_ban_for(rule, rule_context)
346
+ raw = resolve_value(rule.ban_for, rule_context)
347
+ seconds = Integer(raw, exception: false)
348
+ raise ArgumentError, "ban_for resolved to #{raw.inspect}, need at least 1 second" if seconds.nil? || seconds < 1
349
+
350
+ seconds
351
+ end
352
+
353
+ def peek_ban_rule(rule, identifier, rule_context)
354
+ resolved_limit = Integer(resolve_value(rule.limit, rule_context))
355
+ resolved_period = Integer(resolve_value(rule.period, rule_context))
356
+
357
+ count, ttl, ban_ttl = read_ban_with_ttl(
358
+ build_redis_key(rule, identifier),
359
+ build_redis_key(rule, identifier, BAN_KEY_SUFFIX)
360
+ )
361
+
362
+ build_ban_evaluation(rule, resolved_limit, resolved_period, count, ttl, ban_ttl)
363
+ end
364
+
365
+ # A ban blocks whether it was written by this call or an earlier one, so
366
+ # +exceeded+ tracks the ban rather than the count: once the counter window
367
+ # has expired the count can sit below the limit while the ban still holds.
368
+ # reset_at reports when the caller may retry, which is the ban expiry.
369
+ def build_ban_evaluation(rule, resolved_limit, resolved_period, count, ttl, ban_ttl)
370
+ banned = ban_ttl >= 0
371
+ window_remaining = ttl >= 0 ? ttl : resolved_period
372
+
373
+ info = Result::Info.new(
374
+ resolved_limit: resolved_limit, resolved_period: resolved_period,
375
+ count: count,
376
+ # A ban outlives its counter, so count reads 0 once the window has
377
+ # gone. Nothing is remaining while the ban still blocks.
378
+ remaining: banned ? 0 : [resolved_limit - count, 0].max,
379
+ reset_at: Time.now.utc + (banned ? ban_ttl : window_remaining)
380
+ )
381
+
382
+ Result::Evaluation.new(rule: rule, exceeded: banned, info: info)
383
+ end
384
+
223
385
  def build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
224
386
  info = Result::Info.new(
225
387
  resolved_limit: resolved_limit, resolved_period: resolved_period,
@@ -231,12 +393,19 @@ module Labkit
231
393
  Result::Evaluation.new(rule: rule, exceeded: count > resolved_limit, info: info)
232
394
  end
233
395
 
234
- def build_redis_key(rule, identifier)
235
- key = "#{REDIS_KEY_PREFIX}:#{@name}:#{rule.name}"
396
+ # The limiter, rule and characteristics sit inside a Redis hash tag, so a
397
+ # counter and its ban always hash to the same cluster slot. BAN_SCRIPT
398
+ # touches both in one call, and Redis Cluster rejects a script whose keys
399
+ # span slots. +suffix+ is appended outside the tag, so it cannot change
400
+ # the slot and cannot collide with a characteristic value.
401
+ def build_redis_key(rule, identifier, suffix = nil)
402
+ key = "#{REDIS_KEY_PREFIX}:{#{@name}:#{rule.name}"
236
403
  rule.characteristics.each do |char|
237
404
  value = resolve_char_value(char, identifier)
238
- key += ":#{char}:#{encode_char_value(value)}"
405
+ key << ":#{char}:#{encode_char_value(value)}"
239
406
  end
407
+ key << "}"
408
+ key << suffix if suffix
240
409
  key
241
410
  end
242
411
 
@@ -294,6 +463,30 @@ module Labkit
294
463
  end
295
464
  end
296
465
 
466
+ # Atomic ban check + increment + conditional ban write. See BAN_SCRIPT.
467
+ # The returned ban TTL is negative when no ban is in force.
468
+ def ban_incr_with_ttl(counter_key, ban_key, period, cost, limit, ban_for)
469
+ @redis.with do |conn|
470
+ raw_count, ttl, ban_ttl = BAN_SCRIPT.eval(
471
+ conn, keys: [counter_key, ban_key], argv: [period, cost, limit, ban_for]
472
+ )
473
+ [Float(raw_count), Integer(ttl), Integer(ban_ttl)]
474
+ end
475
+ end
476
+
477
+ # Pipelined read of both keys for #peek. No EXPIRE and no SET: peeking a
478
+ # banned identifier must not extend either the window or the ban.
479
+ def read_ban_with_ttl(counter_key, ban_key)
480
+ @redis.with do |conn|
481
+ raw_count, ttl, ban_ttl = conn.pipelined do |pipe|
482
+ pipe.get(counter_key)
483
+ pipe.ttl(counter_key)
484
+ pipe.ttl(ban_key)
485
+ end
486
+ [raw_count.nil? ? 0.0 : Float(raw_count), Integer(ttl), Integer(ban_ttl)]
487
+ end
488
+ end
489
+
297
490
  # Pipelined GET + TTL. No EXPIRE: peek must not extend the window.
298
491
  # A missing key (GET => nil, TTL => -2) is reported as count=0; the
299
492
  # build_evaluation fallback then derives reset_at from the rule period
@@ -341,6 +534,7 @@ module Labkit
341
534
  # loop reached one, or after it finished - so the field is logged as null
342
535
  # rather than omitted, the same way identifier is. A named rule is the rule
343
536
  # whose match or evaluation raised.
537
+ # Never raises: a logging failure must not break fail-open.
344
538
  def log_error(error, identifier, rule = nil)
345
539
  @logger.warn(
346
540
  name: @name,
@@ -350,6 +544,8 @@ module Labkit
350
544
  Labkit::Fields::ERROR_MESSAGE => error.message,
351
545
  identifier: identifier&.to_h
352
546
  )
547
+ rescue StandardError
548
+ nil
353
549
  end
354
550
 
355
551
  def log_missing_count_distinct(rule, identifier)
@@ -362,43 +558,67 @@ module Labkit
362
558
  )
363
559
  end
364
560
 
365
- def report_matched_metrics(evaluation)
366
- Metrics.calls_total.increment(
367
- rate_limiter: @name,
368
- rule: evaluation.rule.name,
369
- action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s
370
- )
371
- Metrics.limit_gauge.set(
372
- { rate_limiter: @name, rule: evaluation.rule.name },
373
- evaluation.info.resolved_limit
561
+ def report_evaluation_metrics(evaluation)
562
+ rule_labels = { rate_limiter: @name, rule: evaluation.rule.name }
563
+
564
+ Metrics.safe_increment(
565
+ :rule_evaluations_total,
566
+ rule_labels.merge(action: evaluation.rule.action.to_s, result: evaluation_result(evaluation)),
567
+ logger: @logger
374
568
  )
375
- Metrics.period_gauge.set(
376
- { rate_limiter: @name, rule: evaluation.rule.name },
377
- evaluation.info.resolved_period
569
+ # Deprecated dual emission - remove together with Metrics.calls_total.
570
+ Metrics.safe_increment(
571
+ :calls_total,
572
+ rule_labels.merge(action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s),
573
+ logger: @logger
378
574
  )
575
+ Metrics.safe_set(:limit_gauge, rule_labels, evaluation.info.resolved_limit, logger: @logger)
576
+ Metrics.safe_set(:period_gauge, rule_labels, evaluation.info.resolved_period, logger: @logger)
577
+ end
578
+
579
+ # An exceeded :log rule reports "log" rather than the "allow" the caller
580
+ # sees, so shadow rules over their limit stay visible.
581
+ # A ban is not a distinct action, so without this a blocking ban would be
582
+ # indistinguishable from an ordinary block. On a ban_for rule exceeded?
583
+ # means the ban is in force, whether this call wrote it or an earlier one
584
+ # did, which is exactly what is worth counting separately.
585
+ def evaluation_result(evaluation)
586
+ return "allow" unless evaluation.exceeded?
587
+ return "banned" if evaluation.rule.ban_for
588
+
589
+ evaluation.rule.action == :limit ? "block" : "log"
379
590
  end
380
591
 
381
- # calls_total carries action="skip" (the rule action, not the :allow the
382
- # caller sees) so bypass traffic stays distinguishable from counted
383
- # allows. No limit/period gauges: a skip rule has no limit to report.
384
592
  def report_skipped_metrics(rule)
385
- Metrics.calls_total.increment(
386
- rate_limiter: @name,
387
- rule: rule.name,
388
- action: "skip"
593
+ Metrics.safe_increment(
594
+ :rule_evaluations_total,
595
+ { rate_limiter: @name, rule: rule.name, action: "skip", result: "skip" },
596
+ logger: @logger
389
597
  )
598
+ # Deprecated dual emission - remove together with Metrics.calls_total.
599
+ Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: rule.name, action: "skip" }, logger: @logger)
390
600
  end
391
601
 
602
+ # Deprecated dual emission - remove together with Metrics.calls_total.
392
603
  def report_unmatched_metrics
393
- Metrics.calls_total.increment(
394
- rate_limiter: @name,
395
- rule: "unmatched",
396
- action: "allow"
604
+ Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: "unmatched", action: "allow" }, logger: @logger)
605
+ end
606
+
607
+ def report_check_metrics(result)
608
+ Metrics.safe_increment(
609
+ :checks_total,
610
+ {
611
+ rate_limiter: @name,
612
+ action: result.action.to_s,
613
+ matched: result.matched?.to_s,
614
+ error: (result.error? || result.degraded?).to_s
615
+ },
616
+ logger: @logger
397
617
  )
398
618
  end
399
619
 
400
620
  def report_error_metrics
401
- Metrics.errors_total.increment(rate_limiter: @name)
621
+ Metrics.safe_increment(:errors_total, { rate_limiter: @name }, logger: @logger)
402
622
  end
403
623
  end
404
624
  end
@@ -45,7 +45,7 @@ module Labkit
45
45
  # the rule definition and the call site colocated.
46
46
  # @return [Result]
47
47
  def check(identifier, cost: 1, rule_context: nil)
48
- id = identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
48
+ id = to_identifier(identifier)
49
49
  @evaluator.check(id, cost: cost, rule_context: rule_context)
50
50
  end
51
51
 
@@ -65,12 +65,33 @@ module Labkit
65
65
  # @param rule_context [Hash, nil] see {#check}
66
66
  # @return [Result]
67
67
  def peek(identifier, rule_context: nil)
68
- id = identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
68
+ id = to_identifier(identifier)
69
69
  @evaluator.peek(id, rule_context: rule_context)
70
70
  end
71
71
 
72
+ # Discards this limiter's state for one identifier: every rule's counter,
73
+ # and any ban written by a rule carrying ban_for.
74
+ #
75
+ # For call sites where a later success should wipe earlier failures, such
76
+ # as an authentication ban cleared by a valid login. Counters otherwise
77
+ # only expire with their window; this is the only way to end one early.
78
+ #
79
+ # Scoped to this limiter and identifier, not to a single rule: a caller
80
+ # clearing after a success knows who succeeded, not which rules matched.
81
+ #
82
+ # @param identifier [Identifier, Hash] caller attributes
83
+ # @return [Integer] number of Redis keys removed; on a Redis error, those
84
+ # removed before it failed
85
+ def clear(identifier)
86
+ @evaluator.clear(to_identifier(identifier))
87
+ end
88
+
72
89
  private
73
90
 
91
+ def to_identifier(identifier)
92
+ identifier.is_a?(Identifier) ? identifier : Identifier.new(identifier)
93
+ end
94
+
74
95
  def validate_name!(name)
75
96
  raise ArgumentError, "name must be a non-empty String" unless name.is_a?(String) && !name.empty?
76
97
  return name if NAME_PATTERN.match?(name)
@@ -1,14 +1,60 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "concurrent-ruby"
4
+
3
5
  module Labkit
4
6
  module RateLimit
5
7
  module Metrics
8
+ # Process-wide once-latch for log_failure; specs reset via make_false.
9
+ FAILURE_LOGGED = Concurrent::AtomicBoolean.new(false)
10
+
6
11
  module_function
7
12
 
8
- # Emitted once per *matched rule*, not once per check: every rule that
9
- # matches is evaluated, so summing this by rate_limiter counts rule
10
- # evaluations rather than requests. rule="unmatched", action="allow" is
11
- # emitted only when no rule matched at all.
13
+ # Metric emission must never affect the caller's outcome. Resolving the
14
+ # metric by name keeps a raising getter inside the rescue.
15
+ def safe_increment(counter, labels, logger: nil)
16
+ public_send(counter).increment(**labels) # rubocop:disable GitlabSecurity/PublicSend
17
+ rescue StandardError => e
18
+ log_failure(e, counter, labels, logger)
19
+ end
20
+
21
+ # Gauge counterpart of safe_increment.
22
+ def safe_set(gauge, labels, value, logger: nil)
23
+ public_send(gauge).set(labels, value) # rubocop:disable GitlabSecurity/PublicSend
24
+ rescue StandardError => e
25
+ log_failure(e, gauge, labels, logger)
26
+ end
27
+
28
+ # make_true returns true only for the flipping caller, so exactly one
29
+ # warn per process. Never raises - a raise here would defeat the
30
+ # callers' rescues.
31
+ def log_failure(error, metric, labels, logger)
32
+ return unless logger && FAILURE_LOGGED.make_true
33
+
34
+ logger.warn(
35
+ name: labels[:rate_limiter],
36
+ metric: metric.to_s,
37
+ Labkit::Fields::ERROR_TYPE => "rate_limit_metrics_error",
38
+ Labkit::Fields::CLASS_NAME => error.class.to_s,
39
+ Labkit::Fields::ERROR_MESSAGE => error.message
40
+ )
41
+ rescue StandardError
42
+ nil
43
+ end
44
+
45
+ # Emitted exactly once per #check call, including calls that fail open;
46
+ # summing by rate_limiter gives the request rate through the limiter.
47
+ def checks_total
48
+ Labkit::Metrics::Client.counter(
49
+ :gitlab_labkit_rate_limiter_checks_total,
50
+ 'Total number of rate limit checks',
51
+ { rate_limiter: nil, action: nil, matched: nil, error: nil }
52
+ )
53
+ end
54
+
55
+ # Deprecated: superseded by checks_total and rule_evaluations_total.
56
+ # Emitted unchanged (once per matched rule, rule="unmatched" when none
57
+ # matched) until consumers migrate; will get removed in a follow-up release.
12
58
  def calls_total
13
59
  Labkit::Metrics::Client.counter(
14
60
  :gitlab_labkit_rate_limiter_calls_total,
@@ -17,6 +63,18 @@ module Labkit
17
63
  )
18
64
  end
19
65
 
66
+ # Emitted once per *evaluated* rule. Only matched rules are evaluated:
67
+ # Unmatched traffic is checks_total{matched="false"}
68
+ def rule_evaluations_total
69
+ Labkit::Metrics::Client.counter(
70
+ :gitlab_labkit_rate_limiter_rule_evaluations_total,
71
+ 'Total number of rate limit rule evaluations',
72
+ { rate_limiter: nil, rule: nil, action: nil, result: nil }
73
+ )
74
+ end
75
+
76
+ # Deprecated: superseded by checks_total{error="true"}. Still emitted
77
+ # because it remains the only error metric for #peek.
20
78
  def errors_total
21
79
  Labkit::Metrics::Client.counter(
22
80
  :gitlab_labkit_rate_limiter_errors_total,
@@ -25,6 +25,8 @@ module Labkit
25
25
  # false). Other rules may also have matched and been counted;
26
26
  # see #evaluations.
27
27
  # error? - true if Redis was unavailable; result fails open (exceeded? is false)
28
+ # degraded? - true if a matched count_distinct rule was skipped by the
29
+ # missing-key fail-open path; the check still completed
28
30
  # info - Result::Info with per-window counters for the reported rule;
29
31
  # nil when matched? is false, error?, or the matched rule is
30
32
  # :skip (no counter exists)
@@ -40,6 +42,7 @@ module Labkit
40
42
  @evaluations = []
41
43
  @skip_rule = nil
42
44
  @error = error
45
+ @degraded = false
43
46
  @most_constraining = nil
44
47
  end
45
48
 
@@ -78,6 +81,17 @@ module Labkit
78
81
  @error
79
82
  end
80
83
 
84
+ # Unlike error?, the check keeps going: the flag marks the outcome as
85
+ # degraded without changing it.
86
+ def degraded!
87
+ @degraded = true
88
+ self
89
+ end
90
+
91
+ def degraded?
92
+ @degraded
93
+ end
94
+
81
95
  def action
82
96
  block? ? :block : :allow
83
97
  end
@@ -119,9 +133,10 @@ module Labkit
119
133
  # exceeded - whether the post-increment count exceeded the resolved limit
120
134
  # info - Result::Info with the per-window counters
121
135
  #
122
- # Evaluations order by constraint: a blocking evaluation (:limit rule over
123
- # its limit) ranks strictly first - it decides the request no matter what
124
- # any other rule reports - then exceeded ones, then fewest remaining.
136
+ # Evaluations order by constraint: a blocking evaluation (a :limit rule over
137
+ # its limit, or one whose ban is still in force) ranks strictly first - it
138
+ # decides the request no matter what any other rule reports - then exceeded
139
+ # ones, then fewest remaining.
125
140
  # remaining floors at 0, so an exceeded :log rule ties with one sitting
126
141
  # exactly on its limit; ranking exceeded ahead keeps a breach from being
127
142
  # hidden by a rule that merely reached its limit.
@@ -14,10 +14,17 @@ module Labkit
14
14
  # period - window in seconds; may be a callable (resolved per check)
15
15
  # action - :limit (enforce; the result blocks when the rule is over
16
16
  # its limit, which also terminates evaluation), :log (count
17
- # and log only, never blocks and never terminates), or
18
- # :skip (bypass: permit and terminate evaluation on match
19
- # without counting; performs no Redis operation, so limit,
20
- # period, characteristics, and count_distinct are inert)
17
+ # and log only, never blocks and never terminates),
18
+ # or :skip (bypass: permit and terminate evaluation on
19
+ # match without counting; performs no Redis operation, so
20
+ # limit, period, characteristics, and count_distinct are
21
+ # inert)
22
+ # ban_for - optional ban duration in seconds. On crossing the limit
23
+ # the rule writes a separate ban key that outlives the
24
+ # counter window, and stops counting while it holds.
25
+ # +action+ still decides who is blocked: :limit enforces
26
+ # the ban, :log records what it would have done. Rejected
27
+ # on :skip. May be a callable, resolved per check.
21
28
  # characteristics - identifier keys used to build the compound Redis counter key
22
29
  # count_distinct - optional Symbol naming an identifier key. When set, the rule
23
30
  # counts the number of distinct values seen for that key within
@@ -28,7 +35,27 @@ module Labkit
28
35
  # +name+ must be a lowercase alphanumeric-and-underscore string of at most 64
29
36
  # characters. It is used as the middle segment of every Redis counter key for
30
37
  # this rule, so changing a rule's name mid-window abandons its in-flight counters.
31
- Rule = Data.define(:name, :match, :limit, :period, :action, :characteristics, :count_distinct) do
38
+ Rule = Data.define(:name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for) do
39
+ # ban_for changes the accounting; action still decides who gets blocked.
40
+ # A :skip rule counts nothing, so it has no limit to cross and no ban to
41
+ # write.
42
+ def self.normalize_ban_for(value, action_sym, count_distinct_sym)
43
+ return nil if value.nil?
44
+
45
+ raise ArgumentError, "ban_for is not valid on a :skip rule" if action_sym == :skip
46
+
47
+ # The ban path counts with INCRBYFLOAT, so a SET-backed rule would
48
+ # silently count calls instead of distinct values.
49
+ raise ArgumentError, "ban_for cannot be combined with count_distinct #{count_distinct_sym.inspect}" if count_distinct_sym
50
+
51
+ return value if value.respond_to?(:call)
52
+ # Whole seconds: the ban is written with SET EX, and anything under a
53
+ # second truncates to 0, which Redis rejects.
54
+ return value if value.is_a?(Numeric) && value >= 1
55
+
56
+ raise ArgumentError, "ban_for must be a Numeric of at least 1 second or a callable, got #{value.inspect}"
57
+ end
58
+
32
59
  def self.normalize_count_distinct(value, characteristics_arr)
33
60
  sym =
34
61
  case value
@@ -44,7 +71,10 @@ module Labkit
44
71
  sym
45
72
  end
46
73
 
47
- def initialize(name:, limit:, period:, characteristics:, match: {}, action: :limit, count_distinct: nil)
74
+ def initialize(
75
+ name:, limit:, period:, characteristics:,
76
+ match: {}, action: :limit, count_distinct: nil, ban_for: nil
77
+ )
48
78
  raise ArgumentError, "name must be a String or Symbol, got #{name.class}" unless name.is_a?(String) || name.is_a?(Symbol)
49
79
 
50
80
  name_str = name.to_s
@@ -59,6 +89,7 @@ module Labkit
59
89
  end
60
90
 
61
91
  characteristics_arr = Array(characteristics).map(&:to_sym).freeze
92
+ count_distinct_sym = self.class.normalize_count_distinct(count_distinct, characteristics_arr)
62
93
 
63
94
  super(
64
95
  name: name_str.freeze,
@@ -67,7 +98,8 @@ module Labkit
67
98
  period: period,
68
99
  action: action_sym,
69
100
  characteristics: characteristics_arr,
70
- count_distinct: self.class.normalize_count_distinct(count_distinct, characteristics_arr)
101
+ count_distinct: count_distinct_sym,
102
+ ban_for: self.class.normalize_ban_for(ban_for, action_sym, count_distinct_sym)
71
103
  )
72
104
  end
73
105
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-labkit
3
3
  version: !ruby/object:Gem::Version
4
- version: 4.4.1
4
+ version: 4.6.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Newdigate
@@ -716,7 +716,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
716
716
  - !ruby/object:Gem::Version
717
717
  version: '0'
718
718
  requirements: []
719
- rubygems_version: 4.0.10
719
+ rubygems_version: 4.0.16
720
720
  specification_version: 4
721
721
  summary: Instrumentation for GitLab
722
722
  test_files: []