gitlab-labkit 4.4.1 → 4.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitlab-ci-asdf-versions.yml +1 -1
- data/.tool-versions +1 -1
- data/lib/labkit/rate_limit/README.md +80 -35
- data/lib/labkit/rate_limit/evaluator.rb +72 -33
- data/lib/labkit/rate_limit/metrics.rb +62 -4
- data/lib/labkit/rate_limit/result.rb +14 -0
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d6850fbd8d4f1d8d81e2369bc5d27aa0638857e9cf32d2847a6c93f9efe84d9f
|
|
4
|
+
data.tar.gz: ac6b4527b4a38f8917e25ed8576628e498e67e20cd4f0d3dfaf9696447cfd9a3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 80803e406ce6e6da82ed72e252c2e6b59349aae34dc128eb1231688ca071a95ff125d93b882cbdd0ece94897f1ca1c7156cb3abba894421ba54d042e56b961c8
|
|
7
|
+
data.tar.gz: 0b8afdf336ae0394ed6dcdea7f597968b3825c13ed01ce1a571a82078532d17e82733aa57a791bf52f26723ad0821be61b785355c25e75479019e605ca71d0de
|
data/.tool-versions
CHANGED
|
@@ -219,7 +219,7 @@ An exceeded rule has `remaining` 0 and therefore outranks any rule still under
|
|
|
219
219
|
its limit. For a `:log` rule that means shadow traffic surfaces to the caller
|
|
220
220
|
as `exceeded? == true` (with `action` still `:allow`) — visible, but unable to
|
|
221
221
|
block. A `:log`-only path that matches therefore returns `matched? == true`
|
|
222
|
-
and
|
|
222
|
+
and its check is counted as `checks_total{matched="true"}`.
|
|
223
223
|
|
|
224
224
|
```mermaid
|
|
225
225
|
flowchart TD
|
|
@@ -227,20 +227,20 @@ flowchart TD
|
|
|
227
227
|
Iter -->|yes| Match{rule.match<br/>all satisfied?}
|
|
228
228
|
Match -->|no| Iter
|
|
229
229
|
Match -->|yes| Skip{rule.action<br/>== :skip?}
|
|
230
|
-
Skip -->|"yes (no Redis op)"| SkipEmit[Emit
|
|
230
|
+
Skip -->|"yes (no Redis op)"| SkipEmit[Emit rule_evaluations_total<br/>action=skip, result=skip]
|
|
231
231
|
SkipEmit --> SkipReturn([Return matched=true<br/>action=:allow])
|
|
232
232
|
Skip -->|no| Eval["INCR Redis counter<br/>(see Redis sequence below)"]
|
|
233
233
|
Eval --> Build[Build Evaluation<br/>resolve limit/period]
|
|
234
234
|
Build --> Add[Add evaluation to Result]
|
|
235
|
-
Add --> Emit[Emit
|
|
235
|
+
Add --> Emit[Emit rule_evaluations_total<br/>+ limit/period gauges]
|
|
236
236
|
Emit --> Act{"result.block?<br/>(:limit rule over limit)"}
|
|
237
237
|
Act -->|yes| Return([Return Result<br/>action=:block])
|
|
238
238
|
Act -->|"no (:log, or :limit under limit)"| Iter
|
|
239
|
-
Iter -->|no more rules|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
Eval -. StandardError .-> Error[Emit errors_total<br/>log warn]
|
|
239
|
+
Iter -->|no more rules| Return2([Return Result reporting<br/>most-constraining evaluation,<br/>or matched=false if none])
|
|
240
|
+
Return --> Check[Emit checks_total<br/>action, matched, error]
|
|
241
|
+
SkipReturn --> Check
|
|
242
|
+
Return2 --> Check
|
|
243
|
+
Eval -. StandardError .-> Error[Emit errors_total +<br/>checks_total error=true<br/>log warn]
|
|
244
244
|
Error --> ReturnErr([Return error=true<br/>action=:allow])
|
|
245
245
|
```
|
|
246
246
|
|
|
@@ -249,23 +249,41 @@ discards the verdicts of the rules already evaluated, even though their counters
|
|
|
249
249
|
were incremented. Those requests are counted but produce no verdict — the
|
|
250
250
|
fail-open trade-off is that a request is never blocked on a partial evaluation.
|
|
251
251
|
|
|
252
|
-
|
|
253
|
-
`
|
|
252
|
+
`rule_evaluations_total` is emitted **per evaluated rule** (including matched
|
|
253
|
+
`:skip` rules); `checks_total` is emitted **exactly once per check**, whatever
|
|
254
|
+
path the evaluation takes — so summing `checks_total` by `rate_limiter` counts
|
|
255
|
+
requests through the limiter, and "no rule matched" is `matched="false"` on the
|
|
256
|
+
check rather than a placeholder rule.
|
|
257
|
+
|
|
258
|
+
Only matched rules are evaluated: a rule whose `match:` conditions are not
|
|
259
|
+
satisfied emits nothing to `rule_evaluations_total` (match-testing is not an
|
|
260
|
+
evaluation), so the counter has no "didn't match" population and needs no
|
|
261
|
+
label for it. A matched `count_distinct` rule skipped by the missing-key
|
|
262
|
+
fail-open also emits nothing here — it was never evaluated; that check is
|
|
263
|
+
visible via `checks_total{error="true"}` and the
|
|
264
|
+
`rate_limit_missing_count_distinct` log, which carries the rule name.
|
|
265
|
+
|
|
266
|
+
During the transition the deprecated `calls_total` counter is additionally
|
|
267
|
+
emitted at every point the diagram emits `rule_evaluations_total` (with its
|
|
268
|
+
historical per-rule semantics, including the `rule="unmatched"` placeholder
|
|
269
|
+
after the loop). It is not shown above to keep the diagram legible; see the
|
|
270
|
+
Metrics table below.
|
|
254
271
|
|
|
255
272
|
### Actions
|
|
256
273
|
|
|
257
274
|
The rule's `action` describes what the rule does; the result's `action`
|
|
258
275
|
describes the outcome — what the caller should do — and is only ever `:allow`
|
|
259
|
-
or `:block`. The counter is always incremented when a rule matches,
|
|
260
|
-
`:skip` rules, which never touch Redis
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
| `:limit` | count against the limit |
|
|
266
|
-
| `:
|
|
267
|
-
| `:log` | count against the limit (observability only) |
|
|
268
|
-
| `:
|
|
276
|
+
or `:block`. The Redis counter is always incremented when a rule matches,
|
|
277
|
+
except for `:skip` rules, which never touch Redis. The `result` label on
|
|
278
|
+
`rule_evaluations_total` records what each evaluation decided:
|
|
279
|
+
|
|
280
|
+
| rule action | what it does | exceeded? | result action | `result` label | terminating? |
|
|
281
|
+
|-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|
|
|
282
|
+
| `:limit` | count against the limit | no | `:allow` | `allow` | no — continue |
|
|
283
|
+
| `:limit` | count against the limit | yes | `:block` | `block` | yes — stop |
|
|
284
|
+
| `:log` | count against the limit (observability only) | no | `:allow` | `allow` | no — continue |
|
|
285
|
+
| `:log` | count against the limit (observability only) | yes | `:allow` | `log` | no — continue |
|
|
286
|
+
| `:skip` | don't count (bypass) | n/a | `:allow` | `skip` | yes — stop |
|
|
269
287
|
|
|
270
288
|
- `:limit` — when exceeded, `Result#action` is `:block` and evaluation
|
|
271
289
|
terminates. Caller should reject the request (e.g. with HTTP 429). When under
|
|
@@ -281,8 +299,8 @@ or `:block`. The counter is always incremented when a rule matches, except for
|
|
|
281
299
|
`Result#action` `:allow` **without any Redis operation**: nothing is
|
|
282
300
|
counted, so `limit`, `period`, `characteristics`, and `count_distinct` are
|
|
283
301
|
inert and the result carries no `info` (`to_response_headers` is `{}`).
|
|
284
|
-
The match is still observable via `
|
|
285
|
-
for bypasses.
|
|
302
|
+
The match is still observable via `rule_evaluations_total{action="skip"}`.
|
|
303
|
+
Use this for bypasses.
|
|
286
304
|
|
|
287
305
|
### Redis keys
|
|
288
306
|
|
|
@@ -375,26 +393,53 @@ safe to merge unconditionally.
|
|
|
375
393
|
|
|
376
394
|
The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
|
|
377
395
|
(Redis connection failure, timeout, OOM in user-supplied callables, …) is
|
|
378
|
-
logged at WARN with `
|
|
379
|
-
`Result` (`matched?` false, `error?` true, `action` `:allow`). The
|
|
380
|
-
`gitlab_labkit_rate_limiter_errors_total` counter is incremented. The caller
|
|
396
|
+
logged at WARN with `error_type: "rate_limit_error"` and returned as an error
|
|
397
|
+
`Result` (`matched?` false, `error?` true, `action` `:allow`). The caller
|
|
381
398
|
should treat the request as allowed.
|
|
382
399
|
|
|
400
|
+
A failed-open `check` is still counted: it emits
|
|
401
|
+
`checks_total{action="allow", matched="false", error="true"}`, so the fraction
|
|
402
|
+
of checks that encountered an error is `checks_total{error="true"}` over
|
|
403
|
+
`checks_total`. `error="true"` also covers a check where a matched
|
|
404
|
+
`count_distinct` rule was skipped because the identifier was missing its
|
|
405
|
+
`count_distinct` key — that check completes (`action` and `matched` describe
|
|
406
|
+
its outcome as usual), so `error="true"` is not exclusively fail-open traffic.
|
|
407
|
+
|
|
408
|
+
The deprecated `gitlab_labkit_rate_limiter_errors_total` counter is still
|
|
409
|
+
incremented on every fail-open (whole-check and per-rule `count_distinct`),
|
|
410
|
+
and remains the only error metric for `peek`, which emits no `checks_total`
|
|
411
|
+
(peek must not inflate the per-check counter). The follow-up MR that removes
|
|
412
|
+
`errors_total` must first decide where `peek` errors go — a dedicated peek
|
|
413
|
+
metric, or logs only.
|
|
414
|
+
|
|
415
|
+
Metric emission itself is best-effort: a failure in the metrics stack never
|
|
416
|
+
alters the verdict or breaks fail-open, and is logged at WARN with
|
|
417
|
+
`error_type: "rate_limit_metrics_error"` (once per process, to avoid
|
|
418
|
+
flooding).
|
|
419
|
+
|
|
383
420
|
## Metrics
|
|
384
421
|
|
|
385
422
|
`Labkit::RateLimit::Metrics` emits the following Prometheus metrics through
|
|
386
423
|
`Labkit::Metrics::Client`:
|
|
387
424
|
|
|
388
|
-
| metric
|
|
389
|
-
|
|
390
|
-
| `
|
|
391
|
-
| `
|
|
392
|
-
| `
|
|
393
|
-
| `
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
`
|
|
425
|
+
| metric | type | labels | meaning |
|
|
426
|
+
|-----------------------------------------------------|---------|----------------------------------------------|----------------------------------------------------------------------|
|
|
427
|
+
| `gitlab_labkit_rate_limiter_checks_total` | counter | `rate_limiter`, `action`, `matched`, `error` | Exactly one increment per `check` call, including fail-open. `action` is what the caller should do (`"allow"` or `"block"`); `matched` and `error` are `"true"`/`"false"`. |
|
|
428
|
+
| `gitlab_labkit_rate_limiter_rule_evaluations_total` | counter | `rate_limiter`, `rule`, `action`, `result` | One increment per evaluated rule (plus one per matched `:skip` rule). `action` is the configured rule action (`"limit"`, `"log"`, `"skip"`); `result` is what the evaluation decided (`"allow"`, `"block"`, `"log"`, `"skip"` — see the Actions table). |
|
|
429
|
+
| `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | **Deprecated** — superseded by `checks_total` + `rule_evaluations_total`. Historical per-rule counter: one increment per counted rule (plus one per matched `:skip` rule), `rule="unmatched", action="allow"` when no rule matched. Emitted unchanged during the transition. |
|
|
430
|
+
| `gitlab_labkit_rate_limiter_errors_total` | counter | `rate_limiter` | **Deprecated** — use `checks_total{error="true"}`. Fail-open events (any `StandardError` in the labkit path); still the only error metric for `peek`. |
|
|
431
|
+
| `gitlab_labkit_rate_limiter_limit` | gauge | `rate_limiter`, `rule` | Resolved limit at the last check (useful when `limit:` is callable). |
|
|
432
|
+
| `gitlab_labkit_rate_limiter_period_seconds` | gauge | `rate_limiter`, `rule` | Resolved period at the last check. |
|
|
433
|
+
|
|
434
|
+
`sum by (rate_limiter) (rate(gitlab_labkit_rate_limiter_checks_total[5m]))` is
|
|
435
|
+
the request rate through a limiter — no exclusions or dedup needed. A single
|
|
436
|
+
`check` call emits **one** `checks_total` increment and as many
|
|
437
|
+
`rule_evaluations_total` increments as rules it evaluated (possibly zero).
|
|
438
|
+
|
|
439
|
+
**Transition:** the deprecated `calls_total` and `errors_total` counters keep
|
|
440
|
+
emitting exactly as before this split, so existing dashboards and alerts stay
|
|
441
|
+
correct while consumers migrate to the new counters. Both are removed in a
|
|
442
|
+
follow-up major release once nothing consumes them.
|
|
398
443
|
|
|
399
444
|
## Dev/test vs production guards
|
|
400
445
|
|
|
@@ -75,13 +75,17 @@ module Labkit
|
|
|
75
75
|
|
|
76
76
|
def check(identifier, cost: 1, rule_context: nil)
|
|
77
77
|
cursor = RuleCursor.new
|
|
78
|
-
check_rules(identifier, cost, rule_context, cursor)
|
|
78
|
+
result = check_rules(identifier, cost, rule_context, cursor)
|
|
79
79
|
rescue StandardError => e
|
|
80
80
|
# Intentionally broad: fail-open applies to any unexpected error (network,
|
|
81
81
|
# timeout, OOM) not only Redis protocol errors.
|
|
82
82
|
report_error_metrics
|
|
83
83
|
log_error(e, identifier, cursor.rule)
|
|
84
|
-
Result.error
|
|
84
|
+
result = Result.error
|
|
85
|
+
ensure
|
|
86
|
+
# StandardError-safe emission, so it cannot mask a propagating error.
|
|
87
|
+
# result is nil when a non-StandardError unwinds: emit nothing then.
|
|
88
|
+
report_check_metrics(result) if result
|
|
85
89
|
end
|
|
86
90
|
|
|
87
91
|
# Read-without-increment counterpart to {#check}. Same matching and Result
|
|
@@ -104,7 +108,7 @@ module Labkit
|
|
|
104
108
|
# - :skip terminates on match without touching Redis. No counter is
|
|
105
109
|
# incremented, so the branch sits before the count_distinct check
|
|
106
110
|
# (identifier completeness is irrelevant to a rule that builds no key).
|
|
107
|
-
#
|
|
111
|
+
# rule_evaluations_total still increments so the bypass stays observable.
|
|
108
112
|
# - a :limit rule over its limit terminates, because the request is
|
|
109
113
|
# rejected and later rules cannot change that. Rules declared after it
|
|
110
114
|
# are neither counted nor evaluated, so a blocked request debits every
|
|
@@ -115,10 +119,14 @@ module Labkit
|
|
|
115
119
|
# most-constraining evaluation (ranking lives in Result::Evaluation#<=>).
|
|
116
120
|
# cost is therefore debited from every matching rule, not just the first.
|
|
117
121
|
#
|
|
122
|
+
# Metrics: each evaluated rule emits rule_evaluations_total (plus the
|
|
123
|
+
# deprecated per-rule calls_total); the per-check checks_total is
|
|
124
|
+
# emitted once in #check. Full contract in the README's Metrics section.
|
|
125
|
+
#
|
|
118
126
|
# SET-mode rules (rule.count_distinct set) that match but whose identifier
|
|
119
|
-
# is missing the count_distinct key fail open + log + bump errors_total
|
|
120
|
-
# the loop continues to the next rule (the rule is
|
|
121
|
-
# rather than aborting the whole evaluation).
|
|
127
|
+
# is missing the count_distinct key fail open + log + bump errors_total +
|
|
128
|
+
# flag the Result, and the loop continues to the next rule (the rule is
|
|
129
|
+
# treated as not applicable rather than aborting the whole evaluation).
|
|
122
130
|
#
|
|
123
131
|
# Error handling stays whole-check (see #check): a raise part-way through
|
|
124
132
|
# discards the results of the rules already evaluated, even though their
|
|
@@ -143,12 +151,13 @@ module Labkit
|
|
|
143
151
|
if rule.count_distinct && missing_count_distinct_value?(rule, identifier)
|
|
144
152
|
log_missing_count_distinct(rule, identifier)
|
|
145
153
|
report_error_metrics
|
|
154
|
+
result.degraded!
|
|
146
155
|
next
|
|
147
156
|
end
|
|
148
157
|
|
|
149
158
|
evaluation = evaluate_rule(rule, identifier, cost, rule_context)
|
|
150
159
|
result.add_evaluation(evaluation)
|
|
151
|
-
|
|
160
|
+
report_evaluation_metrics(evaluation)
|
|
152
161
|
return result if result.block?
|
|
153
162
|
end
|
|
154
163
|
|
|
@@ -157,6 +166,12 @@ module Labkit
|
|
|
157
166
|
|
|
158
167
|
report_unmatched_metrics unless result.matched?
|
|
159
168
|
result
|
|
169
|
+
rescue StandardError => e # binds e for the ensure
|
|
170
|
+
raise
|
|
171
|
+
ensure
|
|
172
|
+
# Keep the rule attributed while an exception unwinds; clear it on
|
|
173
|
+
# every normal exit (early returns included).
|
|
174
|
+
cursor.rule = nil unless e
|
|
160
175
|
end
|
|
161
176
|
|
|
162
177
|
# Mirror of check_rules without metrics or writes. :log rules are read
|
|
@@ -180,9 +195,11 @@ module Labkit
|
|
|
180
195
|
return result if result.block?
|
|
181
196
|
end
|
|
182
197
|
|
|
183
|
-
cursor.rule = nil
|
|
184
|
-
|
|
185
198
|
result
|
|
199
|
+
rescue StandardError => e # binds e for the ensure
|
|
200
|
+
raise
|
|
201
|
+
ensure
|
|
202
|
+
cursor.rule = nil unless e
|
|
186
203
|
end
|
|
187
204
|
|
|
188
205
|
def rule_matches?(rule, identifier)
|
|
@@ -341,6 +358,7 @@ module Labkit
|
|
|
341
358
|
# loop reached one, or after it finished - so the field is logged as null
|
|
342
359
|
# rather than omitted, the same way identifier is. A named rule is the rule
|
|
343
360
|
# whose match or evaluation raised.
|
|
361
|
+
# Never raises: a logging failure must not break fail-open.
|
|
344
362
|
def log_error(error, identifier, rule = nil)
|
|
345
363
|
@logger.warn(
|
|
346
364
|
name: @name,
|
|
@@ -350,6 +368,8 @@ module Labkit
|
|
|
350
368
|
Labkit::Fields::ERROR_MESSAGE => error.message,
|
|
351
369
|
identifier: identifier&.to_h
|
|
352
370
|
)
|
|
371
|
+
rescue StandardError
|
|
372
|
+
nil
|
|
353
373
|
end
|
|
354
374
|
|
|
355
375
|
def log_missing_count_distinct(rule, identifier)
|
|
@@ -362,43 +382,62 @@ module Labkit
|
|
|
362
382
|
)
|
|
363
383
|
end
|
|
364
384
|
|
|
365
|
-
def
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
{ rate_limiter: @name, rule: evaluation.rule.name },
|
|
373
|
-
evaluation.info.resolved_limit
|
|
385
|
+
def report_evaluation_metrics(evaluation)
|
|
386
|
+
rule_labels = { rate_limiter: @name, rule: evaluation.rule.name }
|
|
387
|
+
|
|
388
|
+
Metrics.safe_increment(
|
|
389
|
+
:rule_evaluations_total,
|
|
390
|
+
rule_labels.merge(action: evaluation.rule.action.to_s, result: evaluation_result(evaluation)),
|
|
391
|
+
logger: @logger
|
|
374
392
|
)
|
|
375
|
-
Metrics.
|
|
376
|
-
|
|
377
|
-
|
|
393
|
+
# Deprecated dual emission - remove together with Metrics.calls_total.
|
|
394
|
+
Metrics.safe_increment(
|
|
395
|
+
:calls_total,
|
|
396
|
+
rule_labels.merge(action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s),
|
|
397
|
+
logger: @logger
|
|
378
398
|
)
|
|
399
|
+
Metrics.safe_set(:limit_gauge, rule_labels, evaluation.info.resolved_limit, logger: @logger)
|
|
400
|
+
Metrics.safe_set(:period_gauge, rule_labels, evaluation.info.resolved_period, logger: @logger)
|
|
401
|
+
end
|
|
402
|
+
|
|
403
|
+
# An exceeded :log rule reports "log" rather than the "allow" the caller
|
|
404
|
+
# sees, so shadow rules over their limit stay visible.
|
|
405
|
+
def evaluation_result(evaluation)
|
|
406
|
+
return "allow" unless evaluation.exceeded?
|
|
407
|
+
|
|
408
|
+
evaluation.rule.action == :limit ? "block" : "log"
|
|
379
409
|
end
|
|
380
410
|
|
|
381
|
-
# calls_total carries action="skip" (the rule action, not the :allow the
|
|
382
|
-
# caller sees) so bypass traffic stays distinguishable from counted
|
|
383
|
-
# allows. No limit/period gauges: a skip rule has no limit to report.
|
|
384
411
|
def report_skipped_metrics(rule)
|
|
385
|
-
Metrics.
|
|
386
|
-
|
|
387
|
-
rule: rule.name,
|
|
388
|
-
|
|
412
|
+
Metrics.safe_increment(
|
|
413
|
+
:rule_evaluations_total,
|
|
414
|
+
{ rate_limiter: @name, rule: rule.name, action: "skip", result: "skip" },
|
|
415
|
+
logger: @logger
|
|
389
416
|
)
|
|
417
|
+
# Deprecated dual emission - remove together with Metrics.calls_total.
|
|
418
|
+
Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: rule.name, action: "skip" }, logger: @logger)
|
|
390
419
|
end
|
|
391
420
|
|
|
421
|
+
# Deprecated dual emission - remove together with Metrics.calls_total.
|
|
392
422
|
def report_unmatched_metrics
|
|
393
|
-
Metrics.calls_total
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
423
|
+
Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: "unmatched", action: "allow" }, logger: @logger)
|
|
424
|
+
end
|
|
425
|
+
|
|
426
|
+
def report_check_metrics(result)
|
|
427
|
+
Metrics.safe_increment(
|
|
428
|
+
:checks_total,
|
|
429
|
+
{
|
|
430
|
+
rate_limiter: @name,
|
|
431
|
+
action: result.action.to_s,
|
|
432
|
+
matched: result.matched?.to_s,
|
|
433
|
+
error: (result.error? || result.degraded?).to_s
|
|
434
|
+
},
|
|
435
|
+
logger: @logger
|
|
397
436
|
)
|
|
398
437
|
end
|
|
399
438
|
|
|
400
439
|
def report_error_metrics
|
|
401
|
-
Metrics.errors_total
|
|
440
|
+
Metrics.safe_increment(:errors_total, { rate_limiter: @name }, logger: @logger)
|
|
402
441
|
end
|
|
403
442
|
end
|
|
404
443
|
end
|
|
@@ -1,14 +1,60 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "concurrent-ruby"
|
|
4
|
+
|
|
3
5
|
module Labkit
|
|
4
6
|
module RateLimit
|
|
5
7
|
module Metrics
|
|
8
|
+
# Process-wide once-latch for log_failure; specs reset via make_false.
|
|
9
|
+
FAILURE_LOGGED = Concurrent::AtomicBoolean.new(false)
|
|
10
|
+
|
|
6
11
|
module_function
|
|
7
12
|
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
|
|
11
|
-
|
|
13
|
+
# Metric emission must never affect the caller's outcome. Resolving the
|
|
14
|
+
# metric by name keeps a raising getter inside the rescue.
|
|
15
|
+
def safe_increment(counter, labels, logger: nil)
|
|
16
|
+
public_send(counter).increment(**labels) # rubocop:disable GitlabSecurity/PublicSend
|
|
17
|
+
rescue StandardError => e
|
|
18
|
+
log_failure(e, counter, labels, logger)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# Gauge counterpart of safe_increment.
|
|
22
|
+
def safe_set(gauge, labels, value, logger: nil)
|
|
23
|
+
public_send(gauge).set(labels, value) # rubocop:disable GitlabSecurity/PublicSend
|
|
24
|
+
rescue StandardError => e
|
|
25
|
+
log_failure(e, gauge, labels, logger)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# make_true returns true only for the flipping caller, so exactly one
|
|
29
|
+
# warn per process. Never raises - a raise here would defeat the
|
|
30
|
+
# callers' rescues.
|
|
31
|
+
def log_failure(error, metric, labels, logger)
|
|
32
|
+
return unless logger && FAILURE_LOGGED.make_true
|
|
33
|
+
|
|
34
|
+
logger.warn(
|
|
35
|
+
name: labels[:rate_limiter],
|
|
36
|
+
metric: metric.to_s,
|
|
37
|
+
Labkit::Fields::ERROR_TYPE => "rate_limit_metrics_error",
|
|
38
|
+
Labkit::Fields::CLASS_NAME => error.class.to_s,
|
|
39
|
+
Labkit::Fields::ERROR_MESSAGE => error.message
|
|
40
|
+
)
|
|
41
|
+
rescue StandardError
|
|
42
|
+
nil
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
# Emitted exactly once per #check call, including calls that fail open;
|
|
46
|
+
# summing by rate_limiter gives the request rate through the limiter.
|
|
47
|
+
def checks_total
|
|
48
|
+
Labkit::Metrics::Client.counter(
|
|
49
|
+
:gitlab_labkit_rate_limiter_checks_total,
|
|
50
|
+
'Total number of rate limit checks',
|
|
51
|
+
{ rate_limiter: nil, action: nil, matched: nil, error: nil }
|
|
52
|
+
)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# Deprecated: superseded by checks_total and rule_evaluations_total.
|
|
56
|
+
# Emitted unchanged (once per matched rule, rule="unmatched" when none
|
|
57
|
+
# matched) until consumers migrate; will get removed in a follow-up release.
|
|
12
58
|
def calls_total
|
|
13
59
|
Labkit::Metrics::Client.counter(
|
|
14
60
|
:gitlab_labkit_rate_limiter_calls_total,
|
|
@@ -17,6 +63,18 @@ module Labkit
|
|
|
17
63
|
)
|
|
18
64
|
end
|
|
19
65
|
|
|
66
|
+
# Emitted once per *evaluated* rule. Only matched rules are evaluated:
|
|
67
|
+
# Unmatched traffic is checks_total{matched="false"}
|
|
68
|
+
def rule_evaluations_total
|
|
69
|
+
Labkit::Metrics::Client.counter(
|
|
70
|
+
:gitlab_labkit_rate_limiter_rule_evaluations_total,
|
|
71
|
+
'Total number of rate limit rule evaluations',
|
|
72
|
+
{ rate_limiter: nil, rule: nil, action: nil, result: nil }
|
|
73
|
+
)
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
# Deprecated: superseded by checks_total{error="true"}. Still emitted
|
|
77
|
+
# because it remains the only error metric for #peek.
|
|
20
78
|
def errors_total
|
|
21
79
|
Labkit::Metrics::Client.counter(
|
|
22
80
|
:gitlab_labkit_rate_limiter_errors_total,
|
|
@@ -25,6 +25,8 @@ module Labkit
|
|
|
25
25
|
# false). Other rules may also have matched and been counted;
|
|
26
26
|
# see #evaluations.
|
|
27
27
|
# error? - true if Redis was unavailable; result fails open (exceeded? is false)
|
|
28
|
+
# degraded? - true if a matched count_distinct rule was skipped by the
|
|
29
|
+
# missing-key fail-open path; the check still completed
|
|
28
30
|
# info - Result::Info with per-window counters for the reported rule;
|
|
29
31
|
# nil when matched? is false, error?, or the matched rule is
|
|
30
32
|
# :skip (no counter exists)
|
|
@@ -40,6 +42,7 @@ module Labkit
|
|
|
40
42
|
@evaluations = []
|
|
41
43
|
@skip_rule = nil
|
|
42
44
|
@error = error
|
|
45
|
+
@degraded = false
|
|
43
46
|
@most_constraining = nil
|
|
44
47
|
end
|
|
45
48
|
|
|
@@ -78,6 +81,17 @@ module Labkit
|
|
|
78
81
|
@error
|
|
79
82
|
end
|
|
80
83
|
|
|
84
|
+
# Unlike error?, the check keeps going: the flag marks the outcome as
|
|
85
|
+
# degraded without changing it.
|
|
86
|
+
def degraded!
|
|
87
|
+
@degraded = true
|
|
88
|
+
self
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def degraded?
|
|
92
|
+
@degraded
|
|
93
|
+
end
|
|
94
|
+
|
|
81
95
|
def action
|
|
82
96
|
block? ? :block : :allow
|
|
83
97
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: gitlab-labkit
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 4.
|
|
4
|
+
version: 4.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrew Newdigate
|
|
@@ -716,7 +716,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
716
716
|
- !ruby/object:Gem::Version
|
|
717
717
|
version: '0'
|
|
718
718
|
requirements: []
|
|
719
|
-
rubygems_version: 4.0.
|
|
719
|
+
rubygems_version: 4.0.16
|
|
720
720
|
specification_version: 4
|
|
721
721
|
summary: Instrumentation for GitLab
|
|
722
722
|
test_files: []
|