gitlab-labkit 4.4.1 → 4.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1e7ab6ac58c584a113159e7f42678f508078c83cdeb9a21c86a7f60a5c936b21
4
- data.tar.gz: b4c3abaa02eafb546284dc68a08ba4443da4c1b13aa824c98f51e0dac3e749da
3
+ metadata.gz: d6850fbd8d4f1d8d81e2369bc5d27aa0638857e9cf32d2847a6c93f9efe84d9f
4
+ data.tar.gz: ac6b4527b4a38f8917e25ed8576628e498e67e20cd4f0d3dfaf9696447cfd9a3
5
5
  SHA512:
6
- metadata.gz: 1b032aa2876124669bf1eac1bc09cf67e79b4df4105e95fbc7d416132dd8d5bf1088c49e4c88b304e9a796e119edea60e4c2fae49a70aba050889a928b66b206
7
- data.tar.gz: 91066213316a173023257e30507fdb8c7c0e4e0b360b91dd1efcafebbacd291c82f62e9470798aa86488ae991bb8f81b1c395856ee2e03b537e3ae5a244d78c3
6
+ metadata.gz: 80803e406ce6e6da82ed72e252c2e6b59349aae34dc128eb1231688ca071a95ff125d93b882cbdd0ece94897f1ca1c7156cb3abba894421ba54d042e56b961c8
7
+ data.tar.gz: 0b8afdf336ae0394ed6dcdea7f597968b3825c13ed01ce1a571a82078532d17e82733aa57a791bf52f26723ad0821be61b785355c25e75479019e605ca71d0de
@@ -1,5 +1,5 @@
1
1
  # DO NOT MANUALLY EDIT; Run ./scripts/update-asdf-version-variables.sh to update this
2
2
  variables:
3
- GL_ASDF_RUBY_VERSION: "4.0.5"
3
+ GL_ASDF_RUBY_VERSION: "4.0.6"
4
4
  GL_ASDF_SHELLCHECK_VERSION: "0.11"
5
5
  GL_ASDF_SHFMT_VERSION: "3.13"
data/.tool-versions CHANGED
@@ -1,3 +1,3 @@
1
- ruby 4.0.5
1
+ ruby 4.0.6
2
2
  shfmt 3.13
3
3
  shellcheck 0.11
@@ -219,7 +219,7 @@ An exceeded rule has `remaining` 0 and therefore outranks any rule still under
219
219
  its limit. For a `:log` rule that means shadow traffic surfaces to the caller
220
220
  as `exceeded? == true` (with `action` still `:allow`) — visible, but unable to
221
221
  block. A `:log`-only path that matches therefore returns `matched? == true`
222
- and emits no `rule="unmatched"` metric.
222
+ and its check is counted as `checks_total{matched="true"}`.
223
223
 
224
224
  ```mermaid
225
225
  flowchart TD
@@ -227,20 +227,20 @@ flowchart TD
227
227
  Iter -->|yes| Match{rule.match<br/>all satisfied?}
228
228
  Match -->|no| Iter
229
229
  Match -->|yes| Skip{rule.action<br/>== :skip?}
230
- Skip -->|"yes (no Redis op)"| SkipEmit[Emit calls_total<br/>action=skip]
230
+ Skip -->|"yes (no Redis op)"| SkipEmit[Emit rule_evaluations_total<br/>action=skip, result=skip]
231
231
  SkipEmit --> SkipReturn([Return matched=true<br/>action=:allow])
232
232
  Skip -->|no| Eval["INCR Redis counter<br/>(see Redis sequence below)"]
233
233
  Eval --> Build[Build Evaluation<br/>resolve limit/period]
234
234
  Build --> Add[Add evaluation to Result]
235
- Add --> Emit[Emit calls_total + limit/period gauges]
235
+ Add --> Emit[Emit rule_evaluations_total<br/>+ limit/period gauges]
236
236
  Emit --> Act{"result.block?<br/>(:limit rule over limit)"}
237
237
  Act -->|yes| Return([Return Result<br/>action=:block])
238
238
  Act -->|"no (:log, or :limit under limit)"| Iter
239
- Iter -->|no more rules| Any{any rule<br/>evaluated?}
240
- Any -->|yes| ReturnFold([Return Result reporting<br/>most-constraining evaluation])
241
- Any -->|no| Unmatched[Emit calls_total<br/>rule=unmatched, action=allow]
242
- Unmatched --> ReturnUnmatched([Return matched=false<br/>action=:allow])
243
- Eval -. StandardError .-> Error[Emit errors_total<br/>log warn]
239
+ Iter -->|no more rules| Return2([Return Result reporting<br/>most-constraining evaluation,<br/>or matched=false if none])
240
+ Return --> Check[Emit checks_total<br/>action, matched, error]
241
+ SkipReturn --> Check
242
+ Return2 --> Check
243
+ Eval -. StandardError .-> Error[Emit errors_total +<br/>checks_total error=true<br/>log warn]
244
244
  Error --> ReturnErr([Return error=true<br/>action=:allow])
245
245
  ```
246
246
 
@@ -249,23 +249,41 @@ discards the verdicts of the rules already evaluated, even though their counters
249
249
  were incremented. Those requests are counted but produce no verdict — the
250
250
  fail-open trade-off is that a request is never blocked on a partial evaluation.
251
251
 
252
- Note that `calls_total` is emitted **per matched rule**, so summing it by
253
- `rate_limiter` counts rule evaluations, not requests.
252
+ `rule_evaluations_total` is emitted **per evaluated rule** (including matched
253
+ `:skip` rules); `checks_total` is emitted **exactly once per check**, whatever
254
+ path the evaluation takes — so summing `checks_total` by `rate_limiter` counts
255
+ requests through the limiter, and "no rule matched" is `matched="false"` on the
256
+ check rather than a placeholder rule.
257
+
258
+ Only matched rules are evaluated: a rule whose `match:` conditions are not
259
+ satisfied emits nothing to `rule_evaluations_total` (match-testing is not an
260
+ evaluation), so the counter has no "didn't match" population and needs no
261
+ label for it. A matched `count_distinct` rule skipped by the missing-key
262
+ fail-open also emits nothing here — it was never evaluated; that check is
263
+ visible via `checks_total{error="true"}` and the
264
+ `rate_limit_missing_count_distinct` log, which carries the rule name.
265
+
266
+ During the transition the deprecated `calls_total` counter is additionally
267
+ emitted at every point the diagram emits `rule_evaluations_total` (with its
268
+ historical per-rule semantics, including the `rule="unmatched"` placeholder
269
+ after the loop). It is not shown above to keep the diagram legible; see the
270
+ Metrics table below.
254
271
 
255
272
  ### Actions
256
273
 
257
274
  The rule's `action` describes what the rule does; the result's `action`
258
275
  describes the outcome — what the caller should do — and is only ever `:allow`
259
- or `:block`. The counter is always incremented when a rule matches, except for
260
- `:skip` rules, which never touch Redis:
261
-
262
- | rule action | what it does | exceeded? | result action | terminating? |
263
- |-------------|-----------------------------------------------|-----------|---------------|---------------|
264
- | `:limit` | count against the limit | no | `:allow` | no — continue |
265
- | `:limit` | count against the limit | yes | `:block` | yesstop |
266
- | `:log` | count against the limit (observability only) | no | `:allow` | nocontinue |
267
- | `:log` | count against the limit (observability only) | yes | `:allow` | no — continue |
268
- | `:skip` | don't count (bypass) | n/a | `:allow` | yesstop |
276
+ or `:block`. The Redis counter is always incremented when a rule matches,
277
+ except for `:skip` rules, which never touch Redis. The `result` label on
278
+ `rule_evaluations_total` records what each evaluation decided:
279
+
280
+ | rule action | what it does | exceeded? | result action | `result` label | terminating? |
281
+ |-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|
282
+ | `:limit` | count against the limit | no | `:allow` | `allow` | no continue |
283
+ | `:limit` | count against the limit | yes | `:block` | `block` | yes stop |
284
+ | `:log` | count against the limit (observability only) | no | `:allow` | `allow` | no — continue |
285
+ | `:log` | count against the limit (observability only) | yes | `:allow` | `log` | no continue |
286
+ | `:skip` | don't count (bypass) | n/a | `:allow` | `skip` | yes — stop |
269
287
 
270
288
  - `:limit` — when exceeded, `Result#action` is `:block` and evaluation
271
289
  terminates. Caller should reject the request (e.g. with HTTP 429). When under
@@ -281,8 +299,8 @@ or `:block`. The counter is always incremented when a rule matches, except for
281
299
  `Result#action` `:allow` **without any Redis operation**: nothing is
282
300
  counted, so `limit`, `period`, `characteristics`, and `count_distinct` are
283
301
  inert and the result carries no `info` (`to_response_headers` is `{}`).
284
- The match is still observable via `calls_total{action="skip"}`. Use this
285
- for bypasses.
302
+ The match is still observable via `rule_evaluations_total{action="skip"}`.
303
+ Use this for bypasses.
286
304
 
287
305
  ### Redis keys
288
306
 
@@ -375,26 +393,53 @@ safe to merge unconditionally.
375
393
 
376
394
  The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
377
395
  (Redis connection failure, timeout, OOM in user-supplied callables, …) is
378
- logged at WARN with `message: "rate_limit_error"` and returned as an error
379
- `Result` (`matched?` false, `error?` true, `action` `:allow`). The
380
- `gitlab_labkit_rate_limiter_errors_total` counter is incremented. The caller
396
+ logged at WARN with `error_type: "rate_limit_error"` and returned as an error
397
+ `Result` (`matched?` false, `error?` true, `action` `:allow`). The caller
381
398
  should treat the request as allowed.
382
399
 
400
+ A failed-open `check` is still counted: it emits
401
+ `checks_total{action="allow", matched="false", error="true"}`, so the fraction
402
+ of checks that encountered an error is `checks_total{error="true"}` over
403
+ `checks_total`. `error="true"` also covers a check where a matched
404
+ `count_distinct` rule was skipped because the identifier was missing its
405
+ `count_distinct` key — that check completes (`action` and `matched` describe
406
+ its outcome as usual), so `error="true"` is not exclusively fail-open traffic.
407
+
408
+ The deprecated `gitlab_labkit_rate_limiter_errors_total` counter is still
409
+ incremented on every fail-open (whole-check and per-rule `count_distinct`),
410
+ and remains the only error metric for `peek`, which emits no `checks_total`
411
+ (peek must not inflate the per-check counter). The follow-up MR that removes
412
+ `errors_total` must first decide where `peek` errors go — a dedicated peek
413
+ metric, or logs only.
414
+
415
+ Metric emission itself is best-effort: a failure in the metrics stack never
416
+ alters the verdict or breaks fail-open, and is logged at WARN with
417
+ `error_type: "rate_limit_metrics_error"` (once per process, to avoid
418
+ flooding).
419
+
383
420
  ## Metrics
384
421
 
385
422
  `Labkit::RateLimit::Metrics` emits the following Prometheus metrics through
386
423
  `Labkit::Metrics::Client`:
387
424
 
388
- | metric | type | labels | meaning |
389
- |-------------------------------------------------|---------|-------------------------------------|----------------------------------------------------------------------|
390
- | `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | One increment per counted rule (plus one per matched `:skip` rule). `action` is the rule-level outcome: `"allow"` (under limit), `"limit"` (blocking `:limit` rule), `"log"` (exceeded `:log` rule), `"skip"`. `rule="unmatched", action="allow"` when no rule matched. |
391
- | `gitlab_labkit_rate_limiter_errors_total` | counter | `rate_limiter` | Fail-open events (any `StandardError` in the labkit path). |
392
- | `gitlab_labkit_rate_limiter_limit` | gauge | `rate_limiter`, `rule` | Resolved limit at the last check (useful when `limit:` is callable). |
393
- | `gitlab_labkit_rate_limiter_period_seconds` | gauge | `rate_limiter`, `rule` | Resolved period at the last check. |
394
-
395
- Because every matching rule is counted, a single `check` call can emit
396
- **multiple** `calls_total` increments: one per counted rule (or a single
397
- `rule="unmatched"` increment if nothing matched).
425
+ | metric | type | labels | meaning |
426
+ |-----------------------------------------------------|---------|----------------------------------------------|----------------------------------------------------------------------|
427
+ | `gitlab_labkit_rate_limiter_checks_total` | counter | `rate_limiter`, `action`, `matched`, `error` | Exactly one increment per `check` call, including fail-open. `action` is what the caller should do (`"allow"` or `"block"`); `matched` and `error` are `"true"`/`"false"`. |
428
+ | `gitlab_labkit_rate_limiter_rule_evaluations_total` | counter | `rate_limiter`, `rule`, `action`, `result` | One increment per evaluated rule (plus one per matched `:skip` rule). `action` is the configured rule action (`"limit"`, `"log"`, `"skip"`); `result` is what the evaluation decided (`"allow"`, `"block"`, `"log"`, `"skip"` — see the Actions table). |
429
+ | `gitlab_labkit_rate_limiter_calls_total` | counter | `rate_limiter`, `rule`, `action` | **Deprecated** superseded by `checks_total` + `rule_evaluations_total`. Historical per-rule counter: one increment per counted rule (plus one per matched `:skip` rule), `rule="unmatched", action="allow"` when no rule matched. Emitted unchanged during the transition. |
430
+ | `gitlab_labkit_rate_limiter_errors_total` | counter | `rate_limiter` | **Deprecated** — use `checks_total{error="true"}`. Fail-open events (any `StandardError` in the labkit path); still the only error metric for `peek`. |
431
+ | `gitlab_labkit_rate_limiter_limit` | gauge | `rate_limiter`, `rule` | Resolved limit at the last check (useful when `limit:` is callable). |
432
+ | `gitlab_labkit_rate_limiter_period_seconds` | gauge | `rate_limiter`, `rule` | Resolved period at the last check. |
433
+
434
+ `sum by (rate_limiter) (rate(gitlab_labkit_rate_limiter_checks_total[5m]))` is
435
+ the request rate through a limiter — no exclusions or dedup needed. A single
436
+ `check` call emits **one** `checks_total` increment and as many
437
+ `rule_evaluations_total` increments as rules it evaluated (possibly zero).
438
+
439
+ **Transition:** the deprecated `calls_total` and `errors_total` counters keep
440
+ emitting exactly as before this split, so existing dashboards and alerts stay
441
+ correct while consumers migrate to the new counters. Both are removed in a
442
+ follow-up major release once nothing consumes them.
398
443
 
399
444
  ## Dev/test vs production guards
400
445
 
@@ -75,13 +75,17 @@ module Labkit
75
75
 
76
76
  def check(identifier, cost: 1, rule_context: nil)
77
77
  cursor = RuleCursor.new
78
- check_rules(identifier, cost, rule_context, cursor)
78
+ result = check_rules(identifier, cost, rule_context, cursor)
79
79
  rescue StandardError => e
80
80
  # Intentionally broad: fail-open applies to any unexpected error (network,
81
81
  # timeout, OOM) not only Redis protocol errors.
82
82
  report_error_metrics
83
83
  log_error(e, identifier, cursor.rule)
84
- Result.error
84
+ result = Result.error
85
+ ensure
86
+ # StandardError-safe emission, so it cannot mask a propagating error.
87
+ # result is nil when a non-StandardError unwinds: emit nothing then.
88
+ report_check_metrics(result) if result
85
89
  end
86
90
 
87
91
  # Read-without-increment counterpart to {#check}. Same matching and Result
@@ -104,7 +108,7 @@ module Labkit
104
108
  # - :skip terminates on match without touching Redis. No counter is
105
109
  # incremented, so the branch sits before the count_distinct check
106
110
  # (identifier completeness is irrelevant to a rule that builds no key).
107
- # calls_total still increments so the bypass stays observable.
111
+ # rule_evaluations_total still increments so the bypass stays observable.
108
112
  # - a :limit rule over its limit terminates, because the request is
109
113
  # rejected and later rules cannot change that. Rules declared after it
110
114
  # are neither counted nor evaluated, so a blocked request debits every
@@ -115,10 +119,14 @@ module Labkit
115
119
  # most-constraining evaluation (ranking lives in Result::Evaluation#<=>).
116
120
  # cost is therefore debited from every matching rule, not just the first.
117
121
  #
122
+ # Metrics: each evaluated rule emits rule_evaluations_total (plus the
123
+ # deprecated per-rule calls_total); the per-check checks_total is
124
+ # emitted once in #check. Full contract in the README's Metrics section.
125
+ #
118
126
  # SET-mode rules (rule.count_distinct set) that match but whose identifier
119
- # is missing the count_distinct key fail open + log + bump errors_total, and
120
- # the loop continues to the next rule (the rule is treated as not applicable
121
- # rather than aborting the whole evaluation).
127
+ # is missing the count_distinct key fail open + log + bump errors_total +
128
+ # flag the Result, and the loop continues to the next rule (the rule is
129
+ # treated as not applicable rather than aborting the whole evaluation).
122
130
  #
123
131
  # Error handling stays whole-check (see #check): a raise part-way through
124
132
  # discards the results of the rules already evaluated, even though their
@@ -143,12 +151,13 @@ module Labkit
143
151
  if rule.count_distinct && missing_count_distinct_value?(rule, identifier)
144
152
  log_missing_count_distinct(rule, identifier)
145
153
  report_error_metrics
154
+ result.degraded!
146
155
  next
147
156
  end
148
157
 
149
158
  evaluation = evaluate_rule(rule, identifier, cost, rule_context)
150
159
  result.add_evaluation(evaluation)
151
- report_matched_metrics(evaluation)
160
+ report_evaluation_metrics(evaluation)
152
161
  return result if result.block?
153
162
  end
154
163
 
@@ -157,6 +166,12 @@ module Labkit
157
166
 
158
167
  report_unmatched_metrics unless result.matched?
159
168
  result
169
+ rescue StandardError => e # binds e for the ensure
170
+ raise
171
+ ensure
172
+ # Keep the rule attributed while an exception unwinds; clear it on
173
+ # every normal exit (early returns included).
174
+ cursor.rule = nil unless e
160
175
  end
161
176
 
162
177
  # Mirror of check_rules without metrics or writes. :log rules are read
@@ -180,9 +195,11 @@ module Labkit
180
195
  return result if result.block?
181
196
  end
182
197
 
183
- cursor.rule = nil
184
-
185
198
  result
199
+ rescue StandardError => e # binds e for the ensure
200
+ raise
201
+ ensure
202
+ cursor.rule = nil unless e
186
203
  end
187
204
 
188
205
  def rule_matches?(rule, identifier)
@@ -341,6 +358,7 @@ module Labkit
341
358
  # loop reached one, or after it finished - so the field is logged as null
342
359
  # rather than omitted, the same way identifier is. A named rule is the rule
343
360
  # whose match or evaluation raised.
361
+ # Never raises: a logging failure must not break fail-open.
344
362
  def log_error(error, identifier, rule = nil)
345
363
  @logger.warn(
346
364
  name: @name,
@@ -350,6 +368,8 @@ module Labkit
350
368
  Labkit::Fields::ERROR_MESSAGE => error.message,
351
369
  identifier: identifier&.to_h
352
370
  )
371
+ rescue StandardError
372
+ nil
353
373
  end
354
374
 
355
375
  def log_missing_count_distinct(rule, identifier)
@@ -362,43 +382,62 @@ module Labkit
362
382
  )
363
383
  end
364
384
 
365
- def report_matched_metrics(evaluation)
366
- Metrics.calls_total.increment(
367
- rate_limiter: @name,
368
- rule: evaluation.rule.name,
369
- action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s
370
- )
371
- Metrics.limit_gauge.set(
372
- { rate_limiter: @name, rule: evaluation.rule.name },
373
- evaluation.info.resolved_limit
385
+ def report_evaluation_metrics(evaluation)
386
+ rule_labels = { rate_limiter: @name, rule: evaluation.rule.name }
387
+
388
+ Metrics.safe_increment(
389
+ :rule_evaluations_total,
390
+ rule_labels.merge(action: evaluation.rule.action.to_s, result: evaluation_result(evaluation)),
391
+ logger: @logger
374
392
  )
375
- Metrics.period_gauge.set(
376
- { rate_limiter: @name, rule: evaluation.rule.name },
377
- evaluation.info.resolved_period
393
+ # Deprecated dual emission - remove together with Metrics.calls_total.
394
+ Metrics.safe_increment(
395
+ :calls_total,
396
+ rule_labels.merge(action: (evaluation.exceeded? ? evaluation.rule.action : :allow).to_s),
397
+ logger: @logger
378
398
  )
399
+ Metrics.safe_set(:limit_gauge, rule_labels, evaluation.info.resolved_limit, logger: @logger)
400
+ Metrics.safe_set(:period_gauge, rule_labels, evaluation.info.resolved_period, logger: @logger)
401
+ end
402
+
403
+ # An exceeded :log rule reports "log" rather than the "allow" the caller
404
+ # sees, so shadow rules over their limit stay visible.
405
+ def evaluation_result(evaluation)
406
+ return "allow" unless evaluation.exceeded?
407
+
408
+ evaluation.rule.action == :limit ? "block" : "log"
379
409
  end
380
410
 
381
- # calls_total carries action="skip" (the rule action, not the :allow the
382
- # caller sees) so bypass traffic stays distinguishable from counted
383
- # allows. No limit/period gauges: a skip rule has no limit to report.
384
411
  def report_skipped_metrics(rule)
385
- Metrics.calls_total.increment(
386
- rate_limiter: @name,
387
- rule: rule.name,
388
- action: "skip"
412
+ Metrics.safe_increment(
413
+ :rule_evaluations_total,
414
+ { rate_limiter: @name, rule: rule.name, action: "skip", result: "skip" },
415
+ logger: @logger
389
416
  )
417
+ # Deprecated dual emission - remove together with Metrics.calls_total.
418
+ Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: rule.name, action: "skip" }, logger: @logger)
390
419
  end
391
420
 
421
+ # Deprecated dual emission - remove together with Metrics.calls_total.
392
422
  def report_unmatched_metrics
393
- Metrics.calls_total.increment(
394
- rate_limiter: @name,
395
- rule: "unmatched",
396
- action: "allow"
423
+ Metrics.safe_increment(:calls_total, { rate_limiter: @name, rule: "unmatched", action: "allow" }, logger: @logger)
424
+ end
425
+
426
+ def report_check_metrics(result)
427
+ Metrics.safe_increment(
428
+ :checks_total,
429
+ {
430
+ rate_limiter: @name,
431
+ action: result.action.to_s,
432
+ matched: result.matched?.to_s,
433
+ error: (result.error? || result.degraded?).to_s
434
+ },
435
+ logger: @logger
397
436
  )
398
437
  end
399
438
 
400
439
  def report_error_metrics
401
- Metrics.errors_total.increment(rate_limiter: @name)
440
+ Metrics.safe_increment(:errors_total, { rate_limiter: @name }, logger: @logger)
402
441
  end
403
442
  end
404
443
  end
@@ -1,14 +1,60 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "concurrent-ruby"
4
+
3
5
  module Labkit
4
6
  module RateLimit
5
7
  module Metrics
8
+ # Process-wide once-latch for log_failure; specs reset via make_false.
9
+ FAILURE_LOGGED = Concurrent::AtomicBoolean.new(false)
10
+
6
11
  module_function
7
12
 
8
- # Emitted once per *matched rule*, not once per check: every rule that
9
- # matches is evaluated, so summing this by rate_limiter counts rule
10
- # evaluations rather than requests. rule="unmatched", action="allow" is
11
- # emitted only when no rule matched at all.
13
+ # Metric emission must never affect the caller's outcome. Resolving the
14
+ # metric by name keeps a raising getter inside the rescue.
15
+ def safe_increment(counter, labels, logger: nil)
16
+ public_send(counter).increment(**labels) # rubocop:disable GitlabSecurity/PublicSend
17
+ rescue StandardError => e
18
+ log_failure(e, counter, labels, logger)
19
+ end
20
+
21
+ # Gauge counterpart of safe_increment.
22
+ def safe_set(gauge, labels, value, logger: nil)
23
+ public_send(gauge).set(labels, value) # rubocop:disable GitlabSecurity/PublicSend
24
+ rescue StandardError => e
25
+ log_failure(e, gauge, labels, logger)
26
+ end
27
+
28
+ # make_true returns true only for the flipping caller, so exactly one
29
+ # warn per process. Never raises - a raise here would defeat the
30
+ # callers' rescues.
31
+ def log_failure(error, metric, labels, logger)
32
+ return unless logger && FAILURE_LOGGED.make_true
33
+
34
+ logger.warn(
35
+ name: labels[:rate_limiter],
36
+ metric: metric.to_s,
37
+ Labkit::Fields::ERROR_TYPE => "rate_limit_metrics_error",
38
+ Labkit::Fields::CLASS_NAME => error.class.to_s,
39
+ Labkit::Fields::ERROR_MESSAGE => error.message
40
+ )
41
+ rescue StandardError
42
+ nil
43
+ end
44
+
45
+ # Emitted exactly once per #check call, including calls that fail open;
46
+ # summing by rate_limiter gives the request rate through the limiter.
47
+ def checks_total
48
+ Labkit::Metrics::Client.counter(
49
+ :gitlab_labkit_rate_limiter_checks_total,
50
+ 'Total number of rate limit checks',
51
+ { rate_limiter: nil, action: nil, matched: nil, error: nil }
52
+ )
53
+ end
54
+
55
+ # Deprecated: superseded by checks_total and rule_evaluations_total.
56
+ # Emitted unchanged (once per matched rule, rule="unmatched" when none
57
+ # matched) until consumers migrate; will get removed in a follow-up release.
12
58
  def calls_total
13
59
  Labkit::Metrics::Client.counter(
14
60
  :gitlab_labkit_rate_limiter_calls_total,
@@ -17,6 +63,18 @@ module Labkit
17
63
  )
18
64
  end
19
65
 
66
+ # Emitted once per *evaluated* rule. Only matched rules are evaluated:
67
+ # Unmatched traffic is checks_total{matched="false"}
68
+ def rule_evaluations_total
69
+ Labkit::Metrics::Client.counter(
70
+ :gitlab_labkit_rate_limiter_rule_evaluations_total,
71
+ 'Total number of rate limit rule evaluations',
72
+ { rate_limiter: nil, rule: nil, action: nil, result: nil }
73
+ )
74
+ end
75
+
76
+ # Deprecated: superseded by checks_total{error="true"}. Still emitted
77
+ # because it remains the only error metric for #peek.
20
78
  def errors_total
21
79
  Labkit::Metrics::Client.counter(
22
80
  :gitlab_labkit_rate_limiter_errors_total,
@@ -25,6 +25,8 @@ module Labkit
25
25
  # false). Other rules may also have matched and been counted;
26
26
  # see #evaluations.
27
27
  # error? - true if Redis was unavailable; result fails open (exceeded? is false)
28
+ # degraded? - true if a matched count_distinct rule was skipped by the
29
+ # missing-key fail-open path; the check still completed
28
30
  # info - Result::Info with per-window counters for the reported rule;
29
31
  # nil when matched? is false, error?, or the matched rule is
30
32
  # :skip (no counter exists)
@@ -40,6 +42,7 @@ module Labkit
40
42
  @evaluations = []
41
43
  @skip_rule = nil
42
44
  @error = error
45
+ @degraded = false
43
46
  @most_constraining = nil
44
47
  end
45
48
 
@@ -78,6 +81,17 @@ module Labkit
78
81
  @error
79
82
  end
80
83
 
84
+ # Unlike error?, the check keeps going: the flag marks the outcome as
85
+ # degraded without changing it.
86
+ def degraded!
87
+ @degraded = true
88
+ self
89
+ end
90
+
91
+ def degraded?
92
+ @degraded
93
+ end
94
+
81
95
  def action
82
96
  block? ? :block : :allow
83
97
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-labkit
3
3
  version: !ruby/object:Gem::Version
4
- version: 4.4.1
4
+ version: 4.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Newdigate
@@ -716,7 +716,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
716
716
  - !ruby/object:Gem::Version
717
717
  version: '0'
718
718
  requirements: []
719
- rubygems_version: 4.0.10
719
+ rubygems_version: 4.0.16
720
720
  specification_version: 4
721
721
  summary: Instrumentation for GitLab
722
722
  test_files: []