gitlab-labkit 5.1.2 → 5.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/hooks/precommit-validate.sh +47 -0
- data/.claude/settings.json +15 -0
- data/.copier-answers.yml +2 -1
- data/.gitignore +0 -1
- data/.gitlab/CODEOWNERS +1 -1
- data/.gitlab/duo/agent-config.yml +7 -7
- data/.gitlab-ci.yml +7 -0
- data/.mise.toml +3 -0
- data/.opencode/plugins/precommit-validate.ts +101 -0
- data/.pre-commit-config.yaml +2 -1
- data/.rubocop_todo.yml +2 -17
- data/AGENTS.md +173 -0
- data/CLAUDE.md +1 -0
- data/CODEOWNERS +1 -1
- data/Gemfile +5 -0
- data/README.md +6 -19
- data/gitlab-labkit.gemspec +1 -0
- data/lib/labkit/fields.rb +15 -2
- data/lib/labkit/json_schema/README.md +1 -1
- data/lib/labkit/rate_limit/README.md +82 -7
- data/lib/labkit/rate_limit/matcher.rb +138 -28
- data/lib/labkit/rate_limit/result.rb +32 -9
- data/lib/labkit/rate_limit/rule.rb +14 -4
- data/lib/labkit/rspec/matchers/user_experience_matchers.rb +2 -2
- data/performance.yml +70 -0
- data/scripts/prepare-dev-env.sh +23 -1
- metadata +21 -2
- data/.env.example.sh +0 -7
|
@@ -171,8 +171,8 @@ A `Rule` is a `Data.define` value object with the following fields:
|
|
|
171
171
|
|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
172
172
|
| `name` | Stable identifier used in Redis keys and metric labels. Must match `/\A[a-z0-9_]+\z/`, max 64 chars. Renaming a rule abandons its in-flight counters. |
|
|
173
173
|
| `match` | Hash of identifier key/value predicates that must **all** be satisfied for the rule to apply. Empty hash matches anything. See [Matchers](#matchers). |
|
|
174
|
-
| `limit` | Integer request threshold per `period
|
|
175
|
-
| `period` | Window length in seconds. May be a callable resolved on every check.
|
|
174
|
+
| `limit` | Integer request threshold per `period`, at least 1. May be a callable resolved on every check. |
|
|
175
|
+
| `period` | Window length in seconds, at least 1; a fraction truncates to whole seconds. May be a callable resolved on every check. |
|
|
176
176
|
| `action` | What the rule does when it matches. One of `:limit`, `:log`, `:skip`. Default `:limit`. See [Actions](#actions). |
|
|
177
177
|
| `characteristics` | Array of identifier keys whose values are folded into the Redis counter key. Each unique combination gets its own counter. |
|
|
178
178
|
| `count_distinct` | Optional identifier key. When set, the rule counts distinct values seen for that key within the window, backed by a Redis SET rather than a counter. Must not overlap `characteristics`. |
|
|
@@ -190,6 +190,13 @@ Labkit::RateLimit::Rule.new(
|
|
|
190
190
|
)
|
|
191
191
|
```
|
|
192
192
|
|
|
193
|
+
`Rule.new` raises `ArgumentError` for a static `limit` under 1 or a static
|
|
194
|
+
`period` under one second, except on `:skip` rules where both are inert.
|
|
195
|
+
Callables are not validated: a `limit` resolving to `0` blocks every matching
|
|
196
|
+
request, which is how to express "deny all" dynamically, and a `period`
|
|
197
|
+
resolving under one second reaches `EXPIRE` as `0` and deletes the counter on
|
|
198
|
+
every write.
|
|
199
|
+
|
|
193
200
|
### Matchers
|
|
194
201
|
|
|
195
202
|
A `match` hash gates whether a rule applies. Each value is normalised through
|
|
@@ -200,17 +207,63 @@ A `match` hash gates whether a rule applies. Each value is normalised through
|
|
|
200
207
|
| plain value | `eq` | `match: { user: nil }`, `match: { method: "POST" }` |
|
|
201
208
|
| `Regexp` | `re` | `match: { endpoint: %r{\A/api/} }` |
|
|
202
209
|
| `{ eq: <value> }` | `eq` | `match: { method: { eq: "POST" } }` (YAML-friendly) |
|
|
210
|
+
| `{ ne: <value> }` | `ne` | `match: { method: { ne: "OPTIONS" } }` |
|
|
203
211
|
| `{ re: <source> }`| `re` | `match: { endpoint: { re: '\A/api/' } }` (YAML-friendly) |
|
|
212
|
+
| `{ nre: <source> }`| `nre` | `match: { endpoint: { nre: '\A/api/' } }` |
|
|
204
213
|
| `{ oneOf: <Array> }` | `oneOf` | `match: { user: { oneOf: ["7", "42"] } }` (YAML-friendly) |
|
|
214
|
+
| `{ noneOf: <Array> }` | `noneOf` | `match: { plan: { noneOf: ["premium"] } }` |
|
|
215
|
+
| `Array` of matchers | `all` | `match: { endpoint: [%r{\A/api/}, { nre: 'internal' }] }` |
|
|
205
216
|
|
|
206
|
-
`re`
|
|
207
|
-
used against non-String values (e.g. matching a 503 status against
|
|
217
|
+
`re` and `nre` coerce the identifier value via `#to_s` before matching, so they
|
|
218
|
+
can be used against non-String values (e.g. matching a 503 status against
|
|
219
|
+
`{ re: '^5' }`).
|
|
208
220
|
|
|
209
221
|
`oneOf` is set membership via `Set#include?` (`eql?`/`hash` equality, no
|
|
210
222
|
coercion): it agrees with `eq` for Strings, Symbols, booleans, nil, and
|
|
211
223
|
same-class numerics, but cross-type numerics differ (`1 == 1.0`, yet
|
|
212
|
-
`{ oneOf: [1] }` does not match `1.0`).
|
|
213
|
-
|
|
224
|
+
`{ oneOf: [1] }` does not match `1.0`). `noneOf` is its complement.
|
|
225
|
+
|
|
226
|
+
An Array means **AND**, so one identifier key can carry several predicates:
|
|
227
|
+
|
|
228
|
+
```yaml
|
|
229
|
+
endpoint:
|
|
230
|
+
- nre: '/oauth/'
|
|
231
|
+
- nre: '\A/api/'
|
|
232
|
+
- nre: '\A/-/(?:health|liveness|readiness|metrics)'
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Its entries must be matcher Hashes or bare `Regexp`s, the two shapes that
|
|
236
|
+
already mean "a matcher" on their own. An Array of plain values is rejected.
|
|
237
|
+
A value meant as one matcher, arriving as a list, therefore fails loudly rather
|
|
238
|
+
than becoming a membership test. Membership is spelled `{ oneOf: [...] }`.
|
|
239
|
+
|
|
240
|
+
An Array holds matchers, not other Arrays, so a match value nests three deep at
|
|
241
|
+
most: an Array, a matcher Hash inside it, and that Hash's `oneOf` list. A value
|
|
242
|
+
nesting deeper than `Matcher::MAX_NESTING_DEPTH` is rejected when the rule is
|
|
243
|
+
built.
|
|
244
|
+
|
|
245
|
+
#### Negation and absent keys
|
|
246
|
+
|
|
247
|
+
`ne`, `nre` and `noneOf` are plain complements of their positive forms, with no
|
|
248
|
+
special case for a missing key. The `Evaluator` looks an identifier key up
|
|
249
|
+
rather than requiring it, so an absent key reaches the matcher as `nil`, and a
|
|
250
|
+
complement is therefore **true** for one: a request with no `endpoint` is not a
|
|
251
|
+
request to an excluded endpoint.
|
|
252
|
+
|
|
253
|
+
That widens a rule rather than narrowing it. A rule that should apply only when
|
|
254
|
+
the key is present says so, with a presence matcher alongside the exclusions:
|
|
255
|
+
|
|
256
|
+
```ruby
|
|
257
|
+
match: { endpoint: [/./, { nre: '\A/api/' }] }
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
Since this follows from inversion, a matcher that does accept `nil` inverts the
|
|
261
|
+
other way: `{ ne: nil }`, `{ noneOf: [nil] }` and `{ nre: '^$' }` are all false
|
|
262
|
+
for an absent key, the last because `nil.to_s` is `""`.
|
|
263
|
+
|
|
264
|
+
There is no cap on regex source length. Rules are written by operators, and the
|
|
265
|
+
only bound on a pattern is `Matcher::MATCH_TIMEOUT_SECONDS`, a 50ms wall-clock
|
|
266
|
+
budget applied per compiled `Regexp`.
|
|
214
267
|
|
|
215
268
|
Glob and prefix matchers are intentionally out of scope.
|
|
216
269
|
|
|
@@ -459,7 +512,8 @@ result.degraded? # => true if a matched count_distinct rule was skipped
|
|
|
459
512
|
result.info # => Result::Info or nil
|
|
460
513
|
result.evaluations # => every counted Result::Evaluation, in rule order
|
|
461
514
|
result.to_response_headers
|
|
462
|
-
# => { "RateLimit-
|
|
515
|
+
# => { "RateLimit-Name" => "<rule name>", "RateLimit-Limit" => "...", "RateLimit-Observed" => "...",
|
|
516
|
+
# "RateLimit-Remaining" => "...", "RateLimit-Reset" => "<unix-ts>" }
|
|
463
517
|
```
|
|
464
518
|
|
|
465
519
|
`Result::Info` holds the per-window counter snapshot:
|
|
@@ -475,6 +529,27 @@ result.to_response_headers
|
|
|
475
529
|
`to_response_headers` returns `{}` for an unmatched or error result, so it is
|
|
476
530
|
safe to merge unconditionally.
|
|
477
531
|
|
|
532
|
+
### Response headers
|
|
533
|
+
|
|
534
|
+
`Result#to_response_headers` reports the most constraining evaluation.
|
|
535
|
+
`Result::Evaluation#to_response_headers` builds the same headers for one evaluation, for a caller that picks its own evaluation across several results.
|
|
536
|
+
|
|
537
|
+
| header | present | value |
|
|
538
|
+
|-----------------------|-------------------|-----------------------------------------------------|
|
|
539
|
+
| `RateLimit-Name` | always | The rule's `name`. |
|
|
540
|
+
| `RateLimit-Limit` | always | `resolved_limit` in the rule's period, at least 0. |
|
|
541
|
+
| `RateLimit-Observed` | always | `count`, as an Integer, at least 0. |
|
|
542
|
+
| `RateLimit-Remaining` | always | `remaining`, as an Integer. |
|
|
543
|
+
| `RateLimit-Reset` | always | `reset_at` as a Unix timestamp. |
|
|
544
|
+
| `Retry-After` | blocking only | Seconds to `reset_at`, rounded up, at least 0. |
|
|
545
|
+
| `RateLimit-ResetTime` | blocking only | `reset_at` as an HTTP date. |
|
|
546
|
+
|
|
547
|
+
`Retry-After` and `RateLimit-ResetTime` belong on the 429, so they appear only when the evaluation blocks.
|
|
548
|
+
For a rule with `ban_for`, `reset_at` is the end of the ban, so these headers report when the ban lifts.
|
|
549
|
+
|
|
550
|
+
`RateLimit-Limit`, `RateLimit-Observed` and `RateLimit-Remaining` all count within the rule's `period`.
|
|
551
|
+
The headers do not state the period, so a client that needs it must know the rule.
|
|
552
|
+
|
|
478
553
|
## Fail-open
|
|
479
554
|
|
|
480
555
|
The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
|
|
@@ -12,27 +12,49 @@ module Labkit
|
|
|
12
12
|
# - any plain value (String, Symbol, Integer, ...) -> :eq matcher
|
|
13
13
|
# - a Regexp instance -> :re matcher (Ruby convenience)
|
|
14
14
|
# - { eq: <value> } -> :eq matcher (canonical, YAML-compatible)
|
|
15
|
+
# - { ne: <value> } -> :ne matcher (complement of :eq)
|
|
15
16
|
# - { re: <String|Regexp> } -> :re matcher (canonical, YAML-compatible)
|
|
17
|
+
# - { nre: <String|Regexp> } -> :nre matcher (complement of :re)
|
|
16
18
|
# - { oneOf: <Array> } -> :oneOf matcher (set membership, YAML-compatible)
|
|
19
|
+
# - { noneOf: <Array> } -> :noneOf matcher (complement of :oneOf)
|
|
20
|
+
# - [ <matcher>, <matcher>, ... ] -> :all matcher (every entry must match)
|
|
17
21
|
#
|
|
18
|
-
#
|
|
19
|
-
#
|
|
20
|
-
#
|
|
22
|
+
# An Array means AND, so several predicates can apply to one identifier key.
|
|
23
|
+
# Its entries must be matcher Hashes or bare Regexps, the two shapes that
|
|
24
|
+
# already mean "a matcher" on their own. An Array of plain values is
|
|
25
|
+
# rejected, so a value that was meant as a single matcher but arrived as a
|
|
26
|
+
# list fails loudly instead of silently becoming a membership test. An Array
|
|
27
|
+
# holds matchers rather than other Arrays, so a match value nests at most
|
|
28
|
+
# MAX_NESTING_DEPTH deep; see too_deeply_nested?.
|
|
21
29
|
#
|
|
22
|
-
# :oneOf membership is Set#include? (eql?/hash equality), not
|
|
23
|
-
# cross-type numerics differ (1 == 1.0, but {oneOf: [1]} does
|
|
30
|
+
# :oneOf and :noneOf membership is Set#include? (eql?/hash equality), not
|
|
31
|
+
# :eq's ==, so cross-type numerics differ (1 == 1.0, but {oneOf: [1]} does
|
|
32
|
+
# not match 1.0).
|
|
24
33
|
#
|
|
25
34
|
# Hash-key naming follows the metrics-catalog selector pattern. Glob,
|
|
26
35
|
# prefix, and other matcher kinds are intentionally out of scope here; see
|
|
27
36
|
# gitlab-com/gl-infra/production-engineering#28853 for that follow-up.
|
|
28
37
|
#
|
|
29
|
-
#
|
|
30
|
-
# the regex, so callers can match non-String identifier values such
|
|
31
|
-
# Integer status codes (e.g. {status: { re: "^5" }} against status: 503).
|
|
38
|
+
# The :re and :nre matchers coerce the identifier value via #to_s before
|
|
39
|
+
# applying the regex, so callers can match non-String identifier values such
|
|
40
|
+
# as Integer status codes (e.g. {status: { re: "^5" }} against status: 503).
|
|
41
|
+
#
|
|
42
|
+
# Each negation matcher is the plain complement of its positive form, with
|
|
43
|
+
# no special case for a missing key. Evaluator passes identifier[key], so an
|
|
44
|
+
# absent key arrives as nil, and the complement is therefore usually true
|
|
45
|
+
# for one: a request with no path is not a request to an excluded path. A
|
|
46
|
+
# rule that should apply only when the key is present says so, by adding a
|
|
47
|
+
# presence matcher alongside the exclusions:
|
|
48
|
+
#
|
|
49
|
+
# path: [/./, { nre: '\A/api/' }]
|
|
32
50
|
class Matcher < Data.define(:type, :value)
|
|
33
|
-
KNOWN_HASH_KEYS = %i[eq re oneOf].freeze
|
|
34
|
-
MAX_REGEX_SOURCE_LENGTH = 200
|
|
51
|
+
KNOWN_HASH_KEYS = %i[eq ne re nre oneOf noneOf].freeze
|
|
35
52
|
ERROR_INSPECT_LIMIT = 80
|
|
53
|
+
# No Hash or Array may sit this many levels below the match value, which
|
|
54
|
+
# is level 0. The deepest valid shape is an Array (0) holding a matcher
|
|
55
|
+
# Hash (1) whose :oneOf list (2) holds plain values; see
|
|
56
|
+
# too_deeply_nested?.
|
|
57
|
+
MAX_NESTING_DEPTH = 3
|
|
36
58
|
|
|
37
59
|
# Wall-clock budget for a single #match? call. Without it a match is
|
|
38
60
|
# bounded only by whatever global the host sets (40s in GitLab Rails,
|
|
@@ -45,16 +67,23 @@ module Labkit
|
|
|
45
67
|
# timeouts were wall-clock stalls, since Ruby counts real time and GC
|
|
46
68
|
# pauses on GitLab's git fleet have a p50 of ~72ms. Under 4x CPU
|
|
47
69
|
# oversubscription 50ms still fires on 0.16% of those matches, 5ms 1.33%.
|
|
70
|
+
#
|
|
71
|
+
# This timeout is the only bound on a pattern. There is no cap on regex
|
|
72
|
+
# source length: rules are written by operators rather than end users, and
|
|
73
|
+
# a cap that applied to {re: "..."} but not to a bare Regexp bounded
|
|
74
|
+
# nothing in practice, since callers pass Regexp constants.
|
|
48
75
|
MATCH_TIMEOUT_SECONDS = 0.05
|
|
49
76
|
|
|
50
77
|
def self.build(input)
|
|
78
|
+
raise ArgumentError, nesting_error(input) if too_deeply_nested?(input)
|
|
79
|
+
|
|
51
80
|
case input
|
|
52
81
|
when Regexp
|
|
53
|
-
new(type: :re, value: with_match_timeout(input))
|
|
82
|
+
new(type: :re, value: with_match_timeout(:re, input))
|
|
54
83
|
when Hash
|
|
55
84
|
from_hash(input)
|
|
56
85
|
when Array
|
|
57
|
-
|
|
86
|
+
compose(input)
|
|
58
87
|
else
|
|
59
88
|
new(type: :eq, value: input)
|
|
60
89
|
end
|
|
@@ -64,6 +93,15 @@ module Labkit
|
|
|
64
93
|
raise ArgumentError, invalid_shape_error(input) if input.size != 1
|
|
65
94
|
|
|
66
95
|
type, source = input.first
|
|
96
|
+
|
|
97
|
+
# Identifier rejects keys the same way. Without this, a key such as an
|
|
98
|
+
# Integer or nil leaves NoMethodError for the caller instead of the
|
|
99
|
+
# ArgumentError every other bad shape raises.
|
|
100
|
+
unless type.respond_to?(:to_sym)
|
|
101
|
+
raise ArgumentError,
|
|
102
|
+
"rate-limit match value type key #{truncate_for_error(type)} must be a String or Symbol"
|
|
103
|
+
end
|
|
104
|
+
|
|
67
105
|
type_sym = type.to_sym
|
|
68
106
|
|
|
69
107
|
unless KNOWN_HASH_KEYS.include?(type_sym)
|
|
@@ -77,31 +115,45 @@ module Labkit
|
|
|
77
115
|
|
|
78
116
|
def self.compile(type_sym, source)
|
|
79
117
|
case type_sym
|
|
80
|
-
when :eq
|
|
81
|
-
new(type:
|
|
82
|
-
when :oneOf
|
|
118
|
+
when :eq, :ne
|
|
119
|
+
new(type: type_sym, value: source)
|
|
120
|
+
when :oneOf, :noneOf
|
|
83
121
|
unless source.is_a?(Array)
|
|
84
122
|
raise ArgumentError,
|
|
85
|
-
"rate-limit match value {
|
|
123
|
+
"rate-limit match value {#{type_sym}: ...} must be an Array, got #{truncate_for_error(source)}"
|
|
86
124
|
end
|
|
87
125
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
126
|
+
# {oneOf: []} matches nothing, so the rule never applies and no harm
|
|
127
|
+
# follows. Its complement matches everything, which is an exclusion
|
|
128
|
+
# that excludes nothing: rejected for the same reason an empty Array
|
|
129
|
+
# of matchers is.
|
|
130
|
+
if type_sym == :noneOf && source.empty?
|
|
91
131
|
raise ArgumentError,
|
|
92
|
-
"rate-limit match value {
|
|
132
|
+
"rate-limit match value {noneOf: []} must not be empty, it matches everything"
|
|
93
133
|
end
|
|
94
134
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
raise ArgumentError,
|
|
99
|
-
"rate-limit match value {re: #{truncate_for_error(source)}} failed to compile: #{e.message}"
|
|
100
|
-
end
|
|
135
|
+
new(type: type_sym, value: Set.new(source).freeze)
|
|
136
|
+
when :re, :nre
|
|
137
|
+
new(type: type_sym, value: with_match_timeout(type_sym, source))
|
|
101
138
|
end
|
|
102
139
|
end
|
|
103
140
|
private_class_method :compile
|
|
104
141
|
|
|
142
|
+
# Builds the :all matcher behind the Array form. Entries go back through
|
|
143
|
+
# .build, which is what keeps a nested regex bounded by the match timeout.
|
|
144
|
+
def self.compose(input)
|
|
145
|
+
raise ArgumentError, "rate-limit match value [] must not be empty" if input.empty?
|
|
146
|
+
|
|
147
|
+
unless input.all? { |entry| entry.is_a?(Hash) || entry.is_a?(Regexp) }
|
|
148
|
+
raise ArgumentError,
|
|
149
|
+
"rate-limit match value Array entries must each be a matcher Hash or Regexp, " \
|
|
150
|
+
"got #{truncate_for_error(input)}"
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
new(type: :all, value: input.map { |entry| build(entry) }.freeze)
|
|
154
|
+
end
|
|
155
|
+
private_class_method :compose
|
|
156
|
+
|
|
105
157
|
# Compiles +source+ (a String pattern or an existing Regexp) into a
|
|
106
158
|
# Regexp bounded by MATCH_TIMEOUT_SECONDS.
|
|
107
159
|
#
|
|
@@ -116,7 +168,7 @@ module Labkit
|
|
|
116
168
|
# Recompiling an existing Regexp preserves its source and options
|
|
117
169
|
# (including the fixed-encoding flags) and therefore +#==+; only object
|
|
118
170
|
# identity changes.
|
|
119
|
-
def self.with_match_timeout(source)
|
|
171
|
+
def self.with_match_timeout(key, source)
|
|
120
172
|
return source if source.is_a?(Regexp) && source.timeout
|
|
121
173
|
|
|
122
174
|
if source.is_a?(Regexp)
|
|
@@ -124,12 +176,19 @@ module Labkit
|
|
|
124
176
|
else
|
|
125
177
|
Regexp.new(source, timeout: MATCH_TIMEOUT_SECONDS)
|
|
126
178
|
end
|
|
179
|
+
rescue RegexpError, TypeError => e
|
|
180
|
+
raise ArgumentError,
|
|
181
|
+
"rate-limit match value {#{key}: #{truncate_for_error(source)}} failed to compile: #{e.message}"
|
|
127
182
|
end
|
|
128
183
|
|
|
129
184
|
private_class_method :with_match_timeout
|
|
130
185
|
|
|
131
186
|
def self.invalid_shape_error(input)
|
|
132
|
-
|
|
187
|
+
# No key list here: this fires on the shape, and from_hash names the
|
|
188
|
+
# accepted keys when the shape is right but the key is not. Keeping the
|
|
189
|
+
# fixed part short is also what lets truncate_for_error bound the whole
|
|
190
|
+
# message.
|
|
191
|
+
"rate-limit match value must be a single-key Hash, or an Array of matchers, " \
|
|
133
192
|
"got #{truncate_for_error(input)}"
|
|
134
193
|
end
|
|
135
194
|
private_class_method :invalid_shape_error
|
|
@@ -140,14 +199,65 @@ module Labkit
|
|
|
140
199
|
end
|
|
141
200
|
private_class_method :truncate_for_error
|
|
142
201
|
|
|
202
|
+
def self.nesting_error(input)
|
|
203
|
+
# Does not inspect the input: it is too deep to render, which is the
|
|
204
|
+
# reason this fires. The class is the only hint available cheaply, and
|
|
205
|
+
# the nesting may be in a key or a value.
|
|
206
|
+
"rate-limit match value nests deeper than #{MAX_NESTING_DEPTH} levels (#{input.class})"
|
|
207
|
+
end
|
|
208
|
+
private_class_method :nesting_error
|
|
209
|
+
|
|
210
|
+
# A valid match value nests three deep at most: an Array of matcher
|
|
211
|
+
# Hashes whose :oneOf value is an Array of plain values. Anything past
|
|
212
|
+
# MAX_NESTING_DEPTH is rejected rather than built.
|
|
213
|
+
#
|
|
214
|
+
# The check is iterative because the alternative is finding out through
|
|
215
|
+
# #inspect, which recurses once per level while rendering an error
|
|
216
|
+
# message. Rules are built at boot and SystemStackError is not a
|
|
217
|
+
# StandardError, so that would take the process down rather than reject
|
|
218
|
+
# one rule, and rescuing it is not an option either: the VM aborts on a
|
|
219
|
+
# second overflow in the same process.
|
|
220
|
+
def self.too_deeply_nested?(input)
|
|
221
|
+
# A flat stack of node, depth pairs, so a visit allocates no Array. The
|
|
222
|
+
# pops must mirror the pushes: depth comes off first, then its node.
|
|
223
|
+
stack = [input, 0]
|
|
224
|
+
|
|
225
|
+
until stack.empty?
|
|
226
|
+
depth = stack.pop
|
|
227
|
+
node = stack.pop
|
|
228
|
+
next unless node.is_a?(Hash) || node.is_a?(Array)
|
|
229
|
+
return true if depth >= MAX_NESTING_DEPTH
|
|
230
|
+
|
|
231
|
+
# Keys as well as values: #inspect renders both, so a deeply nested
|
|
232
|
+
# key overflows the same way a deeply nested value does.
|
|
233
|
+
if node.is_a?(Hash)
|
|
234
|
+
node.each { |key, value| stack.push(key, depth + 1, value, depth + 1) }
|
|
235
|
+
else
|
|
236
|
+
node.each { |child| stack.push(child, depth + 1) }
|
|
237
|
+
end
|
|
238
|
+
end
|
|
239
|
+
|
|
240
|
+
false
|
|
241
|
+
end
|
|
242
|
+
private_class_method :too_deeply_nested?
|
|
243
|
+
|
|
143
244
|
def match?(identifier_value)
|
|
144
245
|
case type
|
|
145
246
|
when :eq
|
|
146
247
|
value == identifier_value
|
|
248
|
+
when :ne
|
|
249
|
+
value != identifier_value
|
|
147
250
|
when :oneOf
|
|
148
251
|
value.include?(identifier_value)
|
|
252
|
+
when :noneOf
|
|
253
|
+
# exclude? is ActiveSupport, and nothing in rate_limit requires it.
|
|
254
|
+
!value.include?(identifier_value) # rubocop:disable Rails/NegateInclude
|
|
149
255
|
when :re
|
|
150
256
|
value.match?(identifier_value.to_s)
|
|
257
|
+
when :nre
|
|
258
|
+
!value.match?(identifier_value.to_s)
|
|
259
|
+
when :all
|
|
260
|
+
value.all? { |matcher| matcher.match?(identifier_value) }
|
|
151
261
|
else
|
|
152
262
|
raise ArgumentError, "unknown matcher type: #{type.inspect}"
|
|
153
263
|
end
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "time" # Time#httpdate
|
|
4
|
+
|
|
3
5
|
module Labkit
|
|
4
6
|
module RateLimit
|
|
5
7
|
# Result is the return value of Limiter#check.
|
|
@@ -113,18 +115,13 @@ module Labkit
|
|
|
113
115
|
@most_constraining ||= @evaluations.min
|
|
114
116
|
end
|
|
115
117
|
|
|
116
|
-
# Returns
|
|
117
|
-
#
|
|
118
|
-
#
|
|
119
|
-
# the RateLimit header spec requires integer values.
|
|
118
|
+
# Returns the rate limit response headers of the most constraining
|
|
119
|
+
# evaluation, or {} when no rule matched, a :skip rule matched, or an
|
|
120
|
+
# error occurred. See Result::Evaluation#to_response_headers for the keys.
|
|
120
121
|
def to_response_headers
|
|
121
122
|
return {} unless matched? && !error? && info
|
|
122
123
|
|
|
123
|
-
|
|
124
|
-
"RateLimit-Limit" => info.resolved_limit.to_i.to_s,
|
|
125
|
-
"RateLimit-Remaining" => info.remaining.to_i.to_s,
|
|
126
|
-
"RateLimit-Reset" => info.reset_at.to_i.to_s
|
|
127
|
-
}
|
|
124
|
+
most_constraining.to_response_headers
|
|
128
125
|
end
|
|
129
126
|
end
|
|
130
127
|
|
|
@@ -155,6 +152,32 @@ module Labkit
|
|
|
155
152
|
constraint_rank <=> other.constraint_rank
|
|
156
153
|
end
|
|
157
154
|
|
|
155
|
+
# Returns the rate limit response headers for this evaluation. Values are
|
|
156
|
+
# Strings, and counts are coerced to Integer because the RateLimit header
|
|
157
|
+
# spec requires integer values even when count is fractional.
|
|
158
|
+
#
|
|
159
|
+
# RateLimit-Name (the rule name), RateLimit-Limit, RateLimit-Observed,
|
|
160
|
+
# RateLimit-Remaining and RateLimit-Reset (Unix timestamp) are always
|
|
161
|
+
# present. Limit, Observed and Remaining all count within the rule's own
|
|
162
|
+
# period. A blocking evaluation also gets Retry-After (seconds) and
|
|
163
|
+
# RateLimit-ResetTime (HTTP date), which belong on the 429.
|
|
164
|
+
def to_response_headers
|
|
165
|
+
headers = {}
|
|
166
|
+
headers["RateLimit-Name"] = rule.name
|
|
167
|
+
# Floored at 0 like remaining: a callable limit can resolve below zero
|
|
168
|
+
# to deny all, and an older client could credit a counter below zero.
|
|
169
|
+
headers["RateLimit-Limit"] = [info.resolved_limit.to_i, 0].max.to_s
|
|
170
|
+
headers["RateLimit-Observed"] = [info.count.to_i, 0].max.to_s
|
|
171
|
+
headers["RateLimit-Remaining"] = info.remaining.to_i.to_s
|
|
172
|
+
headers["RateLimit-Reset"] = info.reset_at.to_i.to_s
|
|
173
|
+
return headers unless block?
|
|
174
|
+
|
|
175
|
+
# Rounded up so a client that honours it does not retry before the reset.
|
|
176
|
+
headers["Retry-After"] = [(info.reset_at - Time.now.utc).ceil, 0].max.to_s
|
|
177
|
+
headers["RateLimit-ResetTime"] = info.reset_at.httpdate
|
|
178
|
+
headers
|
|
179
|
+
end
|
|
180
|
+
|
|
158
181
|
protected
|
|
159
182
|
|
|
160
183
|
def constraint_rank
|
|
@@ -10,8 +10,8 @@ module Labkit
|
|
|
10
10
|
# name - stable identifier used in Redis keys and log entries
|
|
11
11
|
# match - hash of identifier key/value pairs that must all match for
|
|
12
12
|
# the rule to apply; empty hash matches any identifier
|
|
13
|
-
# limit - request threshold; may be a callable (resolved per check)
|
|
14
|
-
# period - window in seconds; may be a callable (resolved per check)
|
|
13
|
+
# limit - request threshold, at least 1; may be a callable (resolved per check)
|
|
14
|
+
# period - window in seconds, at least 1; may be a callable (resolved per check)
|
|
15
15
|
# action - :limit (enforce; the result blocks when the rule is over
|
|
16
16
|
# its limit, which also terminates evaluation), :log (count
|
|
17
17
|
# and log only, never blocks and never terminates),
|
|
@@ -56,6 +56,16 @@ module Labkit
|
|
|
56
56
|
raise ArgumentError, "ban_for must be a Numeric of at least 1 second or a callable, got #{value.inspect}"
|
|
57
57
|
end
|
|
58
58
|
|
|
59
|
+
# A static limit under 1 blocks every request, and a period under a second
|
|
60
|
+
# reaches EXPIRE as 0, deleting the counter on every write. A :skip rule
|
|
61
|
+
# counts nothing, and a callable may still resolve to 0 to mean "deny all".
|
|
62
|
+
def self.normalize_numeric(field, value, action_sym)
|
|
63
|
+
return value if action_sym == :skip || value.respond_to?(:call)
|
|
64
|
+
return value if value.is_a?(Numeric) && value >= 1
|
|
65
|
+
|
|
66
|
+
raise ArgumentError, "#{field} must be a Numeric of at least 1 or a callable, got #{value.inspect}"
|
|
67
|
+
end
|
|
68
|
+
|
|
59
69
|
def self.normalize_count_distinct(value, characteristics_arr)
|
|
60
70
|
sym =
|
|
61
71
|
case value
|
|
@@ -94,8 +104,8 @@ module Labkit
|
|
|
94
104
|
super(
|
|
95
105
|
name: name_str.freeze,
|
|
96
106
|
match: match.transform_keys(&:to_sym).transform_values { |v| Matcher.build(v) }.freeze,
|
|
97
|
-
limit: limit,
|
|
98
|
-
period: period,
|
|
107
|
+
limit: self.class.normalize_numeric(:limit, limit, action_sym),
|
|
108
|
+
period: self.class.normalize_numeric(:period, period, action_sym),
|
|
99
109
|
action: action_sym,
|
|
100
110
|
characteristics: characteristics_arr,
|
|
101
111
|
count_distinct: count_distinct_sym,
|
|
@@ -173,7 +173,7 @@ RSpec::Matchers.define :complete_user_experience do |user_experience_id, error:
|
|
|
173
173
|
"Failed to complete user experience '#{user_experience_id}':\n" \
|
|
174
174
|
"expected checkpoint='end' counter to increase by 1, but increased by #{@checkpoint_change}\n" \
|
|
175
175
|
"expected total='error: #{error}' counter to increase by 1, but increased by #{@total_change}\n" \
|
|
176
|
-
"expected apdex='success: #{success}' counter to increase by 1, but increased by #{@apdex_change}"
|
|
176
|
+
"expected apdex='success: #{success}' counter to increase by #{error ? 0 : 1}, but increased by #{@apdex_change}"
|
|
177
177
|
end
|
|
178
178
|
|
|
179
179
|
match_when_negated do |actual|
|
|
@@ -192,7 +192,7 @@ RSpec::Matchers.define :complete_user_experience do |user_experience_id, error:
|
|
|
192
192
|
@total_change = total_after - total_before
|
|
193
193
|
@apdex_change = apdex_after - apdex_before
|
|
194
194
|
|
|
195
|
-
@checkpoint_change.zero? && @total_change.zero? && @apdex_change
|
|
195
|
+
@checkpoint_change.zero? && @total_change.zero? && @apdex_change.zero?
|
|
196
196
|
end
|
|
197
197
|
|
|
198
198
|
failure_message_when_negated do
|
data/performance.yml
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Performance contract for labkit-ruby
|
|
2
|
+
# https://handbook.gitlab.com/handbook/engineering/testing/performance-contracts/
|
|
3
|
+
#
|
|
4
|
+
# This file is the single entry point for performance contract tooling.
|
|
5
|
+
# Customize the sections relevant to your service and remove the rest.
|
|
6
|
+
# Schema reference: https://handbook.gitlab.com/handbook/engineering/testing/performance-contracts/#schema-definition
|
|
7
|
+
|
|
8
|
+
version: "1.0"
|
|
9
|
+
service:
|
|
10
|
+
name: "labkit-ruby"
|
|
11
|
+
description: "Performance contract for labkit-ruby"
|
|
12
|
+
|
|
13
|
+
# ---------------------------------------------------------------------------
|
|
14
|
+
# Frontend / UI contracts (SiteSpeed)
|
|
15
|
+
# Uncomment and configure if this service has a frontend component.
|
|
16
|
+
# Budget files live in the sitespeed-measurement-setup submodule.
|
|
17
|
+
# ---------------------------------------------------------------------------
|
|
18
|
+
# frontend:
|
|
19
|
+
# enabled: true
|
|
20
|
+
# budgets:
|
|
21
|
+
# production: testrunner/sitespeed-measurement-setup/performance/budgets/environments/production.json
|
|
22
|
+
# staging: testrunner/sitespeed-measurement-setup/performance/budgets/environments/staging.json
|
|
23
|
+
# mr: testrunner/sitespeed-measurement-setup/performance/budgets/environments/mr.json
|
|
24
|
+
# teams:
|
|
25
|
+
# labkit-ruby:
|
|
26
|
+
# url_dir: testrunner/sitespeed-measurement-setup/gitlab/desktop/urls
|
|
27
|
+
# budget: testrunner/sitespeed-measurement-setup/performance/budgets/teams/labkit-ruby.json
|
|
28
|
+
# default_budget: mr
|
|
29
|
+
|
|
30
|
+
# ---------------------------------------------------------------------------
|
|
31
|
+
# Backend / Service contracts (k6 / CPT)
|
|
32
|
+
# Define endpoint categories with latency and error-rate targets.
|
|
33
|
+
# Choose a tier as a starting point, then tune from baseline data.
|
|
34
|
+
#
|
|
35
|
+
# Tier 1 (fast reads): p95=100ms, p99=250ms, error_rate=0.001
|
|
36
|
+
# Tier 2 (std reads): p95=500ms, p99=1000ms, error_rate=0.005
|
|
37
|
+
# Tier 3 (writes): p95=1500ms, p99=3000ms, error_rate=0.01
|
|
38
|
+
# Tier 4 (git ops): p95=5000ms, p99=10000ms,error_rate=0.001
|
|
39
|
+
# ---------------------------------------------------------------------------
|
|
40
|
+
# endpoints:
|
|
41
|
+
# fast_reads:
|
|
42
|
+
# description: >
|
|
43
|
+
# Single item lookup by ID. Simulates one indexed DB read.
|
|
44
|
+
# routes:
|
|
45
|
+
# - "GET /api/v1/items/{id}"
|
|
46
|
+
# metrics:
|
|
47
|
+
# latency_p95_ms: 100
|
|
48
|
+
# latency_p99_ms: 250
|
|
49
|
+
# error_rate_threshold: 0.001
|
|
50
|
+
|
|
51
|
+
# ---------------------------------------------------------------------------
|
|
52
|
+
# Resource budgets (informational; enforcement planned for a future iteration)
|
|
53
|
+
# ---------------------------------------------------------------------------
|
|
54
|
+
# resources:
|
|
55
|
+
# memory_limit_mb: 256
|
|
56
|
+
# cpu_limit_cores: 0.5
|
|
57
|
+
# connection_pool_max: 10
|
|
58
|
+
|
|
59
|
+
# ---------------------------------------------------------------------------
|
|
60
|
+
# SLI mapping -- links contract endpoints to Prometheus metrics emitted by
|
|
61
|
+
# LabKit v2. Fill in once your service is instrumented.
|
|
62
|
+
# ---------------------------------------------------------------------------
|
|
63
|
+
# sli_mapping:
|
|
64
|
+
# metrics_namespace: gitlab
|
|
65
|
+
# component: labkit-ruby
|
|
66
|
+
# fast_reads:
|
|
67
|
+
# requests_total_metric: gitlab_http_requests_total
|
|
68
|
+
# duration_metric: gitlab_http_request_duration_seconds
|
|
69
|
+
# endpoint_id_label: "GET /api/v1/items/{id}"
|
|
70
|
+
# feature_category_label: labkit_ruby
|
data/scripts/prepare-dev-env.sh
CHANGED
|
@@ -21,6 +21,28 @@ warn() {
|
|
|
21
21
|
echo >&2 -e "Recommended reading: https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/docs/developer-setup.md"
|
|
22
22
|
}
|
|
23
23
|
|
|
24
|
+
# `mise plugins update` contacts GitHub for every registered plugin: slow, and
|
|
25
|
+
# it adds up when many vendored copies of this script run across many projects
|
|
26
|
+
# on the same machine. Throttle it to once a day using a stamp file in the
|
|
27
|
+
# user's cache directory, deliberately outside the project so that the throttle
|
|
28
|
+
# is shared by every project that vendors this script.
|
|
29
|
+
mise_plugins_update() {
|
|
30
|
+
local stamp_dir="${XDG_CACHE_HOME:-${HOME:-}/.cache}/common-template-copier"
|
|
31
|
+
local stamp_file="${stamp_dir}/mise-plugins-update"
|
|
32
|
+
|
|
33
|
+
# `find -mtime -1` rather than `stat`, which is spelled differently on
|
|
34
|
+
# GNU/BSD. It prints nothing when the stamp is missing or over a day old.
|
|
35
|
+
if [[ -z ${FORCE_MISE_PLUGINS_UPDATE-} ]] && [[ -n $(find "$stamp_file" -mtime -1 2>/dev/null) ]]; then
|
|
36
|
+
echo >&2 "skipping 'mise plugins update' (ran in the last 24h; set FORCE_MISE_PLUGINS_UPDATE=1 to force)"
|
|
37
|
+
return 0
|
|
38
|
+
fi
|
|
39
|
+
|
|
40
|
+
mise plugins update -q
|
|
41
|
+
|
|
42
|
+
# Best effort: a machine without a writable cache dir just updates every time.
|
|
43
|
+
mkdir -p "$stamp_dir" && touch "$stamp_file" || true
|
|
44
|
+
}
|
|
45
|
+
|
|
24
46
|
if command -v mise >/dev/null; then
|
|
25
47
|
echo >&2 -e "mise installed..."
|
|
26
48
|
elif command -v rtx >/dev/null; then
|
|
@@ -53,8 +75,8 @@ fi
|
|
|
53
75
|
|
|
54
76
|
# install mise/asdf dependencies
|
|
55
77
|
echo "installing required plugins with mise install.."
|
|
56
|
-
mise plugins update -q
|
|
57
78
|
mise trust --yes
|
|
79
|
+
mise_plugins_update
|
|
58
80
|
mise install
|
|
59
81
|
|
|
60
82
|
# set PROMPT_COMMAND to empty value for mise if unset
|