gitlab-labkit 5.1.2 → 5.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d272acd766a807cc483a70b13238a55b91dbffd0de51435c3e754df64562176d
4
- data.tar.gz: 2fbc2ff23bbedca43ef1e5c650b3b6c78a0aa4c7ecfec85fec8eae79ef2d2752
3
+ metadata.gz: fb00f476a5fed0be71839727d4fb5d8230f773831c8c8cca40ff8090c76cb2c5
4
+ data.tar.gz: c382bf0eed9e55b283eb90c3956a00c8dbf73ad5819fb22e8dd6a1be931fd9ff
5
5
  SHA512:
6
- metadata.gz: b469df64589bfe387e3d725d1cc6c6a3e769bc078ccb043b8afdf5f3591bae10a5730ecaa87069b1dc49041f50e6c6e92e145989a2a644e2d2f33fe56abd614f
7
- data.tar.gz: 212bf5a04d9a38b4cd64f6b7319179dba2379f3b89b6b1c6070bf64f9f256ea899f6a02ddc35b271dcb6bfb300100627decd388b7b5e3ae5c45f289d2fdd4bfe
6
+ metadata.gz: 5932210330aa7c8c4413de43f8ae710508658846c89f1b7b74e1d8d36af564b61034988a6248bf9795d6343602d4b1579d6a9aff48795cda1163c545377932b9
7
+ data.tar.gz: ddc501d350806e384f122f152f6ceb7066487bc4f41078341f1ae13373631adf52a1f5cd427a69e879482cf3dc1dff75bbfe7cd57dd1237094c595e615acc591
data/.gitlab/CODEOWNERS CHANGED
@@ -1 +1 @@
1
- * @reprazent @andrewn @mkaeppler @ayufan @hmerscher @sankalp_gl @hardikgala @nindurkar @ashs2
1
+ * @reprazent @mkaeppler @ayufan @hmerscher @sankalp_gl @hardikgala @nindurkar @ashs2
data/.gitlab-ci.yml CHANGED
@@ -79,6 +79,13 @@ conformance-spec:
79
79
  variables:
80
80
  LABKIT_RUBY_SHA: $CI_COMMIT_SHA
81
81
  rules:
82
+ # Renovate's project-token bot cannot create pipelines in labkit-spec
83
+ # (GitLab forbids project bots from joining other projects), so its MRs
84
+ # would hard-fail the bridge. Attempt the trigger but do not block; a
85
+ # human re-running the pipeline gates the MR properly (it runs as them).
86
+ - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $GITLAB_USER_LOGIN =~ /^project_\d+_bot/'
87
+ changes: !reference [.conformance-trigger-paths, changes]
88
+ allow_failure: true
82
89
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
83
90
  changes: !reference [.conformance-trigger-paths, changes]
84
91
  trigger:
data/AGENTS.md ADDED
@@ -0,0 +1,136 @@
1
+ # AGENTS.md
2
+
3
+ Instructions for AI agents and automated tooling working in this repository.
4
+
5
+ ## Project context
6
+
7
+ LabKit-Ruby (`gitlab-labkit` on RubyGems) is GitLab's Ruby SDK for cross-project infrastructure concerns: request context and correlation, structured logging, tracing, metrics, SLIs, and rate limiting.
8
+ It is consumed by GitLab Rails and other Ruby services, so every public API is a compatibility commitment.
9
+
10
+ It is one of several LabKit SDKs.
11
+ The others are Go ([labkit](https://gitlab.com/gitlab-org/labkit)), Rust ([labkit-rs](https://gitlab.com/gitlab-org/rust/labkit-rs)), and Python ([labkit-python](https://gitlab.com/gitlab-org/labkit-python)).
12
+ Behaviour that services observe, and anything written to a shared store, must be the same in every SDK.
13
+ The contract lives in [labkit-spec](https://gitlab.com/gitlab-org/quality/tooling/labkit-spec), and its conformance harness is the judge.
14
+
15
+ ### Layout
16
+
17
+ One gem, one `Labkit` namespace, one directory per capability under `lib/labkit/`.
18
+ `lib/gitlab-labkit.rb` autoloads each capability unless the table says otherwise, so requiring the gem stays cheap and callers load only what they touch.
19
+
20
+ | Module | Directory | Notes |
21
+ | --- | --- | --- |
22
+ | `Labkit::Context`, `Labkit::Correlation` | `lib/labkit/context.rb`, `lib/labkit/correlation/` | Request context for logs and propagation |
23
+ | `Labkit::Logging` | `lib/labkit/logging/` | Log sanitisation and JSON formatting |
24
+ | `Labkit::Tracing` | `lib/labkit/tracing/` | OpenTelemetry and legacy OpenTracing |
25
+ | `Labkit::Metrics` | `lib/labkit/metrics/` | Prometheus client wrappers |
26
+ | `Labkit::ApplicationSli`, `Labkit::UserExperienceSli` | `lib/labkit/application_sli/`, `lib/labkit/user_experience_sli/` | Apdex and error-rate SLIs |
27
+ | `Labkit::RateLimit` | `lib/labkit/rate_limit/` | Rules-based, Redis-backed rate limiting, see below |
28
+ | `Labkit::Middleware` | `lib/labkit/middleware/` | Rack and Sidekiq middleware for context, tracing, and SLIs |
29
+ | `Labkit::JsonSchema` | `lib/labkit/json_schema/` | `RefResolver` for remote `$ref` in JSON schemas. Not autoloaded; required by `UserExperienceSli` |
30
+ | `Labkit::FIPS` | `lib/labkit/fips.rb` | FIPS mode detection and OpenSSL digest substitution |
31
+ | `Labkit::System` | `lib/labkit/system.rb` | Monotonic clock helper used by metrics, tracing, and logging |
32
+ | `Labkit::Redis` | `lib/labkit/redis/` | Redis helpers shared by capabilities |
33
+ | `Labkit::Fields` | `lib/labkit/fields.rb` | Generated by labkit-spec CI from `schema/fields.yaml`. Change the field there, not here |
34
+ | `Labkit::RSpec` | `lib/labkit/rspec/` | Matchers for consumers' test suites. Not autoloaded; `require "labkit/rspec/matchers"` |
35
+
36
+ `lib/labkit/*_publisher.rb` hook Excon, HTTPClient, and Net::HTTP to publish outbound request notifications.
37
+
38
+ Modules with their own README (`lib/labkit/<module>/README.md`) document their public API there.
39
+ Update that README in the same MR as a behaviour change.
40
+
41
+ ## Commands
42
+
43
+ ```shell
44
+ bundle install
45
+ bundle exec rake verify # rspec + rubocop, what CI runs
46
+ bundle exec rake fix # rubocop autocorrect
47
+ bundle exec rspec spec/labkit/<module> # one module's specs
48
+ bundle exec rake console # IRB with the gem loaded
49
+ ```
50
+
51
+ Specs that need Redis use the instance at `LABKIT_TEST_REDIS_URL`, default `redis://localhost:6390/0`.
52
+ If nothing answers there and `LABKIT_TEST_REDIS_URL` is unset, the suite starts one with `docker compose up -d redis` and stops it at exit.
53
+ A Redis already listening on that URL is used as is and left running.
54
+ Autostart is off when `LABKIT_TEST_REDIS_URL` is set, even to the default value, when `CI` is set, or when `LABKIT_TEST_REDIS_NO_AUTOSTART=1` is set.
55
+ In those cases, start Redis yourself.
56
+
57
+ Style is `gitlab-styles` via RuboCop with `NewCops: enable`.
58
+ Do not edit `.rubocop_todo.yml` to silence a new offence; fix the code.
59
+ Tool versions come from `.tool-versions` through mise.
60
+
61
+ ## CI
62
+
63
+ - `rspec` runs the suite with a Redis service.
64
+ - `ruby-versions` and `legacy-ruby-versions` repeat it on Ruby 3.4 and 3.3. The versions are set in `.gitlab-ci.yml`, not read from the gemspec.
65
+ - `conformance-spec` triggers a downstream pipeline in labkit-spec that checks out this MR's commit and runs the conformance harness against it.
66
+ It runs when `lib/`, the gemspec, the Gemfile, or `.gitlab-ci.yml` change, and a red downstream run blocks the merge.
67
+ Renovate's bot user cannot start pipelines in labkit-spec, so on its MRs the job has `allow_failure: true` and a human rerun is the gate.
68
+ - Danger runs the shared `gitlab-dangerfiles` checks except changelog and commit message, which semantic-release owns.
69
+
70
+ ## Commits, MRs, and releases
71
+
72
+ Conventional Commits: `feat`, `fix`, `perf`, `revert`, `docs`, `refactor`, `test`, `chore`, `ci`, `style`.
73
+ Scope is the module, for example `fix(rate_limit): ...`.
74
+
75
+ Releases are cut from `master` and the `N-N-stable` branches by semantic-release with the `conventionalcommits` preset, so the commit type decides the version bump.
76
+
77
+ | Type | Release |
78
+ | --- | --- |
79
+ | `feat!`, or any type with a `BREAKING CHANGE:` footer | major |
80
+ | `feat` | minor |
81
+ | `fix`, `perf`, `revert` | patch |
82
+ | `docs`, `refactor`, `test`, `chore`, `ci`, `style` | none |
83
+
84
+ There is no hand-edited changelog.
85
+
86
+ Public APIs stay backward compatible.
87
+ Add keyword arguments with defaults; do not add required parameters or rename public methods.
88
+ Keep an MR to one concern, and separate mechanical changes such as renames from behavioural ones.
89
+
90
+ ## Rate limiting
91
+
92
+ `Labkit::RateLimit` is documented in `lib/labkit/rate_limit/README.md`.
93
+ It is the one capability that shares state across SDKs: services in any language enforce the same limits against the same Redis.
94
+ That makes the Redis key shape and the Lua scripts as much a contract as the HTTP headers, so changes here follow stricter rules than the rest of the gem.
95
+
96
+ ### Where each SDK keeps it
97
+
98
+ | SDK | Repository | Module | Tests | Status |
99
+ | --- | --- | --- | --- | --- |
100
+ | Ruby (this repository) | gitlab-org/ruby/gems/labkit-ruby | `lib/labkit/rate_limit/` | `spec/labkit/rate_limit/` | released |
101
+ | Go | gitlab-org/labkit | `v2/ratelimit/` | `v2/ratelimit/*_test.go`, run with `go -C v2 test ./ratelimit/...` | released |
102
+ | Rust | gitlab-org/rust/labkit-rs | in progress | | not released, skip |
103
+ | Python | gitlab-org/labkit-python | in progress | | not released, skip |
104
+ | Spec | gitlab-org/quality/tooling/labkit-spec | `conformance/README.md` ("v1 decisions"), `conformance/ratelimit_scenarios/*.json`, `conformance/aspect_ratelimit.go` | `conformance/` | judge |
105
+
106
+ ### The three layers
107
+
108
+ Every SDK must agree on three layers.
109
+ This section defines the layers; the current values (key grammar, header names, counting rules, metric names, log fields) live in labkit-spec, in the "v1 decisions" list in `conformance/README.md` and the scenarios under `conformance/ratelimit_scenarios/`.
110
+ If this file and labkit-spec disagree, labkit-spec wins.
111
+
112
+ 1. **Storage contract.**
113
+ Anything written to or read from Redis: key shape and prefix, how values are encoded, TTL rules, and the Lua scripts.
114
+ Two SDKs sharing one Redis with different storage rules keep two counters for one limit.
115
+ Never change this layer in one SDK.
116
+ It changes in labkit-spec first, then every SDK follows.
117
+ 2. **Observable contract.**
118
+ Anything a caller or an operator can see: response status and headers, the counting rule that decides a block, log line messages and field sets, metric families and labels.
119
+ Also changes in labkit-spec first.
120
+ 3. **Behaviour within the contract.**
121
+ Input validation and edge cases the contract already implies, where one SDK does something the contract does not allow.
122
+ These may be fixed SDK by SDK, and every SDK should end up the same.
123
+
124
+ A change that mixes layers is governed by the strictest one: storage, then observable, then behaviour.
125
+ When a change does not fit one layer clearly, say so in the MR instead of picking one.
126
+
127
+ ### Rules for rate-limit changes
128
+
129
+ - A change to layer 1 or 2 opens a labkit-spec MR first and waits for it to merge.
130
+ - A change to layer 3 should be checked against the Go SDK.
131
+ If Go has the same gap, say so in the MR so it can be ported.
132
+ - When a review finds a divergence between SDKs, a conformance scenario in labkit-spec must follow in the same week, so the harness sees the gap next time.
133
+ The author of the MR where the divergence was found owns this.
134
+ Mention the sync agent on the review thread to have it draft the scenario MR; if it cannot, open a labkit-spec issue and link it from the thread.
135
+ - Never edit Lua scripts, key construction, header names, metric names, or log field names to fix a Ruby-only bug.
136
+ If a fix seems to need that, it is a contract change.
data/CODEOWNERS CHANGED
@@ -1,4 +1,4 @@
1
1
  # CODEOWNERS is used to lookup assignees for
2
2
  # Renovate Bot dependency change Merge Requests.
3
3
  # https://docs.renovatebot.com/configuration-options/#assigneesfromcodeowners
4
- * @reprazent @andrewn @ayufan @hmerscher @e_forbes @M_Alvarez @mwoolf @sankalp_gl @hardikgala @nindurkar @ashs2
4
+ * @reprazent @ayufan @hmerscher @e_forbes @M_Alvarez @mwoolf @sankalp_gl @hardikgala @nindurkar @ashs2
@@ -36,6 +36,7 @@ Gem::Specification.new do |spec|
36
36
  spec.add_runtime_dependency "pg_query", ">= 6.1.0", "< 7.0"
37
37
  spec.add_runtime_dependency "prometheus-client-mmap", ">= 1.2", "< 2.0"
38
38
  spec.add_runtime_dependency "redis", "> 3.0.0", "< 6.0.0"
39
+ spec.add_runtime_dependency "thrift", "< 0.25" # Pinned for jaeger-client 1.1.0 compatibility; thrift 0.25 requires transports to implement message_boundaries?
39
40
 
40
41
  # Please maintain alphabetical order for dev dependencies
41
42
  spec.add_development_dependency "excon", "~> 0.78.1"
data/lib/labkit/fields.rb CHANGED
@@ -138,6 +138,17 @@ module Labkit
138
138
  # (e.g. newlines) to prevent log injection.
139
139
  ADDITIONAL_DETAILS = "additional_details"
140
140
 
141
+ # Unrounded complexity score of a GraphQL query. Some fields have a
142
+ # fractional cost (e.g. graphql-ruby's __typename costs 0.2), so the total
143
+ # can be a non-integer value.
144
+ GRAPHQL_COMPLEXITY_RAW = "graphql_complexity_raw"
145
+
146
+ # Number of nodes a GraphQL query returned across all its connection fields.
147
+ # The count is taken after authorization removes the nodes the user can't
148
+ # see. Nested connections add up. The value is 0 when the query selects no
149
+ # connection field.
150
+ GRAPHQL_CONNECTION_NODES = "graphql_connection_nodes"
151
+
141
152
  # Maps each field constant to its logging field variant version.
142
153
  # A version of 0 means the field is not yet assigned to any variant.
143
154
  VARIANT_VERSION = {
@@ -179,6 +190,8 @@ module Labkit
179
190
  ROOT_CALLER_ID => 0,
180
191
  GL_SENT_NOTIFICATION_ID => 0,
181
192
  ADDITIONAL_DETAILS => 0,
193
+ GRAPHQL_COMPLEXITY_RAW => 0,
194
+ GRAPHQL_CONNECTION_NODES => 0,
182
195
  }.freeze
183
196
 
184
197
  # Get the constant name for a field value
@@ -207,8 +220,8 @@ module Labkit
207
220
  Fields::DURATION_S => %w[duration duration_ms elapsed_time actual_duration time_ms total_time gitaly.duration],
208
221
  Fields::REMOTE_IP => %w[ip source_ip ip_address],
209
222
  Fields::HTTP_HOST => %w[hostname request_host gitlab_host kubernetes.host],
210
- Fields::GL_PROJECT_ID => %w[extra.project_id meta.project_id meta.search.project_id job_project_id target_project_id],
211
- Fields::GL_PIPELINE_ID => %w[extra.pipeline_id meta.pipeline_id root_pipeline_id],
223
+ Fields::GL_PROJECT_ID => %w[project_id extra.project_id meta.project_id meta.search.project_id job_project_id target_project_id],
224
+ Fields::GL_PIPELINE_ID => %w[pipeline_id extra.pipeline_id meta.pipeline_id root_pipeline_id],
212
225
  Fields::TIMESTAMP => %w[start_time],
213
226
  Fields::SEVERITY => %w[level],
214
227
  Fields::LOG_MESSAGE => %w[msg custom_message extra.message fields.message graphql.message reason color_message exception.gitaly],
@@ -171,8 +171,8 @@ A `Rule` is a `Data.define` value object with the following fields:
171
171
  |-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
172
172
  | `name` | Stable identifier used in Redis keys and metric labels. Must match `/\A[a-z0-9_]+\z/`, max 64 chars. Renaming a rule abandons its in-flight counters. |
173
173
  | `match` | Hash of identifier key/value predicates that must **all** be satisfied for the rule to apply. Empty hash matches anything. See [Matchers](#matchers). |
174
- | `limit` | Integer request threshold per `period`. May be a callable resolved on every check. |
175
- | `period` | Window length in seconds. May be a callable resolved on every check. |
174
+ | `limit` | Integer request threshold per `period`, at least 1. May be a callable resolved on every check. |
175
+ | `period` | Window length in seconds, at least 1; a fraction truncates to whole seconds. May be a callable resolved on every check. |
176
176
  | `action` | What the rule does when it matches. One of `:limit`, `:log`, `:skip`. Default `:limit`. See [Actions](#actions). |
177
177
  | `characteristics` | Array of identifier keys whose values are folded into the Redis counter key. Each unique combination gets its own counter. |
178
178
  | `count_distinct` | Optional identifier key. When set, the rule counts distinct values seen for that key within the window, backed by a Redis SET rather than a counter. Must not overlap `characteristics`. |
@@ -190,6 +190,13 @@ Labkit::RateLimit::Rule.new(
190
190
  )
191
191
  ```
192
192
 
193
+ `Rule.new` raises `ArgumentError` for a static `limit` under 1 or a static
194
+ `period` under one second, except on `:skip` rules where both are inert.
195
+ Callables are not validated: a `limit` resolving to `0` blocks every matching
196
+ request, which is how to express "deny all" dynamically, and a `period`
197
+ resolving under one second reaches `EXPIRE` as `0` and deletes the counter on
198
+ every write.
199
+
193
200
  ### Matchers
194
201
 
195
202
  A `match` hash gates whether a rule applies. Each value is normalised through
@@ -200,17 +207,63 @@ A `match` hash gates whether a rule applies. Each value is normalised through
200
207
  | plain value | `eq` | `match: { user: nil }`, `match: { method: "POST" }` |
201
208
  | `Regexp` | `re` | `match: { endpoint: %r{\A/api/} }` |
202
209
  | `{ eq: <value> }` | `eq` | `match: { method: { eq: "POST" } }` (YAML-friendly) |
210
+ | `{ ne: <value> }` | `ne` | `match: { method: { ne: "OPTIONS" } }` |
203
211
  | `{ re: <source> }`| `re` | `match: { endpoint: { re: '\A/api/' } }` (YAML-friendly) |
212
+ | `{ nre: <source> }`| `nre` | `match: { endpoint: { nre: '\A/api/' } }` |
204
213
  | `{ oneOf: <Array> }` | `oneOf` | `match: { user: { oneOf: ["7", "42"] } }` (YAML-friendly) |
214
+ | `{ noneOf: <Array> }` | `noneOf` | `match: { plan: { noneOf: ["premium"] } }` |
215
+ | `Array` of matchers | `all` | `match: { endpoint: [%r{\A/api/}, { nre: 'internal' }] }` |
205
216
 
206
- `re` coerces the identifier value via `#to_s` before matching, so it can be
207
- used against non-String values (e.g. matching a 503 status against `{ re: '^5' }`).
217
+ `re` and `nre` coerce the identifier value via `#to_s` before matching, so they
218
+ can be used against non-String values (e.g. matching a 503 status against
219
+ `{ re: '^5' }`).
208
220
 
209
221
  `oneOf` is set membership via `Set#include?` (`eql?`/`hash` equality, no
210
222
  coercion): it agrees with `eq` for Strings, Symbols, booleans, nil, and
211
223
  same-class numerics, but cross-type numerics differ (`1 == 1.0`, yet
212
- `{ oneOf: [1] }` does not match `1.0`). A bare Array is rejected; membership must
213
- be spelled `{ oneOf: [...] }` explicitly.
224
+ `{ oneOf: [1] }` does not match `1.0`). `noneOf` is its complement.
225
+
226
+ An Array means **AND**, so one identifier key can carry several predicates:
227
+
228
+ ```yaml
229
+ endpoint:
230
+ - nre: '/oauth/'
231
+ - nre: '\A/api/'
232
+ - nre: '\A/-/(?:health|liveness|readiness|metrics)'
233
+ ```
234
+
235
+ Its entries must be matcher Hashes or bare `Regexp`s, the two shapes that
236
+ already mean "a matcher" on their own. An Array of plain values is rejected.
237
+ A value meant as one matcher, arriving as a list, therefore fails loudly rather
238
+ than becoming a membership test. Membership is spelled `{ oneOf: [...] }`.
239
+
240
+ An Array holds matchers, not other Arrays, so a match value nests three deep at
241
+ most: an Array, a matcher Hash inside it, and that Hash's `oneOf` list. A value
242
+ nesting deeper than `Matcher::MAX_NESTING_DEPTH` is rejected when the rule is
243
+ built.
244
+
245
+ #### Negation and absent keys
246
+
247
+ `ne`, `nre` and `noneOf` are plain complements of their positive forms, with no
248
+ special case for a missing key. The `Evaluator` looks an identifier key up
249
+ rather than requiring it, so an absent key reaches the matcher as `nil`, and a
250
+ complement is therefore **true** for one: a request with no `endpoint` is not a
251
+ request to an excluded endpoint.
252
+
253
+ That widens a rule rather than narrowing it. A rule that should apply only when
254
+ the key is present says so, with a presence matcher alongside the exclusions:
255
+
256
+ ```ruby
257
+ match: { endpoint: [/./, { nre: '\A/api/' }] }
258
+ ```
259
+
260
+ Since this follows from inversion, a matcher that does accept `nil` inverts the
261
+ other way: `{ ne: nil }`, `{ noneOf: [nil] }` and `{ nre: '^$' }` are all false
262
+ for an absent key, the last because `nil.to_s` is `""`.
263
+
264
+ There is no cap on regex source length. Rules are written by operators, and the
265
+ only bound on a pattern is `Matcher::MATCH_TIMEOUT_SECONDS`, a 50ms wall-clock
266
+ budget applied per compiled `Regexp`.
214
267
 
215
268
  Glob and prefix matchers are intentionally out of scope.
216
269
 
@@ -459,7 +512,8 @@ result.degraded? # => true if a matched count_distinct rule was skipped
459
512
  result.info # => Result::Info or nil
460
513
  result.evaluations # => every counted Result::Evaluation, in rule order
461
514
  result.to_response_headers
462
- # => { "RateLimit-Limit" => "...", "RateLimit-Remaining" => "...", "RateLimit-Reset" => "<unix-ts>" }
515
+ # => { "RateLimit-Name" => "<rule name>", "RateLimit-Limit" => "...", "RateLimit-Observed" => "...",
516
+ # "RateLimit-Remaining" => "...", "RateLimit-Reset" => "<unix-ts>" }
463
517
  ```
464
518
 
465
519
  `Result::Info` holds the per-window counter snapshot:
@@ -475,6 +529,27 @@ result.to_response_headers
475
529
  `to_response_headers` returns `{}` for an unmatched or error result, so it is
476
530
  safe to merge unconditionally.
477
531
 
532
+ ### Response headers
533
+
534
+ `Result#to_response_headers` reports the most constraining evaluation.
535
+ `Result::Evaluation#to_response_headers` builds the same headers for one evaluation, for a caller that picks its own evaluation across several results.
536
+
537
+ | header | present | value |
538
+ |-----------------------|-------------------|-----------------------------------------------------|
539
+ | `RateLimit-Name` | always | The rule's `name`. |
540
+ | `RateLimit-Limit` | always | `resolved_limit` in the rule's period, at least 0. |
541
+ | `RateLimit-Observed` | always | `count`, as an Integer, at least 0. |
542
+ | `RateLimit-Remaining` | always | `remaining`, as an Integer. |
543
+ | `RateLimit-Reset` | always | `reset_at` as a Unix timestamp. |
544
+ | `Retry-After` | blocking only | Seconds to `reset_at`, rounded up, at least 0. |
545
+ | `RateLimit-ResetTime` | blocking only | `reset_at` as an HTTP date. |
546
+
547
+ `Retry-After` and `RateLimit-ResetTime` belong on the 429, so they appear only when the evaluation blocks.
548
+ For a rule with `ban_for`, `reset_at` is the end of the ban, so these headers report when the ban lifts.
549
+
550
+ `RateLimit-Limit`, `RateLimit-Observed` and `RateLimit-Remaining` all count within the rule's `period`.
551
+ The headers do not state the period, so a client that needs it must know the rule.
552
+
478
553
  ## Fail-open
479
554
 
480
555
  The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
@@ -12,27 +12,49 @@ module Labkit
12
12
  # - any plain value (String, Symbol, Integer, ...) -> :eq matcher
13
13
  # - a Regexp instance -> :re matcher (Ruby convenience)
14
14
  # - { eq: <value> } -> :eq matcher (canonical, YAML-compatible)
15
+ # - { ne: <value> } -> :ne matcher (complement of :eq)
15
16
  # - { re: <String|Regexp> } -> :re matcher (canonical, YAML-compatible)
17
+ # - { nre: <String|Regexp> } -> :nre matcher (complement of :re)
16
18
  # - { oneOf: <Array> } -> :oneOf matcher (set membership, YAML-compatible)
19
+ # - { noneOf: <Array> } -> :noneOf matcher (complement of :oneOf)
20
+ # - [ <matcher>, <matcher>, ... ] -> :all matcher (every entry must match)
17
21
  #
18
- # A bare Array is rejected rather than treated as :oneOf, so a value that
19
- # was meant as a single matcher hash but arrived as an Array fails loudly
20
- # instead of silently becoming a membership test.
22
+ # An Array means AND, so several predicates can apply to one identifier key.
23
+ # Its entries must be matcher Hashes or bare Regexps, the two shapes that
24
+ # already mean "a matcher" on their own. An Array of plain values is
25
+ # rejected, so a value that was meant as a single matcher but arrived as a
26
+ # list fails loudly instead of silently becoming a membership test. An Array
27
+ # holds matchers rather than other Arrays, so a match value nests at most
28
+ # MAX_NESTING_DEPTH deep; see too_deeply_nested?.
21
29
  #
22
- # :oneOf membership is Set#include? (eql?/hash equality), not :eq's ==, so
23
- # cross-type numerics differ (1 == 1.0, but {oneOf: [1]} does not match 1.0).
30
+ # :oneOf and :noneOf membership is Set#include? (eql?/hash equality), not
31
+ # :eq's ==, so cross-type numerics differ (1 == 1.0, but {oneOf: [1]} does
32
+ # not match 1.0).
24
33
  #
25
34
  # Hash-key naming follows the metrics-catalog selector pattern. Glob,
26
35
  # prefix, and other matcher kinds are intentionally out of scope here; see
27
36
  # gitlab-com/gl-infra/production-engineering#28853 for that follow-up.
28
37
  #
29
- # An :re matcher coerces the identifier value via #to_s before applying
30
- # the regex, so callers can match non-String identifier values such as
31
- # Integer status codes (e.g. {status: { re: "^5" }} against status: 503).
38
+ # The :re and :nre matchers coerce the identifier value via #to_s before
39
+ # applying the regex, so callers can match non-String identifier values such
40
+ # as Integer status codes (e.g. {status: { re: "^5" }} against status: 503).
41
+ #
42
+ # Each negation matcher is the plain complement of its positive form, with
43
+ # no special case for a missing key. Evaluator passes identifier[key], so an
44
+ # absent key arrives as nil, and the complement is therefore usually true
45
+ # for one: a request with no path is not a request to an excluded path. A
46
+ # rule that should apply only when the key is present says so, by adding a
47
+ # presence matcher alongside the exclusions:
48
+ #
49
+ # path: [/./, { nre: '\A/api/' }]
32
50
  class Matcher < Data.define(:type, :value)
33
- KNOWN_HASH_KEYS = %i[eq re oneOf].freeze
34
- MAX_REGEX_SOURCE_LENGTH = 200
51
+ KNOWN_HASH_KEYS = %i[eq ne re nre oneOf noneOf].freeze
35
52
  ERROR_INSPECT_LIMIT = 80
53
+ # No Hash or Array may sit this many levels below the match value, which
54
+ # is level 0. The deepest valid shape is an Array (0) holding a matcher
55
+ # Hash (1) whose :oneOf list (2) holds plain values; see
56
+ # too_deeply_nested?.
57
+ MAX_NESTING_DEPTH = 3
36
58
 
37
59
  # Wall-clock budget for a single #match? call. Without it a match is
38
60
  # bounded only by whatever global the host sets (40s in GitLab Rails,
@@ -45,16 +67,23 @@ module Labkit
45
67
  # timeouts were wall-clock stalls, since Ruby counts real time and GC
46
68
  # pauses on GitLab's git fleet have a p50 of ~72ms. Under 4x CPU
47
69
  # oversubscription 50ms still fires on 0.16% of those matches, 5ms 1.33%.
70
+ #
71
+ # This timeout is the only bound on a pattern. There is no cap on regex
72
+ # source length: rules are written by operators rather than end users, and
73
+ # a cap that applied to {re: "..."} but not to a bare Regexp bounded
74
+ # nothing in practice, since callers pass Regexp constants.
48
75
  MATCH_TIMEOUT_SECONDS = 0.05
49
76
 
50
77
  def self.build(input)
78
+ raise ArgumentError, nesting_error(input) if too_deeply_nested?(input)
79
+
51
80
  case input
52
81
  when Regexp
53
- new(type: :re, value: with_match_timeout(input))
82
+ new(type: :re, value: with_match_timeout(:re, input))
54
83
  when Hash
55
84
  from_hash(input)
56
85
  when Array
57
- raise ArgumentError, invalid_shape_error(input)
86
+ compose(input)
58
87
  else
59
88
  new(type: :eq, value: input)
60
89
  end
@@ -64,6 +93,15 @@ module Labkit
64
93
  raise ArgumentError, invalid_shape_error(input) if input.size != 1
65
94
 
66
95
  type, source = input.first
96
+
97
+ # Identifier rejects keys the same way. Without this, a key such as an
98
+ # Integer or nil leaves NoMethodError for the caller instead of the
99
+ # ArgumentError every other bad shape raises.
100
+ unless type.respond_to?(:to_sym)
101
+ raise ArgumentError,
102
+ "rate-limit match value type key #{truncate_for_error(type)} must be a String or Symbol"
103
+ end
104
+
67
105
  type_sym = type.to_sym
68
106
 
69
107
  unless KNOWN_HASH_KEYS.include?(type_sym)
@@ -77,31 +115,45 @@ module Labkit
77
115
 
78
116
  def self.compile(type_sym, source)
79
117
  case type_sym
80
- when :eq
81
- new(type: :eq, value: source)
82
- when :oneOf
118
+ when :eq, :ne
119
+ new(type: type_sym, value: source)
120
+ when :oneOf, :noneOf
83
121
  unless source.is_a?(Array)
84
122
  raise ArgumentError,
85
- "rate-limit match value {oneOf: ...} must be an Array, got #{truncate_for_error(source)}"
123
+ "rate-limit match value {#{type_sym}: ...} must be an Array, got #{truncate_for_error(source)}"
86
124
  end
87
125
 
88
- new(type: :oneOf, value: Set.new(source).freeze)
89
- when :re
90
- if source.to_s.length > MAX_REGEX_SOURCE_LENGTH
126
+ # {oneOf: []} matches nothing, so the rule never applies and no harm
127
+ # follows. Its complement matches everything, which is an exclusion
128
+ # that excludes nothing: rejected for the same reason an empty Array
129
+ # of matchers is.
130
+ if type_sym == :noneOf && source.empty?
91
131
  raise ArgumentError,
92
- "rate-limit match value {re: ...} source exceeds #{MAX_REGEX_SOURCE_LENGTH} characters"
132
+ "rate-limit match value {noneOf: []} must not be empty, it matches everything"
93
133
  end
94
134
 
95
- begin
96
- new(type: :re, value: with_match_timeout(source))
97
- rescue RegexpError, TypeError => e
98
- raise ArgumentError,
99
- "rate-limit match value {re: #{truncate_for_error(source)}} failed to compile: #{e.message}"
100
- end
135
+ new(type: type_sym, value: Set.new(source).freeze)
136
+ when :re, :nre
137
+ new(type: type_sym, value: with_match_timeout(type_sym, source))
101
138
  end
102
139
  end
103
140
  private_class_method :compile
104
141
 
142
+ # Builds the :all matcher behind the Array form. Entries go back through
143
+ # .build, which is what keeps a nested regex bounded by the match timeout.
144
+ def self.compose(input)
145
+ raise ArgumentError, "rate-limit match value [] must not be empty" if input.empty?
146
+
147
+ unless input.all? { |entry| entry.is_a?(Hash) || entry.is_a?(Regexp) }
148
+ raise ArgumentError,
149
+ "rate-limit match value Array entries must each be a matcher Hash or Regexp, " \
150
+ "got #{truncate_for_error(input)}"
151
+ end
152
+
153
+ new(type: :all, value: input.map { |entry| build(entry) }.freeze)
154
+ end
155
+ private_class_method :compose
156
+
105
157
  # Compiles +source+ (a String pattern or an existing Regexp) into a
106
158
  # Regexp bounded by MATCH_TIMEOUT_SECONDS.
107
159
  #
@@ -116,7 +168,7 @@ module Labkit
116
168
  # Recompiling an existing Regexp preserves its source and options
117
169
  # (including the fixed-encoding flags) and therefore +#==+; only object
118
170
  # identity changes.
119
- def self.with_match_timeout(source)
171
+ def self.with_match_timeout(key, source)
120
172
  return source if source.is_a?(Regexp) && source.timeout
121
173
 
122
174
  if source.is_a?(Regexp)
@@ -124,12 +176,19 @@ module Labkit
124
176
  else
125
177
  Regexp.new(source, timeout: MATCH_TIMEOUT_SECONDS)
126
178
  end
179
+ rescue RegexpError, TypeError => e
180
+ raise ArgumentError,
181
+ "rate-limit match value {#{key}: #{truncate_for_error(source)}} failed to compile: #{e.message}"
127
182
  end
128
183
 
129
184
  private_class_method :with_match_timeout
130
185
 
131
186
  def self.invalid_shape_error(input)
132
- "rate-limit match value must be a single-key Hash like {re: \"...\"}, {eq: ...} or {oneOf: [...]}, " \
187
+ # No key list here: this fires on the shape, and from_hash names the
188
+ # accepted keys when the shape is right but the key is not. Keeping the
189
+ # fixed part short is also what lets truncate_for_error bound the whole
190
+ # message.
191
+ "rate-limit match value must be a single-key Hash, or an Array of matchers, " \
133
192
  "got #{truncate_for_error(input)}"
134
193
  end
135
194
  private_class_method :invalid_shape_error
@@ -140,14 +199,65 @@ module Labkit
140
199
  end
141
200
  private_class_method :truncate_for_error
142
201
 
202
+ def self.nesting_error(input)
203
+ # Does not inspect the input: it is too deep to render, which is the
204
+ # reason this fires. The class is the only hint available cheaply, and
205
+ # the nesting may be in a key or a value.
206
+ "rate-limit match value nests deeper than #{MAX_NESTING_DEPTH} levels (#{input.class})"
207
+ end
208
+ private_class_method :nesting_error
209
+
210
+ # A valid match value nests three deep at most: an Array of matcher
211
+ # Hashes whose :oneOf value is an Array of plain values. Anything past
212
+ # MAX_NESTING_DEPTH is rejected rather than built.
213
+ #
214
+ # The check is iterative because the alternative is finding out through
215
+ # #inspect, which recurses once per level while rendering an error
216
+ # message. Rules are built at boot and SystemStackError is not a
217
+ # StandardError, so that would take the process down rather than reject
218
+ # one rule, and rescuing it is not an option either: the VM aborts on a
219
+ # second overflow in the same process.
220
+ def self.too_deeply_nested?(input)
221
+ # A flat stack of node, depth pairs, so a visit allocates no Array. The
222
+ # pops must mirror the pushes: depth comes off first, then its node.
223
+ stack = [input, 0]
224
+
225
+ until stack.empty?
226
+ depth = stack.pop
227
+ node = stack.pop
228
+ next unless node.is_a?(Hash) || node.is_a?(Array)
229
+ return true if depth >= MAX_NESTING_DEPTH
230
+
231
+ # Keys as well as values: #inspect renders both, so a deeply nested
232
+ # key overflows the same way a deeply nested value does.
233
+ if node.is_a?(Hash)
234
+ node.each { |key, value| stack.push(key, depth + 1, value, depth + 1) }
235
+ else
236
+ node.each { |child| stack.push(child, depth + 1) }
237
+ end
238
+ end
239
+
240
+ false
241
+ end
242
+ private_class_method :too_deeply_nested?
243
+
143
244
  def match?(identifier_value)
144
245
  case type
145
246
  when :eq
146
247
  value == identifier_value
248
+ when :ne
249
+ value != identifier_value
147
250
  when :oneOf
148
251
  value.include?(identifier_value)
252
+ when :noneOf
253
+ # exclude? is ActiveSupport, and nothing in rate_limit requires it.
254
+ !value.include?(identifier_value) # rubocop:disable Rails/NegateInclude
149
255
  when :re
150
256
  value.match?(identifier_value.to_s)
257
+ when :nre
258
+ !value.match?(identifier_value.to_s)
259
+ when :all
260
+ value.all? { |matcher| matcher.match?(identifier_value) }
151
261
  else
152
262
  raise ArgumentError, "unknown matcher type: #{type.inspect}"
153
263
  end
@@ -1,5 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "time" # Time#httpdate
4
+
3
5
  module Labkit
4
6
  module RateLimit
5
7
  # Result is the return value of Limiter#check.
@@ -113,18 +115,13 @@ module Labkit
113
115
  @most_constraining ||= @evaluations.min
114
116
  end
115
117
 
116
- # Returns RFC-compliant rate limit response headers, or {} when no rule matched or an error occurred.
117
- # Keys: RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset (Unix timestamp).
118
- # remaining is coerced to Integer for header output even when info.remaining is fractional;
119
- # the RateLimit header spec requires integer values.
118
+ # Returns the rate limit response headers of the most constraining
119
+ # evaluation, or {} when no rule matched, a :skip rule matched, or an
120
+ # error occurred. See Result::Evaluation#to_response_headers for the keys.
120
121
  def to_response_headers
121
122
  return {} unless matched? && !error? && info
122
123
 
123
- {
124
- "RateLimit-Limit" => info.resolved_limit.to_i.to_s,
125
- "RateLimit-Remaining" => info.remaining.to_i.to_s,
126
- "RateLimit-Reset" => info.reset_at.to_i.to_s
127
- }
124
+ most_constraining.to_response_headers
128
125
  end
129
126
  end
130
127
 
@@ -155,6 +152,32 @@ module Labkit
155
152
  constraint_rank <=> other.constraint_rank
156
153
  end
157
154
 
155
+ # Returns the rate limit response headers for this evaluation. Values are
156
+ # Strings, and counts are coerced to Integer because the RateLimit header
157
+ # spec requires integer values even when count is fractional.
158
+ #
159
+ # RateLimit-Name (the rule name), RateLimit-Limit, RateLimit-Observed,
160
+ # RateLimit-Remaining and RateLimit-Reset (Unix timestamp) are always
161
+ # present. Limit, Observed and Remaining all count within the rule's own
162
+ # period. A blocking evaluation also gets Retry-After (seconds) and
163
+ # RateLimit-ResetTime (HTTP date), which belong on the 429.
164
+ def to_response_headers
165
+ headers = {}
166
+ headers["RateLimit-Name"] = rule.name
167
+ # Floored at 0 like remaining: a callable limit can resolve below zero
168
+ # to deny all, and an older client could credit a counter below zero.
169
+ headers["RateLimit-Limit"] = [info.resolved_limit.to_i, 0].max.to_s
170
+ headers["RateLimit-Observed"] = [info.count.to_i, 0].max.to_s
171
+ headers["RateLimit-Remaining"] = info.remaining.to_i.to_s
172
+ headers["RateLimit-Reset"] = info.reset_at.to_i.to_s
173
+ return headers unless block?
174
+
175
+ # Rounded up so a client that honours it does not retry before the reset.
176
+ headers["Retry-After"] = [(info.reset_at - Time.now.utc).ceil, 0].max.to_s
177
+ headers["RateLimit-ResetTime"] = info.reset_at.httpdate
178
+ headers
179
+ end
180
+
158
181
  protected
159
182
 
160
183
  def constraint_rank
@@ -10,8 +10,8 @@ module Labkit
10
10
  # name - stable identifier used in Redis keys and log entries
11
11
  # match - hash of identifier key/value pairs that must all match for
12
12
  # the rule to apply; empty hash matches any identifier
13
- # limit - request threshold; may be a callable (resolved per check)
14
- # period - window in seconds; may be a callable (resolved per check)
13
+ # limit - request threshold, at least 1; may be a callable (resolved per check)
14
+ # period - window in seconds, at least 1; may be a callable (resolved per check)
15
15
  # action - :limit (enforce; the result blocks when the rule is over
16
16
  # its limit, which also terminates evaluation), :log (count
17
17
  # and log only, never blocks and never terminates),
@@ -56,6 +56,16 @@ module Labkit
56
56
  raise ArgumentError, "ban_for must be a Numeric of at least 1 second or a callable, got #{value.inspect}"
57
57
  end
58
58
 
59
+ # A static limit under 1 blocks every request, and a period under a second
60
+ # reaches EXPIRE as 0, deleting the counter on every write. A :skip rule
61
+ # counts nothing, and a callable may still resolve to 0 to mean "deny all".
62
+ def self.normalize_numeric(field, value, action_sym)
63
+ return value if action_sym == :skip || value.respond_to?(:call)
64
+ return value if value.is_a?(Numeric) && value >= 1
65
+
66
+ raise ArgumentError, "#{field} must be a Numeric of at least 1 or a callable, got #{value.inspect}"
67
+ end
68
+
59
69
  def self.normalize_count_distinct(value, characteristics_arr)
60
70
  sym =
61
71
  case value
@@ -94,8 +104,8 @@ module Labkit
94
104
  super(
95
105
  name: name_str.freeze,
96
106
  match: match.transform_keys(&:to_sym).transform_values { |v| Matcher.build(v) }.freeze,
97
- limit: limit,
98
- period: period,
107
+ limit: self.class.normalize_numeric(:limit, limit, action_sym),
108
+ period: self.class.normalize_numeric(:period, period, action_sym),
99
109
  action: action_sym,
100
110
  characteristics: characteristics_arr,
101
111
  count_distinct: count_distinct_sym,
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-labkit
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.1.2
4
+ version: 5.6.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Newdigate
@@ -277,6 +277,20 @@ dependencies:
277
277
  - - "<"
278
278
  - !ruby/object:Gem::Version
279
279
  version: 6.0.0
280
+ - !ruby/object:Gem::Dependency
281
+ name: thrift
282
+ requirement: !ruby/object:Gem::Requirement
283
+ requirements:
284
+ - - "<"
285
+ - !ruby/object:Gem::Version
286
+ version: '0.25'
287
+ type: :runtime
288
+ prerelease: false
289
+ version_requirements: !ruby/object:Gem::Requirement
290
+ requirements:
291
+ - - "<"
292
+ - !ruby/object:Gem::Version
293
+ version: '0.25'
280
294
  - !ruby/object:Gem::Dependency
281
295
  name: excon
282
296
  requirement: !ruby/object:Gem::Requirement
@@ -568,6 +582,7 @@ files:
568
582
  - ".rufo"
569
583
  - ".tool-versions"
570
584
  - ".yamllint.yaml"
585
+ - AGENTS.md
571
586
  - CODEOWNERS
572
587
  - CONTRIBUTING.md
573
588
  - Dangerfile