gitlab-labkit 5.8.0 → 5.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: bbffb0883d80c949bfb28ffec2943b9cc89ffb88c2c02656afe99ad777e0f152
4
- data.tar.gz: 93610eb46b3dbbfebe40c1b97d31d07b69af3f5e5ee0f152be29a66eca8da03e
3
+ metadata.gz: 3cf9de3ab607ca2b1f4e24b04a55efc3081556acaf61c8fd4c451fd786aa632a
4
+ data.tar.gz: 6a7d1d5fd0434e9298c41bfa6057f96ae08e37c3ea100f37db6d4879ecea688e
5
5
  SHA512:
6
- metadata.gz: c01235e9277e28adbde6cb2e14cedebbabe78b46a8f7e771664875da94cd59ea3c31e2033b152ea8e23dce3cd8ce84c0c5fd310c9691d1662006df0e87f66b67
7
- data.tar.gz: 87c05f1a0543e7969b35023d227d32cabc87455caa99aeca54a83832a5e40a2968895afb3982c4ecfcadc31d891558ce4abccb249f0758f95903367fd56b82df
6
+ metadata.gz: 931e421f2c52d3d98602f3657088701bb5a48b2f98bd9289becffad20df9d991c3fc1d2f9b03d5ba6cbc26593ca38176227827afc66b973393302e1f789f89cd
7
+ data.tar.gz: b971f78dfcf2529cda540c44e950651876cfd89e5d68af517c10f5013ead19d50c7f603d7ecc582787e790ab6ae1deb46dbc342b010676a48aff8964fb89b109
data/.gitlab-ci.yml CHANGED
@@ -81,6 +81,19 @@ deploy:
81
81
  - Gemfile
82
82
  - .gitlab-ci.yml
83
83
 
84
+ # The generated protobuf code for gitlab.ratelimit.config.v1 is vendored from
85
+ # gitlab-org/protos, because a published gem cannot depend on the protos git gem.
86
+ vendored-proto-check:
87
+ stage: validate
88
+ needs: []
89
+ image: ruby:3.4
90
+ rules:
91
+ - if: $CI_PIPELINE_SOURCE == "merge_request_event"
92
+ - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
93
+ script:
94
+ - ./scripts/vendor-ratelimit-proto.sh
95
+ - git diff --exit-code -- lib/labkit/rate_limit/config/proto/
96
+
84
97
  # Gate SDK changes on the labkit-spec conformance suite, run remotely: trigger
85
98
  # a downstream pipeline in labkit-spec that checks out this pipeline's commit
86
99
  # (LABKIT_RUBY_SHA) and runs the Ruby conformance suite against it, verifying
data/.rubocop.yml CHANGED
@@ -4,5 +4,13 @@ inherit_gem:
4
4
  gitlab-styles:
5
5
  - rubocop-default.yml
6
6
 
7
+ inherit_mode:
8
+ merge:
9
+ - Exclude
10
+
7
11
  AllCops:
8
12
  NewCops: enable
13
+ Exclude:
14
+ # Generated by protoc and vendored from gitlab-org/protos; regenerate with
15
+ # scripts/vendor-ratelimit-proto.sh instead of editing.
16
+ - lib/labkit/rate_limit/config/proto/**/*
data/AGENTS.md CHANGED
@@ -122,48 +122,26 @@ Keep an MR to one concern, and separate mechanical changes such as renames from
122
122
 
123
123
  ## Rate limiting
124
124
 
125
- `Labkit::RateLimit` is documented in `lib/labkit/rate_limit/README.md`.
126
- It is the one capability that shares state across SDKs: services in any language enforce the same limits against the same Redis.
127
- That makes the Redis key shape and the Lua scripts as much a contract as the HTTP headers, so changes here follow stricter rules than the rest of the gem.
128
-
129
- ### Where each SDK keeps it
130
-
131
- | SDK | Repository | Module | Tests | Status |
132
- | --- | --- | --- | --- | --- |
133
- | Ruby (this repository) | gitlab-org/ruby/gems/labkit-ruby | `lib/labkit/rate_limit/` | `spec/labkit/rate_limit/` | released |
134
- | Go | gitlab-org/labkit | `v2/ratelimit/` | `v2/ratelimit/*_test.go`, run with `go -C v2 test ./ratelimit/...` | released |
135
- | Rust | gitlab-org/rust/labkit-rs | in progress | | not released, skip |
136
- | Python | gitlab-org/labkit-python | in progress | | not released, skip |
137
- | Spec | gitlab-org/quality/tooling/labkit-spec | `conformance/README.md` ("v1 decisions"), `conformance/ratelimit_scenarios/*.json`, `conformance/aspect_ratelimit.go` | `conformance/` | judge |
125
+ Services in every language enforce the same rate limits against one shared Redis.
126
+ So these are a contract between all SDKs:
138
127
 
139
- ### The three layers
128
+ - Redis keys, values, and TTLs
129
+ - Lua scripts
130
+ - the counting rule that decides a block
131
+ - the response status and `RateLimit-*` headers
132
+ - metrics and log fields
140
133
 
141
- Every SDK must agree on three layers.
142
- This section defines the layers; the current values (key grammar, header names, counting rules, metric names, log fields) live in labkit-spec, in the "v1 decisions" list in `conformance/README.md` and the scenarios under `conformance/ratelimit_scenarios/`.
134
+ [labkit-spec](https://gitlab.com/gitlab-org/quality/tooling/labkit-spec) owns that contract: the "v1 decisions" in `conformance/README.md` and the scenarios in `conformance/ratelimit_scenarios/`.
143
135
  If this file and labkit-spec disagree, labkit-spec wins.
136
+ `Labkit::RateLimit` is documented in `lib/labkit/rate_limit/README.md`.
137
+
138
+ | SDK | Repository | Rate limiter | Tests |
139
+ | --- | --- | --- | --- |
140
+ | Ruby | [labkit-ruby](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby) | `lib/labkit/rate_limit/` | `bundle exec rspec spec/labkit/rate_limit` |
141
+ | Go | [labkit](https://gitlab.com/gitlab-org/labkit) | `v2/ratelimit/` | `go -C v2 test ./ratelimit/...` |
142
+ | Rust | [labkit-rs](https://gitlab.com/gitlab-org/rust/labkit-rs) | `labkit/src/ratelimit/` | `cargo test -p labkit --features ratelimit-redis --lib ratelimit` |
144
143
 
145
- 1. **Storage contract.**
146
- Anything written to or read from Redis: key shape and prefix, how values are encoded, TTL rules, and the Lua scripts.
147
- Two SDKs sharing one Redis with different storage rules keep two counters for one limit.
148
- Never change this layer in one SDK.
149
- It changes in labkit-spec first, then every SDK follows.
150
- 2. **Observable contract.**
151
- Anything a caller or an operator can see: response status and headers, the counting rule that decides a block, log line messages and field sets, metric families and labels.
152
- Also changes in labkit-spec first.
153
- 3. **Behaviour within the contract.**
154
- Input validation and edge cases the contract already implies, where one SDK does something the contract does not allow.
155
- These may be fixed SDK by SDK, and every SDK should end up the same.
156
-
157
- A change that mixes layers is governed by the strictest one: storage, then observable, then behaviour.
158
- When a change does not fit one layer clearly, say so in the MR instead of picking one.
159
-
160
- ### Rules for rate-limit changes
161
-
162
- - A change to layer 1 or 2 opens a labkit-spec MR first and waits for it to merge.
163
- - A change to layer 3 should be checked against the Go SDK.
164
- If Go has the same gap, say so in the MR so it can be ported.
165
- - When a review finds a divergence between SDKs, a conformance scenario in labkit-spec must follow in the same week, so the harness sees the gap next time.
166
- The author of the MR where the divergence was found owns this.
167
- Mention the sync agent on the review thread to have it draft the scenario MR; if it cannot, open a labkit-spec issue and link it from the thread.
168
- - Never edit Lua scripts, key construction, header names, metric names, or log field names to fix a Ruby-only bug.
169
- If a fix seems to need that, it is a contract change.
144
+ - Change the contract in labkit-spec first, then in every SDK. Merge the labkit-spec MR before any SDK MR. Never change the contract in one SDK only.
145
+ - A fix inside the contract can land in one SDK. Check the other SDKs, and say in the MR if they have the same gap.
146
+ - If you are not sure that a change is inside the contract, treat it as a contract change and say so in the MR.
147
+ - When review finds a gap between SDKs, add a labkit-spec scenario for it in the same week, or open a labkit-spec issue and link it from the thread. The author of the MR where the gap was found owns this.
data/CODEOWNERS CHANGED
@@ -1,4 +1,4 @@
1
1
  # CODEOWNERS is used to lookup assignees for
2
2
  # Renovate Bot dependency change Merge Requests.
3
3
  # https://docs.renovatebot.com/configuration-options/#assigneesfromcodeowners
4
- * @ayufan @hmerscher @e_forbes @M_Alvarez @mwoolf @sankalp_gl @hardikgala @nindurkar @ashs2
4
+ * @ayufan @hmerscher @e_forbes @M_Alvarez @mwoolf @sankalp_gl @hardikgala @nindurkar @ashs2 @lhollinda
data/Gemfile CHANGED
@@ -8,3 +8,9 @@ gemspec
8
8
  # Exporter source: https://gitlab.com/gitlab-org/quality/analytics/test-metrics-exporters
9
9
  # Owned by: @gitlab-org/quality/analytics (group::development analytics)
10
10
  gem 'gitlab-rspec-metrics-exporter', '~> 0.2', groups: [:development, :test]
11
+
12
+ # regexp_parser 2.13.2 made Regexp::Syntax::Base.implements private, and
13
+ # json_schemer 2.5.0 calls it at load time, so every fresh bundle fails on
14
+ # `require "json_schemer"`. Drop the pin once json_schemer or regexp_parser
15
+ # ships a fix: https://github.com/davishmcclurg/json_schemer/issues/238
16
+ gem "regexp_parser", "!= 2.13.2", groups: [:development, :test]
@@ -177,6 +177,7 @@ A `Rule` is a `Data.define` value object with the following fields:
177
177
  | `characteristics` | Array of identifier keys whose values are folded into the Redis counter key. Each unique combination gets its own counter. |
178
178
  | `count_distinct` | Optional identifier key. When set, the rule counts distinct values seen for that key within the window, backed by a Redis SET rather than a counter. Must not overlap `characteristics`. |
179
179
  | `ban_for` | Optional ban duration, whole seconds, minimum 1. Changes the accounting; `action` still decides who is blocked. Rejected on `:skip`. May be a callable resolved on every check. |
180
+ | `admit_counted_values` | Optional, default `false`. Only valid with `count_distinct`, and rejected on `:skip`. When `true`, the rule counts only the values it admits, so values already counted keep passing once the set is full. Rename the rule when changing it. See [Admitting counted values](#admitting-counted-values). |
180
181
 
181
182
  Making `limit` or `period` callable is the supported pattern for
182
183
  runtime-tunable thresholds (e.g. feature flags or database-backed settings):
@@ -193,9 +194,11 @@ Labkit::RateLimit::Rule.new(
193
194
  `Rule.new` raises `ArgumentError` for a static `limit` under 1 or a static
194
195
  `period` under one second, except on `:skip` rules where both are inert.
195
196
  Callables are not validated: a `limit` resolving to `0` blocks every matching
196
- request, which is how to express "deny all" dynamically, and a `period`
197
- resolving under one second reaches `EXPIRE` as `0` and deletes the counter on
198
- every write.
197
+ request, which is how to express "deny all" dynamically. On an
198
+ `admit_counted_values` rule it only denies new values, since values already
199
+ counted in the window still pass; to block those too, put a counter rule whose
200
+ limit resolves to `0` in front of it. A `period` resolving under one second
201
+ reaches `EXPIRE` as `0` and deletes the counter on every write.
199
202
 
200
203
  ### Matchers
201
204
 
@@ -315,7 +318,7 @@ flowchart TD
315
318
  Eval --> Build[Build Evaluation<br/>resolve limit/period]
316
319
  Build --> Add[Add evaluation to Result]
317
320
  Add --> Emit[Emit rule_evaluations_total<br/>+ limit/period gauges]
318
- Emit --> Act{"result.block?<br/>(:limit rule over its limit,<br/>or its ban in force)"}
321
+ Emit --> Act{"result.block?<br/>(:limit rule over its limit,<br/>its ban in force, or an admitting<br/>rule that refused the value)"}
319
322
  Act -->|yes| Return([Return Result<br/>action=:block])
320
323
  Act -->|"no (:log, or under limit)"| Iter
321
324
  Iter -->|no more rules| Return2([Return Result reporting<br/>most-constraining evaluation,<br/>or matched=false if none])
@@ -349,10 +352,11 @@ visible via `checks_total{error="true"}` and the
349
352
 
350
353
  The rule's `action` describes what the rule does; the result's `action`
351
354
  describes the outcome — what the caller should do — and is only ever `:allow`
352
- or `:block`. A matching rule increments its counter, with two exceptions:
353
- `:skip` rules never touch Redis, and a rule carrying `ban_for` whose ban is
354
- already in force short-circuits before incrementing. The `result` label on
355
- `rule_evaluations_total` records what each evaluation decided:
355
+ or `:block`. A matching rule increments its counter, with three exceptions:
356
+ `:skip` rules never touch Redis, a rule carrying `ban_for` whose ban is
357
+ already in force short-circuits before incrementing, and an
358
+ `admit_counted_values` rule adds a value only when it admits it. The `result`
359
+ label on `rule_evaluations_total` records what each evaluation decided:
356
360
 
357
361
  | rule action | what it does | exceeded? | result action | `result` label | terminating? |
358
362
  |-------------|-----------------------------------------------|-----------|---------------|----------------|---------------|
@@ -419,6 +423,60 @@ window, and it stops counting while that ban holds:
419
423
  suit counting discrete failures, such as bad credentials, better than they
420
424
  suit variable-cost resource limits.
421
425
 
426
+ ### Admitting counted values
427
+
428
+ A plain `count_distinct` rule adds every checked value to its set, including
429
+ the values it blocks. Once one value is blocked, the count stays over the
430
+ limit for the rest of the window, so values that were counted earlier are
431
+ blocked too. That suits "block after N distinct values", such as the number
432
+ of users seen from one IP.
433
+
434
+ Set `admit_counted_values: true` to cap how many distinct values can be
435
+ admitted per window instead:
436
+
437
+ ```ruby
438
+ Labkit::RateLimit::Rule.new(
439
+ name: "participants_per_namespace",
440
+ limit: 20,
441
+ period: 86_400,
442
+ characteristics: [:namespace],
443
+ count_distinct: :email_digest,
444
+ admit_counted_values: true
445
+ )
446
+ ```
447
+
448
+ - A value already in the set is allowed and leaves the count unchanged, even
449
+ when the set is full.
450
+ - A new value is added only while the count is below the limit. Otherwise the
451
+ request is blocked and the value is not added, so a blocked value stays
452
+ blocked when it is retried and the count does not grow past the limit.
453
+ - A blocked request reports the set's count and `RateLimit-Remaining: 0`. The
454
+ block comes from the admission, not from `count > limit`, the same way a ban
455
+ decides the block apart from the count.
456
+ - If the limit is lowered during a window, the set can hold more values than
457
+ the new limit. Values already in it are still allowed, and new values are
458
+ blocked until the window ends.
459
+ - `peek` reports what `check` would decide when the identifier has the
460
+ `count_distinct` value, without writing: exceeded only when that value is
461
+ not counted and the count has reached the limit. Without the value, `check`
462
+ skips the rule, but `peek` answers for a value that is not counted yet, so
463
+ it reports exceeded once the count reaches the limit.
464
+ - A `:log` rule does the same accounting, including not adding refused
465
+ values, and only skips the blocking.
466
+ - `cost:` is ignored, as for a plain `count_distinct` rule, so a `cost: 0`
467
+ check still admits a new value.
468
+ - Rules count a request up to the one that blocks it. An admitting rule placed
469
+ before a counter rule that blocks still admits the value, which then takes a
470
+ slot for the rest of the window. Put admitting rules after the counter rules
471
+ in the same limiter.
472
+
473
+ The Redis key and the stored values are the same as for a plain
474
+ `count_distinct` rule, and a plain rule adds every value it checks, including
475
+ the ones it blocks. A rule switched between the two modes keeps those values,
476
+ and they count as admitted until the window ends. Rename the rule when
477
+ changing its mode, which abandons its counters (see [Rule](#rule)), and give a
478
+ rule the same mode in every SDK that writes to it.
479
+
422
480
  ### Redis keys
423
481
 
424
482
  Each matched check writes a key shaped:
@@ -439,7 +497,8 @@ is not extended on subsequent increments, so the window is a true fixed
439
497
  window starting at the first request, not a sliding window.
440
498
 
441
499
  A check is a single `EVALSHA` of `INCR_SCRIPT` (or `SADD_SCRIPT` for
442
- `count_distinct` rules, or `BAN_SCRIPT` for rules carrying `ban_for`). Doing the whole
500
+ `count_distinct` rules, `ADMIT_SCRIPT` for `count_distinct` rules with
501
+ `admit_counted_values`, or `BAN_SCRIPT` for rules carrying `ban_for`). Doing the whole
443
502
  read-modify-write inside Lua means there is no window between the increment
444
503
  and the `EXPIRE` in which a key could be left without a TTL.
445
504
 
@@ -488,9 +547,10 @@ key with no expiry, the TTL only needs reading once — after the increment.
488
547
  That single read distinguishes both cases the script must handle, so there is
489
548
  no reason to read it again before mutating.
490
549
 
491
- `peek` does not use a script: it pipelines `GET` + `TTL` (or `SCARD` + `TTL`)
492
- and never issues `EXPIRE`, so it cannot start or extend a window. A missing
493
- key (`GET → nil`, `TTL → -2`) is reported as `count = 0`, and `build_evaluation`
550
+ `peek` does not use a script: it pipelines `GET` + `TTL` (or `SCARD` + `TTL`,
551
+ plus `SISMEMBER` for an `admit_counted_values` rule when the identifier has the
552
+ value) and never issues `EXPIRE`, so it cannot start or extend a window. A
553
+ missing key (`GET → nil`, `TTL → -2`) is reported as `count = 0`, and `build_evaluation`
494
554
  falls back to the rule's period for `reset_at` since there is no Redis-side
495
555
  window to read.
496
556
 
@@ -503,7 +563,7 @@ the most constraining one (see [Evaluation flow](#evaluation-flow)):
503
563
  ```ruby
504
564
  result.matched? # => true if some rule matched
505
565
  result.skipped? # => true if a matched :skip rule bypassed the check
506
- result.exceeded? # => true if the reported rule's counter > limit
566
+ result.exceeded? # => true if the reported rule is over its limit
507
567
  result.action # => :block | :allow — what the caller should do
508
568
  result.block? # => result.action == :block
509
569
  result.rule # => the reported Rule, or nil
@@ -651,6 +711,39 @@ A logging failure during a flush is logged at WARN with
651
711
  `dropped_pairs`, the number of lines that flush lost. It never raises, so the
652
712
  flush thread keeps running.
653
713
 
714
+ ## Config files
715
+
716
+ `Labkit::RateLimit::Config` loads rules from YAML files that follow the `gitlab.ratelimit.config.v1` schema from [gitlab-org/protos](https://gitlab.com/gitlab-org/protos): the contract an application ships (`AvailableLimiters`) and the rate limits an operator writes (`RateLimits`).
717
+ The generated protobuf classes are vendored under `lib/labkit/rate_limit/config/proto/` and reachable as `Labkit::RateLimit::Config::Proto`.
718
+ Run `scripts/vendor-ratelimit-proto.sh` to update them; CI fails on drift.
719
+
720
+ An application can also install the `gitlab-ratelimit-config-v1` gem from gitlab-org/protos.
721
+ Then labkit uses the gem's schema and compares it with the vendored copy when `Labkit::RateLimit::Config` loads.
722
+ A gem schema that is newer gives a warning.
723
+ A gem schema that lacks or changes a vendored field raises `Labkit::RateLimit::Config::VendoredProto::Mismatch`.
724
+ So upgrade the `gitlab-ratelimit-config-v1` gem in the application first, then the `gitlab-labkit` release that updates the vendored copy.
725
+
726
+ Loading runs in four steps, `parse`, `schema`, `contract` and `rule`, and a failure raises `Labkit::RateLimit::Config::Error` whose `#step` names the step.
727
+ Only the parse step exists so far; the schema, contract and rule steps follow.
728
+ Until the schema step lands, no mode validates a document: strict mode lets an enum number that no `Action` defines through, such as `action: 7`, and lenient mode leaves an unknown `action` name as `ACTION_UNSPECIFIED`.
729
+ The schema step rejects both.
730
+
731
+ ```ruby
732
+ doc = Labkit::RateLimit::Config::Parser.parse(
733
+ "config/rate_limits/rack_request.yaml",
734
+ Labkit::RateLimit::Config::Proto::RateLimits,
735
+ strict: true # CI rejects unknown fields. Production loads with strict: false once the schema step exists
736
+ )
737
+ doc.rate_limits["rack_request"].rules.first.name
738
+ ```
739
+
740
+ The parse step converts each YAML scalar by the type of the field it fills, so `eq: off`, `eq: True` and `eq: 9970` stay strings, `limit: "5"` becomes `5`, `action: "1"` becomes `ACTION_LIMIT`, and a YAML null leaves the field unset.
741
+ A repeated key, also one repeated under its camelCase `json_name`, a merge key (`<<`), an empty file, more than one document, or a non-scalar key fails at step `parse`, in both modes.
742
+ So does a value of the wrong kind for its field, such as a mapping where a scalar belongs, an integer out of range for its field, two fields of one `oneof`, such as `{ eq: a, re: b }`, a null list item or map value, and a document nested deeper than 64 levels.
743
+ Each of these errors names the YAML line.
744
+ An unknown field or an unknown enum name, such as `action: ACTION_LIMT`, fails in strict mode with its YAML line.
745
+ Lenient mode ignores an unknown field and leaves an unknown enum name unset.
746
+
654
747
  ## Dev/test vs production guards
655
748
 
656
749
  `Limiter.new` and `Rule.new` validate names and configuration. In
@@ -0,0 +1,18 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Labkit
4
+ module RateLimit
5
+ module Config
6
+ # Error is raised when a document fails one of the load steps.
7
+ # step is :parse, :schema, :contract or :rule.
8
+ class Error < StandardError
9
+ attr_reader :step
10
+
11
+ def initialize(step, message)
12
+ @step = step
13
+ super("#{step}: #{message}")
14
+ end
15
+ end
16
+ end
17
+ end
18
+ end
@@ -0,0 +1,243 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "json"
4
+ require "psych"
5
+ require "google/protobuf/descriptor_pb"
6
+
7
+ module Labkit
8
+ module RateLimit
9
+ module Config
10
+ # Parser is the parse step: a YAML file into a protobuf message.
11
+ #
12
+ # YAML types do not follow the schema, so each scalar is converted by the
13
+ # type of the field it fills, the way protoyaml does in Go: a string field
14
+ # keeps the scalar text (`off`, `True` and `9970` stay strings), an integer
15
+ # or enum field gets an Integer when the text is one, quoted or not, and
16
+ # the YAML null forms mean the field is absent. Psych keeps the last value
17
+ # of a repeated key, so repeated keys are rejected here, also when one
18
+ # spelling is the snake_case field name and the other its camelCase
19
+ # json_name.
20
+ #
21
+ # This step checks the shape of a document, not its values. An enum
22
+ # number that no Action defines, such as `action: 7`, passes in both
23
+ # modes, and in lenient mode protobuf drops an unknown enum name together
24
+ # with the unknown fields, so a rule with a misspelt action reaches the
25
+ # schema step as ACTION_UNSPECIFIED. The schema step rejects or drops
26
+ # both.
27
+ module Parser
28
+ INT32 = (-(2**31))...(2**31)
29
+ UINT32 = 0...(2**32)
30
+ INT64 = (-(2**63))...(2**63)
31
+ UINT64 = 0...(2**64)
32
+ INTEGER_RANGES = {
33
+ int32: INT32, sint32: INT32, sfixed32: INT32, enum: INT32,
34
+ uint32: UINT32, fixed32: UINT32,
35
+ int64: INT64, sint64: INT64, sfixed64: INT64,
36
+ uint64: UINT64, fixed64: UINT64
37
+ }.freeze
38
+ INTEGER = /\A[-+]?\d+\z/
39
+ NULLS = ["", "~", "null", "Null", "NULL"].freeze
40
+ # The protobuf JSON decoder stops at 64 levels of objects and arrays.
41
+ # Mappings and lists are counted here so the error names the YAML line
42
+ # and a deeper document cannot exhaust the stack.
43
+ MAX_DEPTH = 64
44
+
45
+ class << self
46
+ # @param path [String] YAML file
47
+ # @param klass [Class] a Proto message class, AvailableLimiters or RateLimits
48
+ # @param strict [Boolean] when true an unknown field is an error
49
+ # @return [Google::Protobuf::AbstractMessage]
50
+ # @raise [Error] with step :parse
51
+ def parse(path, klass, strict:)
52
+ tree = tree_for(document_root(path), klass.descriptor, strict: strict, depth: 0)
53
+ klass.decode_json(JSON.generate(tree), ignore_unknown_fields: !strict)
54
+ rescue Psych::SyntaxError, SystemCallError => e
55
+ # Both messages already name the file.
56
+ raise Error.new(:parse, e.message)
57
+ rescue Google::Protobuf::ParseError, Psych::Exception, JSON::JSONError => e
58
+ raise Error.new(:parse, "#{path}: #{e.message}")
59
+ end
60
+
61
+ private
62
+
63
+ # One YAML document per file. Psych would silently keep only the first
64
+ # of several, and an empty file has no root to walk.
65
+ def document_root(path)
66
+ documents = Psych.parse_stream(File.read(path), filename: path).children
67
+ raise Psych::Exception, "the file holds no YAML document" if documents.empty?
68
+ raise Psych::Exception, "the file holds #{documents.size} YAML documents, expected 1" if documents.size > 1
69
+
70
+ documents.first.root
71
+ end
72
+
73
+ # One value for one slot. type is a field type Symbol, a message
74
+ # Descriptor or an EnumDescriptor. A node of the wrong kind is
75
+ # rejected here so the error names the YAML line, not an offset into
76
+ # the generated JSON. depth counts the mappings and lists above node.
77
+ def tree_for(node, type, strict:, depth:)
78
+ expects_message = type.is_a?(Google::Protobuf::Descriptor)
79
+
80
+ case node
81
+ when Psych::Nodes::Mapping
82
+ fail_at(node, "expected a scalar, found a mapping") unless expects_message
83
+ mapping_for(node, type, strict: strict, depth: depth)
84
+ when Psych::Nodes::Sequence
85
+ fail_at(node, "expected a single value, found a list")
86
+ when Psych::Nodes::Scalar
87
+ return nil if null?(node)
88
+
89
+ fail_at(node, "expected a mapping, found a scalar") if expects_message
90
+ scalar_for(node, type, strict: strict)
91
+ when Psych::Nodes::Alias
92
+ fail_at(node, "aliases are not supported")
93
+ end
94
+ end
95
+
96
+ def mapping_for(node, descriptor, strict:, depth:)
97
+ check_depth!(node, depth)
98
+ entries = node.children.each_slice(2).map do |key, value|
99
+ name = scalar_key(key)
100
+ field = field_for(descriptor, name)
101
+ fail_at(key, "unknown field #{name.inspect}") if strict && field.nil?
102
+ [name, key, field, value]
103
+ end
104
+ reject_repeated_keys!(entries.map { |name, key, field, _| [field ? field.name : name, key] })
105
+ reject_oneof_conflicts!(descriptor, entries)
106
+
107
+ maps = map_fields(descriptor)
108
+ # An unknown field is not walked: strict mode rejected its name
109
+ # above and lenient mode ignores it, so a newer schema may put
110
+ # anything there.
111
+ entries.to_h do |name, _, field, value|
112
+ next [name, nil] unless field
113
+
114
+ [name, field_value_for(value, field, map: maps.include?(field.name), strict: strict, depth: depth + 1)]
115
+ end
116
+ end
117
+
118
+ # Descriptor#options, which carries the map_entry flag, arrived after
119
+ # google-protobuf 3.25.0, so a blank message tells a map from a list.
120
+ # A message type keeps one Descriptor object, so the cache is bounded
121
+ # by the number of message types.
122
+ def map_fields(descriptor)
123
+ (@map_fields ||= {})[descriptor] ||= begin
124
+ blank = descriptor.msgclass.new
125
+ descriptor.select { |field| blank[field.name].is_a?(Google::Protobuf::Map) }.map(&:name).freeze
126
+ end
127
+ end
128
+
129
+ # A mapping key must be a plain scalar; a sequence, mapping or alias
130
+ # as a key has no field to fill. A merge key is rejected in both
131
+ # modes: lenient mode would ignore it as an unknown field and drop
132
+ # the merged fields with no error.
133
+ def scalar_key(node)
134
+ fail_at(node, "a mapping key must be a scalar") unless node.is_a?(Psych::Nodes::Scalar)
135
+ fail_at(node, "merge keys (<<) are not supported") if node.plain && node.value == "<<"
136
+
137
+ node.value
138
+ end
139
+
140
+ # The error points at the repeated key, not at the start of the mapping.
141
+ def reject_repeated_keys!(named_keys)
142
+ seen = {}
143
+ named_keys.each do |name, key|
144
+ fail_at(key, "duplicate key #{name.inspect}") if seen[name]
145
+ seen[name] = true
146
+ end
147
+ end
148
+
149
+ # Two fields of one oneof, such as two Matcher operators, would
150
+ # otherwise fail in the JSON decoder with no YAML line.
151
+ def reject_oneof_conflicts!(descriptor, entries)
152
+ descriptor.each_oneof do |oneof|
153
+ names = oneof.map(&:name)
154
+ set = entries.select { |_, _, field, value| field && names.include?(field.name) && !null?(value) }
155
+ next if set.size < 2
156
+
157
+ fail_at(set[1][1], "#{set[0][2].name} and #{set[1][2].name} are both set; set only one field of #{oneof.name}")
158
+ end
159
+ end
160
+
161
+ def check_depth!(node, depth)
162
+ fail_at(node, "nesting deeper than #{MAX_DEPTH} levels") if depth >= MAX_DEPTH
163
+ end
164
+
165
+ def fail_at(node, message)
166
+ raise Psych::Exception, "line #{node.start_line + 1}: #{message}"
167
+ end
168
+
169
+ # Files are snake_case; protojson also accepts the camelCase json_name.
170
+ def field_for(descriptor, name)
171
+ descriptor.lookup(name) || descriptor.find { |field| field.json_name == name }
172
+ end
173
+
174
+ def field_value_for(node, field, map:, strict:, depth:)
175
+ return map_for(node, field, strict: strict, depth: depth) if map
176
+ return tree_for(node, element_type(field), strict: strict, depth: depth) unless field.label == :repeated
177
+ return nil if null?(node)
178
+
179
+ fail_at(node, "expected a list") unless node.is_a?(Psych::Nodes::Sequence)
180
+ check_depth!(node, depth)
181
+ node.children.map do |child|
182
+ fail_at(child, "a list item cannot be null") if null?(child)
183
+
184
+ tree_for(child, element_type(field), strict: strict, depth: depth + 1)
185
+ end
186
+ end
187
+
188
+ def map_for(node, field, strict:, depth:)
189
+ return nil if null?(node)
190
+
191
+ fail_at(node, "expected a mapping") unless node.is_a?(Psych::Nodes::Mapping)
192
+ check_depth!(node, depth)
193
+ pairs = node.children.each_slice(2)
194
+ reject_repeated_keys!(pairs.map { |key, _| [scalar_key(key), key] })
195
+ value_type = element_type(field.subtype.lookup("value"))
196
+ pairs.to_h do |key, value|
197
+ fail_at(value, "a map value cannot be null") if null?(value)
198
+
199
+ [scalar_key(key), tree_for(value, value_type, strict: strict, depth: depth + 1)]
200
+ end
201
+ end
202
+
203
+ # A message or enum field passes its descriptor down, so a scalar
204
+ # can be checked against the enum's names.
205
+ def element_type(field)
206
+ %i[message enum].include?(field.type) ? field.subtype : field.type
207
+ end
208
+
209
+ def null?(node)
210
+ node.is_a?(Psych::Nodes::Scalar) && node.plain && NULLS.include?(node.value)
211
+ end
212
+
213
+ # Integer text is converted quoted or not: protojson takes a quoted
214
+ # number for an integer field but not for an enum. The range is
215
+ # checked here so the error names the YAML line.
216
+ def scalar_for(node, type, strict:)
217
+ enum = type if type.is_a?(Google::Protobuf::EnumDescriptor)
218
+ kind = enum ? :enum : type
219
+ range = INTEGER_RANGES[kind]
220
+ if range && INTEGER.match?(node.value)
221
+ value = Integer(node.value, 10)
222
+ fail_at(node, "#{node.value} is out of range for #{kind}") unless range.cover?(value)
223
+ return value
224
+ end
225
+
226
+ # Lenient mode passes an unknown name on, and protobuf drops it
227
+ # with the unknown fields, so a newer schema's value still loads.
228
+ fail_at(node, "unknown value #{node.value.inspect} for enum #{enum.name}") if enum && strict && enum.lookup_name(node.value.to_sym).nil?
229
+
230
+ return node.value unless node.plain
231
+
232
+ case kind
233
+ when :bool
234
+ { "true" => true, "false" => false }.fetch(node.value, node.value)
235
+ else
236
+ node.value
237
+ end
238
+ end
239
+ end
240
+ end
241
+ end
242
+ end
243
+ end
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+ # Vendored from gitlab-org/protos at gen/ruby/gitlab/ratelimit/config/v1/v0.1.0, gen/ruby/gitlab/ratelimit/config/v1/lib/gitlab/ratelimit/config/v1/config_pb.rb.
3
+ # Do not edit. Run scripts/vendor-ratelimit-proto.sh to update.
4
+ # Generated by the protocol buffer compiler. DO NOT EDIT!
5
+ # source: gitlab/ratelimit/config/v1/config.proto
6
+
7
+ require 'google/protobuf'
8
+ require 'labkit/rate_limit/config/vendored_proto'
9
+
10
+
11
+ descriptor_data = "\n\'gitlab/ratelimit/config/v1/config.proto\x12\x1agitlab.ratelimit.config.v1\"\xf7\x01\n\x11\x41vailableLimiters\x12s\n\x12\x61vailable_limiters\x18\x01 \x03(\x0b\x32\x44.gitlab.ratelimit.config.v1.AvailableLimiters.AvailableLimitersEntryR\x11\x61vailableLimiters\x1am\n\x16\x41vailableLimitersEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12=\n\x05value\x18\x02 \x01(\x0b\x32\'.gitlab.ratelimit.config.v1.LimiterSpecR\x05value:\x02\x38\x01\"\xd2\x01\n\x0bLimiterSpec\x12 \n\x0b\x64\x65scription\x18\x01 \x01(\tR\x0b\x64\x65scription\x12\x31\n\x14\x61vailable_properties\x18\x02 \x03(\tR\x13\x61vailableProperties\x12\x45\n\rdefault_rules\x18\x03 \x03(\x0b\x32 .gitlab.ratelimit.config.v1.RuleR\x0c\x64\x65\x66\x61ultRules\x12\'\n\x0fprotected_rules\x18\x04 \x03(\tR\x0eprotectedRules\"\xca\x01\n\nRateLimits\x12W\n\x0brate_limits\x18\x01 \x03(\x0b\x32\x36.gitlab.ratelimit.config.v1.RateLimits.RateLimitsEntryR\nrateLimits\x1a\x63\n\x0fRateLimitsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12:\n\x05value\x18\x02 \x01(\x0b\x32$.gitlab.ratelimit.config.v1.RuleListR\x05value:\x02\x38\x01\"B\n\x08RuleList\x12\x36\n\x05rules\x18\x01 \x03(\x0b\x32 .gitlab.ratelimit.config.v1.RuleR\x05rules\"\x9a\x04\n\x04Rule\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12 \n\x0b\x64\x65scription\x18\x02 \x01(\tR\x0b\x64\x65scription\x12:\n\x06\x61\x63tion\x18\x03 \x01(\x0e\x32\".gitlab.ratelimit.config.v1.ActionR\x06\x61\x63tion\x12(\n\x0f\x63haracteristics\x18\x04 \x03(\tR\x0f\x63haracteristics\x12\x41\n\x05match\x18\x05 \x03(\x0b\x32+.gitlab.ratelimit.config.v1.Rule.MatchEntryR\x05match\x12\x19\n\x05limit\x18\x06 \x01(\rH\x00R\x05limit\x88\x01\x01\x12\x1e\n\x08period_s\x18\x07 \x01(\rH\x01R\x07periodS\x88\x01\x01\x12%\n\x0e\x63ount_distinct\x18\x08 \x01(\tR\rcountDistinct\x12\x1f\n\tban_for_s\x18\t \x01(\rH\x02R\x07\x62\x61nForS\x88\x01\x01\x12\x1f\n\x08position\x18\n \x01(\rH\x03R\x08position\x88\x01\x01\x1a]\n\nMatchEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x39\n\x05value\x18\x02 \x01(\x0b\x32#.gitlab.ratelimit.config.v1.MatcherR\x05value:\x02\x38\x01\x42\x08\n\x06_limitB\x0b\n\t_period_sB\x0c\n\n_ban_for_sB\x0b\n\t_position\"\x93\x03\n\x13PublishRulesRequest\x12`\n\x0brate_limits\x18\x01 \x03(\x0b\x32?.gitlab.ratelimit.config.v1.PublishRulesRequest.RateLimitsEntryR\nrateLimits\x12S\n\x06remove\x18\x02 \x03(\x0b\x32;.gitlab.ratelimit.config.v1.PublishRulesRequest.RemoveEntryR\x06remove\x1a\x63\n\x0fRateLimitsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12:\n\x05value\x18\x02 \x01(\x0b\x32$.gitlab.ratelimit.config.v1.RuleListR\x05value:\x02\x38\x01\x1a`\n\x0bRemoveEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12;\n\x05value\x18\x02 \x01(\x0b\x32%.gitlab.ratelimit.config.v1.RuleNamesR\x05value:\x02\x38\x01\"!\n\tRuleNames\x12\x14\n\x05names\x18\x01 \x03(\tR\x05names\"\x9a\x02\n\x07Matcher\x12\x10\n\x02\x65q\x18\x01 \x01(\tH\x00R\x02\x65q\x12\x10\n\x02ne\x18\x02 \x01(\tH\x00R\x02ne\x12\x10\n\x02re\x18\x03 \x01(\tH\x00R\x02re\x12\x12\n\x03nre\x18\x04 \x01(\tH\x00R\x03nre\x12?\n\x06one_of\x18\x05 \x01(\x0b\x32&.gitlab.ratelimit.config.v1.StringListH\x00R\x05oneOf\x12\x41\n\x07none_of\x18\x06 \x01(\x0b\x32&.gitlab.ratelimit.config.v1.StringListH\x00R\x06noneOf\x12;\n\x03\x61ll\x18\x07 \x01(\x0b\x32\'.gitlab.ratelimit.config.v1.MatcherListH\x00R\x03\x61llB\x04\n\x02op\"$\n\nStringList\x12\x16\n\x06values\x18\x01 \x03(\tR\x06values\"N\n\x0bMatcherList\x12?\n\x08matchers\x18\x01 \x03(\x0b\x32#.gitlab.ratelimit.config.v1.MatcherR\x08matchers*S\n\x06\x41\x63tion\x12\x16\n\x12\x41\x43TION_UNSPECIFIED\x10\x00\x12\x10\n\x0c\x41\x43TION_LIMIT\x10\x01\x12\x0e\n\nACTION_LOG\x10\x02\x12\x0f\n\x0b\x41\x43TION_SKIP\x10\x03\x62\x06proto3"
12
+
13
+ pool = ::Google::Protobuf::DescriptorPool.generated_pool
14
+ return unless Labkit::RateLimit::Config::VendoredProto.register(pool, descriptor_data)
15
+
16
+ module Gitlab
17
+ module Ratelimit
18
+ module Config
19
+ module V1
20
+ AvailableLimiters = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.AvailableLimiters").msgclass
21
+ LimiterSpec = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.LimiterSpec").msgclass
22
+ RateLimits = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.RateLimits").msgclass
23
+ RuleList = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.RuleList").msgclass
24
+ Rule = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.Rule").msgclass
25
+ PublishRulesRequest = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.PublishRulesRequest").msgclass
26
+ RuleNames = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.RuleNames").msgclass
27
+ Matcher = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.Matcher").msgclass
28
+ StringList = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.StringList").msgclass
29
+ MatcherList = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.MatcherList").msgclass
30
+ Action = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitlab.ratelimit.config.v1.Action").enummodule
31
+ end
32
+ end
33
+ end
34
+ end
@@ -0,0 +1,105 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "google/protobuf/descriptor_pb"
4
+
5
+ module Labkit
6
+ module RateLimit
7
+ module Config
8
+ # VendoredProto registers the vendored gitlab.ratelimit.config.v1 file in
9
+ # the global descriptor pool. The gitlab-ratelimit-config-v1 gem from
10
+ # gitlab-org/protos registers the same file name, unguarded, and a second
11
+ # registration raises. So the gem is loaded first whenever it is
12
+ # installed, and the vendored copy is then compared with it: the same
13
+ # schema means the gem's registration is used, a different one is a boot
14
+ # error, because two versions of one schema in one process would make the
15
+ # loader read fields the other copy does not know.
16
+ module VendoredProto
17
+ # The gem registered a schema that differs from the vendored copy, or
18
+ # one this google-protobuf cannot compare.
19
+ Mismatch = Class.new(StandardError)
20
+
21
+ PACKAGE = "gitlab.ratelimit.config.v1"
22
+ PROBE = "#{PACKAGE}.RateLimits".freeze
23
+ GEM_FILE = "gitlab/ratelimit/config/v1/config_pb"
24
+
25
+ class << self
26
+ # @param pool [Google::Protobuf::DescriptorPool]
27
+ # @param serialized [String] a serialized FileDescriptorProto
28
+ # @return [Boolean] true when this call registered the file and the
29
+ # vendored file must define its constants; false when the gem's
30
+ # registration is in use and the vendored file must stop loading
31
+ # @raise [Mismatch] when the gem registered a different schema
32
+ def register(pool, serialized)
33
+ load_gem_first
34
+
35
+ existing = pool.lookup(PROBE)
36
+ if existing.nil?
37
+ pool.add_serialized_file(serialized)
38
+ return true
39
+ end
40
+
41
+ check_same_schema!(existing.file_descriptor, serialized)
42
+ false
43
+ end
44
+
45
+ private
46
+
47
+ # Ruby's require loads the gem's file once, so whichever side comes
48
+ # first, the gem registers before the vendored copy is compared. Only
49
+ # an absent gem is expected; a LoadError from inside the gem's own
50
+ # file is a real failure and propagates.
51
+ def load_gem_first
52
+ require GEM_FILE
53
+ rescue ::LoadError => e
54
+ raise unless e.path == GEM_FILE
55
+ end
56
+
57
+ # The gem may be ahead of the vendored copy: an additive release adds
58
+ # fields the loader does not read, and that is fine. The gem may not
59
+ # be behind it: the loader would then read a field the gem's schema
60
+ # does not have.
61
+ def check_same_schema!(file, serialized)
62
+ # FileDescriptor#to_proto arrived in google-protobuf 4.30. The gem
63
+ # keeps its 3.25 floor for consumers that never reach this path, so
64
+ # the requirement is raised here, where it applies.
65
+ raise Mismatch, "#{PACKAGE} is registered by another gem and this google-protobuf cannot compare schemas; rate limit config needs google-protobuf >= 4.30" unless file.respond_to?(:to_proto)
66
+
67
+ registered = signature(file.to_proto)
68
+ vendored = signature(Google::Protobuf::FileDescriptorProto.decode(serialized))
69
+ missing = vendored.reject { |path, sig| registered[path] == sig }.keys
70
+ detail = "lacks or changes #{missing.join(', ')}; update the gitlab-labkit vendored copy or the gem"
71
+ raise Mismatch, "#{PACKAGE} is registered by another gem with a schema that #{detail}" unless missing.empty?
72
+
73
+ extra = registered.keys - vendored.keys
74
+ warn "gitlab-labkit: #{PACKAGE} from another gem is a newer schema (#{extra.join(', ')}); the vendored copy is older" unless extra.empty?
75
+ end
76
+
77
+ # "Message.field" => its wire and API shape, "Enum.VALUE" => number,
78
+ # for every message, nested message and enum. proto3 optional decides
79
+ # whether has_x? exists, the oneof name which fields exclude each
80
+ # other, and json_name what the parser accepts as a key.
81
+ def signature(file)
82
+ out = {}
83
+ walk = lambda do |messages, enums, prefix|
84
+ enums.each { |e| e.value.each { |val| out["#{prefix}#{e.name}.#{val.name}"] = val.number } }
85
+ messages.each do |m|
86
+ m.field.each { |f| out["#{prefix}#{m.name}.#{f.name}"] = field_signature(f, m) }
87
+ walk.call(m.nested_type, m.enum_type, "#{prefix}#{m.name}.")
88
+ end
89
+ end
90
+ walk.call(file.message_type, file.enum_type, "")
91
+ out
92
+ end
93
+
94
+ # protoc numbers oneofs in declaration order, and every proto3
95
+ # optional field adds a synthetic one, so an additive field shifts
96
+ # the index of every oneof after it. The name does not move.
97
+ def field_signature(field, message)
98
+ oneof = field.has_oneof_index? ? message.oneof_decl[field.oneof_index].name : nil
99
+ [field.number, field.type, field.label, field.type_name, field.json_name, field.proto3_optional, oneof]
100
+ end
101
+ end
102
+ end
103
+ end
104
+ end
105
+ end
@@ -0,0 +1,22 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "labkit/rate_limit/config/vendored_proto"
4
+ require "labkit/rate_limit/config/proto/config_pb"
5
+
6
+ module Labkit
7
+ module RateLimit
8
+ # Config loads rate limit rules from YAML files that follow the
9
+ # gitlab.ratelimit.config.v1 schema: the contract an application ships
10
+ # (AvailableLimiters) and the rate limits an operator writes (RateLimits).
11
+ # Loading runs in four steps, parse, schema, contract and rule, and a
12
+ # failure names its step so CI output and the conformance corpus agree
13
+ # across SDKs.
14
+ module Config
15
+ # The generated protobuf classes, vendored from gitlab-org/protos.
16
+ Proto = Gitlab::Ratelimit::Config::V1
17
+
18
+ autoload :Error, "labkit/rate_limit/config/error"
19
+ autoload :Parser, "labkit/rate_limit/config/parser"
20
+ end
21
+ end
22
+ end
@@ -67,6 +67,40 @@ module Labkit
67
67
  return {count, ttl_after}
68
68
  LUA
69
69
 
70
+ # Atomic admission for count_distinct rules carrying admit_counted_values.
71
+ # A member already in the set is admitted without changing it; a new one
72
+ # is added only while SCARD is below the limit. A refused member is never
73
+ # added, so it stays refused on retry and the count does not grow past the
74
+ # limit. Returns post-state {count, TTL, admitted}.
75
+ #
76
+ # The same self-healing as SADD_SCRIPT applies to keys left without TTL.
77
+ # EXPIRE on a missing key does nothing, so a refusal on a missing key (a
78
+ # limit resolving to 0) writes nothing.
79
+ ADMIT_SCRIPT = Labkit::Redis::Script.new(<<~LUA)
80
+ local ttl = ARGV[1]
81
+ local member = ARGV[2]
82
+ local limit = tonumber(ARGV[3])
83
+
84
+ local count = redis.call('SCARD', KEYS[1])
85
+ local admitted = 1
86
+ if redis.call('SISMEMBER', KEYS[1], member) == 0 then
87
+ if count < limit then
88
+ redis.call('SADD', KEYS[1], member)
89
+ count = count + 1
90
+ else
91
+ admitted = 0
92
+ end
93
+ end
94
+
95
+ local ttl_after = redis.call('TTL', KEYS[1])
96
+ if ttl_after < 0 then
97
+ redis.call('EXPIRE', KEYS[1], ttl)
98
+ ttl_after = tonumber(ttl)
99
+ end
100
+
101
+ return {count, ttl_after, admitted}
102
+ LUA
103
+
70
104
  # Atomic ban check + INCR + conditional ban write, for rules carrying
71
105
  # ban_for.
72
106
  #
@@ -262,9 +296,10 @@ module Labkit
262
296
  # here too: check can report a :log rule's evaluation, so excluding them
263
297
  # would make peek answer a different question than check.
264
298
  #
265
- # peek does not need the count_distinct identifier key - it reads SCARD on
266
- # the rule-keyed compound key, which contains the cardinality across all
267
- # members. So missing-key fail-open does not apply here.
299
+ # peek does not require the count_distinct identifier key: it reads SCARD
300
+ # on the rule-keyed compound key, so missing-key fail-open does not apply.
301
+ # An admitting rule also looks the value up when it is present (see
302
+ # peek_admission_rule).
268
303
  def peek_rules(identifier, rule_context, cursor)
269
304
  result = Result.new
270
305
 
@@ -302,6 +337,8 @@ module Labkit
302
337
  resolved_limit = Integer(resolve_value(rule.limit, rule_context))
303
338
  resolved_period = Integer(resolve_value(rule.period, rule_context))
304
339
 
340
+ return evaluate_admission_rule(rule, redis_key, identifier, resolved_limit, resolved_period) if rule.admit_counted_values
341
+
305
342
  # cost is ignored for count_distinct rules: SADD is binary (a member is
306
343
  # either added or not), and the post-add count is SCARD regardless.
307
344
  count, ttl =
@@ -314,6 +351,15 @@ module Labkit
314
351
  build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
315
352
  end
316
353
 
354
+ # The count does not grow past the limit in this mode, so the script's
355
+ # admission decides the block, the way a ban decides it apart from the count.
356
+ # cost is ignored, as for a plain count_distinct rule.
357
+ def evaluate_admission_rule(rule, redis_key, identifier, resolved_limit, resolved_period)
358
+ count, ttl, admitted = admit_with_ttl(redis_key, identifier[rule.count_distinct], resolved_limit, resolved_period)
359
+
360
+ build_evaluation(rule, resolved_limit, resolved_period, count, ttl, exceeded: !admitted)
361
+ end
362
+
317
363
  def peek_rule(rule, identifier, rule_context)
318
364
  return peek_ban_rule(rule, identifier, rule_context) if rule.ban_for
319
365
 
@@ -321,10 +367,22 @@ module Labkit
321
367
  resolved_limit = Integer(resolve_value(rule.limit, rule_context))
322
368
  resolved_period = Integer(resolve_value(rule.period, rule_context))
323
369
 
370
+ return peek_admission_rule(rule, redis_key, identifier, resolved_limit, resolved_period) if rule.admit_counted_values
371
+
324
372
  count, ttl = rule.count_distinct ? scard_with_ttl(redis_key) : read_with_ttl(redis_key)
325
373
  build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
326
374
  end
327
375
 
376
+ # An admission is decided before the script writes anything, so with the
377
+ # value peek reports exactly what check would. Without it, check skips the
378
+ # rule, but peek answers for a value that is not counted yet.
379
+ def peek_admission_rule(rule, redis_key, identifier, resolved_limit, resolved_period)
380
+ value = identifier[rule.count_distinct] unless missing_count_distinct_value?(rule, identifier)
381
+ count, ttl, counted = scard_with_ttl(redis_key, member: value)
382
+
383
+ build_evaluation(rule, resolved_limit, resolved_period, count, ttl, exceeded: !counted && count >= resolved_limit)
384
+ end
385
+
328
386
  def evaluate_ban_rule(rule, identifier, cost, rule_context)
329
387
  resolved_limit = Integer(resolve_value(rule.limit, rule_context))
330
388
  resolved_period = Integer(resolve_value(rule.period, rule_context))
@@ -382,7 +440,7 @@ module Labkit
382
440
  Result::Evaluation.new(rule: rule, exceeded: banned, info: info)
383
441
  end
384
442
 
385
- def build_evaluation(rule, resolved_limit, resolved_period, count, ttl)
443
+ def build_evaluation(rule, resolved_limit, resolved_period, count, ttl, exceeded: count > resolved_limit)
386
444
  info = Result::Info.new(
387
445
  resolved_limit: resolved_limit, resolved_period: resolved_period,
388
446
  count: count,
@@ -390,12 +448,13 @@ module Labkit
390
448
  reset_at: Time.now.utc + (ttl >= 0 ? ttl : resolved_period)
391
449
  )
392
450
 
393
- Result::Evaluation.new(rule: rule, exceeded: count > resolved_limit, info: info)
451
+ Result::Evaluation.new(rule: rule, exceeded: exceeded, info: info)
394
452
  end
395
453
 
396
454
  # RateLimit-Remaining must not exceed RateLimit-Limit, and a counter can
397
455
  # sit below zero if an older client credited it. A limit resolving to a
398
- # non-positive value means "deny all", so the ceiling floors at zero too.
456
+ # non-positive value means "deny all" ("deny new values" on an
457
+ # admit_counted_values rule), so the ceiling floors at zero too.
399
458
  def remaining_of(resolved_limit, count)
400
459
  ceiling = [resolved_limit, 0].max
401
460
  (ceiling - count).clamp(0, ceiling)
@@ -523,17 +582,28 @@ module Labkit
523
582
  end
524
583
  end
525
584
 
526
- # Pipelined SCARD + TTL. SCARD on a missing key returns 0, so no
527
- # explicit nil handling is needed (unlike GET in read_with_ttl).
528
- # SCARD is integer-valued; coerced to Float for type uniformity with
529
- # the INCR path so callers see a consistent count type.
530
- def scard_with_ttl(redis_key)
585
+ # Atomic admission. See ADMIT_SCRIPT.
586
+ def admit_with_ttl(redis_key, member, limit, period)
587
+ member_str = encode_char_value(member.to_s)
588
+ @redis.with do |conn|
589
+ raw_count, ttl, admitted = ADMIT_SCRIPT.eval(conn, keys: [redis_key], argv: [period, member_str, limit])
590
+ [Float(raw_count), Integer(ttl), admitted == 1]
591
+ end
592
+ end
593
+
594
+ # Pipelined SCARD + TTL, plus SISMEMBER when given a member to look up
595
+ # (admitting rules), returned as the third element. SCARD on a missing key
596
+ # returns 0, so no explicit nil handling is needed (unlike GET in
597
+ # read_with_ttl). SCARD is integer-valued; coerced to Float for type
598
+ # uniformity with the INCR path so callers see a consistent count type.
599
+ def scard_with_ttl(redis_key, member: nil)
531
600
  @redis.with do |conn|
532
- scard, ttl = conn.pipelined do |pipe|
601
+ scard, ttl, counted = conn.pipelined do |pipe|
533
602
  pipe.scard(redis_key)
534
603
  pipe.ttl(redis_key)
604
+ pipe.sismember(redis_key, encode_char_value(member.to_s)) unless member.nil?
535
605
  end
536
- [Float(scard), ttl]
606
+ [Float(scard), ttl, counted == true]
537
607
  end
538
608
  end
539
609
 
@@ -35,6 +35,8 @@ module Labkit
35
35
  # @param cost [Numeric] amount to add to the counter. Defaults to 1
36
36
  # (count-mode). Pass a non-1 Numeric for cost-mode counters such as
37
37
  # resource-usage limits; passing 0 reads the counter without writing.
38
+ # count_distinct rules ignore cost, so a cost of 0 still adds the value
39
+ # (or admits it, on an admit_counted_values rule).
38
40
  # A negative cost would credit quota back, so the check fails open and
39
41
  # logs instead of counting.
40
42
  # @param rule_context [Hash, nil] optional per-request context passed to
@@ -60,8 +62,10 @@ module Labkit
60
62
  #
61
63
  # When the underlying Redis key does not exist yet, the result reports
62
64
  # count=0, exceeded=false, and remaining=resolved_limit; matched? is
63
- # still true because the rule applied. On Redis error the result fails
64
- # open identically to {#check}.
65
+ # still true because the rule applied. The exception is an
66
+ # admit_counted_values rule whose limit resolves to 0 or less, which
67
+ # refuses every new value and so reports exceeded. On Redis error the result
68
+ # fails open identically to {#check}.
65
69
  #
66
70
  # @param identifier [Identifier, Hash] caller attributes for this request
67
71
  # @param rule_context [Hash, nil] see {#check}
@@ -15,7 +15,8 @@ module Labkit
15
15
  # from the same evaluations are not #==.
16
16
  #
17
17
  # matched? - true if at least one rule's match conditions were satisfied
18
- # exceeded? - true if the reported evaluation's counter exceeded its limit
18
+ # exceeded? - true if the reported evaluation is over its limit (see
19
+ # Result::Evaluation#exceeded)
19
20
  # action - the outcome: what the caller should do
20
21
  # :block = a :limit rule matched and is over its limit
21
22
  # :allow = everything else, including an exceeded :log rule
@@ -127,7 +128,8 @@ module Labkit
127
128
 
128
129
  # The outcome of counting one matched rule.
129
130
  # rule - the evaluated Rule
130
- # exceeded - whether the post-increment count exceeded the resolved limit
131
+ # exceeded - whether the rule is over its limit: the post-increment count
132
+ # exceeded it, a ban holds, or an admitting rule refused the value
131
133
  # info - Result::Info with the per-window counters
132
134
  #
133
135
  # Evaluations order by constraint: a blocking evaluation (a :limit rule over
@@ -165,7 +167,8 @@ module Labkit
165
167
  headers = {}
166
168
  headers["RateLimit-Name"] = rule.name
167
169
  # Floored at 0 like remaining: a callable limit can resolve below zero
168
- # to deny all, and an older client could credit a counter below zero.
170
+ # to deny all (deny new values on an admit_counted_values rule), and an
171
+ # older client could credit a counter below zero.
169
172
  headers["RateLimit-Limit"] = [info.resolved_limit.to_i, 0].max.to_s
170
173
  headers["RateLimit-Observed"] = [info.count.to_i, 0].max.to_s
171
174
  headers["RateLimit-Remaining"] = info.remaining.to_i.to_s
@@ -31,11 +31,22 @@ module Labkit
31
31
  # the (characteristics-bucketed) period, backed by a Redis SET.
32
32
  # When nil (default), the rule counts the number of calls,
33
33
  # backed by INCR. The named key must not overlap +characteristics+.
34
+ # admit_counted_values - optional Boolean, only valid with +count_distinct+ and
35
+ # rejected on :skip.
36
+ # When true, the rule counts only the values it admits: a value
37
+ # already in the window's set is allowed even when the set is
38
+ # full, and a new value is added only while the count is below
39
+ # the limit, so a value this rule refuses is never counted.
40
+ # Rules count a request up to the one that blocks it, so put
41
+ # admitting rules after counter rules. Default false; nil
42
+ # means not set.
34
43
  #
35
44
  # +name+ must be a lowercase alphanumeric-and-underscore string of at most 64
36
45
  # characters. It is used as the middle segment of every Redis counter key for
37
46
  # this rule, so changing a rule's name mid-window abandons its in-flight counters.
38
- Rule = Data.define(:name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for) do
47
+ Rule = Data.define(
48
+ :name, :match, :limit, :period, :action, :characteristics, :count_distinct, :ban_for, :admit_counted_values
49
+ ) do
39
50
  # ban_for changes the accounting; action still decides who gets blocked.
40
51
  # A :skip rule counts nothing, so it has no limit to cross and no ban to
41
52
  # write.
@@ -58,7 +69,8 @@ module Labkit
58
69
 
59
70
  # A static limit under 1 blocks every request, and a period under a second
60
71
  # reaches EXPIRE as 0, deleting the counter on every write. A :skip rule
61
- # counts nothing, and a callable may still resolve to 0 to mean "deny all".
72
+ # counts nothing, and a callable may still resolve to 0 to mean "deny all"
73
+ # ("deny new values" on an admit_counted_values rule).
62
74
  def self.normalize_numeric(field, value, action_sym)
63
75
  return value if action_sym == :skip || value.respond_to?(:call)
64
76
  return value if value.is_a?(Numeric) && value >= 1
@@ -66,6 +78,19 @@ module Labkit
66
78
  raise ArgumentError, "#{field} must be a Numeric of at least 1 or a callable, got #{value.inspect}"
67
79
  end
68
80
 
81
+ # Admission only means something for a SET-backed rule, where a value can
82
+ # already be counted. A counter has no members to admit, and a :skip rule
83
+ # counts nothing.
84
+ def self.normalize_admit_counted_values(value, action_sym, count_distinct_sym)
85
+ # nil means "not set", like count_distinct and ban_for.
86
+ value = false if value.nil?
87
+ raise ArgumentError, "admit_counted_values must be true, false, or nil, got #{value.inspect}" unless [true, false].include?(value)
88
+ raise ArgumentError, "admit_counted_values is not valid on a :skip rule" if value && action_sym == :skip
89
+ raise ArgumentError, "admit_counted_values requires count_distinct" if value && count_distinct_sym.nil?
90
+
91
+ value
92
+ end
93
+
69
94
  def self.normalize_count_distinct(value, characteristics_arr)
70
95
  sym =
71
96
  case value
@@ -81,9 +106,9 @@ module Labkit
81
106
  sym
82
107
  end
83
108
 
84
- def initialize(
109
+ def initialize( # rubocop:disable Metrics/ParameterLists -- one keyword per Rule field; the public API grows by keywords with defaults
85
110
  name:, limit:, period:, characteristics:,
86
- match: {}, action: :limit, count_distinct: nil, ban_for: nil
111
+ match: {}, action: :limit, count_distinct: nil, ban_for: nil, admit_counted_values: false
87
112
  )
88
113
  raise ArgumentError, "name must be a String or Symbol, got #{name.class}" unless name.is_a?(String) || name.is_a?(Symbol)
89
114
 
@@ -109,7 +134,8 @@ module Labkit
109
134
  action: action_sym,
110
135
  characteristics: characteristics_arr,
111
136
  count_distinct: count_distinct_sym,
112
- ban_for: self.class.normalize_ban_for(ban_for, action_sym, count_distinct_sym)
137
+ ban_for: self.class.normalize_ban_for(ban_for, action_sym, count_distinct_sym),
138
+ admit_counted_values: self.class.normalize_admit_counted_values(admit_counted_values, action_sym, count_distinct_sym)
113
139
  )
114
140
  end
115
141
  end
@@ -26,6 +26,7 @@ module Labkit
26
26
  autoload :Limiter, "labkit/rate_limit/limiter"
27
27
  autoload :Metrics, "labkit/rate_limit/metrics"
28
28
  autoload :BlockSummary, "labkit/rate_limit/block_summary"
29
+ autoload :Config, "labkit/rate_limit/config"
29
30
 
30
31
  class << self
31
32
  def configure
@@ -0,0 +1,36 @@
1
+ #!/usr/bin/env bash
2
+ # Copies the generated Ruby code for gitlab.ratelimit.config.v1 from gitlab-org/protos.
3
+ # A published gem cannot depend on the protos Bundler git gem, so the file is
4
+ # vendored and CI checks that it matches PROTOS_REF (job vendored-proto-check).
5
+ set -euo pipefail
6
+
7
+ # Pinned protos module tag. Bump it when a newer schema release is needed.
8
+ PROTOS_REF="${PROTOS_REF:-gen/ruby/gitlab/ratelimit/config/v1/v0.1.0}"
9
+ SOURCE_PATH="gen/ruby/gitlab/ratelimit/config/v1/lib/gitlab/ratelimit/config/v1/config_pb.rb"
10
+ TARGET="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib/labkit/rate_limit/config/proto/config_pb.rb"
11
+
12
+ url="https://gitlab.com/gitlab-org/protos/-/raw/${PROTOS_REF}/${SOURCE_PATH}"
13
+ tmp="$(mktemp)"
14
+ trap 'rm -f "$tmp"' EXIT
15
+ curl --fail --silent --show-error --location "$url" -o "$tmp"
16
+
17
+ # The generated file registers gitlab/ratelimit/config/v1/config.proto in the
18
+ # global descriptor pool. The protos gem registers the same file name, so the
19
+ # vendored copy goes through VendoredProto.register, which loads the gem first
20
+ # when it is installed and then compares schemas. When the gem's registration
21
+ # is in use the file returns early, so its constants are not defined twice.
22
+ guard='return unless Labkit::RateLimit::Config::VendoredProto.register(pool, descriptor_data)'
23
+ {
24
+ echo "# frozen_string_literal: true"
25
+ echo "# Vendored from gitlab-org/protos at ${PROTOS_REF}, ${SOURCE_PATH}."
26
+ echo "# Do not edit. Run scripts/vendor-ratelimit-proto.sh to update."
27
+ grep -v '^# frozen_string_literal: true$' "$tmp" |
28
+ sed -e "s|^require 'google/protobuf'\$|require 'google/protobuf'\nrequire 'labkit/rate_limit/config/vendored_proto'|" \
29
+ -e "s|^pool.add_serialized_file(descriptor_data)\$|${guard}|"
30
+ } >"$TARGET"
31
+ if ! grep -q 'VendoredProto.register' "$TARGET" ||
32
+ ! grep -qx "require 'labkit/rate_limit/config/vendored_proto'" "$TARGET"; then
33
+ echo "vendor: the generated file no longer requires VendoredProto or registers the descriptor as expected; update the sed in this script" >&2
34
+ exit 1
35
+ fi
36
+ echo "vendored ${SOURCE_PATH}@${PROTOS_REF} -> ${TARGET#"$PWD"/}"
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: gitlab-labkit
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.8.0
4
+ version: 5.10.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrew Newdigate
@@ -570,7 +570,6 @@ files:
570
570
  - ".gitlab-ci-asdf-versions.yml"
571
571
  - ".gitlab-ci-other-versions.yml"
572
572
  - ".gitlab-ci.yml"
573
- - ".gitlab/CODEOWNERS"
574
573
  - ".gitlab/duo/agent-config.yml"
575
574
  - ".gitlab/merge_request_templates/default.md"
576
575
  - ".gitleaks.toml"
@@ -652,6 +651,11 @@ files:
652
651
  - lib/labkit/rate_limit.rb
653
652
  - lib/labkit/rate_limit/README.md
654
653
  - lib/labkit/rate_limit/block_summary.rb
654
+ - lib/labkit/rate_limit/config.rb
655
+ - lib/labkit/rate_limit/config/error.rb
656
+ - lib/labkit/rate_limit/config/parser.rb
657
+ - lib/labkit/rate_limit/config/proto/config_pb.rb
658
+ - lib/labkit/rate_limit/config/vendored_proto.rb
655
659
  - lib/labkit/rate_limit/configuration.rb
656
660
  - lib/labkit/rate_limit/evaluator.rb
657
661
  - lib/labkit/rate_limit/identifier.rb
@@ -720,6 +724,7 @@ files:
720
724
  - renovate.json
721
725
  - scripts/prepare-dev-env.sh
722
726
  - scripts/update-asdf-version-variables.sh
727
+ - scripts/vendor-ratelimit-proto.sh
723
728
  homepage: https://gitlab.com/gitlab-org/labkit-ruby
724
729
  licenses:
725
730
  - MIT
data/.gitlab/CODEOWNERS DELETED
@@ -1 +0,0 @@
1
- * @mkaeppler @ayufan @hmerscher @sankalp_gl @hardikgala @nindurkar @ashs2