gitlab-labkit 5.1.2 → 5.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/hooks/precommit-validate.sh +48 -0
- data/.claude/settings.json +15 -0
- data/.copier-answers.yml +2 -1
- data/.gitignore +0 -1
- data/.gitlab/CODEOWNERS +1 -1
- data/.gitlab/duo/agent-config.yml +7 -7
- data/.gitlab-ci.yml +24 -3
- data/.mise.toml +3 -0
- data/.opencode/plugins/precommit-validate.ts +226 -0
- data/.pre-commit-config.yaml +3 -2
- data/.rubocop_todo.yml +2 -17
- data/AGENTS.md +173 -0
- data/CLAUDE.md +1 -0
- data/CODEOWNERS +1 -1
- data/CONTRIBUTING.md +1 -1
- data/Gemfile +5 -0
- data/README.md +6 -19
- data/gitlab-labkit.gemspec +1 -0
- data/lib/labkit/fields.rb +29 -2
- data/lib/labkit/json_schema/README.md +1 -1
- data/lib/labkit/rate_limit/README.md +82 -7
- data/lib/labkit/rate_limit/matcher.rb +138 -28
- data/lib/labkit/rate_limit/result.rb +32 -9
- data/lib/labkit/rate_limit/rule.rb +14 -4
- data/lib/labkit/rspec/matchers/user_experience_matchers.rb +2 -2
- data/performance.yml +70 -0
- data/scripts/prepare-dev-env.sh +24 -1
- metadata +21 -2
- data/.env.example.sh +0 -7
data/AGENTS.md
ADDED
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
<<<<<<< before updating
|
|
2
|
+
# AGENTS.md
|
|
3
|
+
|
|
4
|
+
Instructions for AI agents and automated tooling working in this repository.
|
|
5
|
+
|
|
6
|
+
## Project context
|
|
7
|
+
|
|
8
|
+
LabKit-Ruby (`gitlab-labkit` on RubyGems) is GitLab's Ruby SDK for cross-project infrastructure concerns: request context and correlation, structured logging, tracing, metrics, SLIs, and rate limiting.
|
|
9
|
+
It is consumed by GitLab Rails and other Ruby services, so every public API is a compatibility commitment.
|
|
10
|
+
|
|
11
|
+
It is one of several LabKit SDKs.
|
|
12
|
+
The others are Go ([labkit](https://gitlab.com/gitlab-org/labkit)), Rust ([labkit-rs](https://gitlab.com/gitlab-org/rust/labkit-rs)), and Python ([labkit-python](https://gitlab.com/gitlab-org/labkit-python)).
|
|
13
|
+
Behaviour that services observe, and anything written to a shared store, must be the same in every SDK.
|
|
14
|
+
The contract lives in [labkit-spec](https://gitlab.com/gitlab-org/quality/tooling/labkit-spec), and its conformance harness is the judge.
|
|
15
|
+
|
|
16
|
+
### Layout
|
|
17
|
+
|
|
18
|
+
One gem, one `Labkit` namespace, one directory per capability under `lib/labkit/`.
|
|
19
|
+
`lib/gitlab-labkit.rb` autoloads each capability unless the table says otherwise, so requiring the gem stays cheap and callers load only what they touch.
|
|
20
|
+
|
|
21
|
+
| Module | Directory | Notes |
|
|
22
|
+
| --- | --- | --- |
|
|
23
|
+
| `Labkit::Context`, `Labkit::Correlation` | `lib/labkit/context.rb`, `lib/labkit/correlation/` | Request context for logs and propagation |
|
|
24
|
+
| `Labkit::Logging` | `lib/labkit/logging/` | Log sanitisation and JSON formatting |
|
|
25
|
+
| `Labkit::Tracing` | `lib/labkit/tracing/` | OpenTelemetry and legacy OpenTracing |
|
|
26
|
+
| `Labkit::Metrics` | `lib/labkit/metrics/` | Prometheus client wrappers |
|
|
27
|
+
| `Labkit::ApplicationSli`, `Labkit::UserExperienceSli` | `lib/labkit/application_sli/`, `lib/labkit/user_experience_sli/` | Apdex and error-rate SLIs |
|
|
28
|
+
| `Labkit::RateLimit` | `lib/labkit/rate_limit/` | Rules-based, Redis-backed rate limiting, see below |
|
|
29
|
+
| `Labkit::Middleware` | `lib/labkit/middleware/` | Rack and Sidekiq middleware for context, tracing, and SLIs |
|
|
30
|
+
| `Labkit::JsonSchema` | `lib/labkit/json_schema/` | `RefResolver` for remote `$ref` in JSON schemas. Not autoloaded; required by `UserExperienceSli` |
|
|
31
|
+
| `Labkit::FIPS` | `lib/labkit/fips.rb` | FIPS mode detection and OpenSSL digest substitution |
|
|
32
|
+
| `Labkit::System` | `lib/labkit/system.rb` | Monotonic clock helper used by metrics, tracing, and logging |
|
|
33
|
+
| `Labkit::Redis` | `lib/labkit/redis/` | Redis helpers shared by capabilities |
|
|
34
|
+
| `Labkit::Fields` | `lib/labkit/fields.rb` | Generated by labkit-spec CI from `schema/fields.yaml`. Change the field there, not here |
|
|
35
|
+
| `Labkit::RSpec` | `lib/labkit/rspec/` | Matchers for consumers' test suites. Not autoloaded; `require "labkit/rspec/matchers"` |
|
|
36
|
+
|
|
37
|
+
`lib/labkit/*_publisher.rb` hook Excon, HTTPClient, and Net::HTTP to publish outbound request notifications.
|
|
38
|
+
|
|
39
|
+
Modules with their own README (`lib/labkit/<module>/README.md`) document their public API there.
|
|
40
|
+
Update that README in the same MR as a behaviour change.
|
|
41
|
+
|
|
42
|
+
## Commands
|
|
43
|
+
|
|
44
|
+
```shell
|
|
45
|
+
bundle install
|
|
46
|
+
bundle exec rake verify # rspec + rubocop, what CI runs
|
|
47
|
+
bundle exec rake fix # rubocop autocorrect
|
|
48
|
+
bundle exec rspec spec/labkit/<module> # one module's specs
|
|
49
|
+
bundle exec rake console # IRB with the gem loaded
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Specs that need Redis use the instance at `LABKIT_TEST_REDIS_URL`, default `redis://localhost:6390/0`.
|
|
53
|
+
If nothing answers there and `LABKIT_TEST_REDIS_URL` is unset, the suite starts one with `docker compose up -d redis` and stops it at exit.
|
|
54
|
+
A Redis already listening on that URL is used as is and left running.
|
|
55
|
+
Autostart is off when `LABKIT_TEST_REDIS_URL` is set, even to the default value, when `CI` is set, or when `LABKIT_TEST_REDIS_NO_AUTOSTART=1` is set.
|
|
56
|
+
In those cases, start Redis yourself.
|
|
57
|
+
|
|
58
|
+
Style is `gitlab-styles` via RuboCop with `NewCops: enable`.
|
|
59
|
+
Do not edit `.rubocop_todo.yml` to silence a new offence; fix the code.
|
|
60
|
+
Tool versions come from `.tool-versions` through mise.
|
|
61
|
+
|
|
62
|
+
## CI
|
|
63
|
+
|
|
64
|
+
- `rspec` runs the suite with a Redis service.
|
|
65
|
+
- `ruby-versions` and `legacy-ruby-versions` repeat it on Ruby 3.4 and 3.3. The versions are set in `.gitlab-ci.yml`, not read from the gemspec.
|
|
66
|
+
- `conformance-spec` triggers a downstream pipeline in labkit-spec that checks out this MR's commit and runs the conformance harness against it.
|
|
67
|
+
It runs when `lib/`, the gemspec, the Gemfile, or `.gitlab-ci.yml` change, and a red downstream run blocks the merge.
|
|
68
|
+
Renovate's bot user cannot start pipelines in labkit-spec, so on its MRs the job has `allow_failure: true` and a human rerun is the gate.
|
|
69
|
+
- Danger runs the shared `gitlab-dangerfiles` checks except changelog and commit message, which semantic-release owns.
|
|
70
|
+
|
|
71
|
+
## Commits, MRs, and releases
|
|
72
|
+
|
|
73
|
+
Conventional Commits: `feat`, `fix`, `perf`, `revert`, `docs`, `refactor`, `test`, `chore`, `ci`, `style`.
|
|
74
|
+
Scope is the module, for example `fix(rate_limit): ...`.
|
|
75
|
+
|
|
76
|
+
Releases are cut from `master` and the `N-N-stable` branches by semantic-release with the `conventionalcommits` preset, so the commit type decides the version bump.
|
|
77
|
+
|
|
78
|
+
| Type | Release |
|
|
79
|
+
| --- | --- |
|
|
80
|
+
| `feat!`, or any type with a `BREAKING CHANGE:` footer | major |
|
|
81
|
+
| `feat` | minor |
|
|
82
|
+
| `fix`, `perf`, `revert` | patch |
|
|
83
|
+
| `docs`, `refactor`, `test`, `chore`, `ci`, `style` | none |
|
|
84
|
+
|
|
85
|
+
There is no hand-edited changelog.
|
|
86
|
+
|
|
87
|
+
Public APIs stay backward compatible.
|
|
88
|
+
Add keyword arguments with defaults; do not add required parameters or rename public methods.
|
|
89
|
+
Keep an MR to one concern, and separate mechanical changes such as renames from behavioural ones.
|
|
90
|
+
|
|
91
|
+
## Rate limiting
|
|
92
|
+
|
|
93
|
+
`Labkit::RateLimit` is documented in `lib/labkit/rate_limit/README.md`.
|
|
94
|
+
It is the one capability that shares state across SDKs: services in any language enforce the same limits against the same Redis.
|
|
95
|
+
That makes the Redis key shape and the Lua scripts as much a contract as the HTTP headers, so changes here follow stricter rules than the rest of the gem.
|
|
96
|
+
|
|
97
|
+
### Where each SDK keeps it
|
|
98
|
+
|
|
99
|
+
| SDK | Repository | Module | Tests | Status |
|
|
100
|
+
| --- | --- | --- | --- | --- |
|
|
101
|
+
| Ruby (this repository) | gitlab-org/ruby/gems/labkit-ruby | `lib/labkit/rate_limit/` | `spec/labkit/rate_limit/` | released |
|
|
102
|
+
| Go | gitlab-org/labkit | `v2/ratelimit/` | `v2/ratelimit/*_test.go`, run with `go -C v2 test ./ratelimit/...` | released |
|
|
103
|
+
| Rust | gitlab-org/rust/labkit-rs | in progress | | not released, skip |
|
|
104
|
+
| Python | gitlab-org/labkit-python | in progress | | not released, skip |
|
|
105
|
+
| Spec | gitlab-org/quality/tooling/labkit-spec | `conformance/README.md` ("v1 decisions"), `conformance/ratelimit_scenarios/*.json`, `conformance/aspect_ratelimit.go` | `conformance/` | judge |
|
|
106
|
+
|
|
107
|
+
### The three layers
|
|
108
|
+
|
|
109
|
+
Every SDK must agree on three layers.
|
|
110
|
+
This section defines the layers; the current values (key grammar, header names, counting rules, metric names, log fields) live in labkit-spec, in the "v1 decisions" list in `conformance/README.md` and the scenarios under `conformance/ratelimit_scenarios/`.
|
|
111
|
+
If this file and labkit-spec disagree, labkit-spec wins.
|
|
112
|
+
|
|
113
|
+
1. **Storage contract.**
|
|
114
|
+
Anything written to or read from Redis: key shape and prefix, how values are encoded, TTL rules, and the Lua scripts.
|
|
115
|
+
Two SDKs sharing one Redis with different storage rules keep two counters for one limit.
|
|
116
|
+
Never change this layer in one SDK.
|
|
117
|
+
It changes in labkit-spec first, then every SDK follows.
|
|
118
|
+
2. **Observable contract.**
|
|
119
|
+
Anything a caller or an operator can see: response status and headers, the counting rule that decides a block, log line messages and field sets, metric families and labels.
|
|
120
|
+
Also changes in labkit-spec first.
|
|
121
|
+
3. **Behaviour within the contract.**
|
|
122
|
+
Input validation and edge cases the contract already implies, where one SDK does something the contract does not allow.
|
|
123
|
+
These may be fixed SDK by SDK, and every SDK should end up the same.
|
|
124
|
+
|
|
125
|
+
A change that mixes layers is governed by the strictest one: storage, then observable, then behaviour.
|
|
126
|
+
When a change does not fit one layer clearly, say so in the MR instead of picking one.
|
|
127
|
+
|
|
128
|
+
### Rules for rate-limit changes
|
|
129
|
+
|
|
130
|
+
- A change to layer 1 or 2 opens a labkit-spec MR first and waits for it to merge.
|
|
131
|
+
- A change to layer 3 should be checked against the Go SDK.
|
|
132
|
+
If Go has the same gap, say so in the MR so it can be ported.
|
|
133
|
+
- When a review finds a divergence between SDKs, a conformance scenario in labkit-spec must follow in the same week, so the harness sees the gap next time.
|
|
134
|
+
The author of the MR where the divergence was found owns this.
|
|
135
|
+
Mention the sync agent on the review thread to have it draft the scenario MR; if it cannot, open a labkit-spec issue and link it from the thread.
|
|
136
|
+
- Never edit Lua scripts, key construction, header names, metric names, or log field names to fix a Ruby-only bug.
|
|
137
|
+
If a fix seems to need that, it is a contract change.
|
|
138
|
+
=======
|
|
139
|
+
# Agent guidelines
|
|
140
|
+
|
|
141
|
+
This project uses [Conventional Commits](https://www.conventionalcommits.org/).
|
|
142
|
+
Commit messages drive automated [semantic releases](https://semantic-release.gitbook.io/)
|
|
143
|
+
(see `.releaserc.json`), so use `feat:`, `fix:`, etc. prefixes to ensure the
|
|
144
|
+
correct version bump and changelog entry.
|
|
145
|
+
|
|
146
|
+
## AI-authored GitLab comments
|
|
147
|
+
|
|
148
|
+
Wrap the full body of any comment, note, or reply you post to GitLab (issues,
|
|
149
|
+
merge requests, epics — via `glab`/`glab api`) in `<:robot:>` / `</:robot:>`
|
|
150
|
+
tags (GitLab emoji shortcode, renders as 🤖), so readers can tell at a glance
|
|
151
|
+
that it's AI-generated:
|
|
152
|
+
|
|
153
|
+
```
|
|
154
|
+
<:robot:>
|
|
155
|
+
|
|
156
|
+
...full comment body...
|
|
157
|
+
|
|
158
|
+
</:robot:>
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
This applies to newly posted comments. It does not apply to code, commit
|
|
162
|
+
messages, or MR/issue descriptions unless asked.
|
|
163
|
+
|
|
164
|
+
<!--
|
|
165
|
+
####################################################################
|
|
166
|
+
# 🌊✨ PLEASE ADD PROJECT SPECIFIC RULES BELOW THIS LINE ✨🌊 #
|
|
167
|
+
# The section above is managed by the common-template-copier #
|
|
168
|
+
# template and is updated automatically on `copier update`. #
|
|
169
|
+
# Add your project-specific instructions below this line so #
|
|
170
|
+
# they are preserved when the template is updated. #
|
|
171
|
+
####################################################################
|
|
172
|
+
-->
|
|
173
|
+
>>>>>>> after updating
|
data/CLAUDE.md
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
@AGENTS.md
|
data/CODEOWNERS
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
# CODEOWNERS is used to lookup assignees for
|
|
2
2
|
# Renovate Bot dependency change Merge Requests.
|
|
3
3
|
# https://docs.renovatebot.com/configuration-options/#assigneesfromcodeowners
|
|
4
|
-
* @
|
|
4
|
+
* @ayufan @hmerscher @e_forbes @M_Alvarez @mwoolf @sankalp_gl @hardikgala @nindurkar @ashs2
|
data/CONTRIBUTING.md
CHANGED
|
@@ -6,7 +6,7 @@ Except for the license granted herein to GitLab Inc. and recipients of software
|
|
|
6
6
|
distributed by GitLab Inc., you reserve all right, title, and interest in and to
|
|
7
7
|
your Contributions.
|
|
8
8
|
|
|
9
|
-
All contributions are subject to the Developer Certificate of Origin and license set out at [docs.gitlab.com/
|
|
9
|
+
All contributions are subject to the Developer Certificate of Origin and license set out at [docs.gitlab.com/legal/developer_certificate_of_origin](https://docs.gitlab.com/legal/developer_certificate_of_origin/).
|
|
10
10
|
|
|
11
11
|
_This notice should stay as the first item in the CONTRIBUTING.md file._
|
|
12
12
|
|
data/Gemfile
CHANGED
|
@@ -3,3 +3,8 @@
|
|
|
3
3
|
source "https://rubygems.org"
|
|
4
4
|
|
|
5
5
|
gemspec
|
|
6
|
+
|
|
7
|
+
# Exports RSpec results to the DX Observer when GLCI_OBSERVER_AUTH_TOKEN is set.
|
|
8
|
+
# Exporter source: https://gitlab.com/gitlab-org/quality/analytics/test-metrics-exporters
|
|
9
|
+
# Owned by: @gitlab-org/quality/analytics (group::development analytics)
|
|
10
|
+
gem 'gitlab-rspec-metrics-exporter', '~> 0.2', groups: [:development, :test]
|
data/README.md
CHANGED
|
@@ -80,27 +80,14 @@ pass.
|
|
|
80
80
|
|
|
81
81
|
Please also review the [development section of the LabKit (go) README](https://gitlab.com/gitlab-org/labkit#developing-labkit) for details of the LabKit architectural philosophy.
|
|
82
82
|
|
|
83
|
-
To work on some of the scripts we use for releasing a new version,
|
|
84
|
-
make sure to add a new `.env.sh`.
|
|
85
|
-
|
|
86
|
-
```console
|
|
87
|
-
cp .env.example.sh .env.sh`
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
Inside `.env.sh`, add a personal acccess token for the `CHANGELOG_GITLAB_TOKEN`
|
|
91
|
-
environment variable. Next source the file:
|
|
92
|
-
|
|
93
|
-
```console
|
|
94
|
-
. .env.sh
|
|
95
|
-
```
|
|
96
|
-
|
|
97
83
|
### Releasing a new version
|
|
98
84
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
85
|
+
Releases are cut automatically by
|
|
86
|
+
[semantic-release](https://github.com/semantic-release/semantic-release)
|
|
87
|
+
when changes are merged to `master` or to an `N-N-stable` branch. The
|
|
88
|
+
[Conventional Commits](https://www.conventionalcommits.org/) type of each
|
|
89
|
+
commit decides the version bump (see [AGENTS.md](AGENTS.md)), and the
|
|
90
|
+
release notes are generated from those commit messages.
|
|
104
91
|
|
|
105
92
|
The new version will automatically be published to `gitlab-labkit` on
|
|
106
93
|
[rubygems](https://rubygems.org/gems/gitlab-labkit) when the pipeline
|
data/gitlab-labkit.gemspec
CHANGED
|
@@ -36,6 +36,7 @@ Gem::Specification.new do |spec|
|
|
|
36
36
|
spec.add_runtime_dependency "pg_query", ">= 6.1.0", "< 7.0"
|
|
37
37
|
spec.add_runtime_dependency "prometheus-client-mmap", ">= 1.2", "< 2.0"
|
|
38
38
|
spec.add_runtime_dependency "redis", "> 3.0.0", "< 6.0.0"
|
|
39
|
+
spec.add_runtime_dependency "thrift", "< 0.25" # Pinned for jaeger-client 1.1.0 compatibility; thrift 0.25 requires transports to implement message_boundaries?
|
|
39
40
|
|
|
40
41
|
# Please maintain alphabetical order for dev dependencies
|
|
41
42
|
spec.add_development_dependency "excon", "~> 0.78.1"
|
data/lib/labkit/fields.rb
CHANGED
|
@@ -16,6 +16,12 @@ module Labkit
|
|
|
16
16
|
# GitLab username.
|
|
17
17
|
GL_USER_NAME = "gl_user_name"
|
|
18
18
|
|
|
19
|
+
# Type of key used for authentication (e.g. "deploy_key", "key").
|
|
20
|
+
GL_KEY_TYPE = "gl_key_type"
|
|
21
|
+
|
|
22
|
+
# Numeric ID of the key used for authentication.
|
|
23
|
+
GL_KEY_ID = "gl_key_id"
|
|
24
|
+
|
|
19
25
|
# Duo Workflow definition identifier (e.g. "analytics_agent/v1"). Identifies
|
|
20
26
|
# which Duo Workflow agent originated a request, enabling per-agent
|
|
21
27
|
# filtering in Kibana and Grafana.
|
|
@@ -138,12 +144,30 @@ module Labkit
|
|
|
138
144
|
# (e.g. newlines) to prevent log injection.
|
|
139
145
|
ADDITIONAL_DETAILS = "additional_details"
|
|
140
146
|
|
|
147
|
+
# Unrounded complexity score of a GraphQL query. Some fields have a
|
|
148
|
+
# fractional cost (e.g. graphql-ruby's __typename costs 0.2), so the total
|
|
149
|
+
# can be a non-integer value.
|
|
150
|
+
GRAPHQL_COMPLEXITY_RAW = "graphql_complexity_raw"
|
|
151
|
+
|
|
152
|
+
# Number of nodes a GraphQL query returned across all its connection fields.
|
|
153
|
+
# The count is taken after authorization removes the nodes the user can't
|
|
154
|
+
# see. Nested connections add up. The value is 0 when the query selects no
|
|
155
|
+
# connection field.
|
|
156
|
+
GRAPHQL_CONNECTION_NODES = "graphql_connection_nodes"
|
|
157
|
+
|
|
158
|
+
# Protocol or subsystem that produced an access log line (e.g. "http",
|
|
159
|
+
# "grpc"). Emitted by the LabKit access-logging middleware so a single
|
|
160
|
+
# service's HTTP and gRPC access logs can be told apart.
|
|
161
|
+
SYSTEM = "system"
|
|
162
|
+
|
|
141
163
|
# Maps each field constant to its logging field variant version.
|
|
142
164
|
# A version of 0 means the field is not yet assigned to any variant.
|
|
143
165
|
VARIANT_VERSION = {
|
|
144
166
|
CORRELATION_ID => 1,
|
|
145
167
|
GL_USER_ID => 1,
|
|
146
168
|
GL_USER_NAME => 1,
|
|
169
|
+
GL_KEY_TYPE => 0,
|
|
170
|
+
GL_KEY_ID => 0,
|
|
147
171
|
DUO_WORKFLOW_DEFINITION => 0,
|
|
148
172
|
DUO_WORKFLOW_ID => 0,
|
|
149
173
|
ERROR_TYPE => 0,
|
|
@@ -179,6 +203,9 @@ module Labkit
|
|
|
179
203
|
ROOT_CALLER_ID => 0,
|
|
180
204
|
GL_SENT_NOTIFICATION_ID => 0,
|
|
181
205
|
ADDITIONAL_DETAILS => 0,
|
|
206
|
+
GRAPHQL_COMPLEXITY_RAW => 0,
|
|
207
|
+
GRAPHQL_CONNECTION_NODES => 0,
|
|
208
|
+
SYSTEM => 0,
|
|
182
209
|
}.freeze
|
|
183
210
|
|
|
184
211
|
# Get the constant name for a field value
|
|
@@ -207,8 +234,8 @@ module Labkit
|
|
|
207
234
|
Fields::DURATION_S => %w[duration duration_ms elapsed_time actual_duration time_ms total_time gitaly.duration],
|
|
208
235
|
Fields::REMOTE_IP => %w[ip source_ip ip_address],
|
|
209
236
|
Fields::HTTP_HOST => %w[hostname request_host gitlab_host kubernetes.host],
|
|
210
|
-
Fields::GL_PROJECT_ID => %w[extra.project_id meta.project_id meta.search.project_id job_project_id target_project_id],
|
|
211
|
-
Fields::GL_PIPELINE_ID => %w[extra.pipeline_id meta.pipeline_id root_pipeline_id],
|
|
237
|
+
Fields::GL_PROJECT_ID => %w[project_id extra.project_id meta.project_id meta.search.project_id job_project_id target_project_id],
|
|
238
|
+
Fields::GL_PIPELINE_ID => %w[pipeline_id extra.pipeline_id meta.pipeline_id root_pipeline_id],
|
|
212
239
|
Fields::TIMESTAMP => %w[start_time],
|
|
213
240
|
Fields::SEVERITY => %w[level],
|
|
214
241
|
Fields::LOG_MESSAGE => %w[msg custom_message extra.message fields.message graphql.message reason color_message exception.gitaly],
|
|
@@ -66,7 +66,7 @@ Labkit::JsonSchema::RefResolver.cache.clear
|
|
|
66
66
|
|
|
67
67
|
## Use Case Example
|
|
68
68
|
|
|
69
|
-
The `RefResolver` is used in the User Experience SLI Registry [lib/labkit/user_experience_sli/registry.rb](
|
|
69
|
+
The `RefResolver` is used in the User Experience SLI Registry [lib/labkit/user_experience_sli/registry.rb](../user_experience_sli/registry.rb) to validate user experience definitions against a JSON schema.
|
|
70
70
|
|
|
71
71
|
## Implementation Details
|
|
72
72
|
|
|
@@ -171,8 +171,8 @@ A `Rule` is a `Data.define` value object with the following fields:
|
|
|
171
171
|
|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
172
172
|
| `name` | Stable identifier used in Redis keys and metric labels. Must match `/\A[a-z0-9_]+\z/`, max 64 chars. Renaming a rule abandons its in-flight counters. |
|
|
173
173
|
| `match` | Hash of identifier key/value predicates that must **all** be satisfied for the rule to apply. Empty hash matches anything. See [Matchers](#matchers). |
|
|
174
|
-
| `limit` | Integer request threshold per `period
|
|
175
|
-
| `period` | Window length in seconds. May be a callable resolved on every check.
|
|
174
|
+
| `limit` | Integer request threshold per `period`, at least 1. May be a callable resolved on every check. |
|
|
175
|
+
| `period` | Window length in seconds, at least 1; a fraction truncates to whole seconds. May be a callable resolved on every check. |
|
|
176
176
|
| `action` | What the rule does when it matches. One of `:limit`, `:log`, `:skip`. Default `:limit`. See [Actions](#actions). |
|
|
177
177
|
| `characteristics` | Array of identifier keys whose values are folded into the Redis counter key. Each unique combination gets its own counter. |
|
|
178
178
|
| `count_distinct` | Optional identifier key. When set, the rule counts distinct values seen for that key within the window, backed by a Redis SET rather than a counter. Must not overlap `characteristics`. |
|
|
@@ -190,6 +190,13 @@ Labkit::RateLimit::Rule.new(
|
|
|
190
190
|
)
|
|
191
191
|
```
|
|
192
192
|
|
|
193
|
+
`Rule.new` raises `ArgumentError` for a static `limit` under 1 or a static
|
|
194
|
+
`period` under one second, except on `:skip` rules where both are inert.
|
|
195
|
+
Callables are not validated: a `limit` resolving to `0` blocks every matching
|
|
196
|
+
request, which is how to express "deny all" dynamically, and a `period`
|
|
197
|
+
resolving under one second reaches `EXPIRE` as `0` and deletes the counter on
|
|
198
|
+
every write.
|
|
199
|
+
|
|
193
200
|
### Matchers
|
|
194
201
|
|
|
195
202
|
A `match` hash gates whether a rule applies. Each value is normalised through
|
|
@@ -200,17 +207,63 @@ A `match` hash gates whether a rule applies. Each value is normalised through
|
|
|
200
207
|
| plain value | `eq` | `match: { user: nil }`, `match: { method: "POST" }` |
|
|
201
208
|
| `Regexp` | `re` | `match: { endpoint: %r{\A/api/} }` |
|
|
202
209
|
| `{ eq: <value> }` | `eq` | `match: { method: { eq: "POST" } }` (YAML-friendly) |
|
|
210
|
+
| `{ ne: <value> }` | `ne` | `match: { method: { ne: "OPTIONS" } }` |
|
|
203
211
|
| `{ re: <source> }`| `re` | `match: { endpoint: { re: '\A/api/' } }` (YAML-friendly) |
|
|
212
|
+
| `{ nre: <source> }`| `nre` | `match: { endpoint: { nre: '\A/api/' } }` |
|
|
204
213
|
| `{ oneOf: <Array> }` | `oneOf` | `match: { user: { oneOf: ["7", "42"] } }` (YAML-friendly) |
|
|
214
|
+
| `{ noneOf: <Array> }` | `noneOf` | `match: { plan: { noneOf: ["premium"] } }` |
|
|
215
|
+
| `Array` of matchers | `all` | `match: { endpoint: [%r{\A/api/}, { nre: 'internal' }] }` |
|
|
205
216
|
|
|
206
|
-
`re`
|
|
207
|
-
used against non-String values (e.g. matching a 503 status against
|
|
217
|
+
`re` and `nre` coerce the identifier value via `#to_s` before matching, so they
|
|
218
|
+
can be used against non-String values (e.g. matching a 503 status against
|
|
219
|
+
`{ re: '^5' }`).
|
|
208
220
|
|
|
209
221
|
`oneOf` is set membership via `Set#include?` (`eql?`/`hash` equality, no
|
|
210
222
|
coercion): it agrees with `eq` for Strings, Symbols, booleans, nil, and
|
|
211
223
|
same-class numerics, but cross-type numerics differ (`1 == 1.0`, yet
|
|
212
|
-
`{ oneOf: [1] }` does not match `1.0`).
|
|
213
|
-
|
|
224
|
+
`{ oneOf: [1] }` does not match `1.0`). `noneOf` is its complement.
|
|
225
|
+
|
|
226
|
+
An Array means **AND**, so one identifier key can carry several predicates:
|
|
227
|
+
|
|
228
|
+
```yaml
|
|
229
|
+
endpoint:
|
|
230
|
+
- nre: '/oauth/'
|
|
231
|
+
- nre: '\A/api/'
|
|
232
|
+
- nre: '\A/-/(?:health|liveness|readiness|metrics)'
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Its entries must be matcher Hashes or bare `Regexp`s, the two shapes that
|
|
236
|
+
already mean "a matcher" on their own. An Array of plain values is rejected.
|
|
237
|
+
A value meant as one matcher, arriving as a list, therefore fails loudly rather
|
|
238
|
+
than becoming a membership test. Membership is spelled `{ oneOf: [...] }`.
|
|
239
|
+
|
|
240
|
+
An Array holds matchers, not other Arrays, so a match value nests three deep at
|
|
241
|
+
most: an Array, a matcher Hash inside it, and that Hash's `oneOf` list. A value
|
|
242
|
+
nesting deeper than `Matcher::MAX_NESTING_DEPTH` is rejected when the rule is
|
|
243
|
+
built.
|
|
244
|
+
|
|
245
|
+
#### Negation and absent keys
|
|
246
|
+
|
|
247
|
+
`ne`, `nre` and `noneOf` are plain complements of their positive forms, with no
|
|
248
|
+
special case for a missing key. The `Evaluator` looks an identifier key up
|
|
249
|
+
rather than requiring it, so an absent key reaches the matcher as `nil`, and a
|
|
250
|
+
complement is therefore **true** for one: a request with no `endpoint` is not a
|
|
251
|
+
request to an excluded endpoint.
|
|
252
|
+
|
|
253
|
+
That widens a rule rather than narrowing it. A rule that should apply only when
|
|
254
|
+
the key is present says so, with a presence matcher alongside the exclusions:
|
|
255
|
+
|
|
256
|
+
```ruby
|
|
257
|
+
match: { endpoint: [/./, { nre: '\A/api/' }] }
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
Since this follows from inversion, a matcher that does accept `nil` inverts the
|
|
261
|
+
other way: `{ ne: nil }`, `{ noneOf: [nil] }` and `{ nre: '^$' }` are all false
|
|
262
|
+
for an absent key, the last because `nil.to_s` is `""`.
|
|
263
|
+
|
|
264
|
+
There is no cap on regex source length. Rules are written by operators, and the
|
|
265
|
+
only bound on a pattern is `Matcher::MATCH_TIMEOUT_SECONDS`, a 50ms wall-clock
|
|
266
|
+
budget applied per compiled `Regexp`.
|
|
214
267
|
|
|
215
268
|
Glob and prefix matchers are intentionally out of scope.
|
|
216
269
|
|
|
@@ -459,7 +512,8 @@ result.degraded? # => true if a matched count_distinct rule was skipped
|
|
|
459
512
|
result.info # => Result::Info or nil
|
|
460
513
|
result.evaluations # => every counted Result::Evaluation, in rule order
|
|
461
514
|
result.to_response_headers
|
|
462
|
-
# => { "RateLimit-
|
|
515
|
+
# => { "RateLimit-Name" => "<rule name>", "RateLimit-Limit" => "...", "RateLimit-Observed" => "...",
|
|
516
|
+
# "RateLimit-Remaining" => "...", "RateLimit-Reset" => "<unix-ts>" }
|
|
463
517
|
```
|
|
464
518
|
|
|
465
519
|
`Result::Info` holds the per-window counter snapshot:
|
|
@@ -475,6 +529,27 @@ result.to_response_headers
|
|
|
475
529
|
`to_response_headers` returns `{}` for an unmatched or error result, so it is
|
|
476
530
|
safe to merge unconditionally.
|
|
477
531
|
|
|
532
|
+
### Response headers
|
|
533
|
+
|
|
534
|
+
`Result#to_response_headers` reports the most constraining evaluation.
|
|
535
|
+
`Result::Evaluation#to_response_headers` builds the same headers for one evaluation, for a caller that picks its own evaluation across several results.
|
|
536
|
+
|
|
537
|
+
| header | present | value |
|
|
538
|
+
|-----------------------|-------------------|-----------------------------------------------------|
|
|
539
|
+
| `RateLimit-Name` | always | The rule's `name`. |
|
|
540
|
+
| `RateLimit-Limit` | always | `resolved_limit` in the rule's period, at least 0. |
|
|
541
|
+
| `RateLimit-Observed` | always | `count`, as an Integer, at least 0. |
|
|
542
|
+
| `RateLimit-Remaining` | always | `remaining`, as an Integer. |
|
|
543
|
+
| `RateLimit-Reset` | always | `reset_at` as a Unix timestamp. |
|
|
544
|
+
| `Retry-After` | blocking only | Seconds to `reset_at`, rounded up, at least 0. |
|
|
545
|
+
| `RateLimit-ResetTime` | blocking only | `reset_at` as an HTTP date. |
|
|
546
|
+
|
|
547
|
+
`Retry-After` and `RateLimit-ResetTime` belong on the 429, so they appear only when the evaluation blocks.
|
|
548
|
+
For a rule with `ban_for`, `reset_at` is the end of the ban, so these headers report when the ban lifts.
|
|
549
|
+
|
|
550
|
+
`RateLimit-Limit`, `RateLimit-Observed` and `RateLimit-Remaining` all count within the rule's `period`.
|
|
551
|
+
The headers do not state the period, so a client that needs it must know the rule.
|
|
552
|
+
|
|
478
553
|
## Fail-open
|
|
479
554
|
|
|
480
555
|
The evaluator wraps `check` and `peek` in a broad rescue. Any `StandardError`
|