perfgate 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.rspec +3 -0
- data/.rubocop.yml +25 -0
- data/CHANGELOG.md +18 -0
- data/CONTRIBUTING.md +73 -0
- data/LICENSE +201 -0
- data/NOT_FINALIZED.md +70 -0
- data/README.md +86 -0
- data/ROADMAP.md +125 -0
- data/Rakefile +12 -0
- data/SECURITY.md +63 -0
- data/docs/README.md +7 -0
- data/docs/architecture.md +125 -0
- data/docs/compatibility.md +49 -0
- data/docs/launch-article.md +97 -0
- data/docs/onboarding.md +122 -0
- data/docs/telemetry.md +81 -0
- data/examples/rails-rspec-app/.github/workflows/baseline.yml +48 -0
- data/examples/rails-rspec-app/README.md +38 -0
- data/examples/rails-rspec-app/spec/jobs/invoice_job_spec.rb +15 -0
- data/examples/rails-rspec-app/spec/requests/checkout_spec.rb +24 -0
- data/exe/perfgate +7 -0
- data/lib/perfgate/cli/compare_command.rb +91 -0
- data/lib/perfgate/cli/run_command.rb +123 -0
- data/lib/perfgate/cli/run_comparison_reporter.rb +77 -0
- data/lib/perfgate/cli.rb +60 -0
- data/lib/perfgate/comparison/deterministic_metric_decision.rb +34 -0
- data/lib/perfgate/comparison/diagnostics.rb +70 -0
- data/lib/perfgate/comparison/engine.rb +79 -0
- data/lib/perfgate/comparison/metric_change.rb +84 -0
- data/lib/perfgate/comparison/metric_decision.rb +39 -0
- data/lib/perfgate/comparison/statistical_metric_decision.rb +75 -0
- data/lib/perfgate/comparison/workload_comparison.rb +98 -0
- data/lib/perfgate/config/defaults.rb +70 -0
- data/lib/perfgate/config/env_overrides.rb +54 -0
- data/lib/perfgate/config/schema.rb +53 -0
- data/lib/perfgate/config/validator.rb +64 -0
- data/lib/perfgate/config.rb +136 -0
- data/lib/perfgate/errors.rb +20 -0
- data/lib/perfgate/execution/process_runner.rb +71 -0
- data/lib/perfgate/execution/runner.rb +60 -0
- data/lib/perfgate/execution/sample_context.rb +66 -0
- data/lib/perfgate/fingerprints/compatibility.rb +46 -0
- data/lib/perfgate/fingerprints/components.rb +98 -0
- data/lib/perfgate/fingerprints/workload_definition.rb +30 -0
- data/lib/perfgate/instrumentation/allocations.rb +20 -0
- data/lib/perfgate/instrumentation/duration.rb +20 -0
- data/lib/perfgate/instrumentation/gc.rb +30 -0
- data/lib/perfgate/instrumentation/sql_activity.rb +50 -0
- data/lib/perfgate/instrumentation.rb +36 -0
- data/lib/perfgate/metrics/.gitkeep +0 -0
- data/lib/perfgate/policy/engine.rb +73 -0
- data/lib/perfgate/rails/.gitkeep +0 -0
- data/lib/perfgate/report/console.rb +57 -0
- data/lib/perfgate/report/markdown.rb +101 -0
- data/lib/perfgate/reporting/.gitkeep +0 -0
- data/lib/perfgate/rspec/discovery.rb +31 -0
- data/lib/perfgate/rspec/id_resolver.rb +30 -0
- data/lib/perfgate/rspec/workload_builder.rb +48 -0
- data/lib/perfgate/rspec.rb +14 -0
- data/lib/perfgate/serialization/run_result.rb +58 -0
- data/lib/perfgate/statistics/mann_whitney_u.rb +84 -0
- data/lib/perfgate/statistics/summary.rb +60 -0
- data/lib/perfgate/storage/adapter.rb +24 -0
- data/lib/perfgate/storage/archive.rb +69 -0
- data/lib/perfgate/storage/filesystem.rb +121 -0
- data/lib/perfgate/telemetry/.gitkeep +0 -0
- data/lib/perfgate/version.rb +5 -0
- data/lib/perfgate/workloads/registry.rb +50 -0
- data/lib/perfgate/workloads/workload.rb +26 -0
- data/lib/perfgate.rb +46 -0
- data/perfgate.gemspec +41 -0
- data/schemas/comparison-result-v1.schema.json +7 -0
- data/schemas/run-result-v1.schema.json +7 -0
- data/sig/perfgate.rbs +4 -0
- metadata +139 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 100020429feb022487b41383ee3fdab69d2274a07851a33ac205c05120e24199
|
|
4
|
+
data.tar.gz: 889de864cd8145a014665eafeb9311f4a358358b0b61da85f380361210cf4ff6
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: a1f83f58d455a7ee3705b7696ba67617d6ddb3f96d8c9ee871223b4fc9fc4c825997d8add9f33670fcbdab52055a394c3d4b9f4746d844a02e9dcdc9f178cf01
|
|
7
|
+
data.tar.gz: 3e5e768960362b02652201712d61656132bfa70c9f78d7ad5c560cba5ad8b4bae9338d4bf9b07a2543cbdaa65ab873a43d7fc37fb685aec8ba52818a6e2d0a70
|
data/.rspec
ADDED
data/.rubocop.yml
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
AllCops:
|
|
2
|
+
TargetRubyVersion: 3.1
|
|
3
|
+
NewCops: enable
|
|
4
|
+
# Exploratory Milestone 0 spikes are throwaway scripts, not held to the
|
|
5
|
+
# same design/style bar as the gem itself.
|
|
6
|
+
Exclude:
|
|
7
|
+
- "spikes/**/*"
|
|
8
|
+
|
|
9
|
+
Style/StringLiterals:
|
|
10
|
+
Enabled: true
|
|
11
|
+
EnforcedStyle: double_quotes
|
|
12
|
+
|
|
13
|
+
Style/StringLiteralsInInterpolation:
|
|
14
|
+
Enabled: true
|
|
15
|
+
EnforcedStyle: double_quotes
|
|
16
|
+
|
|
17
|
+
Layout/LineLength:
|
|
18
|
+
Max: 120
|
|
19
|
+
|
|
20
|
+
# RSpec example groups naturally group many small `it` blocks together;
|
|
21
|
+
# the default limit is tuned for regular method bodies, not spec files.
|
|
22
|
+
Metrics/BlockLength:
|
|
23
|
+
Exclude:
|
|
24
|
+
- "spec/**/*"
|
|
25
|
+
- "*.gemspec"
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
## [Unreleased]
|
|
2
|
+
|
|
3
|
+
## [0.1.0] - 2026-08-26
|
|
4
|
+
|
|
5
|
+
- Core execution engine: CLI, configuration loader, RSpec discovery
|
|
6
|
+
via `perfgate: true` metadata, `Perfgate.measure`, process-isolated
|
|
7
|
+
warmup/samples, and a filesystem result bundle.
|
|
8
|
+
- Rails metrics: SQL query count/duration, allocations, and GC
|
|
9
|
+
diagnostics, correctly isolated to the measurement block.
|
|
10
|
+
- Comparison and policy: execution fingerprints, a compatibility
|
|
11
|
+
engine, Mann-Whitney statistics, practical thresholds, PASS/WARN/FAIL
|
|
12
|
+
decisions with CI exit codes, and regression-injection test coverage.
|
|
13
|
+
- CI experience: deterministic diagnostics, console and Markdown
|
|
14
|
+
reports, `perfgate run --compare/--format` for a single-step CI
|
|
15
|
+
comparison, portable `perfgate-run-<run-id>.tar.gz` archives, and a
|
|
16
|
+
documented GitHub Actions workflow.
|
|
17
|
+
- Release readiness: security policy, contribution guide,
|
|
18
|
+
compatibility matrix, and telemetry specification.
|
data/CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# Contributing to Baseline
|
|
2
|
+
|
|
3
|
+
Thanks for considering a contribution. Baseline is early and the scope
|
|
4
|
+
is deliberately narrow -- see
|
|
5
|
+
[baseline_oss_mvp_technical_spec_and_roadmap.md](../baseline_oss_mvp_technical_spec_and_roadmap.md)
|
|
6
|
+
for the authoritative product and technical specification. If you're
|
|
7
|
+
proposing something not covered there, please open an issue to discuss
|
|
8
|
+
scope before sending a large pull request.
|
|
9
|
+
|
|
10
|
+
## Development setup
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
git clone <your fork>
|
|
14
|
+
cd baseline
|
|
15
|
+
bin/setup
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
This installs dependencies via Bundler. `bin/console` gives you an
|
|
19
|
+
interactive prompt with the gem loaded.
|
|
20
|
+
|
|
21
|
+
## Running the checks
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
bundle exec rspec # unit tests
|
|
25
|
+
bundle exec rubocop # style and complexity checks
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Both must pass before a pull request will be merged; `rake` (no
|
|
29
|
+
arguments) runs both.
|
|
30
|
+
|
|
31
|
+
## Coding conventions
|
|
32
|
+
|
|
33
|
+
- This project has a firm house rule: when RuboCop flags a file for
|
|
34
|
+
length or complexity (`Metrics/MethodLength`, `Metrics/ClassLength`,
|
|
35
|
+
`Metrics/ModuleLength`, etc.), **extract a smaller method or a new
|
|
36
|
+
collaborator class/module** rather than raising the limit in
|
|
37
|
+
`.rubocop.yml`. Several modules in `lib/baseline/comparison/` and
|
|
38
|
+
`lib/baseline/cli/` exist specifically because of this rule -- follow
|
|
39
|
+
that pattern.
|
|
40
|
+
- Favor small, single-purpose classes and modules over large ones.
|
|
41
|
+
Keep public interfaces documented with a short module/class comment
|
|
42
|
+
explaining the "why", not just the "what".
|
|
43
|
+
- New behavior should ship with unit tests. Regression-injection style
|
|
44
|
+
tests (seed a known regression, assert Baseline detects it) are
|
|
45
|
+
especially valuable for comparison/policy logic.
|
|
46
|
+
- Don't add a new runtime dependency without discussing it first --
|
|
47
|
+
the spec favors using Ruby/Rails standard library and already-present
|
|
48
|
+
gems (e.g. `rubygems/package` and `zlib` for archives) over adding
|
|
49
|
+
new ones.
|
|
50
|
+
|
|
51
|
+
## Commit and PR style
|
|
52
|
+
|
|
53
|
+
- Write commit messages and PR descriptions the way you'd want to read
|
|
54
|
+
them in `git log`: an imperative-mood subject line, and a body that
|
|
55
|
+
explains *why* when it's not obvious from the diff.
|
|
56
|
+
- Keep pull requests scoped to one milestone or one deliverable where
|
|
57
|
+
possible; it makes review much faster.
|
|
58
|
+
- Reference the relevant spec section (e.g. "spec 20.3") in the PR
|
|
59
|
+
description when implementing a specific documented behavior.
|
|
60
|
+
|
|
61
|
+
## Reporting bugs
|
|
62
|
+
|
|
63
|
+
Open a GitHub issue with:
|
|
64
|
+
|
|
65
|
+
- Baseline version, Ruby version, Rails version, RSpec version;
|
|
66
|
+
- your `baseline.yml` (redacted if needed);
|
|
67
|
+
- the command you ran and its full output;
|
|
68
|
+
- what you expected vs. what happened.
|
|
69
|
+
|
|
70
|
+
## Reporting security issues
|
|
71
|
+
|
|
72
|
+
Please don't file security vulnerabilities as public issues -- see
|
|
73
|
+
[SECURITY.md](SECURITY.md) instead.
|
data/LICENSE
ADDED
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
6
|
+
|
|
7
|
+
1. Definitions.
|
|
8
|
+
|
|
9
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
10
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
11
|
+
|
|
12
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
13
|
+
the copyright owner that is granting the License.
|
|
14
|
+
|
|
15
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
16
|
+
other entities that control, are controlled by, or are under common
|
|
17
|
+
control with that entity. For the purposes of this definition,
|
|
18
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
19
|
+
direction or management of such entity, whether by contract or
|
|
20
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
21
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
22
|
+
|
|
23
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
24
|
+
exercising permissions granted by this License.
|
|
25
|
+
|
|
26
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
27
|
+
including but not limited to software source code, documentation
|
|
28
|
+
source, and configuration files.
|
|
29
|
+
|
|
30
|
+
"Object" form shall mean any form resulting from mechanical
|
|
31
|
+
transformation or translation of a Source form, including but
|
|
32
|
+
not limited to compiled object code, generated documentation,
|
|
33
|
+
and conversions to other media types.
|
|
34
|
+
|
|
35
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
36
|
+
Object form, made available under the License, as indicated by a
|
|
37
|
+
copyright notice that is included in or attached to the work
|
|
38
|
+
(an example is provided in the Appendix below).
|
|
39
|
+
|
|
40
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
41
|
+
form, that is based on (or derived from) the Work and for which the
|
|
42
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
43
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
44
|
+
of this License, Derivative Works shall not include works that remain
|
|
45
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
46
|
+
the Work and Derivative Works thereof.
|
|
47
|
+
|
|
48
|
+
"Contribution" shall mean any work of authorship, including
|
|
49
|
+
the original version of the Work and any modifications or additions
|
|
50
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
51
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
52
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
53
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
54
|
+
means any form of electronic, verbal, or written communication sent
|
|
55
|
+
to the Licensor or its representatives, including but not limited to
|
|
56
|
+
communication on electronic mailing lists, source code control systems,
|
|
57
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
58
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
59
|
+
excluding communication that is conspicuously marked or otherwise
|
|
60
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
61
|
+
|
|
62
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
63
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
64
|
+
subsequently incorporated within the Work.
|
|
65
|
+
|
|
66
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
67
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
68
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
69
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
70
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
71
|
+
Work and such Derivative Works in Source or Object form.
|
|
72
|
+
|
|
73
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
74
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
75
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
76
|
+
(except as stated in this section) patent license to make, have made,
|
|
77
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
78
|
+
where such license applies only to those patent claims licensable
|
|
79
|
+
by such Contributor that are necessarily infringed by their
|
|
80
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
81
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
82
|
+
institute patent litigation against any entity (including a
|
|
83
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
84
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
85
|
+
or contributory patent infringement, then any patent licenses
|
|
86
|
+
granted to You under this License for that Work shall terminate
|
|
87
|
+
as of the date such litigation is filed.
|
|
88
|
+
|
|
89
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
90
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
91
|
+
modifications, and in Source or Object form, provided that You
|
|
92
|
+
meet the following conditions:
|
|
93
|
+
|
|
94
|
+
(a) You must give any other recipients of the Work or
|
|
95
|
+
Derivative Works a copy of this License; and
|
|
96
|
+
|
|
97
|
+
(b) You must cause any modified files to carry prominent notices
|
|
98
|
+
stating that You changed the files; and
|
|
99
|
+
|
|
100
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
101
|
+
that You distribute, all copyright, patent, trademark, and
|
|
102
|
+
attribution notices from the Source form of the Work,
|
|
103
|
+
excluding those notices that do not pertain to any part of
|
|
104
|
+
the Derivative Works; and
|
|
105
|
+
|
|
106
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
107
|
+
distribution, then any Derivative Works that You distribute must
|
|
108
|
+
include a readable copy of the attribution notices contained
|
|
109
|
+
within such NOTICE file, excluding those notices that do not
|
|
110
|
+
pertain to any part of the Derivative Works, in at least one
|
|
111
|
+
of the following places: within a NOTICE text file distributed
|
|
112
|
+
as part of the Derivative Works; within the Source form or
|
|
113
|
+
documentation, if provided along with the Derivative Works; or,
|
|
114
|
+
within a display generated by the Derivative Works, if and
|
|
115
|
+
wherever such third-party notices normally appear. The contents
|
|
116
|
+
of the NOTICE file are for informational purposes only and
|
|
117
|
+
do not modify the License. You may add Your own attribution
|
|
118
|
+
notices within Derivative Works that You distribute, alongside
|
|
119
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
120
|
+
that such additional attribution notices cannot be construed
|
|
121
|
+
as modifying the License.
|
|
122
|
+
|
|
123
|
+
You may add Your own copyright statement to Your modifications and
|
|
124
|
+
may provide additional or different license terms and conditions
|
|
125
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
126
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
127
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
128
|
+
the conditions stated in this License.
|
|
129
|
+
|
|
130
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
131
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
132
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
133
|
+
this License, without any additional terms or conditions.
|
|
134
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
135
|
+
the terms of any separate license agreement you may have executed
|
|
136
|
+
with Licensor regarding such Contributions.
|
|
137
|
+
|
|
138
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
139
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
140
|
+
except as required for reasonable and customary use in describing the
|
|
141
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
142
|
+
|
|
143
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
144
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
145
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
146
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
147
|
+
implied, including, without limitation, any warranties or conditions
|
|
148
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
149
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
150
|
+
appropriateness of using or redistributing the Work and assume any
|
|
151
|
+
risks associated with Your exercise of permissions under this License.
|
|
152
|
+
|
|
153
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
154
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
155
|
+
unless required by applicable law (such as deliberate and grossly
|
|
156
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
157
|
+
liable to You for damages, including any direct, indirect, special,
|
|
158
|
+
incidental, or consequential damages of any character arising as a
|
|
159
|
+
result of this License or out of the use or inability to use the
|
|
160
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
161
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
162
|
+
other commercial damages or losses), even if such Contributor
|
|
163
|
+
has been advised of the possibility of such damages.
|
|
164
|
+
|
|
165
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
166
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
167
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
168
|
+
or other liability obligations and/or rights consistent with this
|
|
169
|
+
License. However, in accepting such obligations, You may act only
|
|
170
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
171
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
172
|
+
defend, and hold each Contributor harmless for any liability
|
|
173
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
174
|
+
of your accepting any such warranty or additional liability.
|
|
175
|
+
|
|
176
|
+
END OF TERMS AND CONDITIONS
|
|
177
|
+
|
|
178
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
179
|
+
|
|
180
|
+
To apply the Apache License to your work, attach the following
|
|
181
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
182
|
+
replaced with your own identifying information. (Don't include
|
|
183
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
184
|
+
comment syntax for the file format. We also recommend that a
|
|
185
|
+
file or class name and description of purpose be included on the
|
|
186
|
+
same "printed page" as the copyright notice for easier
|
|
187
|
+
identification within third-party archives.
|
|
188
|
+
|
|
189
|
+
Copyright 2026 Baseline contributors
|
|
190
|
+
|
|
191
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
192
|
+
you may not use this file except in compliance with the License.
|
|
193
|
+
You may obtain a copy of the License at
|
|
194
|
+
|
|
195
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
196
|
+
|
|
197
|
+
Unless required by applicable law or agreed to in writing, software
|
|
198
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
199
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
200
|
+
See the License for the specific language governing permissions and
|
|
201
|
+
limitations under the License.
|
data/NOT_FINALIZED.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Not Finalized
|
|
2
|
+
|
|
3
|
+
## CLI commands not yet built
|
|
4
|
+
|
|
5
|
+
| Command | What it should do | Spec section |
|
|
6
|
+
|---|---|---|
|
|
7
|
+
| `perfgate init` | Scaffold `baselined.yml` with commented defaults and an example workload block | ยง6.1 |
|
|
8
|
+
| `perfgate doctor` | Pre-flight check: Ruby/Rails version, RSpec integration present, storage path writable, config valid | Phase 2 |
|
|
9
|
+
| `perfgate schema` | Print the canonical JSON Schema for run-result or comparison-result to stdout | Phase 2 |
|
|
10
|
+
| `perfgate report` | Re-render a saved comparison result in any supported format without re-running workloads | Phase 2 |
|
|
11
|
+
|
|
12
|
+
`init` and `doctor` directly affect first-run experience and are the highest priority.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## Memory metric not implemented
|
|
17
|
+
|
|
18
|
+
`metrics.memory` config key exists and is accepted by the schema validator,
|
|
19
|
+
but the collector is absent. `Instrumentation::REGISTRY` has no `:memory` entry.
|
|
20
|
+
|
|
21
|
+
File: `lib/baselined/instrumentation.rb:19` โ comment says "not yet implemented".
|
|
22
|
+
|
|
23
|
+
Safe to ship as-is (the config key silently has no effect), but the docs
|
|
24
|
+
(`docs/compatibility.md`) describe it as opt-in and experimental โ should either
|
|
25
|
+
implement it or document clearly that it is deferred.
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## README status note is stale
|
|
30
|
+
|
|
31
|
+
`README.md` still reads:
|
|
32
|
+
|
|
33
|
+
> **Status:** Early development. `run` and `compare` are implemented and
|
|
34
|
+
> covered by tests; `init`, `report`, `doctor`, and `schema` are not yet built.
|
|
35
|
+
|
|
36
|
+
This needs to be updated before publishing โ either to reflect what is missing
|
|
37
|
+
or removed once `init`/`doctor` are built.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## CI matrix gap
|
|
42
|
+
|
|
43
|
+
`docs/compatibility.md` lists Ruby 3.2โ3.4 and Rails 7.1/7.2/8.0 as supported,
|
|
44
|
+
but the repo's own CI (`github/workflows/main.yml`) only tests Ruby 3.2.2 + SQLite.
|
|
45
|
+
|
|
46
|
+
No PostgreSQL or MySQL adapter tests. No Rails 7.2/8.0 matrix row.
|
|
47
|
+
|
|
48
|
+
Not a hard blocker for a first release, but should be noted in the release
|
|
49
|
+
announcement and closed before claiming broad compatibility.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## No GitHub remote
|
|
54
|
+
|
|
55
|
+
The git repo at `baseline/` has commits and no remote.
|
|
56
|
+
`spec.homepage` in the gemspec points to `https://github.com/baseline-oss/perfgate`
|
|
57
|
+
which does not exist yet.
|
|
58
|
+
|
|
59
|
+
Must push before publishing so RubyGems.org can resolve the source link and
|
|
60
|
+
`spec.files` (`git ls-files`) captures everything correctly.
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## CHANGELOG not updated for release
|
|
65
|
+
|
|
66
|
+
The `[Unreleased]` block contains all milestone work but has never been
|
|
67
|
+
promoted to a dated `[0.1.0]` entry. The existing `[0.1.0]` line at the
|
|
68
|
+
bottom just says "Initial release".
|
|
69
|
+
|
|
70
|
+
Needs a proper dated entry before tagging and pushing.
|
data/README.md
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
# Perfgate
|
|
2
|
+
|
|
3
|
+
Perfgate is an open-source, CI-native performance assurance tool for Ruby on
|
|
4
|
+
Rails applications. It converts selected RSpec examples into repeatable
|
|
5
|
+
performance workloads, measures application-level signals (duration, SQL
|
|
6
|
+
activity, allocations, GC), compares a pull request against a compatible
|
|
7
|
+
default-branch baseline, and produces a clear PASS/WARN/FAIL merge-gate
|
|
8
|
+
decision.
|
|
9
|
+
|
|
10
|
+
> Did this change introduce a material, reproducible performance regression?
|
|
11
|
+
|
|
12
|
+
## Installation
|
|
13
|
+
|
|
14
|
+
Add to your Gemfile:
|
|
15
|
+
|
|
16
|
+
```ruby
|
|
17
|
+
gem "perfgate", group: :test
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Usage
|
|
21
|
+
|
|
22
|
+
Tag an RSpec example with `perfgate: true` and wrap the part you want
|
|
23
|
+
measured in `Perfgate.measure`:
|
|
24
|
+
|
|
25
|
+
```ruby
|
|
26
|
+
RSpec.describe "Checkout", type: :request, perfgate: true do
|
|
27
|
+
it "creates an order" do
|
|
28
|
+
sign_in(create(:user))
|
|
29
|
+
cart = create(:cart, :with_line_items)
|
|
30
|
+
|
|
31
|
+
Perfgate.measure { post "/checkout", params: { cart_id: cart.id } }
|
|
32
|
+
|
|
33
|
+
expect(response).to have_http_status(:created)
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Then run it:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
bundle exec perfgate run --output .perfgate/current
|
|
42
|
+
bundle exec perfgate compare --baseline .perfgate/main --candidate .perfgate/current
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
`perfgate run` also accepts `--compare PATH` to run and compare against a
|
|
46
|
+
reference bundle in one step, and `--format markdown` to render a Markdown
|
|
47
|
+
report instead of the console summary. Combined, this is what a CI job needs:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
bundle exec perfgate run \
|
|
51
|
+
--output .perfgate/current \
|
|
52
|
+
--compare .perfgate/reference \
|
|
53
|
+
--format markdown
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
This writes a `summary.md` file into `--output` alongside the run bundle, so
|
|
57
|
+
it can be published as a GitHub Actions job summary. See
|
|
58
|
+
[docs/onboarding.md](docs/onboarding.md) for a full walkthrough and
|
|
59
|
+
[examples/rails-rspec-app](examples/rails-rspec-app) for a full example
|
|
60
|
+
workflow, including the artifact download/upload steps that carry a baseline
|
|
61
|
+
result between CI runs.
|
|
62
|
+
|
|
63
|
+
## Docs
|
|
64
|
+
|
|
65
|
+
- [Architecture](docs/architecture.md) - layered pipeline, key design decisions, how to add a metric
|
|
66
|
+
- [Onboarding guide](docs/onboarding.md) - first-run walkthrough for an existing Rails/RSpec app
|
|
67
|
+
- [Compatibility matrix](docs/compatibility.md) - supported Ruby, Rails, and database versions
|
|
68
|
+
- [Telemetry contract](docs/telemetry.md) - what is (and is never) collected
|
|
69
|
+
- [Roadmap](ROADMAP.md) - what's done, what's next, what's out of scope
|
|
70
|
+
|
|
71
|
+
## Development
|
|
72
|
+
|
|
73
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then,
|
|
74
|
+
run `rake spec` to run the tests. You can also run `bin/console` for an
|
|
75
|
+
interactive prompt that will allow you to experiment.
|
|
76
|
+
|
|
77
|
+
## Contributing
|
|
78
|
+
|
|
79
|
+
Bug reports and pull requests are welcome on [GitHub](https://github.com/a11ejandro/perfgate).
|
|
80
|
+
See the contribution guide for scope, principles, and the implementation
|
|
81
|
+
constraints that pull requests are expected to follow.
|
|
82
|
+
|
|
83
|
+
## License
|
|
84
|
+
|
|
85
|
+
Perfgate is available as open source under the terms of the
|
|
86
|
+
[Apache License 2.0](LICENSE).
|
data/ROADMAP.md
ADDED
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
# Roadmap
|
|
2
|
+
|
|
3
|
+
This roadmap tracks work toward a public gem release and beyond. Items within
|
|
4
|
+
each phase are roughly prioritized; the order within a phase is not fixed.
|
|
5
|
+
|
|
6
|
+
Status key: โ
done ยท ๐ง in progress ยท โฌ not started
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## Phase 1 โ Implemented (MVP core)
|
|
11
|
+
|
|
12
|
+
โ
`perfgate run` โ discovers `:baseline`-tagged RSpec examples, runs warmups
|
|
13
|
+
+ measured samples in subprocess isolation, writes a versioned JSON result bundle
|
|
14
|
+
|
|
15
|
+
โ
`perfgate compare` โ loads two result bundles, validates fingerprint
|
|
16
|
+
compatibility, produces per-workload PASS/WARN/FAIL/INCOMPARABLE decisions,
|
|
17
|
+
saves a comparison result, exits with CI-meaningful codes
|
|
18
|
+
|
|
19
|
+
โ
Instrumentation โ wall-clock duration (monotonic), SQL query count and
|
|
20
|
+
cumulative duration (ActiveSupport::Notifications), Ruby object allocations
|
|
21
|
+
(GC.stat delta), GC activity
|
|
22
|
+
|
|
23
|
+
โ
Statistics โ Mann-Whitney U with tie correction, extended summary
|
|
24
|
+
(min/p50/p95/p99/max/count)
|
|
25
|
+
|
|
26
|
+
โ
Fingerprinting โ environment fingerprint (Ruby version, Rails version,
|
|
27
|
+
platform, dataset version) + per-workload definition hash; compatibility
|
|
28
|
+
engine that blocks incomparable runs
|
|
29
|
+
|
|
30
|
+
โ
Regression policy โ statistical + practical-floor dual guard; noise-ratio
|
|
31
|
+
downgrade; deterministic SQL comparison; configurable thresholds
|
|
32
|
+
|
|
33
|
+
โ
Console and Markdown reporters โ table-format console output; GitHub-flavored
|
|
34
|
+
Markdown for job summaries and PR comments
|
|
35
|
+
|
|
36
|
+
โ
`baseline run --compare PATH --format markdown` โ one-step CI command
|
|
37
|
+
|
|
38
|
+
โ
Portable archives โ `export_archive`/`import_archive` for `.tar.gz` artifact
|
|
39
|
+
hand-off between CI jobs; path-traversal rejection on import
|
|
40
|
+
|
|
41
|
+
โ
GitHub Actions example workflow โ upload/download artifact pattern for
|
|
42
|
+
default-branch โ PR comparison
|
|
43
|
+
|
|
44
|
+
โ
Docs โ CONTRIBUTING, SECURITY, compatibility matrix, telemetry contract,
|
|
45
|
+
onboarding guide
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Phase 2 โ Pre-release polish
|
|
50
|
+
|
|
51
|
+
โฌ `perfgate init` โ scaffold `baseline.yml` with commented defaults and an
|
|
52
|
+
example workload block; the spec describes it but it is not yet built
|
|
53
|
+
|
|
54
|
+
โฌ `perfgate doctor` โ pre-flight check: Ruby/Rails version, RSpec integration
|
|
55
|
+
presence, storage directory writability, config validity
|
|
56
|
+
|
|
57
|
+
โฌ `perfgate schema` โ print the canonical JSON Schema for run-result or
|
|
58
|
+
comparison-result to stdout; useful for tooling integration
|
|
59
|
+
|
|
60
|
+
โฌ `perfgate report` โ re-render a saved comparison result in any supported
|
|
61
|
+
format without re-running workloads
|
|
62
|
+
|
|
63
|
+
โฌ Expanded CI matrix โ test against Ruby 3.2/3.3/3.4 and
|
|
64
|
+
Rails 7.1/7.2/8.0 in CI (currently only Ruby 3.2.2 + SQLite)
|
|
65
|
+
|
|
66
|
+
โฌ PostgreSQL and MySQL smoke tests โ verify SQL instrumentation behaves
|
|
67
|
+
identically across adapters
|
|
68
|
+
|
|
69
|
+
โฌ RubyGems.org release โ cut `v0.1.0`, add gem badge to README, publish
|
|
70
|
+
to rubygems.org
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Phase 3 โ Developer experience
|
|
75
|
+
|
|
76
|
+
โฌ `baseline accept` โ mark a known-regression comparison as accepted so the
|
|
77
|
+
next default-branch run becomes the new reference without manual file moves
|
|
78
|
+
|
|
79
|
+
โฌ Per-workload threshold overrides in `baseline.yml` โ allow tighter or
|
|
80
|
+
looser thresholds for specific workload IDs
|
|
81
|
+
|
|
82
|
+
โฌ Memory delta metric (opt-in) โ process RSS delta per sample; gated behind
|
|
83
|
+
`metrics.memory: true` because RSS is noisy on most platforms
|
|
84
|
+
|
|
85
|
+
โฌ Sidekiq adapter โ first-class `Perfgate.measure { MyWorker.drain }` helper
|
|
86
|
+
that suppresses Sidekiq's own threading and logging noise
|
|
87
|
+
|
|
88
|
+
โฌ VS Code / RubyMine run-configuration snippets in the example app
|
|
89
|
+
|
|
90
|
+
โฌ Interactive `baseline run --watch` for local iteration
|
|
91
|
+
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
## Phase 4 โ Ecosystem and integrations
|
|
95
|
+
|
|
96
|
+
โฌ GitHub PR comment integration โ post or update a comparison summary comment
|
|
97
|
+
on the PR (requires a GitHub token; strictly opt-in)
|
|
98
|
+
|
|
99
|
+
โฌ Telemetry (opt-in) โ anonymous, privacy-preserving usage data; schema and
|
|
100
|
+
contract already documented in `docs/telemetry.md`; no data is sent until
|
|
101
|
+
this is built and the user opts in
|
|
102
|
+
|
|
103
|
+
โฌ Minitest adapter โ extend discovery and execution to work with Minitest
|
|
104
|
+
test suites
|
|
105
|
+
|
|
106
|
+
โฌ GitLab CI / Bitbucket Pipelines example workflows
|
|
107
|
+
|
|
108
|
+
โฌ SARIF output format โ machine-readable regression report for GitHub
|
|
109
|
+
Advanced Security or similar tooling
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Out of scope (MVP and foreseeable future)
|
|
114
|
+
|
|
115
|
+
These are explicitly not on the roadmap for the open-source gem:
|
|
116
|
+
|
|
117
|
+
- Baseline Cloud / hosted dashboards / long-term history
|
|
118
|
+
- User accounts or billing
|
|
119
|
+
- Multi-repository analytics
|
|
120
|
+
- Production APM or continuous profiling
|
|
121
|
+
- Automatic workload discovery (without explicit `:baseline` tag)
|
|
122
|
+
- Browser / E2E measurement
|
|
123
|
+
- Distributed or concurrency load testing
|
|
124
|
+
- AI-generated diagnoses or automatic code attribution
|
|
125
|
+
- Non-Ruby languages
|