perfgate 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.rspec +3 -0
- data/.rubocop.yml +25 -0
- data/CHANGELOG.md +18 -0
- data/CONTRIBUTING.md +73 -0
- data/LICENSE +201 -0
- data/NOT_FINALIZED.md +70 -0
- data/README.md +86 -0
- data/ROADMAP.md +125 -0
- data/Rakefile +12 -0
- data/SECURITY.md +63 -0
- data/docs/README.md +7 -0
- data/docs/architecture.md +125 -0
- data/docs/compatibility.md +49 -0
- data/docs/launch-article.md +97 -0
- data/docs/onboarding.md +122 -0
- data/docs/telemetry.md +81 -0
- data/examples/rails-rspec-app/.github/workflows/baseline.yml +48 -0
- data/examples/rails-rspec-app/README.md +38 -0
- data/examples/rails-rspec-app/spec/jobs/invoice_job_spec.rb +15 -0
- data/examples/rails-rspec-app/spec/requests/checkout_spec.rb +24 -0
- data/exe/perfgate +7 -0
- data/lib/perfgate/cli/compare_command.rb +91 -0
- data/lib/perfgate/cli/run_command.rb +123 -0
- data/lib/perfgate/cli/run_comparison_reporter.rb +77 -0
- data/lib/perfgate/cli.rb +60 -0
- data/lib/perfgate/comparison/deterministic_metric_decision.rb +34 -0
- data/lib/perfgate/comparison/diagnostics.rb +70 -0
- data/lib/perfgate/comparison/engine.rb +79 -0
- data/lib/perfgate/comparison/metric_change.rb +84 -0
- data/lib/perfgate/comparison/metric_decision.rb +39 -0
- data/lib/perfgate/comparison/statistical_metric_decision.rb +75 -0
- data/lib/perfgate/comparison/workload_comparison.rb +98 -0
- data/lib/perfgate/config/defaults.rb +70 -0
- data/lib/perfgate/config/env_overrides.rb +54 -0
- data/lib/perfgate/config/schema.rb +53 -0
- data/lib/perfgate/config/validator.rb +64 -0
- data/lib/perfgate/config.rb +136 -0
- data/lib/perfgate/errors.rb +20 -0
- data/lib/perfgate/execution/process_runner.rb +71 -0
- data/lib/perfgate/execution/runner.rb +60 -0
- data/lib/perfgate/execution/sample_context.rb +66 -0
- data/lib/perfgate/fingerprints/compatibility.rb +46 -0
- data/lib/perfgate/fingerprints/components.rb +98 -0
- data/lib/perfgate/fingerprints/workload_definition.rb +30 -0
- data/lib/perfgate/instrumentation/allocations.rb +20 -0
- data/lib/perfgate/instrumentation/duration.rb +20 -0
- data/lib/perfgate/instrumentation/gc.rb +30 -0
- data/lib/perfgate/instrumentation/sql_activity.rb +50 -0
- data/lib/perfgate/instrumentation.rb +36 -0
- data/lib/perfgate/metrics/.gitkeep +0 -0
- data/lib/perfgate/policy/engine.rb +73 -0
- data/lib/perfgate/rails/.gitkeep +0 -0
- data/lib/perfgate/report/console.rb +57 -0
- data/lib/perfgate/report/markdown.rb +101 -0
- data/lib/perfgate/reporting/.gitkeep +0 -0
- data/lib/perfgate/rspec/discovery.rb +31 -0
- data/lib/perfgate/rspec/id_resolver.rb +30 -0
- data/lib/perfgate/rspec/workload_builder.rb +48 -0
- data/lib/perfgate/rspec.rb +14 -0
- data/lib/perfgate/serialization/run_result.rb +58 -0
- data/lib/perfgate/statistics/mann_whitney_u.rb +84 -0
- data/lib/perfgate/statistics/summary.rb +60 -0
- data/lib/perfgate/storage/adapter.rb +24 -0
- data/lib/perfgate/storage/archive.rb +69 -0
- data/lib/perfgate/storage/filesystem.rb +121 -0
- data/lib/perfgate/telemetry/.gitkeep +0 -0
- data/lib/perfgate/version.rb +5 -0
- data/lib/perfgate/workloads/registry.rb +50 -0
- data/lib/perfgate/workloads/workload.rb +26 -0
- data/lib/perfgate.rb +46 -0
- data/perfgate.gemspec +41 -0
- data/schemas/comparison-result-v1.schema.json +7 -0
- data/schemas/run-result-v1.schema.json +7 -0
- data/sig/perfgate.rbs +4 -0
- metadata +139 -0
data/SECURITY.md
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# Security Policy
|
|
2
|
+
|
|
3
|
+
## Supported versions
|
|
4
|
+
|
|
5
|
+
Baseline is pre-1.0. Security fixes are made against the latest release
|
|
6
|
+
on the default branch; there is no long-term support branch yet.
|
|
7
|
+
|
|
8
|
+
| Version | Supported |
|
|
9
|
+
| ------- | --------- |
|
|
10
|
+
| latest | yes |
|
|
11
|
+
| < 0.1 | no |
|
|
12
|
+
|
|
13
|
+
## Reporting a vulnerability
|
|
14
|
+
|
|
15
|
+
Please do not open a public GitHub issue for suspected security
|
|
16
|
+
vulnerabilities. Instead, use GitHub's private
|
|
17
|
+
[vulnerability reporting](https://github.com/baseline-oss/perfgate/security/advisories/new)
|
|
18
|
+
feature, or email the maintainer directly at bendthe@gmail.com with:
|
|
19
|
+
|
|
20
|
+
- a description of the issue and its potential impact;
|
|
21
|
+
- steps to reproduce, or a minimal repro case;
|
|
22
|
+
- the Baseline version, Ruby version, and Rails version involved.
|
|
23
|
+
|
|
24
|
+
You should receive an acknowledgment within 5 business days. We'll work
|
|
25
|
+
with you to understand and confirm the issue, agree on a disclosure
|
|
26
|
+
timeline, and credit you in the release notes unless you'd prefer to
|
|
27
|
+
stay anonymous.
|
|
28
|
+
|
|
29
|
+
## What's in scope
|
|
30
|
+
|
|
31
|
+
- The `baseline` gem's CLI, execution engine, comparison/policy logic,
|
|
32
|
+
and filesystem storage adapter.
|
|
33
|
+
- Archive import handling (path traversal, malformed archives).
|
|
34
|
+
- Anything that could cause Baseline to execute untrusted code,
|
|
35
|
+
deserialize untrusted data unsafely, or exfiltrate source code, SQL,
|
|
36
|
+
or environment values it isn't supposed to touch (see the data
|
|
37
|
+
handling guarantees in the technical specification, section 22).
|
|
38
|
+
|
|
39
|
+
## What's out of scope
|
|
40
|
+
|
|
41
|
+
- The example Rails application under `examples/`, which is
|
|
42
|
+
illustrative only and not meant to be run as a real service.
|
|
43
|
+
- Vulnerabilities that require the attacker to already control
|
|
44
|
+
`baseline.yml` or the workload specs in a repository that has
|
|
45
|
+
chosen to run Baseline (i.e. arbitrary Ruby code a repository owner
|
|
46
|
+
chose to execute in their own CI).
|
|
47
|
+
|
|
48
|
+
## Baseline's security posture
|
|
49
|
+
|
|
50
|
+
By design, Baseline:
|
|
51
|
+
|
|
52
|
+
- makes no mandatory network requests;
|
|
53
|
+
- never sends source code, SQL text, bind values, or environment
|
|
54
|
+
values off the machine it runs on;
|
|
55
|
+
- treats imported result bundles and archives as untrusted input,
|
|
56
|
+
parses them as JSON (never `Marshal` or other Ruby object
|
|
57
|
+
deserialization formats), and rejects path traversal in archive
|
|
58
|
+
entries;
|
|
59
|
+
- keeps telemetry opt-in and disabled by default (see
|
|
60
|
+
[docs/telemetry.md](docs/telemetry.md)).
|
|
61
|
+
|
|
62
|
+
A vulnerability report that Baseline violates one of these guarantees
|
|
63
|
+
is always in scope, even if it wasn't listed above.
|
data/docs/README.md
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# Baseline Documentation
|
|
2
|
+
|
|
3
|
+
This directory will hold user-facing documentation (installation, CLI
|
|
4
|
+
reference, configuration reference, CI integration guides) as the gem is
|
|
5
|
+
implemented. Until then, refer to
|
|
6
|
+
[../baseline_oss_mvp_technical_spec_and_roadmap.md](../../baseline_oss_mvp_technical_spec_and_roadmap.md)
|
|
7
|
+
for the authoritative specification.
|
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
# Architecture
|
|
2
|
+
|
|
3
|
+
Baseline is a layered pipeline: each layer has a single responsibility and hands
|
|
4
|
+
off a well-typed value to the next. No layer reaches backwards.
|
|
5
|
+
|
|
6
|
+
```
|
|
7
|
+
RSpec examples
|
|
8
|
+
│
|
|
9
|
+
▼
|
|
10
|
+
┌─────────────┐
|
|
11
|
+
│ Discovery │ rspec/discovery.rb, rspec/workload_builder.rb
|
|
12
|
+
│ │ Finds :baseline-tagged examples; builds Workload objects
|
|
13
|
+
└──────┬──────┘
|
|
14
|
+
│ []Workload
|
|
15
|
+
▼
|
|
16
|
+
┌─────────────┐
|
|
17
|
+
│ Execution │ execution/runner.rb, execution/process_runner.rb
|
|
18
|
+
│ │ Warmup → measured samples; isolates each run in a subprocess
|
|
19
|
+
└──────┬──────┘
|
|
20
|
+
│ raw sample arrays (ns integers)
|
|
21
|
+
▼
|
|
22
|
+
┌──────────────────┐
|
|
23
|
+
│ Instrumentation │ instrumentation/{duration,sql_activity,allocations,gc}.rb
|
|
24
|
+
│ │ Wraps Perfgate.measure { } with collectors for each metric
|
|
25
|
+
└────────┬─────────┘
|
|
26
|
+
│ SampleContext per metric
|
|
27
|
+
▼
|
|
28
|
+
┌────────────────┐
|
|
29
|
+
│ Statistics │ statistics/{summary,mann_whitney_u}.rb
|
|
30
|
+
│ │ min/p50/p95/max; Mann-Whitney U for regression significance
|
|
31
|
+
└───────┬────────┘
|
|
32
|
+
│ Summary structs
|
|
33
|
+
▼
|
|
34
|
+
┌──────────────────┐
|
|
35
|
+
│ Fingerprints │ fingerprints/{components,workload_definition,compatibility}.rb
|
|
36
|
+
│ │ Environment + workload hashes; compatibility gate that blocks
|
|
37
|
+
│ │ incomparable runs before any metric decision is made
|
|
38
|
+
└────────┬─────────┘
|
|
39
|
+
│ FingerprintResult
|
|
40
|
+
▼
|
|
41
|
+
┌──────────────────────┐
|
|
42
|
+
│ Comparison::Engine │ comparison/{engine,workload_comparison,metric_decision,
|
|
43
|
+
│ │ statistical_metric_decision,deterministic_metric_decision,
|
|
44
|
+
│ │ diagnostics}.rb
|
|
45
|
+
│ │
|
|
46
|
+
│ Per-metric: │ Duration/allocations → statistical (Mann-Whitney + floor)
|
|
47
|
+
│ │ SQL count → deterministic (exact delta)
|
|
48
|
+
│ │ GC → informational only
|
|
49
|
+
└──────────┬───────────┘
|
|
50
|
+
│ ComparisonResult
|
|
51
|
+
▼
|
|
52
|
+
┌──────────────────┐
|
|
53
|
+
│ Policy::Engine │ policy/engine.rb
|
|
54
|
+
│ │ Maps workload-level PASS/WARN/FAIL/INCOMPARABLE →
|
|
55
|
+
│ │ overall status + exit code (0–5 per spec §21)
|
|
56
|
+
└──────────┬───────┘
|
|
57
|
+
│ PolicyResult (status, exit_code)
|
|
58
|
+
▼
|
|
59
|
+
┌──────────────────┐
|
|
60
|
+
│ Report │ report/{console,markdown}.rb
|
|
61
|
+
│ │ Human-readable console table or GitHub-flavored Markdown
|
|
62
|
+
└──────────┬───────┘
|
|
63
|
+
│ String
|
|
64
|
+
▼
|
|
65
|
+
stdout / summary.md
|
|
66
|
+
|
|
67
|
+
────────────────────────────────────────────────────────
|
|
68
|
+
Cross-cutting concerns (not in the pipeline)
|
|
69
|
+
────────────────────────────────────────────────────────
|
|
70
|
+
|
|
71
|
+
Config config/{schema,validator,defaults,env_overrides}.rb
|
|
72
|
+
Single Config object loaded once; env vars layer on top of YAML.
|
|
73
|
+
|
|
74
|
+
Storage storage/{adapter,filesystem,archive}.rb
|
|
75
|
+
Filesystem adapter writes versioned JSON bundles.
|
|
76
|
+
import/export_archive produce .tar.gz for CI artifact hand-off.
|
|
77
|
+
|
|
78
|
+
Serialization serialization/run_result.rb
|
|
79
|
+
RunResult ↔ JSON; forwards-compatible with schema versioning.
|
|
80
|
+
|
|
81
|
+
CLI cli/{run_command,compare_command,run_comparison_reporter}.rb
|
|
82
|
+
Thin dispatcher; each subcommand is a callable object.
|
|
83
|
+
`baseline run --compare PATH --format markdown` is the canonical
|
|
84
|
+
one-step CI command.
|
|
85
|
+
|
|
86
|
+
Errors errors.rb
|
|
87
|
+
Typed error hierarchy; CLI maps each class to an exit code.
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Key design decisions
|
|
91
|
+
|
|
92
|
+
**Subprocess isolation.** Each workload runs in a forked child process (or a
|
|
93
|
+
fresh subprocess when fork is unavailable). This prevents metric leakage between
|
|
94
|
+
workloads and matches the Rails parallel-test-runner constraint that SQLite
|
|
95
|
+
requires a file-backed database.
|
|
96
|
+
|
|
97
|
+
**Fingerprint-first.** Compatibility is checked before any metric comparison. If
|
|
98
|
+
the environment fingerprint is incompatible the whole comparison is INCOMPARABLE;
|
|
99
|
+
if an individual workload's definition changed it is flagged as modified. This
|
|
100
|
+
prevents silent comparisons across incompatible runs.
|
|
101
|
+
|
|
102
|
+
**Two comparison strategies.** Continuous metrics (duration, allocations) use
|
|
103
|
+
Mann-Whitney U so random OS noise does not produce false alarms. SQL query count
|
|
104
|
+
is deterministic — it should not vary between runs on the same code, so any
|
|
105
|
+
change is significant.
|
|
106
|
+
|
|
107
|
+
**Practical floor beats statistics alone.** A statistically-significant
|
|
108
|
+
difference that is smaller than `minimum_absolute_ms` (default 10 ms) is
|
|
109
|
+
downgraded from FAIL to WARN. A sub-noise-ratio change is downgraded further
|
|
110
|
+
to PASS. This prevents microscopic regressions from blocking PRs.
|
|
111
|
+
|
|
112
|
+
**One source of truth for thresholds.** `baseline.yml` controls every
|
|
113
|
+
comparison and policy knob. Environment variables may override values for CI
|
|
114
|
+
parameterisation but cannot introduce new keys.
|
|
115
|
+
|
|
116
|
+
## Adding a new metric
|
|
117
|
+
|
|
118
|
+
1. Add a collector in `instrumentation/` that captures `before`/`after` values
|
|
119
|
+
and returns a delta in `SampleContext`.
|
|
120
|
+
2. Register it in `Instrumentation` and add it to `RunResult`'s schema.
|
|
121
|
+
3. Choose a comparison strategy (statistical or deterministic) and add a rule
|
|
122
|
+
to `Comparison::Engine`.
|
|
123
|
+
4. Add a policy threshold key to `Config::Schema` with a safe default that
|
|
124
|
+
makes the metric informational-only until the user opts in.
|
|
125
|
+
5. Update `Report::Console` and `Report::Markdown` to surface it.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Compatibility Matrix
|
|
2
|
+
|
|
3
|
+
Baseline's target support matrix, per the technical specification
|
|
4
|
+
(section 23):
|
|
5
|
+
|
|
6
|
+
| Component | Target |
|
|
7
|
+
| ----------------- | ---------------------------------------- |
|
|
8
|
+
| Ruby | 3.2, 3.3, 3.4 |
|
|
9
|
+
| Rails | 7.1, 7.2, 8.0 |
|
|
10
|
+
| RSpec Core | 3.12+ |
|
|
11
|
+
| Database | PostgreSQL and MySQL, via Active Record |
|
|
12
|
+
| CI runner | Linux (GitHub Actions) |
|
|
13
|
+
| Local development | macOS, best-effort |
|
|
14
|
+
| Windows | Not supported in the MVP |
|
|
15
|
+
|
|
16
|
+
## What this repository's own CI currently exercises
|
|
17
|
+
|
|
18
|
+
This is a gap to close before a public release, not a promise already
|
|
19
|
+
kept. As of Milestone 5, this repository's own test suite
|
|
20
|
+
(`.github/workflows/ruby.yml`) only runs:
|
|
21
|
+
|
|
22
|
+
- Ruby 3.2.2
|
|
23
|
+
- ActiveRecord 7.1 + SQLite (used by the SQL instrumentation specs,
|
|
24
|
+
which exercise real `ActiveSupport::Notifications` events rather
|
|
25
|
+
than stubs)
|
|
26
|
+
- No MySQL, no PostgreSQL, no Rails 7.2/8.0, no Ruby 3.3/3.4
|
|
27
|
+
|
|
28
|
+
Closing this gap means:
|
|
29
|
+
|
|
30
|
+
- adding a Ruby version matrix (3.2, 3.3, 3.4) to the CI workflow;
|
|
31
|
+
- adding a Rails version matrix (7.1, 7.2, 8.0) via Appraisal or a
|
|
32
|
+
similar Gemfile-matrix approach;
|
|
33
|
+
- adding a PostgreSQL and a MySQL service to CI and running the SQL
|
|
34
|
+
instrumentation specs against both, not just SQLite;
|
|
35
|
+
- verifying `perfgate doctor` (once implemented) correctly classifies
|
|
36
|
+
any of the above outside this matrix as incompatible rather than
|
|
37
|
+
silently comparing.
|
|
38
|
+
|
|
39
|
+
## Fingerprint compatibility, not just supported versions
|
|
40
|
+
|
|
41
|
+
Being in the target matrix is necessary but not sufficient for two
|
|
42
|
+
runs to be compared. `Fingerprints::Compatibility` (Milestone 3) is
|
|
43
|
+
the actual gate: it compares Ruby engine/version, Rails version,
|
|
44
|
+
Baseline's own major version, database adapter/version, and the
|
|
45
|
+
workload's own definition and dataset hashes, and marks a comparison
|
|
46
|
+
`incompatible` if any of the "strict" fields differ. The matrix above
|
|
47
|
+
describes what Baseline is *tested against* -- the fingerprint
|
|
48
|
+
mechanism is what protects a specific comparison from ever silently
|
|
49
|
+
running across incompatible environments.
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# Introducing Baseline: a CI-native performance gate for Rails, built on RSpec
|
|
2
|
+
|
|
3
|
+
*Draft launch article. Adjust tone, add real screenshots/output, and
|
|
4
|
+
link a public repository before publishing.*
|
|
5
|
+
|
|
6
|
+
## The problem
|
|
7
|
+
|
|
8
|
+
Most Rails teams find out about a performance regression one of two
|
|
9
|
+
ways: a customer complains, or an on-call engineer gets paged. Load
|
|
10
|
+
testing exists, but it's usually a separate, heavyweight process that
|
|
11
|
+
runs occasionally, not on every pull request. The result is that a
|
|
12
|
+
change that quietly adds an N+1 query, or turns a fast endpoint into a
|
|
13
|
+
slow one, often ships and sits in production for weeks before anyone
|
|
14
|
+
notices.
|
|
15
|
+
|
|
16
|
+
Meanwhile, your team already writes RSpec examples that exercise the
|
|
17
|
+
exact code paths that matter -- the checkout flow, the search endpoint,
|
|
18
|
+
the background job that processes an order. Those examples know how to
|
|
19
|
+
set up the right data and call the right code. What they don't do is
|
|
20
|
+
tell you whether that code got slower.
|
|
21
|
+
|
|
22
|
+
## What Baseline does
|
|
23
|
+
|
|
24
|
+
Baseline turns selected RSpec examples into repeatable performance
|
|
25
|
+
workloads. It runs each one several times in an isolated process,
|
|
26
|
+
measures wall-clock duration, SQL query count and duration, and object
|
|
27
|
+
allocations, and produces a versioned result bundle. On a pull request,
|
|
28
|
+
it compares that bundle against a result from your default branch and
|
|
29
|
+
answers one question:
|
|
30
|
+
|
|
31
|
+
> Did this change introduce a material, reproducible performance
|
|
32
|
+
> regression?
|
|
33
|
+
|
|
34
|
+
The answer comes back as a clear PASS, WARN, or FAIL, with a
|
|
35
|
+
console summary and a Markdown report explaining *why*:
|
|
36
|
+
|
|
37
|
+
```text
|
|
38
|
+
Baseline Performance Assurance
|
|
39
|
+
|
|
40
|
+
Overall: FAIL
|
|
41
|
+
Baseline: main@1a2b3c4
|
|
42
|
+
Candidate: feature/checkout@9d8e7f6
|
|
43
|
+
|
|
44
|
+
✗ checkout.create_order
|
|
45
|
+
Duration 281 ms → 337 ms +19.9% FAIL
|
|
46
|
+
SQL queries 14 → 19 +5 FAIL
|
|
47
|
+
SQL duration 51 ms → 73 ms +43.1% FAIL
|
|
48
|
+
Allocations 18.4k → 19.1k +3.8% PASS
|
|
49
|
+
|
|
50
|
+
Likely signal:
|
|
51
|
+
SQL query count increased by 5.
|
|
52
|
+
|
|
53
|
+
Compatibility: compatible
|
|
54
|
+
Samples: 8 baseline / 8 candidate
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Why not just look at duration?
|
|
58
|
+
|
|
59
|
+
Wall-clock duration on a shared CI runner is noisy. Two runs of
|
|
60
|
+
identical code can easily differ by 10-20% just from scheduling noise.
|
|
61
|
+
Baseline treats duration as one signal among several, downgrades
|
|
62
|
+
low-confidence results instead of crying wolf, and refuses to compare
|
|
63
|
+
runs from environments it isn't confident are equivalent -- a different
|
|
64
|
+
Ruby or Rails version, a changed workload definition, or an
|
|
65
|
+
incompatible dataset all mark a comparison `incompatible` rather than
|
|
66
|
+
silently producing a misleading result.
|
|
67
|
+
|
|
68
|
+
## Built for CI, not a hosted product
|
|
69
|
+
|
|
70
|
+
There's no account to create and no dashboard to log into. Baseline
|
|
71
|
+
stores its result bundles as plain, versioned JSON on your own
|
|
72
|
+
filesystem or CI artifact storage. A documented GitHub Actions workflow
|
|
73
|
+
downloads your default branch's last result, runs your workloads, and
|
|
74
|
+
publishes a job summary -- all with `GITHUB_TOKEN`, no third-party
|
|
75
|
+
service in the loop.
|
|
76
|
+
|
|
77
|
+
## Try it
|
|
78
|
+
|
|
79
|
+
```ruby
|
|
80
|
+
# Gemfile
|
|
81
|
+
gem "perfgate", group: :test
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Tag an existing request spec or job spec with `baseline: true`, wrap the
|
|
85
|
+
part you care about in `Perfgate.measure { ... }`, and you have your
|
|
86
|
+
first workload. See
|
|
87
|
+
[docs/onboarding.md](onboarding.md) for a full walkthrough, including
|
|
88
|
+
wiring up the GitHub Actions workflow.
|
|
89
|
+
|
|
90
|
+
## Where this is going
|
|
91
|
+
|
|
92
|
+
Baseline is early. The MVP focuses on Rails + RSpec, GitHub Actions,
|
|
93
|
+
and a conservative, explainable comparison engine over a broad feature
|
|
94
|
+
set. We'd rather earn trust on a narrow surface than ship something
|
|
95
|
+
that produces confusing or noisy results. If you try it and hit a
|
|
96
|
+
false positive, a confusing report, or a missing feature, please open
|
|
97
|
+
an issue -- that feedback is exactly what shapes the next milestone.
|
data/docs/onboarding.md
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
# Design-Partner Onboarding Guide
|
|
2
|
+
|
|
3
|
+
This is the walkthrough for an early adopter team installing Baseline
|
|
4
|
+
for the first time, aimed squarely at Milestone 5's exit criterion:
|
|
5
|
+
completing installation without the maintainer touching your repo.
|
|
6
|
+
|
|
7
|
+
It reflects what's actually implemented today. `perfgate init`,
|
|
8
|
+
`report`, `doctor`, and `schema` are not built yet -- everything below
|
|
9
|
+
uses only `perfgate run` and `perfgate compare`.
|
|
10
|
+
|
|
11
|
+
## 1. Add the gem
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
# Gemfile
|
|
15
|
+
gem "perfgate", path: "../baseline", group: :test # or a git ref, until published
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
bundle install
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
No `baseline.yml` is required to get started -- a missing config file
|
|
23
|
+
is treated as pure defaults (8 samples, 2 warmup iterations, all
|
|
24
|
+
metrics enabled). Add one later once you want to tune thresholds or
|
|
25
|
+
policy.
|
|
26
|
+
|
|
27
|
+
## 2. Tag your first workload
|
|
28
|
+
|
|
29
|
+
Pick one existing request spec, job spec, or similar RSpec example
|
|
30
|
+
that exercises a code path you care about. Add `baseline: true` to its
|
|
31
|
+
metadata, and wrap only the part you want measured in
|
|
32
|
+
`Perfgate.measure`:
|
|
33
|
+
|
|
34
|
+
```ruby
|
|
35
|
+
RSpec.describe "Checkout", type: :request, baseline: true do
|
|
36
|
+
it "creates an order" do
|
|
37
|
+
sign_in(create(:user))
|
|
38
|
+
cart = create(:cart, :with_line_items)
|
|
39
|
+
|
|
40
|
+
Perfgate.measure do
|
|
41
|
+
post "/checkout", params: { cart_id: cart.id }
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
expect(response).to have_http_status(:created)
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Only code inside `Perfgate.measure` is timed and has its SQL/allocation
|
|
50
|
+
metrics collected -- sign-in, fixture creation, and response assertions
|
|
51
|
+
outside the block are excluded on purpose (spec section 9.3).
|
|
52
|
+
|
|
53
|
+
See [examples/rails-rspec-app/spec/requests/checkout_spec.rb](../examples/rails-rspec-app/spec/requests/checkout_spec.rb)
|
|
54
|
+
and [.../spec/jobs/invoice_job_spec.rb](../examples/rails-rspec-app/spec/jobs/invoice_job_spec.rb)
|
|
55
|
+
for a request-spec and a job-spec example side by side.
|
|
56
|
+
|
|
57
|
+
## 3. Run it locally
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
bundle exec baseline run --output .baseline/current
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
This discovers every `baseline: true`-tagged example, runs its warmup
|
|
64
|
+
+ samples in an isolated process, and writes a versioned result bundle
|
|
65
|
+
to `.baseline/current/runs/<run-id>/`.
|
|
66
|
+
|
|
67
|
+
## 4. Compare two runs
|
|
68
|
+
|
|
69
|
+
Run it again (ideally after making a change you'd expect to matter),
|
|
70
|
+
then compare:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
bundle exec baseline compare \
|
|
74
|
+
--baseline .baseline/current \
|
|
75
|
+
--candidate .baseline/new-run \
|
|
76
|
+
--output .baseline/comparisons
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
You'll get a console report with a PASS/WARN/FAIL decision per metric,
|
|
80
|
+
an overall decision, and a nonzero exit code on FAIL -- see the report
|
|
81
|
+
format in the main [README](../README.md#usage).
|
|
82
|
+
|
|
83
|
+
## 5. Wire up CI
|
|
84
|
+
|
|
85
|
+
Copy [examples/rails-rspec-app/.github/workflows/baseline.yml](../examples/rails-rspec-app/.github/workflows/baseline.yml)
|
|
86
|
+
into `.github/workflows/` in your repository. It:
|
|
87
|
+
|
|
88
|
+
1. downloads the last `baseline-main` artifact (if one exists yet);
|
|
89
|
+
2. runs `baseline run --compare .baseline/reference --format markdown`,
|
|
90
|
+
which runs your workloads and compares them in one step;
|
|
91
|
+
3. publishes the resulting `summary.md` to the GitHub job summary;
|
|
92
|
+
4. re-uploads the artifact when building `main`, so the next PR has
|
|
93
|
+
something to compare against.
|
|
94
|
+
|
|
95
|
+
The very first run on a repository will have nothing to compare
|
|
96
|
+
against yet -- `baseline run --compare` detects the missing baseline
|
|
97
|
+
and reports it as a warning rather than failing the build. After the
|
|
98
|
+
first successful `main` build, every subsequent PR compares against
|
|
99
|
+
it.
|
|
100
|
+
|
|
101
|
+
## 6. Reading your first result
|
|
102
|
+
|
|
103
|
+
- **PASS**: no metric regressed beyond its configured threshold with
|
|
104
|
+
statistical confidence. Merge as usual.
|
|
105
|
+
- **WARN**: something changed, but not enough to be treated as a
|
|
106
|
+
blocking regression (e.g. a metric moved but wasn't statistically
|
|
107
|
+
significant, or the workload/environment changed in a way the
|
|
108
|
+
default policy doesn't block on). Worth a look, not a blocker.
|
|
109
|
+
- **FAIL**: a metric both changed by more than its practical threshold
|
|
110
|
+
*and* is statistically significant given the sample noise. The
|
|
111
|
+
console/Markdown report's "Likely signal" line names the most
|
|
112
|
+
probable contributing metric (e.g. "SQL query count increased by
|
|
113
|
+
5") -- it's a deterministic hint, not a root-cause diagnosis.
|
|
114
|
+
|
|
115
|
+
## Getting help
|
|
116
|
+
|
|
117
|
+
If something doesn't work as described here, or the report doesn't
|
|
118
|
+
make sense, please open an issue (see
|
|
119
|
+
[CONTRIBUTING.md](../CONTRIBUTING.md)) rather than working around it
|
|
120
|
+
silently -- unclear reports and rough edges in exactly this kind of
|
|
121
|
+
first-run experience are the most valuable thing for us to hear about
|
|
122
|
+
right now.
|
data/docs/telemetry.md
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Telemetry Specification
|
|
2
|
+
|
|
3
|
+
Baseline collects **no telemetry by default**. This document specifies
|
|
4
|
+
the opt-in telemetry payload as designed in the technical specification
|
|
5
|
+
(section 22), so that anyone considering enabling it -- or auditing
|
|
6
|
+
whether Baseline is safe to run in a sensitive environment -- can see
|
|
7
|
+
exactly what would be sent and what never would be, before any
|
|
8
|
+
transmission code exists.
|
|
9
|
+
|
|
10
|
+
## Current status
|
|
11
|
+
|
|
12
|
+
As of this release, telemetry is **not implemented**. The
|
|
13
|
+
`telemetry.enabled` configuration key exists and defaults to `false`
|
|
14
|
+
(see `lib/baseline/config/defaults.rb`), but no code path currently
|
|
15
|
+
reads it to make a network request. This document describes the
|
|
16
|
+
contract that any future telemetry implementation must honor.
|
|
17
|
+
|
|
18
|
+
## Opt-in only
|
|
19
|
+
|
|
20
|
+
- Telemetry is off unless a user explicitly sets `telemetry.enabled:
|
|
21
|
+
true` in `baseline.yml`, or an equivalent explicit environment
|
|
22
|
+
override.
|
|
23
|
+
- There is no implicit opt-in through usage, installation, or CI
|
|
24
|
+
execution.
|
|
25
|
+
- Baseline must function identically, with no missing features and no
|
|
26
|
+
degraded behavior, whether telemetry is enabled or not.
|
|
27
|
+
- A telemetry send failure (network error, timeout, malformed
|
|
28
|
+
response) must never affect Baseline's exit code, comparison result,
|
|
29
|
+
or CLI output. Telemetry is best-effort and fire-and-forget.
|
|
30
|
+
|
|
31
|
+
## Permitted payload fields
|
|
32
|
+
|
|
33
|
+
If enabled, a telemetry event may only ever contain:
|
|
34
|
+
|
|
35
|
+
- Baseline version
|
|
36
|
+
- Ruby version
|
|
37
|
+
- Rails version
|
|
38
|
+
- RSpec version
|
|
39
|
+
- operating system
|
|
40
|
+
- CI provider
|
|
41
|
+
- number of workloads
|
|
42
|
+
- enabled metric names
|
|
43
|
+
- command success/failure category
|
|
44
|
+
- an anonymous installation ID (randomly generated, not derived from
|
|
45
|
+
any repository or organization identifier)
|
|
46
|
+
|
|
47
|
+
## Never transmitted, under any configuration
|
|
48
|
+
|
|
49
|
+
- repository name or URL
|
|
50
|
+
- organization name
|
|
51
|
+
- source file names
|
|
52
|
+
- workload names
|
|
53
|
+
- metric *values* (durations, SQL counts, allocation counts, etc.)
|
|
54
|
+
- SQL text or bind values
|
|
55
|
+
- environment variable values
|
|
56
|
+
- commit SHA
|
|
57
|
+
- IP-derived geolocation, beyond whatever is inherent to receiving an
|
|
58
|
+
HTTP request (Baseline itself never resolves or stores this)
|
|
59
|
+
|
|
60
|
+
## Why this split
|
|
61
|
+
|
|
62
|
+
The permitted fields are enough to answer aggregate product questions
|
|
63
|
+
("which Ruby/Rails versions are people actually running Baseline
|
|
64
|
+
against?", "does the CLI usually succeed or fail?") without being able
|
|
65
|
+
to reconstruct anything about a specific codebase, its performance
|
|
66
|
+
characteristics, or its data. This mirrors the broader security and
|
|
67
|
+
privacy requirements in spec section 22: no source code leaves the
|
|
68
|
+
process, and no mandatory network requests exist regardless of the
|
|
69
|
+
telemetry setting.
|
|
70
|
+
|
|
71
|
+
## Implementing telemetry (future work)
|
|
72
|
+
|
|
73
|
+
When telemetry transmission is implemented, it must:
|
|
74
|
+
|
|
75
|
+
1. Be added behind the existing `telemetry.enabled` flag, defaulting
|
|
76
|
+
to `false`.
|
|
77
|
+
2. Serialize only the permitted fields above, ideally validated by a
|
|
78
|
+
JSON schema the way run/comparison results already are.
|
|
79
|
+
3. Fail silently (log at most, never raise) on any transmission error.
|
|
80
|
+
4. Be documented in this file, including the exact endpoint and
|
|
81
|
+
retention policy, before it ships in a release.
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
name: Baseline
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
pull_request:
|
|
5
|
+
push:
|
|
6
|
+
branches: [main]
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
baseline:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
steps:
|
|
12
|
+
- uses: actions/checkout@v4
|
|
13
|
+
|
|
14
|
+
- uses: ruby/setup-ruby@v1
|
|
15
|
+
with:
|
|
16
|
+
bundler-cache: true
|
|
17
|
+
|
|
18
|
+
# The MVP has no artifact-lookup helper (spec 19.1), so this just
|
|
19
|
+
# grabs whatever the main branch's most recent workflow run
|
|
20
|
+
# uploaded under this fixed artifact name. continue-on-error
|
|
21
|
+
# covers the very first run on a repository, where no such
|
|
22
|
+
# artifact exists yet -- baseline run then falls back to treating
|
|
23
|
+
# the comparison as a missing baseline instead of failing.
|
|
24
|
+
- name: Download default-branch Baseline
|
|
25
|
+
if: github.event_name == 'pull_request'
|
|
26
|
+
uses: actions/download-artifact@v4
|
|
27
|
+
with:
|
|
28
|
+
name: baseline-main
|
|
29
|
+
path: .baseline/reference
|
|
30
|
+
continue-on-error: true
|
|
31
|
+
|
|
32
|
+
- name: Run Baseline
|
|
33
|
+
run: |
|
|
34
|
+
bundle exec baseline run \
|
|
35
|
+
--output .baseline/current \
|
|
36
|
+
--compare .baseline/reference \
|
|
37
|
+
--format markdown
|
|
38
|
+
|
|
39
|
+
- name: Publish summary
|
|
40
|
+
if: always()
|
|
41
|
+
run: cat .baseline/current/summary.md >> "$GITHUB_STEP_SUMMARY"
|
|
42
|
+
|
|
43
|
+
- name: Upload default-branch Baseline
|
|
44
|
+
if: github.ref == 'refs/heads/main'
|
|
45
|
+
uses: actions/upload-artifact@v4
|
|
46
|
+
with:
|
|
47
|
+
name: baseline-main
|
|
48
|
+
path: .baseline/current
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Example Rails + RSpec App
|
|
2
|
+
|
|
3
|
+
Placeholder for the example Rails application used to demonstrate and
|
|
4
|
+
integration-test Baseline (see section 8 and Prompt 2 of the technical
|
|
5
|
+
specification). The full app is not yet scaffolded.
|
|
6
|
+
|
|
7
|
+
`spec/requests/checkout_spec.rb` and `spec/jobs/invoice_job_spec.rb`
|
|
8
|
+
are illustrative workload examples (Milestone 2 deliverables: a
|
|
9
|
+
request-spec example and a job-spec example, spec section 13.7). They
|
|
10
|
+
show the intended `baseline:`-tagged, `Perfgate.measure`-wrapped shape
|
|
11
|
+
once a real Rails app backs this directory; they are not executed by
|
|
12
|
+
CI yet since there's no app for them to run against.
|
|
13
|
+
|
|
14
|
+
## CI usage
|
|
15
|
+
|
|
16
|
+
`.github/workflows/baseline.yml` reproduces the conceptual workflow
|
|
17
|
+
from spec section 19.1: it downloads whatever `baseline-main` artifact
|
|
18
|
+
the last successful main-branch build published, runs the current
|
|
19
|
+
branch's workloads and compares them against it in one step (`baseline
|
|
20
|
+
run --output .baseline/current --compare .baseline/reference --format
|
|
21
|
+
markdown`), publishes the resulting `summary.md` to the job summary,
|
|
22
|
+
and re-uploads the artifact when building main itself.
|
|
23
|
+
|
|
24
|
+
This is deliberately the MVP-level version the spec calls for:
|
|
25
|
+
|
|
26
|
+
- No GitHub App or API-based artifact lookup -- just the fixed
|
|
27
|
+
`baseline-main` artifact name via `actions/download-artifact`.
|
|
28
|
+
`continue-on-error: true` covers the very first run, before any such
|
|
29
|
+
artifact exists; `baseline run --compare` then reports a missing
|
|
30
|
+
baseline instead of crashing.
|
|
31
|
+
- No provenance checking (spec 19.3) beyond what
|
|
32
|
+
`actions/download-artifact` already gives you for free -- there's no
|
|
33
|
+
verification here that the artifact came from a successful run, the
|
|
34
|
+
configured default branch, or a commit at or before the PR's base.
|
|
35
|
+
A real deployment should tighten this with a small composite action
|
|
36
|
+
once one exists.
|
|
37
|
+
- Job summary only (spec 19.2); no sticky PR comment or check
|
|
38
|
+
annotation yet.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Illustrative job-spec workload (spec section 13.7): explicit in-process
|
|
4
|
+
# job execution, with per-example overrides of the default sample/warmup
|
|
5
|
+
# counts (spec section 9.3) for a more expensive workload. See the note
|
|
6
|
+
# in ../requests/checkout_spec.rb about this file's placeholder status.
|
|
7
|
+
RSpec.describe InvoiceJob, type: :job, perfgate: { samples: 5, warmup: 1 } do
|
|
8
|
+
it "generates an invoice for a completed order" do
|
|
9
|
+
order = create(:order, :completed, line_item_count: 50)
|
|
10
|
+
|
|
11
|
+
Perfgate.measure { described_class.perform_now(order.id) }
|
|
12
|
+
|
|
13
|
+
expect(order.reload.invoice).to be_present
|
|
14
|
+
end
|
|
15
|
+
end
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Illustrative request-spec workload (spec sections 9.1 and 13.7). This
|
|
4
|
+
# file is not currently run by CI: examples/rails-rspec-app is a
|
|
5
|
+
# placeholder until a real Rails app is scaffolded (see the top-level
|
|
6
|
+
# README in this directory). It shows the intended shape of a
|
|
7
|
+
# request-spec workload once that app exists.
|
|
8
|
+
#
|
|
9
|
+
# `baseline: true` opts the example into measurement with the project's
|
|
10
|
+
# default samples/warmup/metrics (spec section 9.3). `Perfgate.measure`
|
|
11
|
+
# scopes SQL/allocation/GC collection to exactly the request under test,
|
|
12
|
+
# excluding sign-in, fixture setup, and response-body assertions.
|
|
13
|
+
RSpec.describe "Checkout", type: :request, baseline: true do
|
|
14
|
+
it "creates an order" do
|
|
15
|
+
sign_in(create(:user))
|
|
16
|
+
cart = create(:cart, :with_line_items)
|
|
17
|
+
|
|
18
|
+
Perfgate.measure do
|
|
19
|
+
post "/checkout", params: { cart_id: cart.id }
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
expect(response).to have_http_status(:created)
|
|
23
|
+
end
|
|
24
|
+
end
|