perfgate 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (76) hide show
  1. checksums.yaml +7 -0
  2. data/.rspec +3 -0
  3. data/.rubocop.yml +25 -0
  4. data/CHANGELOG.md +18 -0
  5. data/CONTRIBUTING.md +73 -0
  6. data/LICENSE +201 -0
  7. data/NOT_FINALIZED.md +70 -0
  8. data/README.md +86 -0
  9. data/ROADMAP.md +125 -0
  10. data/Rakefile +12 -0
  11. data/SECURITY.md +63 -0
  12. data/docs/README.md +7 -0
  13. data/docs/architecture.md +125 -0
  14. data/docs/compatibility.md +49 -0
  15. data/docs/launch-article.md +97 -0
  16. data/docs/onboarding.md +122 -0
  17. data/docs/telemetry.md +81 -0
  18. data/examples/rails-rspec-app/.github/workflows/baseline.yml +48 -0
  19. data/examples/rails-rspec-app/README.md +38 -0
  20. data/examples/rails-rspec-app/spec/jobs/invoice_job_spec.rb +15 -0
  21. data/examples/rails-rspec-app/spec/requests/checkout_spec.rb +24 -0
  22. data/exe/perfgate +7 -0
  23. data/lib/perfgate/cli/compare_command.rb +91 -0
  24. data/lib/perfgate/cli/run_command.rb +123 -0
  25. data/lib/perfgate/cli/run_comparison_reporter.rb +77 -0
  26. data/lib/perfgate/cli.rb +60 -0
  27. data/lib/perfgate/comparison/deterministic_metric_decision.rb +34 -0
  28. data/lib/perfgate/comparison/diagnostics.rb +70 -0
  29. data/lib/perfgate/comparison/engine.rb +79 -0
  30. data/lib/perfgate/comparison/metric_change.rb +84 -0
  31. data/lib/perfgate/comparison/metric_decision.rb +39 -0
  32. data/lib/perfgate/comparison/statistical_metric_decision.rb +75 -0
  33. data/lib/perfgate/comparison/workload_comparison.rb +98 -0
  34. data/lib/perfgate/config/defaults.rb +70 -0
  35. data/lib/perfgate/config/env_overrides.rb +54 -0
  36. data/lib/perfgate/config/schema.rb +53 -0
  37. data/lib/perfgate/config/validator.rb +64 -0
  38. data/lib/perfgate/config.rb +136 -0
  39. data/lib/perfgate/errors.rb +20 -0
  40. data/lib/perfgate/execution/process_runner.rb +71 -0
  41. data/lib/perfgate/execution/runner.rb +60 -0
  42. data/lib/perfgate/execution/sample_context.rb +66 -0
  43. data/lib/perfgate/fingerprints/compatibility.rb +46 -0
  44. data/lib/perfgate/fingerprints/components.rb +98 -0
  45. data/lib/perfgate/fingerprints/workload_definition.rb +30 -0
  46. data/lib/perfgate/instrumentation/allocations.rb +20 -0
  47. data/lib/perfgate/instrumentation/duration.rb +20 -0
  48. data/lib/perfgate/instrumentation/gc.rb +30 -0
  49. data/lib/perfgate/instrumentation/sql_activity.rb +50 -0
  50. data/lib/perfgate/instrumentation.rb +36 -0
  51. data/lib/perfgate/metrics/.gitkeep +0 -0
  52. data/lib/perfgate/policy/engine.rb +73 -0
  53. data/lib/perfgate/rails/.gitkeep +0 -0
  54. data/lib/perfgate/report/console.rb +57 -0
  55. data/lib/perfgate/report/markdown.rb +101 -0
  56. data/lib/perfgate/reporting/.gitkeep +0 -0
  57. data/lib/perfgate/rspec/discovery.rb +31 -0
  58. data/lib/perfgate/rspec/id_resolver.rb +30 -0
  59. data/lib/perfgate/rspec/workload_builder.rb +48 -0
  60. data/lib/perfgate/rspec.rb +14 -0
  61. data/lib/perfgate/serialization/run_result.rb +58 -0
  62. data/lib/perfgate/statistics/mann_whitney_u.rb +84 -0
  63. data/lib/perfgate/statistics/summary.rb +60 -0
  64. data/lib/perfgate/storage/adapter.rb +24 -0
  65. data/lib/perfgate/storage/archive.rb +69 -0
  66. data/lib/perfgate/storage/filesystem.rb +121 -0
  67. data/lib/perfgate/telemetry/.gitkeep +0 -0
  68. data/lib/perfgate/version.rb +5 -0
  69. data/lib/perfgate/workloads/registry.rb +50 -0
  70. data/lib/perfgate/workloads/workload.rb +26 -0
  71. data/lib/perfgate.rb +46 -0
  72. data/perfgate.gemspec +41 -0
  73. data/schemas/comparison-result-v1.schema.json +7 -0
  74. data/schemas/run-result-v1.schema.json +7 -0
  75. data/sig/perfgate.rbs +4 -0
  76. metadata +139 -0
data/SECURITY.md ADDED
@@ -0,0 +1,63 @@
1
+ # Security Policy
2
+
3
+ ## Supported versions
4
+
5
+ Baseline is pre-1.0. Security fixes are made against the latest release
6
+ on the default branch; there is no long-term support branch yet.
7
+
8
+ | Version | Supported |
9
+ | ------- | --------- |
10
+ | latest | yes |
11
+ | < 0.1 | no |
12
+
13
+ ## Reporting a vulnerability
14
+
15
+ Please do not open a public GitHub issue for suspected security
16
+ vulnerabilities. Instead, use GitHub's private
17
+ [vulnerability reporting](https://github.com/baseline-oss/perfgate/security/advisories/new)
18
+ feature, or email the maintainer directly at bendthe@gmail.com with:
19
+
20
+ - a description of the issue and its potential impact;
21
+ - steps to reproduce, or a minimal repro case;
22
+ - the Baseline version, Ruby version, and Rails version involved.
23
+
24
+ You should receive an acknowledgment within 5 business days. We'll work
25
+ with you to understand and confirm the issue, agree on a disclosure
26
+ timeline, and credit you in the release notes unless you'd prefer to
27
+ stay anonymous.
28
+
29
+ ## What's in scope
30
+
31
+ - The `baseline` gem's CLI, execution engine, comparison/policy logic,
32
+ and filesystem storage adapter.
33
+ - Archive import handling (path traversal, malformed archives).
34
+ - Anything that could cause Baseline to execute untrusted code,
35
+ deserialize untrusted data unsafely, or exfiltrate source code, SQL,
36
+ or environment values it isn't supposed to touch (see the data
37
+ handling guarantees in the technical specification, section 22).
38
+
39
+ ## What's out of scope
40
+
41
+ - The example Rails application under `examples/`, which is
42
+ illustrative only and not meant to be run as a real service.
43
+ - Vulnerabilities that require the attacker to already control
44
+ `baseline.yml` or the workload specs in a repository that has
45
+ chosen to run Baseline (i.e. arbitrary Ruby code a repository owner
46
+ chose to execute in their own CI).
47
+
48
+ ## Baseline's security posture
49
+
50
+ By design, Baseline:
51
+
52
+ - makes no mandatory network requests;
53
+ - never sends source code, SQL text, bind values, or environment
54
+ values off the machine it runs on;
55
+ - treats imported result bundles and archives as untrusted input,
56
+ parses them as JSON (never `Marshal` or other Ruby object
57
+ deserialization formats), and rejects path traversal in archive
58
+ entries;
59
+ - keeps telemetry opt-in and disabled by default (see
60
+ [docs/telemetry.md](docs/telemetry.md)).
61
+
62
+ A vulnerability report that Baseline violates one of these guarantees
63
+ is always in scope, even if it wasn't listed above.
data/docs/README.md ADDED
@@ -0,0 +1,7 @@
1
+ # Baseline Documentation
2
+
3
+ This directory will hold user-facing documentation (installation, CLI
4
+ reference, configuration reference, CI integration guides) as the gem is
5
+ implemented. Until then, refer to
6
+ [../baseline_oss_mvp_technical_spec_and_roadmap.md](../../baseline_oss_mvp_technical_spec_and_roadmap.md)
7
+ for the authoritative specification.
@@ -0,0 +1,125 @@
1
+ # Architecture
2
+
3
+ Baseline is a layered pipeline: each layer has a single responsibility and hands
4
+ off a well-typed value to the next. No layer reaches backwards.
5
+
6
+ ```
7
+ RSpec examples
8
+
9
+
10
+ ┌─────────────┐
11
+ │ Discovery │ rspec/discovery.rb, rspec/workload_builder.rb
12
+ │ │ Finds :baseline-tagged examples; builds Workload objects
13
+ └──────┬──────┘
14
+ │ []Workload
15
+
16
+ ┌─────────────┐
17
+ │ Execution │ execution/runner.rb, execution/process_runner.rb
18
+ │ │ Warmup → measured samples; isolates each run in a subprocess
19
+ └──────┬──────┘
20
+ │ raw sample arrays (ns integers)
21
+
22
+ ┌──────────────────┐
23
+ │ Instrumentation │ instrumentation/{duration,sql_activity,allocations,gc}.rb
24
+ │ │ Wraps Perfgate.measure { } with collectors for each metric
25
+ └────────┬─────────┘
26
+ │ SampleContext per metric
27
+
28
+ ┌────────────────┐
29
+ │ Statistics │ statistics/{summary,mann_whitney_u}.rb
30
+ │ │ min/p50/p95/max; Mann-Whitney U for regression significance
31
+ └───────┬────────┘
32
+ │ Summary structs
33
+
34
+ ┌──────────────────┐
35
+ │ Fingerprints │ fingerprints/{components,workload_definition,compatibility}.rb
36
+ │ │ Environment + workload hashes; compatibility gate that blocks
37
+ │ │ incomparable runs before any metric decision is made
38
+ └────────┬─────────┘
39
+ │ FingerprintResult
40
+
41
+ ┌──────────────────────┐
42
+ │ Comparison::Engine │ comparison/{engine,workload_comparison,metric_decision,
43
+ │ │ statistical_metric_decision,deterministic_metric_decision,
44
+ │ │ diagnostics}.rb
45
+ │ │
46
+ │ Per-metric: │ Duration/allocations → statistical (Mann-Whitney + floor)
47
+ │ │ SQL count → deterministic (exact delta)
48
+ │ │ GC → informational only
49
+ └──────────┬───────────┘
50
+ │ ComparisonResult
51
+
52
+ ┌──────────────────┐
53
+ │ Policy::Engine │ policy/engine.rb
54
+ │ │ Maps workload-level PASS/WARN/FAIL/INCOMPARABLE →
55
+ │ │ overall status + exit code (0–5 per spec §21)
56
+ └──────────┬───────┘
57
+ │ PolicyResult (status, exit_code)
58
+
59
+ ┌──────────────────┐
60
+ │ Report │ report/{console,markdown}.rb
61
+ │ │ Human-readable console table or GitHub-flavored Markdown
62
+ └──────────┬───────┘
63
+ │ String
64
+
65
+ stdout / summary.md
66
+
67
+ ────────────────────────────────────────────────────────
68
+ Cross-cutting concerns (not in the pipeline)
69
+ ────────────────────────────────────────────────────────
70
+
71
+ Config config/{schema,validator,defaults,env_overrides}.rb
72
+ Single Config object loaded once; env vars layer on top of YAML.
73
+
74
+ Storage storage/{adapter,filesystem,archive}.rb
75
+ Filesystem adapter writes versioned JSON bundles.
76
+ import/export_archive produce .tar.gz for CI artifact hand-off.
77
+
78
+ Serialization serialization/run_result.rb
79
+ RunResult ↔ JSON; forwards-compatible with schema versioning.
80
+
81
+ CLI cli/{run_command,compare_command,run_comparison_reporter}.rb
82
+ Thin dispatcher; each subcommand is a callable object.
83
+ `baseline run --compare PATH --format markdown` is the canonical
84
+ one-step CI command.
85
+
86
+ Errors errors.rb
87
+ Typed error hierarchy; CLI maps each class to an exit code.
88
+ ```
89
+
90
+ ## Key design decisions
91
+
92
+ **Subprocess isolation.** Each workload runs in a forked child process (or a
93
+ fresh subprocess when fork is unavailable). This prevents metric leakage between
94
+ workloads and matches the Rails parallel-test-runner constraint that SQLite
95
+ requires a file-backed database.
96
+
97
+ **Fingerprint-first.** Compatibility is checked before any metric comparison. If
98
+ the environment fingerprint is incompatible the whole comparison is INCOMPARABLE;
99
+ if an individual workload's definition changed it is flagged as modified. This
100
+ prevents silent comparisons across incompatible runs.
101
+
102
+ **Two comparison strategies.** Continuous metrics (duration, allocations) use
103
+ Mann-Whitney U so random OS noise does not produce false alarms. SQL query count
104
+ is deterministic — it should not vary between runs on the same code, so any
105
+ change is significant.
106
+
107
+ **Practical floor beats statistics alone.** A statistically-significant
108
+ difference that is smaller than `minimum_absolute_ms` (default 10 ms) is
109
+ downgraded from FAIL to WARN. A sub-noise-ratio change is downgraded further
110
+ to PASS. This prevents microscopic regressions from blocking PRs.
111
+
112
+ **One source of truth for thresholds.** `baseline.yml` controls every
113
+ comparison and policy knob. Environment variables may override values for CI
114
+ parameterisation but cannot introduce new keys.
115
+
116
+ ## Adding a new metric
117
+
118
+ 1. Add a collector in `instrumentation/` that captures `before`/`after` values
119
+ and returns a delta in `SampleContext`.
120
+ 2. Register it in `Instrumentation` and add it to `RunResult`'s schema.
121
+ 3. Choose a comparison strategy (statistical or deterministic) and add a rule
122
+ to `Comparison::Engine`.
123
+ 4. Add a policy threshold key to `Config::Schema` with a safe default that
124
+ makes the metric informational-only until the user opts in.
125
+ 5. Update `Report::Console` and `Report::Markdown` to surface it.
@@ -0,0 +1,49 @@
1
+ # Compatibility Matrix
2
+
3
+ Baseline's target support matrix, per the technical specification
4
+ (section 23):
5
+
6
+ | Component | Target |
7
+ | ----------------- | ---------------------------------------- |
8
+ | Ruby | 3.2, 3.3, 3.4 |
9
+ | Rails | 7.1, 7.2, 8.0 |
10
+ | RSpec Core | 3.12+ |
11
+ | Database | PostgreSQL and MySQL, via Active Record |
12
+ | CI runner | Linux (GitHub Actions) |
13
+ | Local development | macOS, best-effort |
14
+ | Windows | Not supported in the MVP |
15
+
16
+ ## What this repository's own CI currently exercises
17
+
18
+ This is a gap to close before a public release, not a promise already
19
+ kept. As of Milestone 5, this repository's own test suite
20
+ (`.github/workflows/ruby.yml`) only runs:
21
+
22
+ - Ruby 3.2.2
23
+ - ActiveRecord 7.1 + SQLite (used by the SQL instrumentation specs,
24
+ which exercise real `ActiveSupport::Notifications` events rather
25
+ than stubs)
26
+ - No MySQL, no PostgreSQL, no Rails 7.2/8.0, no Ruby 3.3/3.4
27
+
28
+ Closing this gap means:
29
+
30
+ - adding a Ruby version matrix (3.2, 3.3, 3.4) to the CI workflow;
31
+ - adding a Rails version matrix (7.1, 7.2, 8.0) via Appraisal or a
32
+ similar Gemfile-matrix approach;
33
+ - adding a PostgreSQL and a MySQL service to CI and running the SQL
34
+ instrumentation specs against both, not just SQLite;
35
+ - verifying `perfgate doctor` (once implemented) correctly classifies
36
+ any of the above outside this matrix as incompatible rather than
37
+ silently comparing.
38
+
39
+ ## Fingerprint compatibility, not just supported versions
40
+
41
+ Being in the target matrix is necessary but not sufficient for two
42
+ runs to be compared. `Fingerprints::Compatibility` (Milestone 3) is
43
+ the actual gate: it compares Ruby engine/version, Rails version,
44
+ Baseline's own major version, database adapter/version, and the
45
+ workload's own definition and dataset hashes, and marks a comparison
46
+ `incompatible` if any of the "strict" fields differ. The matrix above
47
+ describes what Baseline is *tested against* -- the fingerprint
48
+ mechanism is what protects a specific comparison from ever silently
49
+ running across incompatible environments.
@@ -0,0 +1,97 @@
1
+ # Introducing Baseline: a CI-native performance gate for Rails, built on RSpec
2
+
3
+ *Draft launch article. Adjust tone, add real screenshots/output, and
4
+ link a public repository before publishing.*
5
+
6
+ ## The problem
7
+
8
+ Most Rails teams find out about a performance regression one of two
9
+ ways: a customer complains, or an on-call engineer gets paged. Load
10
+ testing exists, but it's usually a separate, heavyweight process that
11
+ runs occasionally, not on every pull request. The result is that a
12
+ change that quietly adds an N+1 query, or turns a fast endpoint into a
13
+ slow one, often ships and sits in production for weeks before anyone
14
+ notices.
15
+
16
+ Meanwhile, your team already writes RSpec examples that exercise the
17
+ exact code paths that matter -- the checkout flow, the search endpoint,
18
+ the background job that processes an order. Those examples know how to
19
+ set up the right data and call the right code. What they don't do is
20
+ tell you whether that code got slower.
21
+
22
+ ## What Baseline does
23
+
24
+ Baseline turns selected RSpec examples into repeatable performance
25
+ workloads. It runs each one several times in an isolated process,
26
+ measures wall-clock duration, SQL query count and duration, and object
27
+ allocations, and produces a versioned result bundle. On a pull request,
28
+ it compares that bundle against a result from your default branch and
29
+ answers one question:
30
+
31
+ > Did this change introduce a material, reproducible performance
32
+ > regression?
33
+
34
+ The answer comes back as a clear PASS, WARN, or FAIL, with a
35
+ console summary and a Markdown report explaining *why*:
36
+
37
+ ```text
38
+ Baseline Performance Assurance
39
+
40
+ Overall: FAIL
41
+ Baseline: main@1a2b3c4
42
+ Candidate: feature/checkout@9d8e7f6
43
+
44
+ ✗ checkout.create_order
45
+ Duration 281 ms → 337 ms +19.9% FAIL
46
+ SQL queries 14 → 19 +5 FAIL
47
+ SQL duration 51 ms → 73 ms +43.1% FAIL
48
+ Allocations 18.4k → 19.1k +3.8% PASS
49
+
50
+ Likely signal:
51
+ SQL query count increased by 5.
52
+
53
+ Compatibility: compatible
54
+ Samples: 8 baseline / 8 candidate
55
+ ```
56
+
57
+ ## Why not just look at duration?
58
+
59
+ Wall-clock duration on a shared CI runner is noisy. Two runs of
60
+ identical code can easily differ by 10-20% just from scheduling noise.
61
+ Baseline treats duration as one signal among several, downgrades
62
+ low-confidence results instead of crying wolf, and refuses to compare
63
+ runs from environments it isn't confident are equivalent -- a different
64
+ Ruby or Rails version, a changed workload definition, or an
65
+ incompatible dataset all mark a comparison `incompatible` rather than
66
+ silently producing a misleading result.
67
+
68
+ ## Built for CI, not a hosted product
69
+
70
+ There's no account to create and no dashboard to log into. Baseline
71
+ stores its result bundles as plain, versioned JSON on your own
72
+ filesystem or CI artifact storage. A documented GitHub Actions workflow
73
+ downloads your default branch's last result, runs your workloads, and
74
+ publishes a job summary -- all with `GITHUB_TOKEN`, no third-party
75
+ service in the loop.
76
+
77
+ ## Try it
78
+
79
+ ```ruby
80
+ # Gemfile
81
+ gem "perfgate", group: :test
82
+ ```
83
+
84
+ Tag an existing request spec or job spec with `baseline: true`, wrap the
85
+ part you care about in `Perfgate.measure { ... }`, and you have your
86
+ first workload. See
87
+ [docs/onboarding.md](onboarding.md) for a full walkthrough, including
88
+ wiring up the GitHub Actions workflow.
89
+
90
+ ## Where this is going
91
+
92
+ Baseline is early. The MVP focuses on Rails + RSpec, GitHub Actions,
93
+ and a conservative, explainable comparison engine over a broad feature
94
+ set. We'd rather earn trust on a narrow surface than ship something
95
+ that produces confusing or noisy results. If you try it and hit a
96
+ false positive, a confusing report, or a missing feature, please open
97
+ an issue -- that feedback is exactly what shapes the next milestone.
@@ -0,0 +1,122 @@
1
+ # Design-Partner Onboarding Guide
2
+
3
+ This is the walkthrough for an early adopter team installing Baseline
4
+ for the first time, aimed squarely at Milestone 5's exit criterion:
5
+ completing installation without the maintainer touching your repo.
6
+
7
+ It reflects what's actually implemented today. `perfgate init`,
8
+ `report`, `doctor`, and `schema` are not built yet -- everything below
9
+ uses only `perfgate run` and `perfgate compare`.
10
+
11
+ ## 1. Add the gem
12
+
13
+ ```ruby
14
+ # Gemfile
15
+ gem "perfgate", path: "../baseline", group: :test # or a git ref, until published
16
+ ```
17
+
18
+ ```bash
19
+ bundle install
20
+ ```
21
+
22
+ No `baseline.yml` is required to get started -- a missing config file
23
+ is treated as pure defaults (8 samples, 2 warmup iterations, all
24
+ metrics enabled). Add one later once you want to tune thresholds or
25
+ policy.
26
+
27
+ ## 2. Tag your first workload
28
+
29
+ Pick one existing request spec, job spec, or similar RSpec example
30
+ that exercises a code path you care about. Add `baseline: true` to its
31
+ metadata, and wrap only the part you want measured in
32
+ `Perfgate.measure`:
33
+
34
+ ```ruby
35
+ RSpec.describe "Checkout", type: :request, baseline: true do
36
+ it "creates an order" do
37
+ sign_in(create(:user))
38
+ cart = create(:cart, :with_line_items)
39
+
40
+ Perfgate.measure do
41
+ post "/checkout", params: { cart_id: cart.id }
42
+ end
43
+
44
+ expect(response).to have_http_status(:created)
45
+ end
46
+ end
47
+ ```
48
+
49
+ Only code inside `Perfgate.measure` is timed and has its SQL/allocation
50
+ metrics collected -- sign-in, fixture creation, and response assertions
51
+ outside the block are excluded on purpose (spec section 9.3).
52
+
53
+ See [examples/rails-rspec-app/spec/requests/checkout_spec.rb](../examples/rails-rspec-app/spec/requests/checkout_spec.rb)
54
+ and [.../spec/jobs/invoice_job_spec.rb](../examples/rails-rspec-app/spec/jobs/invoice_job_spec.rb)
55
+ for a request-spec and a job-spec example side by side.
56
+
57
+ ## 3. Run it locally
58
+
59
+ ```bash
60
+ bundle exec baseline run --output .baseline/current
61
+ ```
62
+
63
+ This discovers every `baseline: true`-tagged example, runs its warmup
64
+ + samples in an isolated process, and writes a versioned result bundle
65
+ to `.baseline/current/runs/<run-id>/`.
66
+
67
+ ## 4. Compare two runs
68
+
69
+ Run it again (ideally after making a change you'd expect to matter),
70
+ then compare:
71
+
72
+ ```bash
73
+ bundle exec baseline compare \
74
+ --baseline .baseline/current \
75
+ --candidate .baseline/new-run \
76
+ --output .baseline/comparisons
77
+ ```
78
+
79
+ You'll get a console report with a PASS/WARN/FAIL decision per metric,
80
+ an overall decision, and a nonzero exit code on FAIL -- see the report
81
+ format in the main [README](../README.md#usage).
82
+
83
+ ## 5. Wire up CI
84
+
85
+ Copy [examples/rails-rspec-app/.github/workflows/baseline.yml](../examples/rails-rspec-app/.github/workflows/baseline.yml)
86
+ into `.github/workflows/` in your repository. It:
87
+
88
+ 1. downloads the last `baseline-main` artifact (if one exists yet);
89
+ 2. runs `baseline run --compare .baseline/reference --format markdown`,
90
+ which runs your workloads and compares them in one step;
91
+ 3. publishes the resulting `summary.md` to the GitHub job summary;
92
+ 4. re-uploads the artifact when building `main`, so the next PR has
93
+ something to compare against.
94
+
95
+ The very first run on a repository will have nothing to compare
96
+ against yet -- `baseline run --compare` detects the missing baseline
97
+ and reports it as a warning rather than failing the build. After the
98
+ first successful `main` build, every subsequent PR compares against
99
+ it.
100
+
101
+ ## 6. Reading your first result
102
+
103
+ - **PASS**: no metric regressed beyond its configured threshold with
104
+ statistical confidence. Merge as usual.
105
+ - **WARN**: something changed, but not enough to be treated as a
106
+ blocking regression (e.g. a metric moved but wasn't statistically
107
+ significant, or the workload/environment changed in a way the
108
+ default policy doesn't block on). Worth a look, not a blocker.
109
+ - **FAIL**: a metric both changed by more than its practical threshold
110
+ *and* is statistically significant given the sample noise. The
111
+ console/Markdown report's "Likely signal" line names the most
112
+ probable contributing metric (e.g. "SQL query count increased by
113
+ 5") -- it's a deterministic hint, not a root-cause diagnosis.
114
+
115
+ ## Getting help
116
+
117
+ If something doesn't work as described here, or the report doesn't
118
+ make sense, please open an issue (see
119
+ [CONTRIBUTING.md](../CONTRIBUTING.md)) rather than working around it
120
+ silently -- unclear reports and rough edges in exactly this kind of
121
+ first-run experience are the most valuable thing for us to hear about
122
+ right now.
data/docs/telemetry.md ADDED
@@ -0,0 +1,81 @@
1
+ # Telemetry Specification
2
+
3
+ Baseline collects **no telemetry by default**. This document specifies
4
+ the opt-in telemetry payload as designed in the technical specification
5
+ (section 22), so that anyone considering enabling it -- or auditing
6
+ whether Baseline is safe to run in a sensitive environment -- can see
7
+ exactly what would be sent and what never would be, before any
8
+ transmission code exists.
9
+
10
+ ## Current status
11
+
12
+ As of this release, telemetry is **not implemented**. The
13
+ `telemetry.enabled` configuration key exists and defaults to `false`
14
+ (see `lib/baseline/config/defaults.rb`), but no code path currently
15
+ reads it to make a network request. This document describes the
16
+ contract that any future telemetry implementation must honor.
17
+
18
+ ## Opt-in only
19
+
20
+ - Telemetry is off unless a user explicitly sets `telemetry.enabled:
21
+ true` in `baseline.yml`, or an equivalent explicit environment
22
+ override.
23
+ - There is no implicit opt-in through usage, installation, or CI
24
+ execution.
25
+ - Baseline must function identically, with no missing features and no
26
+ degraded behavior, whether telemetry is enabled or not.
27
+ - A telemetry send failure (network error, timeout, malformed
28
+ response) must never affect Baseline's exit code, comparison result,
29
+ or CLI output. Telemetry is best-effort and fire-and-forget.
30
+
31
+ ## Permitted payload fields
32
+
33
+ If enabled, a telemetry event may only ever contain:
34
+
35
+ - Baseline version
36
+ - Ruby version
37
+ - Rails version
38
+ - RSpec version
39
+ - operating system
40
+ - CI provider
41
+ - number of workloads
42
+ - enabled metric names
43
+ - command success/failure category
44
+ - an anonymous installation ID (randomly generated, not derived from
45
+ any repository or organization identifier)
46
+
47
+ ## Never transmitted, under any configuration
48
+
49
+ - repository name or URL
50
+ - organization name
51
+ - source file names
52
+ - workload names
53
+ - metric *values* (durations, SQL counts, allocation counts, etc.)
54
+ - SQL text or bind values
55
+ - environment variable values
56
+ - commit SHA
57
+ - IP-derived geolocation, beyond whatever is inherent to receiving an
58
+ HTTP request (Baseline itself never resolves or stores this)
59
+
60
+ ## Why this split
61
+
62
+ The permitted fields are enough to answer aggregate product questions
63
+ ("which Ruby/Rails versions are people actually running Baseline
64
+ against?", "does the CLI usually succeed or fail?") without being able
65
+ to reconstruct anything about a specific codebase, its performance
66
+ characteristics, or its data. This mirrors the broader security and
67
+ privacy requirements in spec section 22: no source code leaves the
68
+ process, and no mandatory network requests exist regardless of the
69
+ telemetry setting.
70
+
71
+ ## Implementing telemetry (future work)
72
+
73
+ When telemetry transmission is implemented, it must:
74
+
75
+ 1. Be added behind the existing `telemetry.enabled` flag, defaulting
76
+ to `false`.
77
+ 2. Serialize only the permitted fields above, ideally validated by a
78
+ JSON schema the way run/comparison results already are.
79
+ 3. Fail silently (log at most, never raise) on any transmission error.
80
+ 4. Be documented in this file, including the exact endpoint and
81
+ retention policy, before it ships in a release.
@@ -0,0 +1,48 @@
1
+ name: Baseline
2
+
3
+ on:
4
+ pull_request:
5
+ push:
6
+ branches: [main]
7
+
8
+ jobs:
9
+ baseline:
10
+ runs-on: ubuntu-latest
11
+ steps:
12
+ - uses: actions/checkout@v4
13
+
14
+ - uses: ruby/setup-ruby@v1
15
+ with:
16
+ bundler-cache: true
17
+
18
+ # The MVP has no artifact-lookup helper (spec 19.1), so this just
19
+ # grabs whatever the main branch's most recent workflow run
20
+ # uploaded under this fixed artifact name. continue-on-error
21
+ # covers the very first run on a repository, where no such
22
+ # artifact exists yet -- baseline run then falls back to treating
23
+ # the comparison as a missing baseline instead of failing.
24
+ - name: Download default-branch Baseline
25
+ if: github.event_name == 'pull_request'
26
+ uses: actions/download-artifact@v4
27
+ with:
28
+ name: baseline-main
29
+ path: .baseline/reference
30
+ continue-on-error: true
31
+
32
+ - name: Run Baseline
33
+ run: |
34
+ bundle exec baseline run \
35
+ --output .baseline/current \
36
+ --compare .baseline/reference \
37
+ --format markdown
38
+
39
+ - name: Publish summary
40
+ if: always()
41
+ run: cat .baseline/current/summary.md >> "$GITHUB_STEP_SUMMARY"
42
+
43
+ - name: Upload default-branch Baseline
44
+ if: github.ref == 'refs/heads/main'
45
+ uses: actions/upload-artifact@v4
46
+ with:
47
+ name: baseline-main
48
+ path: .baseline/current
@@ -0,0 +1,38 @@
1
+ # Example Rails + RSpec App
2
+
3
+ Placeholder for the example Rails application used to demonstrate and
4
+ integration-test Baseline (see section 8 and Prompt 2 of the technical
5
+ specification). The full app is not yet scaffolded.
6
+
7
+ `spec/requests/checkout_spec.rb` and `spec/jobs/invoice_job_spec.rb`
8
+ are illustrative workload examples (Milestone 2 deliverables: a
9
+ request-spec example and a job-spec example, spec section 13.7). They
10
+ show the intended `baseline:`-tagged, `Perfgate.measure`-wrapped shape
11
+ once a real Rails app backs this directory; they are not executed by
12
+ CI yet since there's no app for them to run against.
13
+
14
+ ## CI usage
15
+
16
+ `.github/workflows/baseline.yml` reproduces the conceptual workflow
17
+ from spec section 19.1: it downloads whatever `baseline-main` artifact
18
+ the last successful main-branch build published, runs the current
19
+ branch's workloads and compares them against it in one step (`baseline
20
+ run --output .baseline/current --compare .baseline/reference --format
21
+ markdown`), publishes the resulting `summary.md` to the job summary,
22
+ and re-uploads the artifact when building main itself.
23
+
24
+ This is deliberately the MVP-level version the spec calls for:
25
+
26
+ - No GitHub App or API-based artifact lookup -- just the fixed
27
+ `baseline-main` artifact name via `actions/download-artifact`.
28
+ `continue-on-error: true` covers the very first run, before any such
29
+ artifact exists; `baseline run --compare` then reports a missing
30
+ baseline instead of crashing.
31
+ - No provenance checking (spec 19.3) beyond what
32
+ `actions/download-artifact` already gives you for free -- there's no
33
+ verification here that the artifact came from a successful run, the
34
+ configured default branch, or a commit at or before the PR's base.
35
+ A real deployment should tighten this with a small composite action
36
+ once one exists.
37
+ - Job summary only (spec 19.2); no sticky PR comment or check
38
+ annotation yet.
@@ -0,0 +1,15 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Illustrative job-spec workload (spec section 13.7): explicit in-process
4
+ # job execution, with per-example overrides of the default sample/warmup
5
+ # counts (spec section 9.3) for a more expensive workload. See the note
6
+ # in ../requests/checkout_spec.rb about this file's placeholder status.
7
+ RSpec.describe InvoiceJob, type: :job, perfgate: { samples: 5, warmup: 1 } do
8
+ it "generates an invoice for a completed order" do
9
+ order = create(:order, :completed, line_item_count: 50)
10
+
11
+ Perfgate.measure { described_class.perform_now(order.id) }
12
+
13
+ expect(order.reload.invoice).to be_present
14
+ end
15
+ end
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Illustrative request-spec workload (spec sections 9.1 and 13.7). This
4
+ # file is not currently run by CI: examples/rails-rspec-app is a
5
+ # placeholder until a real Rails app is scaffolded (see the top-level
6
+ # README in this directory). It shows the intended shape of a
7
+ # request-spec workload once that app exists.
8
+ #
9
+ # `baseline: true` opts the example into measurement with the project's
10
+ # default samples/warmup/metrics (spec section 9.3). `Perfgate.measure`
11
+ # scopes SQL/allocation/GC collection to exactly the request under test,
12
+ # excluding sign-in, fixture setup, and response-body assertions.
13
+ RSpec.describe "Checkout", type: :request, baseline: true do
14
+ it "creates an order" do
15
+ sign_in(create(:user))
16
+ cart = create(:cart, :with_line_items)
17
+
18
+ Perfgate.measure do
19
+ post "/checkout", params: { cart_id: cart.id }
20
+ end
21
+
22
+ expect(response).to have_http_status(:created)
23
+ end
24
+ end