haveapi 0.30.0 → 0.30.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/Gemfile +1 -0
- data/benchmarks/documentation.rb +277 -0
- data/benchmarks/endpoints.rb +730 -0
- data/benchmarks/performance_budgets.json +1354 -0
- data/benchmarks/performance_compare.rb +579 -0
- data/doc/client-examples.md +233 -24
- data/doc/performance.md +272 -0
- data/haveapi.gemspec +1 -1
- data/lib/haveapi/documentation_clients.rb +158 -35
- data/lib/haveapi/documentation_context.rb +42 -12
- data/lib/haveapi/documentation_schema.rb +73 -0
- data/lib/haveapi/server.rb +5 -0
- data/lib/haveapi/version.rb +1 -1
- data/lib/haveapi/views/version_page.erb +2 -2
- data/spec/client_examples/context_performance_spec.rb +72 -0
- data/spec/client_examples/context_spec.rb +206 -0
- data/spec/client_examples/dynamic_js_spec.rb +1 -0
- data/spec/client_examples/profile_mount_spec.rb +215 -0
- data/spec/client_examples/profiles_spec.rb +202 -32
- data/spec/client_examples/structural_schema_spec.rb +48 -0
- data/spec/documentation/client_examples_spec.rb +10 -0
- data/spec/documentation/performance_spec.rb +79 -0
- data/spec/performance/comparator_spec.rb +342 -0
- data/spec/performance/endpoint_contract_spec.rb +226 -0
- data/spec/performance/endpoint_scaling_spec.rb +82 -0
- data/spec/performance/event_baseline_spec.rb +279 -0
- data/spec/performance/json_activation_spec.rb +151 -0
- data/spec/performance/release_lineage_spec.rb +431 -0
- data/spec/performance/runner_spec.rb +476 -0
- data/spec/performance/sqlite_spec.rb +194 -0
- data/spec/performance/support/live_worker.rb +44 -0
- data/spec/performance/support/samples.rb +72 -0
- data/test_support/documentation_performance_fixture.rb +106 -0
- data/test_support/performance_endpoint_fixture.rb +350 -0
- data/test_support/performance_sqlite_fixture.rb +384 -0
- metadata +26 -3
data/doc/client-examples.md
CHANGED
|
@@ -15,9 +15,96 @@ profile; ordinary servers do not depend on either generator.
|
|
|
15
15
|
|
|
16
16
|
## Configure a generated SDK
|
|
17
17
|
|
|
18
|
-
Register profiles before mounting the server.
|
|
19
|
-
|
|
20
|
-
|
|
18
|
+
Register profiles before mounting the server. For an SDK that matches the current
|
|
19
|
+
API, configure its package identity:
|
|
20
|
+
|
|
21
|
+
```ruby
|
|
22
|
+
api.documentation_clients('1.0',
|
|
23
|
+
go: {
|
|
24
|
+
import_path: 'example.org/widget/client'
|
|
25
|
+
},
|
|
26
|
+
rust: {
|
|
27
|
+
crate_name: 'widget-client'
|
|
28
|
+
}
|
|
29
|
+
)
|
|
30
|
+
api.mount('/api/')
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
`generator_version` defaults to the server's `HaveAPI::VERSION`. Install the
|
|
34
|
+
optional `haveapi-go-client` or `haveapi-rust-client` generator gem at that
|
|
35
|
+
version. An explicit `generator_version` requires the installed generator to
|
|
36
|
+
report that version; it never downloads one. It is independent of the SDK and
|
|
37
|
+
API versions.
|
|
38
|
+
|
|
39
|
+
At the end of mount, the server builds one private structural model per configured
|
|
40
|
+
API version. The model includes authentication resources, routes and changes from
|
|
41
|
+
post-mount hooks, so canonical names and parent arguments use the complete API
|
|
42
|
+
declaration. The private copy excludes examples, prose and parameter defaults.
|
|
43
|
+
HTML requests use the current user's authorized description, fields and samples;
|
|
44
|
+
they never expose the private model or its initialization diagnostics. Without
|
|
45
|
+
a configured profile, the server loads no optional generator and shows no Go or
|
|
46
|
+
Rust panels.
|
|
47
|
+
|
|
48
|
+
Automatic profiles assume that the installed SDK matches the current server's
|
|
49
|
+
generation contract. A package name or version does not prove its contents.
|
|
50
|
+
An older SDK can lack new actions or use different names after a collision or
|
|
51
|
+
route change. Check actual SDK compatibility before advertising it; use a saved
|
|
52
|
+
schema override below when the SDK represents an older or restricted API.
|
|
53
|
+
|
|
54
|
+
### Optional SDK versions and sources
|
|
55
|
+
|
|
56
|
+
Set `module_version: 'v1.2.3'` for Go or `crate_version: '1.2.3'` for Rust when a
|
|
57
|
+
specific package version is needed. Omit the field or use nil to let the user's
|
|
58
|
+
package manager select a release. The server does not query a registry or claim
|
|
59
|
+
to have resolved a version. Omitted versions produce these installation hints:
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
go get example.org/widget/client@latest
|
|
63
|
+
cargo add widget-client
|
|
64
|
+
# Blocking Rust variant:
|
|
65
|
+
cargo add widget-client --features blocking
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Go resolves `@latest` by its normal rules. Cargo selects a compatible release
|
|
69
|
+
according to its resolver, manifest and toolchain; existing lockfiles still
|
|
70
|
+
apply. Explicit Cargo versions remain version requirements, without an added
|
|
71
|
+
exact `=` pin. Latest selection can change over time; supply a version to control
|
|
72
|
+
the selection. Cargo never receives a literal `version = "latest"`. Rust
|
|
73
|
+
examples also show their Tokio and serde_json requirements.
|
|
74
|
+
|
|
75
|
+
For an unpublished Rust SDK, supply an HTTPS Git URL and a full commit SHA:
|
|
76
|
+
|
|
77
|
+
```ruby
|
|
78
|
+
api.documentation_clients('7.0',
|
|
79
|
+
rust: {
|
|
80
|
+
crate_name: 'vpsadmin-client',
|
|
81
|
+
source: {
|
|
82
|
+
git: 'https://github.com/vpsfreecz/vpsadmin-rust-client',
|
|
83
|
+
rev: 'ff876abe17e3301da62c1b6afbda055ec89db9d2'
|
|
84
|
+
}
|
|
85
|
+
}
|
|
86
|
+
)
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
The dependency hint uses that source instead of a registry release:
|
|
90
|
+
|
|
91
|
+
```toml
|
|
92
|
+
# Cargo.toml
|
|
93
|
+
vpsadmin-client = { git = "https://github.com/vpsfreecz/vpsadmin-rust-client", rev = "ff876abe17e3301da62c1b6afbda055ec89db9d2" }
|
|
94
|
+
# The blocking variant also sets features = ["blocking"].
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
`source` accepts only `git` and `rev`. URLs must be absolute HTTPS without
|
|
98
|
+
credentials, query strings, fragments or control characters; revisions must be
|
|
99
|
+
40 hexadecimal characters. Branches, tags, local paths and custom registries are
|
|
100
|
+
not supported. An explicit crate version additionally constrains the selected
|
|
101
|
+
source package. The server never fetches the repository or runs these user
|
|
102
|
+
installation commands. These settings do not regenerate or publish an SDK and
|
|
103
|
+
do not establish that this particular SDK matches a given API surface.
|
|
104
|
+
|
|
105
|
+
### Saved-schema overrides
|
|
106
|
+
|
|
107
|
+
The existing file-based profiles remain supported:
|
|
21
108
|
|
|
22
109
|
```ruby
|
|
23
110
|
api.documentation_clients('1.0',
|
|
@@ -33,24 +120,39 @@ api.documentation_clients('1.0',
|
|
|
33
120
|
generator_version: '0.30.0'
|
|
34
121
|
}
|
|
35
122
|
)
|
|
36
|
-
api.mount('/api/')
|
|
37
123
|
```
|
|
38
124
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
125
|
+
Paths name trusted absolute local files. `description_path` is Go's saved
|
|
126
|
+
machine-readable description, not HTML. `snapshot_path` is Rust's saved schema
|
|
127
|
+
and capture metadata. They select an exact SDK shape in place of the current
|
|
128
|
+
server declaration. Keep the files private and retain them with the SDK release.
|
|
129
|
+
|
|
130
|
+
Rust's legacy `generator_config_path` supplies the crate name/version and full
|
|
131
|
+
packaging configuration. It is mutually exclusive with inline `crate_name` and
|
|
132
|
+
`crate_version`, and can be paired with either a saved snapshot or automatic
|
|
133
|
+
schema. A saved snapshot can instead use inline crate identity. `source` works
|
|
134
|
+
with either identity form. The server validates the full legacy TOML; normal
|
|
135
|
+
inline documentation needs no license, compiler or formatter settings.
|
|
136
|
+
|
|
137
|
+
Saved files are read and validated during registration. Automatic models are
|
|
138
|
+
initialized at the end of mount. Missing files or generators, actual generator
|
|
139
|
+
version mismatches, unknown fields, conflicting identities and invalid schemas
|
|
140
|
+
fail startup. Mounting also rejects unavailable API versions and mismatched Rust
|
|
141
|
+
snapshot versions. Failed mounting publishes no profiles and leaves no usable
|
|
142
|
+
server application. An older explicitly pinned generator can serve a suitable
|
|
143
|
+
legacy file profile, but automatic/inline Rust mode requires the new factory
|
|
144
|
+
interface and never guesses a fallback.
|
|
145
|
+
|
|
146
|
+
A saved SDK can still serve examples after unrelated live additions. Each sample
|
|
147
|
+
must match its saved route, layout and supplied parameter contracts; unavailable
|
|
148
|
+
or stale samples show a limitation. Hidden actions, fields, defaults and samples
|
|
149
|
+
remain hidden from the current user. The server never fetches descriptions,
|
|
150
|
+
regenerates packages or runs a compiler/formatter during an HTML request.
|
|
151
|
+
|
|
152
|
+
Old profile settings work on the new server. Inline settings require the new
|
|
153
|
+
server code; rolling back that code also requires restoring compatible settings.
|
|
154
|
+
Mixed server workers can show different documentation examples without changing
|
|
155
|
+
wire requests or persisted data.
|
|
54
156
|
|
|
55
157
|
## Renderer extensions
|
|
56
158
|
|
|
@@ -59,8 +161,20 @@ Existing renderer positional constructors and `init`, `auth(method, desc)` and
|
|
|
59
161
|
The registry binds a request-local `HaveAPI::ClientExample::Context` through
|
|
60
162
|
`with_context`. It holds the selected version, URLs including the API mount,
|
|
61
163
|
the filtered description, and the current resource, action and sample.
|
|
62
|
-
|
|
63
|
-
|
|
164
|
+
Construction copies and freezes Hash, Array and String data without freezing
|
|
165
|
+
the caller's containers. Each request owns its description snapshot.
|
|
166
|
+
`context.with(...)` creates a distinct frozen context, sharing unchanged fields
|
|
167
|
+
by identity and normalizing only explicitly supplied fields. A shallow-frozen
|
|
168
|
+
input container still needs copying; explicitly supplying an existing owned Hash,
|
|
169
|
+
Array or String also copies it. Renderers must treat context-owned data as
|
|
170
|
+
read-only.
|
|
171
|
+
|
|
172
|
+
The profile is a shared reference; registered profiles and their adapters are
|
|
173
|
+
frozen separately. Other object types retain their existing reference semantics,
|
|
174
|
+
so context construction is not an arbitrary-object deep-freezing contract.
|
|
175
|
+
Subclasses of `Context` continue deriving through their constructors, preserving
|
|
176
|
+
custom initialization and extra keywords; the sharing optimization applies to
|
|
177
|
+
the exact base class.
|
|
64
178
|
|
|
65
179
|
Renderers may return a String or nil through their existing methods. New
|
|
66
180
|
renderers return either `{status: :code, code: source}` or
|
|
@@ -71,9 +185,12 @@ overview and every panel before rendering.
|
|
|
71
185
|
|
|
72
186
|
Generated-client adapters belong to their generator gems and consume context
|
|
73
187
|
readers without depending on the server gem. They resolve names using the
|
|
74
|
-
|
|
75
|
-
HTML request. Keep adapter rendering read-only: configured adapters are frozen
|
|
76
|
-
and shared, while
|
|
188
|
+
complete private generator model and never run a compiler, formatter or generator
|
|
189
|
+
in an HTML request. Keep adapter rendering read-only: configured adapters are frozen
|
|
190
|
+
and shared, while authorized description snapshots belong to individual requests.
|
|
191
|
+
Derived contexts within one request share unchanged owned data. Supplying a
|
|
192
|
+
resource, action or sample still copies that supplied subtree, so a resource with
|
|
193
|
+
many actions can remain costly to render.
|
|
77
194
|
|
|
78
195
|
## Verification
|
|
79
196
|
|
|
@@ -88,6 +205,98 @@ against the real request encoder. It requires the locked Python environment
|
|
|
88
205
|
prepared by `make release-deps` and is included in `make test`. The ordinary
|
|
89
206
|
Ruby renderer checks run without the Python package environment.
|
|
90
207
|
|
|
208
|
+
Deterministic context and HTML regressions run in the ordinary server suite.
|
|
209
|
+
They check description sharing, allocation growth and isolation between users
|
|
210
|
+
without a wall-clock threshold. The Rust adapter suite separately checks that
|
|
211
|
+
normalization avoids unrelated description descendants.
|
|
212
|
+
|
|
213
|
+
### Documentation benchmark
|
|
214
|
+
|
|
215
|
+
Prepare the server bundle in its declared Nix environment, then run the opt-in
|
|
216
|
+
harness from the repository root:
|
|
217
|
+
|
|
218
|
+
```sh
|
|
219
|
+
nix develop .#server-ruby --command bash -c \
|
|
220
|
+
'make --no-print-directory benchmark-documentation > /tmp/documentation.json'
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
The target synchronizes shared documentation before running the harness. Standard
|
|
224
|
+
output is JSON; shell/dependency preparation happens before the redirected command.
|
|
225
|
+
No network server, database, generated SDK compilation or formatter is involved.
|
|
226
|
+
The synthetic fixture uses Basic users `admin` and `member`, both with the local
|
|
227
|
+
test password `fixture-only`.
|
|
228
|
+
|
|
229
|
+
Defaults measure two shapes with eight ordinary actions each: eight resources
|
|
230
|
+
with one action, and one resource with eight actions. Each ordinary action also
|
|
231
|
+
has a privileged sample; the first resource has an additional admin-only action.
|
|
232
|
+
Context lanes derive a prepared parent with a changed profile or a constant small
|
|
233
|
+
sample. The HTML lane performs one real Rack request per trial, with all available
|
|
234
|
+
dynamic renderer panels. Use a minimal smoke run or a larger repeated campaign:
|
|
235
|
+
|
|
236
|
+
```sh
|
|
237
|
+
make benchmark-documentation \
|
|
238
|
+
BENCHMARK_DOCUMENTATION_ARGS='--shape resources --size 1 --warmup 0 --trials 1 --iterations 1'
|
|
239
|
+
make benchmark-documentation \
|
|
240
|
+
BENCHMARK_DOCUMENTATION_ARGS='--shape both --size 64 --samples 2 --parameters 8 --warmup 2 --trials 7 --iterations 500' \
|
|
241
|
+
> /tmp/documentation-large.json
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Run these `make` commands inside the prepared shell. `--shape custom --resources
|
|
245
|
+
N --actions N` controls both dimensions independently; `--lanes context` isolates
|
|
246
|
+
derivation and `--lanes html` isolates requests. `--user member` selects restricted
|
|
247
|
+
help. `--help` lists all options. `--rust` enables the fixture's matching Rust
|
|
248
|
+
profile, includes both Rust variants in HTML, and adds separate async/blocking
|
|
249
|
+
action and authentication adapter lanes. It requires the optional generator's
|
|
250
|
+
Ruby dependencies to be available; ordinary benchmark runs need none of them.
|
|
251
|
+
Prepare both component environments using `nix develop .#server-ruby` and
|
|
252
|
+
`nix develop .#client-rust`. The server bundle alone does not activate the Rust
|
|
253
|
+
dependencies. In the top-level Nix shell, combine the prepared local gem stores:
|
|
254
|
+
|
|
255
|
+
```sh
|
|
256
|
+
env -u RUBYOPT -u RUBYLIB -u BUNDLE_BIN_PATH -u BUNDLE_GEMFILE \
|
|
257
|
+
-u BUNDLE_LOCKFILE -u BUNDLE_PATH -u BUNDLE_FROZEN -u BUNDLER_SETUP \
|
|
258
|
+
GEM_PATH="$PWD/.gems:$PWD/clients/rust/.gems:$(ruby -e 'puts Gem.default_dir')" \
|
|
259
|
+
make benchmark-documentation BENCHMARK_DOCUMENTATION_RUBY=ruby \
|
|
260
|
+
BENCHMARK_DOCUMENTATION_ARGS='--rust --shape both --size 8 --warmup 1 --trials 3 --iterations 100' \
|
|
261
|
+
> /tmp/documentation-rust.json
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
This command runs outside Bundler and records the gem versions actually
|
|
265
|
+
activated. Retain that list alongside both component locks and use the same
|
|
266
|
+
prepared inputs for both revisions. Missing optional dependencies fail the run;
|
|
267
|
+
Rust lanes are never silently omitted. See the
|
|
268
|
+
[Rust adapter guide](../clients/rust/docs/documentation.md#verification).
|
|
269
|
+
|
|
270
|
+
Every lane reports each trial and the median wall seconds, process CPU seconds
|
|
271
|
+
and allocated objects, with the operation count per trial. Wall time uses a
|
|
272
|
+
monotonic clock. A full GC runs before each trial, outside its timer; GC triggered
|
|
273
|
+
by the measured work remains included. Mount/profile construction has separate
|
|
274
|
+
setup metrics. Ownership construction, description selection, warmups, output
|
|
275
|
+
validation, hashing and JSON serialization are outside lane measurements.
|
|
276
|
+
Reports include fixture dimensions, renderer IDs, profile mode, Ruby runtime,
|
|
277
|
+
host, Git HEAD/status/diff digest, loaded source digests, dependency versions and
|
|
278
|
+
available lockfile/harness/fixture digests. HTML and Rust outputs include status, bytes and
|
|
279
|
+
SHA-256; failed or missing output causes a nonzero exit.
|
|
280
|
+
|
|
281
|
+
For a before/after comparison, retain one exact harness and fixture and point
|
|
282
|
+
them at each implementation with `--source-root /absolute/checkout`. Set up
|
|
283
|
+
equivalent dependencies and runtime first. An explicit `--source-revision LABEL`
|
|
284
|
+
can supplement the detected HEAD and is required for an exported source tree.
|
|
285
|
+
Retain exact heads, commands, raw JSON, lock/input identities and host identity.
|
|
286
|
+
Compare dimensions, renderer modes, statuses, bytes and body digests before
|
|
287
|
+
comparing medians. Context `description_shared` deliberately distinguishes the
|
|
288
|
+
old copying implementation. Investigate any output difference; record exact
|
|
289
|
+
volatile-field normalization if needed rather than silently dropping mismatches.
|
|
290
|
+
|
|
291
|
+
These measurements describe the synthetic workload on the recorded host. They
|
|
292
|
+
do not establish production latency. The opt-in harness has no timing threshold;
|
|
293
|
+
the [paired endpoint performance suite](performance.md) supplies required
|
|
294
|
+
quick/weekly CI timing, allocation and query gates, including documentation
|
|
295
|
+
OPTIONS and both HTML shapes. It retains baseline/self noise and blocks
|
|
296
|
+
inconclusive results. Ownership, traversal and allocation regressions cover this
|
|
297
|
+
failure class, while concentrated-resource costs and other future regressions
|
|
298
|
+
still need representative measurements.
|
|
299
|
+
|
|
91
300
|
Each client also has acceptance commands that check executable snippets against
|
|
92
301
|
installed packages or a generated SDK. These use
|
|
93
302
|
rendered source against disposable fixtures; syntax checks alone do not
|
data/doc/performance.md
ADDED
|
@@ -0,0 +1,272 @@
|
|
|
1
|
+
# Endpoint performance checks
|
|
2
|
+
|
|
3
|
+
The Ruby performance suite sends real Rack requests through routing,
|
|
4
|
+
authentication, authorization, typed validation and serialization. It covers
|
|
5
|
+
public, Basic and token requests, valid and invalid typed input, nested resources,
|
|
6
|
+
lists, SQLite pagination and eager loading, and documentation OPTIONS/HTML.
|
|
7
|
+
Deterministic specs check response contracts, isolation, allocation growth and
|
|
8
|
+
query/authorization counts. Paired timing runs compare exact baseline and
|
|
9
|
+
candidate sources with one neutral harness and third-party dependency graph.
|
|
10
|
+
|
|
11
|
+
These are synthetic in-process workloads. SQLite results describe the fixture's
|
|
12
|
+
query behavior; they do not measure production database, network server or
|
|
13
|
+
vpsAdmin latency. The checks detect specific regressions and do not guarantee
|
|
14
|
+
every future performance characteristic.
|
|
15
|
+
|
|
16
|
+
## Local commands
|
|
17
|
+
|
|
18
|
+
Use the repository's Nix environments and prepare dependencies before running
|
|
19
|
+
measurements. `nix develop .#server-ruby` prepares the server bundle; the top-level
|
|
20
|
+
`nix develop` prepares contributor tools. Both profiles render full Rust-enabled
|
|
21
|
+
pages and need the Ruby dependencies from `nix develop .#client-rust`. SDK
|
|
22
|
+
generation, Cargo and formatting are outside these measurements. Resolved
|
|
23
|
+
component lockfiles remain local library inputs and must not be committed.
|
|
24
|
+
|
|
25
|
+
Run deterministic specs in the prepared top-level shell:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
make test-performance
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The runner consumes the same neutral fixtures for both implementation roots,
|
|
32
|
+
using a fresh process for each sample. Keep the harness in a candidate repository
|
|
33
|
+
checkout containing the exact baseline and candidate Git objects. Export both
|
|
34
|
+
source revisions separately, retain their archive hashes and source manifests,
|
|
35
|
+
and run `make doc` in each exported root. Do not point one process at both roots.
|
|
36
|
+
Then run outside Bundler with the prepared gem stores:
|
|
37
|
+
|
|
38
|
+
```sh
|
|
39
|
+
env -u RUBYOPT -u RUBYLIB -u BUNDLE_BIN_PATH -u BUNDLE_GEMFILE \
|
|
40
|
+
-u BUNDLE_LOCKFILE -u BUNDLE_PATH -u BUNDLE_FROZEN -u BUNDLER_SETUP \
|
|
41
|
+
GEM_PATH="$PWD/.gems:$PWD/clients/rust/.gems:$(ruby -e 'puts Gem.default_dir')" \
|
|
42
|
+
make performance PERFORMANCE_ARGS='--baseline-root /absolute/baseline --baseline-sha BASELINE_FULL_SHA --candidate-root /absolute/candidate --candidate-sha CANDIDATE_FULL_SHA --profile quick --budgets servers/ruby/benchmarks/performance_budgets.json --output /absolute/new-evidence-directory'
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
Run inside the prepared top-level Nix shell, from the candidate checkout.
|
|
46
|
+
Replace the paths and full 40-character lowercase SHAs. Use `--profile weekly`
|
|
47
|
+
for the expanded workload. The runner verifies source paths, source identities,
|
|
48
|
+
response contracts and dependency/runtime identities before accepting a result.
|
|
49
|
+
An incompatible graph or fixture contract fails; prepare a separately reviewed
|
|
50
|
+
environment/reference migration instead of mixing versions or relaxing checks.
|
|
51
|
+
|
|
52
|
+
To reclassify retained samples without making requests:
|
|
53
|
+
|
|
54
|
+
```sh
|
|
55
|
+
make performance-compare \
|
|
56
|
+
PERFORMANCE_ARGS='--input /absolute/evidence/samples.json --budgets servers/ruby/benchmarks/performance_budgets.json --output /absolute/new-comparison-directory'
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
The commands invoke `servers/ruby/benchmarks/endpoints.rb run` and `compare`
|
|
60
|
+
directly. `run` retains `samples.json`, `report.json` and process diagnostics.
|
|
61
|
+
`compare` reads compact samples format version 2; unsupported versions are
|
|
62
|
+
rejected without conversion. Historical evidence remains attributable to its
|
|
63
|
+
recorded implementation. `report.json` contains source/profile identity,
|
|
64
|
+
overall outcome and per-workload results, without raw samples or response bodies.
|
|
65
|
+
Exit codes are 0 for a pass
|
|
66
|
+
or an inconclusive timing result, 1 for a regression and 3 for errors, including
|
|
67
|
+
missing workloads, wrong responses, inputs or provenance. An inconclusive report
|
|
68
|
+
remains explicitly inconclusive; a successful process exit does not establish a
|
|
69
|
+
measured performance pass. Deterministic failures and regressions block CI.
|
|
70
|
+
Keep the original evidence and diagnose failures or insufficient timing signal;
|
|
71
|
+
there is no automatic retry loop or baseline update.
|
|
72
|
+
|
|
73
|
+
## Timing and work gates
|
|
74
|
+
|
|
75
|
+
Quick workloads include seven API cases, SQLite page/includes cases, and
|
|
76
|
+
documentation OPTIONS plus both HTML shapes at size 8. List and includes quick
|
|
77
|
+
cases use 32 rows; pagination uses page size 32 over 128 rows. Weekly adds
|
|
78
|
+
member/admin variants, larger payloads, list/includes output up to 1,024 rows,
|
|
79
|
+
page size 32 over up to 10,000 SQLite rows, documentation shapes at size 16 and
|
|
80
|
+
Rust async/blocking action/authentication examples. An output of 1,024 rows uses
|
|
81
|
+
an unpaginated relation; it is not an invalid request above the pagination limit.
|
|
82
|
+
|
|
83
|
+
Both profiles also render complete Rust-enabled HTML pages for admin and member
|
|
84
|
+
users. Quick uses 16 resources; weekly uses 32. Each resource has four actions,
|
|
85
|
+
two samples per action and four parameters. These cases validate the full page,
|
|
86
|
+
Rust panels, authorization boundaries and operation counts. They use relative
|
|
87
|
+
allocation and deterministic checks without inventing fixed-reference allocation
|
|
88
|
+
ceilings for the new shapes. Existing calibrated workload budgets remain intact.
|
|
89
|
+
|
|
90
|
+
These shapes exercise complete pages with several megabytes of HTML, so a
|
|
91
|
+
single request provides substantial measurement work. The weekly shape doubles
|
|
92
|
+
the resource count. Actual batch durations and output sizes belong in each
|
|
93
|
+
run's evidence; these synthetic shapes do not define production latency targets.
|
|
94
|
+
|
|
95
|
+
| Measurement policy | Quick | Weekly |
|
|
96
|
+
| --- | --- | --- |
|
|
97
|
+
| Baseline request work per batch | 1 second | 2 seconds |
|
|
98
|
+
| Minimum adequate observed batch | 750 ms | 1.5 seconds |
|
|
99
|
+
| Baseline/self pairs | 3 | 3 |
|
|
100
|
+
| Baseline/candidate pairs | 5 | 7 |
|
|
101
|
+
| Required agreement | 4 of 5 | 6 of 7 |
|
|
102
|
+
| Relative wall/CPU regression budget | 20% | 10% |
|
|
103
|
+
| Minimum detectable batch increase | 50 ms | 50 ms |
|
|
104
|
+
|
|
105
|
+
A baseline pilot selects a common count of 1–100,000 requests for both revisions.
|
|
106
|
+
One bounded calibration adjustment can increase an undersized baseline batch.
|
|
107
|
+
Reports retain observed duration, chosen count, adjustment and any cap limitation;
|
|
108
|
+
a capped batch below the adequacy threshold is insufficient timing evidence.
|
|
109
|
+
Cheap workloads have two warmup batches of about 100 ms each, outside the
|
|
110
|
+
measurement. If the baseline pilot already meets the profile's work target with
|
|
111
|
+
one expensive request, both revisions use count 1 and no extra request warmups.
|
|
112
|
+
That case measures the first complete request after equivalent fixture setup;
|
|
113
|
+
the candidate cannot choose a different warmup policy.
|
|
114
|
+
|
|
115
|
+
Wall and process CPU time are compared separately after normalization per request.
|
|
116
|
+
The minimum detectable increase is the larger of the profile's relative budget
|
|
117
|
+
and 50 ms divided by the common request count. Paired baseline/self observations
|
|
118
|
+
retain the measured noise. A consistent increase beyond the budget and noise
|
|
119
|
+
fails. Reports distinguish ordinary variation from insufficient timing signal.
|
|
120
|
+
Inconclusive timing remains labeled as such; it does not trigger automatic
|
|
121
|
+
workflow retries. The agreement rules are engineering thresholds, not statistical
|
|
122
|
+
confidence levels.
|
|
123
|
+
|
|
124
|
+
Candidate order alternates. Setup, mount, schema, seeds, token issuance, reset,
|
|
125
|
+
response parsing/digests and counter checks stay outside the timed loop. GC
|
|
126
|
+
remains enabled, including collections triggered by measured requests. A
|
|
127
|
+
30-second watchdog surrounds each request batch, including pilot and warmups.
|
|
128
|
+
This also bounds every individual request without instrumenting the cheap request
|
|
129
|
+
loop. It is supplementary runaway protection, not a regression budget. Setup has
|
|
130
|
+
a separate 120-second timeout. Supervision records source, workload and phase and terminates
|
|
131
|
+
and reaps stuck workers.
|
|
132
|
+
|
|
133
|
+
Within each measurement or warmup, the worker validates each distinct raw body
|
|
134
|
+
once for that scenario and retains an immutable proof of its parsed values and
|
|
135
|
+
output semantics. Every operation still checks the raw digest and exact bytes,
|
|
136
|
+
HTTP status and content type, called index, and execution/authorization/query
|
|
137
|
+
records. A hash collision cannot reuse a proof. Proofs are discarded at the
|
|
138
|
+
sample boundary and never shared between users, scenarios or source revisions.
|
|
139
|
+
Samples retain metrics, one uniform validated counter record with its operation
|
|
140
|
+
count, and distinct response examples with occurrence counts and raw bodies.
|
|
141
|
+
Replay checks those counts, counters, examples and raw/canonical identities,
|
|
142
|
+
including comparison mode, verified own version and source/environment/inputs.
|
|
143
|
+
It compares response occurrence totals across paired samples; it does not retain
|
|
144
|
+
request order or independently reprove each live authorization/counter check.
|
|
145
|
+
The compact artifact is a summary, not an execution transcript.
|
|
146
|
+
|
|
147
|
+
Allocation growth above baseline median plus the larger of 10% or 32 objects
|
|
148
|
+
per request fails. Additional fixed-reference ceilings use measured accepted
|
|
149
|
+
counts with 25% headroom plus 32 objects/request. Exact SQL query and required
|
|
150
|
+
authorization counts prevent a fast but incomplete operation from passing.
|
|
151
|
+
Scaling bounds reflect actual output work; growing responses need not allocate
|
|
152
|
+
a constant amount. Budget changes require measured justification and review.
|
|
153
|
+
Missing or uncalibrated budgets for the existing calibrated workloads fail rather
|
|
154
|
+
than disabling the gate.
|
|
155
|
+
|
|
156
|
+
Both profiles use the already recorded weekly Rust dependency closure as their
|
|
157
|
+
expected runtime identity. Quick includes the Rust manifests, gemspec, resolved
|
|
158
|
+
lock and actually loaded dependencies too. Baseline and candidate must match
|
|
159
|
+
that environment exactly; unavailable or changed identities fail. The shared
|
|
160
|
+
comparison entry point activates the recorded native JSON 2.21.2 before its
|
|
161
|
+
first require, including in unbundled workers. A newer visible or already loaded
|
|
162
|
+
JSON must not silently replace that dependency; a missing or conflicting version
|
|
163
|
+
fails during activation. Historical
|
|
164
|
+
quick allocation calibration and all existing reference/budget bytes are retained
|
|
165
|
+
unchanged, without claiming that the original quick calibration loaded Rust.
|
|
166
|
+
|
|
167
|
+
The root, Ruby server and Rust generator development Gemfiles declare JSON
|
|
168
|
+
2.21.2 so fresh setup installs the recorded native dependency. Comparisons
|
|
169
|
+
accept only that exact declaration change in these three files. They retain
|
|
170
|
+
the original manifest contents, hashes and declaration evidence, then compare
|
|
171
|
+
normalized contents. Replay verifies the transformation from the retained raw
|
|
172
|
+
inputs. Other dependency requirements and manifest contents must match.
|
|
173
|
+
|
|
174
|
+
## GitHub workflow and baseline selection
|
|
175
|
+
|
|
176
|
+
The `servers/ruby performance` workflow runs on pushes, ordinary pull requests,
|
|
177
|
+
manual dispatch and Sunday at 03:00 UTC. It always creates the `Performance gate`
|
|
178
|
+
aggregate check. Relevant-path detection happens after event resolution, inside
|
|
179
|
+
the workflow; an irrelevant valid change can succeed without running benchmarks.
|
|
180
|
+
Source/dependency/fixture/budget/workflow/flake changes trigger measurement.
|
|
181
|
+
Missing event/source evidence fails even when no relevant path can be established.
|
|
182
|
+
|
|
183
|
+
| Event | Candidate | Baseline |
|
|
184
|
+
| --- | --- | --- |
|
|
185
|
+
| Pull request | Exact event head SHA | Merge-base of head and exact event base SHA |
|
|
186
|
+
| Feature push, including a new branch | Event `after` | Merge-base with captured default-branch SHA; compares the whole feature |
|
|
187
|
+
| Default-branch push | Event `after` | Event `before`, covering the whole pushed batch |
|
|
188
|
+
| `haveapi-0.30` push with verified maintenance lineage | Event `after` | Event `before`, covering the whole pushed batch |
|
|
189
|
+
| Feature push with verified maintenance lineage | Event `after` | Merge-base with captured `origin/haveapi-0.30` head |
|
|
190
|
+
| Weekly | Exact scheduled default-branch event SHA | Fixed accepted reference below |
|
|
191
|
+
| Manual with a complete explicit pair | Supplied full candidate SHA | Supplied full baseline SHA |
|
|
192
|
+
| Manual without either SHA | Captured default SHA | Fixed accepted reference below |
|
|
193
|
+
|
|
194
|
+
The immutable accepted reference is
|
|
195
|
+
`b85ec364057f536940d689b09e612512ffffaf82`. Required ancestry is checked. The
|
|
196
|
+
reference and budgets never move automatically. Deleted branches, initial
|
|
197
|
+
default pushes without a predecessor, missing commits, partial manual pairs,
|
|
198
|
+
unrelated history or equal baseline/candidate fail. PRs use the head commit,
|
|
199
|
+
never the synthetic merge commit; default pushes use `before`, never `HEAD^`.
|
|
200
|
+
The weekly profile always requires that fixed baseline, including an explicit
|
|
201
|
+
manual pair. An explicit quick pair may select another ancestor.
|
|
202
|
+
|
|
203
|
+
### Maintained 0.30 release comparisons
|
|
204
|
+
|
|
205
|
+
The original fixed reference remains the measured baseline for weekly runs.
|
|
206
|
+
Its maintenance counterpart is
|
|
207
|
+
`8b7abc097db49893da9cc8fa4395648901fec71d`. The resolver and direct runner accept
|
|
208
|
+
this one nonliteral ancestry relationship only when the counterpart is a literal
|
|
209
|
+
candidate ancestor and the complete binary tree delta from the original reference
|
|
210
|
+
equals the published 0.30.1 metadata commit
|
|
211
|
+
`1c3c98f0d77e0b23c8da1a8937f25c5799864a3b` relative to its parent. They verify
|
|
212
|
+
the exact four trees and canonical full-index delta hash, retain both raw diffs
|
|
213
|
+
and bind the receipt to
|
|
214
|
+
the candidate SHA. Cherry-pick trailers do not establish this proof. A different
|
|
215
|
+
mapping needs a new review; the reference and budget calibration never change.
|
|
216
|
+
Diff rendering uses full blob IDs and explicit formatting options, independent
|
|
217
|
+
of repository-dependent abbreviation and user diff display settings. Ordinary
|
|
218
|
+
master ancestry is checked before probing the maintenance mapping, so a master
|
|
219
|
+
clone does not need maintenance objects for that path.
|
|
220
|
+
Unrelated quick baselines still fail. Maintenance feature selection records both
|
|
221
|
+
the captured default and release heads; missing release refs fail closed.
|
|
222
|
+
|
|
223
|
+
These verified maintenance comparisons reuse the candidate's prepared environment
|
|
224
|
+
for both sources. In addition to the JSON preparation rule above, they normalize
|
|
225
|
+
the shared `0.30.x` patch identity, matching
|
|
226
|
+
local Ruby component VERSION assignments and exactly synchronized internal
|
|
227
|
+
`haveapi-client` requirements. All remaining bytes in dependency manifests, gemspecs,
|
|
228
|
+
Ruby requirements and flake inputs must match, as must the resolved third-party
|
|
229
|
+
runtime graph and retained lock bytes. Raw manifest contents and hashes remain
|
|
230
|
+
bound to each verified source tree alongside their normalized hashes. Replay
|
|
231
|
+
recomputes the transformation and rejects missing, inconsistent or changed inputs.
|
|
232
|
+
|
|
233
|
+
The HTML comparison may normalize the own-version segment of the documentation
|
|
234
|
+
templates' `https://webui.haveapi.org/v<own-version>/#...` anchor hrefs and
|
|
235
|
+
the version in the exact framework-generated HaveAPI footer. The footer must
|
|
236
|
+
occur once and match the verified source's own VERSION; literal examples, other
|
|
237
|
+
footer text and URLs remain exact. Validation runs after the timed loop. Complete raw bodies, byte counts and hashes remain in
|
|
238
|
+
the samples, together with the normalized comparison digest and source version.
|
|
239
|
+
Replay checks these again from the raw body. The exception applies only to the
|
|
240
|
+
HTML documentation workloads on the proven maintenance lineage. JSON/API
|
|
241
|
+
values, other URLs and hosts, example code, authorization markers, panels and all
|
|
242
|
+
other output remain exact. A version change cannot hide a response difference.
|
|
243
|
+
|
|
244
|
+
The workflow uses ephemeral GitHub-hosted Ubuntu workers, `contents: read`,
|
|
245
|
+
disabled checkout credential persistence, ordinary `pull_request` events and no
|
|
246
|
+
secrets, publication context or PR comments. The comparison job has a hard
|
|
247
|
+
15-minute quick limit and 45-minute weekly limit. Measured page sizes and sample
|
|
248
|
+
counts must leave room for preparation, ordinary workloads and evidence
|
|
249
|
+
validation within those limits. Actual hosted campaign duration requires its own
|
|
250
|
+
run receipt; the page probe alone does not prove it. A timeout blocks and needs
|
|
251
|
+
diagnosis. Superseded runs are cancelled only within
|
|
252
|
+
the same event/branch or PR concurrency group.
|
|
253
|
+
|
|
254
|
+
Success, regression, inconclusive and preparation-failure artifacts are retained
|
|
255
|
+
for 30 days. They include event selection, exact SHAs and source trees/archive
|
|
256
|
+
hashes, matched dependency manifests, resolved locks, runtime/input graphs, raw
|
|
257
|
+
samples, reports, process diagnostics and runner exit. The aggregate rejects
|
|
258
|
+
failed, cancelled or unexpectedly skipped required work and requires the retained
|
|
259
|
+
report. A completed comparison can retain inconclusive timing diagnostics while
|
|
260
|
+
its deterministic checks pass; the report keeps that distinction visible.
|
|
261
|
+
Repository rules determine whether this aggregate check is required for merging;
|
|
262
|
+
the workflow does not change branch protection or repository permissions.
|
|
263
|
+
|
|
264
|
+
A newly added workflow's manual dispatch may be unavailable until its definition
|
|
265
|
+
exists on the default branch. Preserve a dispatch refusal and run the same weekly
|
|
266
|
+
CLI locally if needed; disclose that hosted weekly execution remains unverified.
|
|
267
|
+
Workflow enablement alone is not a reason to integrate source into the default
|
|
268
|
+
branch. Push verification still checks the exact published candidate.
|
|
269
|
+
|
|
270
|
+
The separate [documentation benchmark](client-examples.md#documentation-benchmark)
|
|
271
|
+
remains available for diagnosis and custom page shapes. Its JSON trials/medians
|
|
272
|
+
are useful measurements; the paired endpoint runner owns required timing gates.
|
data/haveapi.gemspec
CHANGED
|
@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
|
|
|
15
15
|
s.required_ruby_version = ">= #{File.read('../../.ruby-version').strip}"
|
|
16
16
|
|
|
17
17
|
s.add_dependency 'activesupport', '>= 7.1'
|
|
18
|
-
s.add_dependency 'haveapi-client', '~> 0.30.
|
|
18
|
+
s.add_dependency 'haveapi-client', '~> 0.30.2'
|
|
19
19
|
s.add_dependency 'i18n', '>= 1.6', '< 2'
|
|
20
20
|
s.add_dependency 'json'
|
|
21
21
|
s.add_dependency 'mail'
|