haveapi 0.30.2 → 0.31.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/benchmarks/Gemfile +30 -0
- data/benchmarks/Gemfile.lock +296 -0
- data/benchmarks/endpoints.rb +77 -21
- data/benchmarks/performance_compare.rb +204 -19
- data/benchmarks/shell.nix +21 -0
- data/benchmarks/spec/client_examples_spec.rb +216 -0
- data/doc/client-examples.md +88 -35
- data/doc/performance.md +163 -61
- data/haveapi.gemspec +1 -1
- data/lib/haveapi/client_examples/ts_generated_client.rb +36 -0
- data/lib/haveapi/documentation_clients.rb +49 -6
- data/lib/haveapi/version.rb +1 -1
- data/lib/haveapi/views/version_page.erb +2 -2
- data/spec/client_examples/profile_mount_spec.rb +45 -0
- data/spec/client_examples/ts_generated_client_spec.rb +53 -0
- data/spec/client_examples/typescript_profiles_spec.rb +113 -0
- data/spec/documentation/client_examples_spec.rb +2 -2
- data/spec/performance/comparator_spec.rb +136 -6
- data/spec/performance/event_baseline_spec.rb +185 -30
- data/spec/performance/release_lineage_spec.rb +53 -14
- data/spec/performance/runner_spec.rb +136 -0
- data/spec/performance/source_versions_spec.rb +168 -0
- data/test_support/all_client_performance_fixture.rb +84 -0
- metadata +12 -3
data/doc/client-examples.md
CHANGED
|
@@ -2,20 +2,23 @@
|
|
|
2
2
|
|
|
3
3
|
The Ruby server renders examples from the current user's API description.
|
|
4
4
|
JavaScript and TypeScript examples cover callback and async clients; Python
|
|
5
|
-
examples cover sync and async clients. These clients need no server
|
|
5
|
+
examples cover sync and async clients. These clients need no server
|
|
6
|
+
configuration.
|
|
6
7
|
Ruby, PHP, CLI, curl and HTTP examples are also available. A sample
|
|
7
8
|
that a client cannot represent shows a limitation instead of dropping input or
|
|
8
9
|
inventing a successful result.
|
|
9
10
|
|
|
10
|
-
Go and
|
|
11
|
-
are independent of HaveAPI and the API version. Without a
|
|
11
|
+
Go, Rust and generated TypeScript tabs require a profile for a generated SDK.
|
|
12
|
+
Their package versions are independent of HaveAPI and the API version. Without a
|
|
13
|
+
profile, the server
|
|
12
14
|
hides those clients from the overview and all initialization, authentication and
|
|
13
15
|
action panels. Install the matching generator gem only when configuring its
|
|
14
|
-
profile; ordinary servers do not depend on
|
|
16
|
+
profile; ordinary servers do not depend on these generators.
|
|
15
17
|
|
|
16
18
|
## Configure a generated SDK
|
|
17
19
|
|
|
18
|
-
Register profiles before mounting the server. For an SDK that matches the
|
|
20
|
+
Register profiles before mounting the server. For an SDK that matches the
|
|
21
|
+
current
|
|
19
22
|
API, configure its package identity:
|
|
20
23
|
|
|
21
24
|
```ruby
|
|
@@ -25,25 +28,34 @@ api.documentation_clients('1.0',
|
|
|
25
28
|
},
|
|
26
29
|
rust: {
|
|
27
30
|
crate_name: 'widget-client'
|
|
31
|
+
},
|
|
32
|
+
typescript: {
|
|
33
|
+
package_name: '@example/widget-client'
|
|
28
34
|
}
|
|
29
35
|
)
|
|
30
36
|
api.mount('/api/')
|
|
31
37
|
```
|
|
32
38
|
|
|
33
39
|
`generator_version` defaults to the server's `HaveAPI::VERSION`. Install the
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
40
|
+
matching generator in the server's Ruby environment: `haveapi-go-client`,
|
|
41
|
+
`haveapi-rust-client`, or the source-distributed `haveapi-typescript-client`.
|
|
42
|
+
See the [TypeScript generator guide](../clients/typescript/README.md) for its
|
|
43
|
+
setup. An explicit `generator_version` requires the installed generator to
|
|
44
|
+
report
|
|
45
|
+
that version; the server never downloads one. This version is independent of the
|
|
46
|
+
SDK and API versions.
|
|
47
|
+
|
|
48
|
+
At the end of mount, the server builds one private structural model per
|
|
49
|
+
configured
|
|
50
|
+
API version. The model includes authentication resources, routes and changes
|
|
51
|
+
from
|
|
41
52
|
post-mount hooks, so canonical names and parent arguments use the complete API
|
|
42
53
|
declaration. The private copy excludes examples, prose and parameter defaults.
|
|
43
54
|
HTML requests use the current user's authorized description, fields and samples;
|
|
44
55
|
they never expose the private model or its initialization diagnostics. Without
|
|
45
|
-
a configured profile, the server loads no optional generator and
|
|
46
|
-
Rust panels.
|
|
56
|
+
a configured profile, the server loads no optional generator and hides the Go,
|
|
57
|
+
Rust and generated TypeScript panels. The callback and async TypeScript
|
|
58
|
+
examples remain separate and need no profile.
|
|
47
59
|
|
|
48
60
|
Automatic profiles assume that the installed SDK matches the current server's
|
|
49
61
|
generation contract. A package name or version does not prove its contents.
|
|
@@ -53,8 +65,9 @@ schema override below when the SDK represents an older or restricted API.
|
|
|
53
65
|
|
|
54
66
|
### Optional SDK versions and sources
|
|
55
67
|
|
|
56
|
-
Set `module_version: 'v1.2.3'` for Go
|
|
57
|
-
|
|
68
|
+
Set `module_version: 'v1.2.3'` for Go, `crate_version: '1.2.3'` for Rust or
|
|
69
|
+
`package_version: '1.2.3'` for TypeScript when a specific package version is
|
|
70
|
+
needed. Omit the field or use nil to let the user's
|
|
58
71
|
package manager select a release. The server does not query a registry or claim
|
|
59
72
|
to have resolved a version. Omitted versions produce these installation hints:
|
|
60
73
|
|
|
@@ -63,15 +76,25 @@ go get example.org/widget/client@latest
|
|
|
63
76
|
cargo add widget-client
|
|
64
77
|
# Blocking Rust variant:
|
|
65
78
|
cargo add widget-client --features blocking
|
|
79
|
+
npm install @example/widget-client haveapi-client@0.30.2
|
|
66
80
|
```
|
|
67
81
|
|
|
68
82
|
Go resolves `@latest` by its normal rules. Cargo selects a compatible release
|
|
69
83
|
according to its resolver, manifest and toolchain; existing lockfiles still
|
|
70
84
|
apply. Explicit Cargo versions remain version requirements, without an added
|
|
71
|
-
exact `=` pin. Latest selection can change over time; supply a version to
|
|
85
|
+
exact `=` pin. Latest selection can change over time; supply a version to
|
|
86
|
+
control
|
|
72
87
|
the selection. Cargo never receives a literal `version = "latest"`. Rust
|
|
73
88
|
examples also show their Tokio and serde_json requirements.
|
|
74
89
|
|
|
90
|
+
TypeScript accepts ordinary lowercase npm package names, including scoped names,
|
|
91
|
+
and optional concrete SemVer versions. Tags, version ranges, Git URLs, local
|
|
92
|
+
paths
|
|
93
|
+
and custom factories are not profile options. Configure a package that users can
|
|
94
|
+
install; the server does not check its availability or publish it. Generated
|
|
95
|
+
snippets use the package's public `createClient` export and the published
|
|
96
|
+
`haveapi-client` 0.30.2 runtime.
|
|
97
|
+
|
|
75
98
|
For an unpublished Rust SDK, supply an HTTPS Git URL and a full commit SHA:
|
|
76
99
|
|
|
77
100
|
```ruby
|
|
@@ -118,13 +141,20 @@ api.documentation_clients('1.0',
|
|
|
118
141
|
snapshot_path: '/srv/widget-sdk/api-1.0.snapshot.json',
|
|
119
142
|
generator_config_path: '/srv/widget-sdk/generator.toml',
|
|
120
143
|
generator_version: '0.30.0'
|
|
144
|
+
},
|
|
145
|
+
typescript: {
|
|
146
|
+
package_name: '@example/widget-client',
|
|
147
|
+
package_version: '1.2.3',
|
|
148
|
+
snapshot_path: '/srv/widget-sdk/api-1.0.snapshot.json',
|
|
149
|
+
generator_version: '0.30.0'
|
|
121
150
|
}
|
|
122
151
|
)
|
|
123
152
|
```
|
|
124
153
|
|
|
125
154
|
Paths name trusted absolute local files. `description_path` is Go's saved
|
|
126
|
-
machine-readable description, not HTML.
|
|
127
|
-
and capture metadata.
|
|
155
|
+
machine-readable description, not HTML. For Rust and TypeScript, `snapshot_path`
|
|
156
|
+
selects the saved schema and capture metadata. These files select an exact SDK
|
|
157
|
+
shape in place of the current
|
|
128
158
|
server declaration. Keep the files private and retain them with the SDK release.
|
|
129
159
|
|
|
130
160
|
Rust's legacy `generator_config_path` supplies the crate name/version and full
|
|
@@ -138,7 +168,8 @@ Saved files are read and validated during registration. Automatic models are
|
|
|
138
168
|
initialized at the end of mount. Missing files or generators, actual generator
|
|
139
169
|
version mismatches, unknown fields, conflicting identities and invalid schemas
|
|
140
170
|
fail startup. Mounting also rejects unavailable API versions and mismatched Rust
|
|
141
|
-
snapshot versions. Failed mounting publishes no profiles and
|
|
171
|
+
and TypeScript snapshot versions. Failed mounting publishes no profiles and
|
|
172
|
+
leaves no usable
|
|
142
173
|
server application. An older explicitly pinned generator can serve a suitable
|
|
143
174
|
legacy file profile, but automatic/inline Rust mode requires the new factory
|
|
144
175
|
interface and never guesses a fallback.
|
|
@@ -165,7 +196,8 @@ Construction copies and freezes Hash, Array and String data without freezing
|
|
|
165
196
|
the caller's containers. Each request owns its description snapshot.
|
|
166
197
|
`context.with(...)` creates a distinct frozen context, sharing unchanged fields
|
|
167
198
|
by identity and normalizing only explicitly supplied fields. A shallow-frozen
|
|
168
|
-
input container still needs copying; explicitly supplying an existing owned
|
|
199
|
+
input container still needs copying; explicitly supplying an existing owned
|
|
200
|
+
Hash,
|
|
169
201
|
Array or String also copies it. Renderers must treat context-owned data as
|
|
170
202
|
read-only.
|
|
171
203
|
|
|
@@ -185,16 +217,21 @@ overview and every panel before rendering.
|
|
|
185
217
|
|
|
186
218
|
Generated-client adapters belong to their generator gems and consume context
|
|
187
219
|
readers without depending on the server gem. They resolve names using the
|
|
188
|
-
complete private generator model and never run a compiler, formatter or
|
|
189
|
-
|
|
190
|
-
|
|
220
|
+
complete private generator model and never run a compiler, formatter or
|
|
221
|
+
generator
|
|
222
|
+
in an HTML request. Keep adapter rendering read-only: configured adapters are
|
|
223
|
+
frozen
|
|
224
|
+
and shared, while authorized description snapshots belong to individual
|
|
225
|
+
requests.
|
|
191
226
|
Derived contexts within one request share unchanged owned data. Supplying a
|
|
192
|
-
resource, action or sample still copies that supplied subtree, so a resource
|
|
227
|
+
resource, action or sample still copies that supplied subtree, so a resource
|
|
228
|
+
with
|
|
193
229
|
many actions can remain costly to render.
|
|
194
230
|
|
|
195
231
|
## Verification
|
|
196
232
|
|
|
197
|
-
`make test-documentation` checks renderers, contexts, profiles, rendered HTML
|
|
233
|
+
`make test-documentation` checks renderers, contexts, profiles, rendered HTML
|
|
234
|
+
and
|
|
198
235
|
local assets. Prepare the component bundles and JavaScript development
|
|
199
236
|
dependencies with `make release-deps` first. The target builds the JavaScript
|
|
200
237
|
package before compiling TypeScript examples against its public declarations;
|
|
@@ -210,6 +247,13 @@ They check description sharing, allocation growth and isolation between users
|
|
|
210
247
|
without a wall-clock threshold. The Rust adapter suite separately checks that
|
|
211
248
|
normalization avoids unrelated description descendants.
|
|
212
249
|
|
|
250
|
+
The additional required `make test-performance-clients` contract checks every
|
|
251
|
+
registered renderer and mode with real generated profiles, bounded per-render
|
|
252
|
+
work and allocations, and complete-page allocation scaling. It runs in the
|
|
253
|
+
locked, unbundled benchmark environment, separately from the ordinary server
|
|
254
|
+
bundle. See [performance testing](performance.md#all-client-documentation-contract)
|
|
255
|
+
for preparation, exact coverage and CI behavior.
|
|
256
|
+
|
|
213
257
|
### Documentation benchmark
|
|
214
258
|
|
|
215
259
|
Prepare the server bundle in its declared Nix environment, then run the opt-in
|
|
@@ -220,8 +264,10 @@ nix develop .#server-ruby --command bash -c \
|
|
|
220
264
|
'make --no-print-directory benchmark-documentation > /tmp/documentation.json'
|
|
221
265
|
```
|
|
222
266
|
|
|
223
|
-
The target synchronizes shared documentation before running the harness.
|
|
224
|
-
|
|
267
|
+
The target synchronizes shared documentation before running the harness.
|
|
268
|
+
Standard
|
|
269
|
+
output is JSON; shell/dependency preparation happens before the redirected
|
|
270
|
+
command.
|
|
225
271
|
No network server, database, generated SDK compilation or formatter is involved.
|
|
226
272
|
The synthetic fixture uses Basic users `admin` and `member`, both with the local
|
|
227
273
|
test password `fixture-only`.
|
|
@@ -229,8 +275,10 @@ test password `fixture-only`.
|
|
|
229
275
|
Defaults measure two shapes with eight ordinary actions each: eight resources
|
|
230
276
|
with one action, and one resource with eight actions. Each ordinary action also
|
|
231
277
|
has a privileged sample; the first resource has an additional admin-only action.
|
|
232
|
-
Context lanes derive a prepared parent with a changed profile or a constant
|
|
233
|
-
|
|
278
|
+
Context lanes derive a prepared parent with a changed profile or a constant
|
|
279
|
+
small
|
|
280
|
+
sample. The HTML lane performs one real Rack request per trial, with all
|
|
281
|
+
available
|
|
234
282
|
dynamic renderer panels. Use a minimal smoke run or a larger repeated campaign:
|
|
235
283
|
|
|
236
284
|
```sh
|
|
@@ -242,8 +290,10 @@ make benchmark-documentation \
|
|
|
242
290
|
```
|
|
243
291
|
|
|
244
292
|
Run these `make` commands inside the prepared shell. `--shape custom --resources
|
|
245
|
-
N --actions N` controls both dimensions independently; `--lanes context`
|
|
246
|
-
|
|
293
|
+
N --actions N` controls both dimensions independently; `--lanes context`
|
|
294
|
+
isolates
|
|
295
|
+
derivation and `--lanes html` isolates requests. `--user member` selects
|
|
296
|
+
restricted
|
|
247
297
|
help. `--help` lists all options. `--rust` enables the fixture's matching Rust
|
|
248
298
|
profile, includes both Rust variants in HTML, and adds separate async/blocking
|
|
249
299
|
action and authentication adapter lanes. It requires the optional generator's
|
|
@@ -269,13 +319,16 @@ Rust lanes are never silently omitted. See the
|
|
|
269
319
|
|
|
270
320
|
Every lane reports each trial and the median wall seconds, process CPU seconds
|
|
271
321
|
and allocated objects, with the operation count per trial. Wall time uses a
|
|
272
|
-
monotonic clock. A full GC runs before each trial, outside its timer; GC
|
|
322
|
+
monotonic clock. A full GC runs before each trial, outside its timer; GC
|
|
323
|
+
triggered
|
|
273
324
|
by the measured work remains included. Mount/profile construction has separate
|
|
274
325
|
setup metrics. Ownership construction, description selection, warmups, output
|
|
275
326
|
validation, hashing and JSON serialization are outside lane measurements.
|
|
276
327
|
Reports include fixture dimensions, renderer IDs, profile mode, Ruby runtime,
|
|
277
|
-
host, Git HEAD/status/diff digest, loaded source digests, dependency versions
|
|
278
|
-
|
|
328
|
+
host, Git HEAD/status/diff digest, loaded source digests, dependency versions
|
|
329
|
+
and
|
|
330
|
+
available lockfile/harness/fixture digests. HTML and Rust outputs include
|
|
331
|
+
status, bytes and
|
|
279
332
|
SHA-256; failed or missing output causes a nonzero exit.
|
|
280
333
|
|
|
281
334
|
For a before/after comparison, retain one exact harness and fixture and point
|
data/doc/performance.md
CHANGED
|
@@ -13,41 +13,97 @@ query behavior; they do not measure production database, network server or
|
|
|
13
13
|
vpsAdmin latency. The checks detect specific regressions and do not guarantee
|
|
14
14
|
every future performance characteristic.
|
|
15
15
|
|
|
16
|
-
##
|
|
16
|
+
## All-client documentation contract
|
|
17
|
+
|
|
18
|
+
The required performance workflow first runs `make test-performance-clients`
|
|
19
|
+
against the selected candidate in the prepared, unbundled benchmark shell:
|
|
20
|
+
|
|
21
|
+
```sh
|
|
22
|
+
export HAVEAPI_BENCHMARK_GEM_HOME="$PWD/servers/ruby/benchmarks/.gems"
|
|
23
|
+
nix-shell servers/ruby/benchmarks/shell.nix --run 'make test-performance-clients'
|
|
24
|
+
```
|
|
17
25
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
26
|
+
Prepare the frozen benchmark bundle as described below before running this
|
|
27
|
+
command. It loads the actual Go, Rust and TypeScript generator sources and fails
|
|
28
|
+
if a dependency or profile is missing. It needs no Go, Rust or TypeScript
|
|
29
|
+
compiler. The ordinary `make test-performance` remains a separate server-bundle
|
|
30
|
+
check; both checks are required.
|
|
31
|
+
|
|
32
|
+
The explicit coverage matrix must equal the live renderer registry. Its 15
|
|
33
|
+
entries include every callback, async, blocking and generated variant. Each
|
|
34
|
+
entry renders initialization, Basic, token and OAuth2 authentication, public and
|
|
35
|
+
protected actions, and a failed action sample. Basic authentication and action
|
|
36
|
+
samples must produce code for every renderer. Generated TypeScript's OAuth2
|
|
37
|
+
limitation and the Ruby library/CLI's existing unsupported-OAuth2 comments have
|
|
38
|
+
explicit expectations. New renderer IDs or providers require reviewed coverage;
|
|
39
|
+
they cannot silently disappear from measurement.
|
|
40
|
+
|
|
41
|
+
Direct checks hold the selected action, sample and provider constant while adding
|
|
42
|
+
128 unrelated resource subtrees. Setup and context ownership copying happen
|
|
43
|
+
outside measurement. Three batches of eight renders check allocations, and a
|
|
44
|
+
separate Ruby-call count checks traversal growth; each allows 32 additional
|
|
45
|
+
objects or calls per render. Output must remain identical and inputs unchanged.
|
|
46
|
+
|
|
47
|
+
Complete Rack HTML requests enable all three generated profiles. Checks compare
|
|
48
|
+
every registered initialization/authentication/action panel with real renderer
|
|
49
|
+
output, including successful and failed samples, and exercise admin/member/admin
|
|
50
|
+
isolation through the same adapters. Allocation measurements use pages with
|
|
51
|
+
2, 4 and 8 resources, each with two actions and three admin-visible samples per
|
|
52
|
+
action, plus one admin-only action. Member pages hide the admin sample and action.
|
|
53
|
+
The added-resource allocation increment may grow by at most 2.5 times when the
|
|
54
|
+
resource increment doubles. This permits linear panel/output growth while
|
|
55
|
+
rejecting the measured whole-description-per-example copy regression. The
|
|
56
|
+
fixture uses scalar fields so Go action panels exercise supported code paths.
|
|
57
|
+
|
|
58
|
+
The log retains the named registry, kinds, allocations, Ruby-call counts and
|
|
59
|
+
page byte counts. These deterministic checks measure the selected candidate.
|
|
60
|
+
They use no wall-clock threshold and provide no historical timing result. The
|
|
61
|
+
paired timing suite below retains its original workloads, fixed reference and budgets; its
|
|
62
|
+
reference predates generated TypeScript support. Relevant pushes and pull
|
|
63
|
+
requests, manual runs and scheduled runs must pass the all-client contract before
|
|
64
|
+
the paired comparison. Changes under any client directory trigger the gate.
|
|
24
65
|
|
|
25
|
-
|
|
66
|
+
## Local commands
|
|
67
|
+
|
|
68
|
+
The benchmark has its own third-party tool bundle and pinned Nix shell in
|
|
69
|
+
`servers/ruby/benchmarks/`. This keeps contributor Nix updates separate from the
|
|
70
|
+
calibrated measurement runtime. Prepare the ordinary frozen bundle once:
|
|
26
71
|
|
|
27
72
|
```sh
|
|
28
|
-
|
|
73
|
+
export HAVEAPI_BENCHMARK_GEM_HOME="$PWD/servers/ruby/benchmarks/.gems"
|
|
74
|
+
nix-shell servers/ruby/benchmarks/shell.nix --run '
|
|
75
|
+
set -e
|
|
76
|
+
gem install bundler --version 4.0.23 --no-document
|
|
77
|
+
BUNDLE_IGNORE_CONFIG=1 BUNDLE_FROZEN=1 BUNDLE_PATH__SYSTEM=1 \
|
|
78
|
+
BUNDLE_GEMFILE="$PWD/servers/ruby/benchmarks/Gemfile" bundle _4.0.23_ install
|
|
79
|
+
'
|
|
29
80
|
```
|
|
30
81
|
|
|
31
|
-
The
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
82
|
+
The tool Gemfile installs no HaveAPI package. Its tracked lock is an application
|
|
83
|
+
lock; component library locks remain local contributor inputs. Launch the runner
|
|
84
|
+
outside Bundler so both exported sources use the same Ruby and gem paths. The
|
|
85
|
+
shell clears inherited Bundler settings, `RUBYOPT`, `RUBYLIB` and
|
|
86
|
+
`RUBYGEMS_GEMDEPS`.
|
|
87
|
+
|
|
88
|
+
Run deterministic specs in the prepared contributor shell with
|
|
89
|
+
`make test-performance`. For a comparison, keep the neutral harness in a candidate
|
|
90
|
+
checkout containing both exact Git objects. Export the baseline and candidate
|
|
91
|
+
separately, retain their archive hashes and manifests, and run `make doc` in each
|
|
92
|
+
exported root. Then run from the candidate checkout:
|
|
37
93
|
|
|
38
94
|
```sh
|
|
39
|
-
|
|
40
|
-
-
|
|
41
|
-
|
|
42
|
-
make performance PERFORMANCE_ARGS='--baseline-root /absolute/baseline --baseline-sha BASELINE_FULL_SHA --candidate-root /absolute/candidate --candidate-sha CANDIDATE_FULL_SHA --profile quick --budgets servers/ruby/benchmarks/performance_budgets.json --output /absolute/new-evidence-directory'
|
|
95
|
+
nix-shell servers/ruby/benchmarks/shell.nix --run '
|
|
96
|
+
make performance PERFORMANCE_ARGS="--baseline-root /absolute/baseline --baseline-sha BASELINE_FULL_SHA --candidate-root /absolute/candidate --candidate-sha CANDIDATE_FULL_SHA --profile quick --budgets servers/ruby/benchmarks/performance_budgets.json --output /absolute/new-evidence-directory"
|
|
97
|
+
'
|
|
43
98
|
```
|
|
44
99
|
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
environment
|
|
100
|
+
Replace paths and full 40-character lowercase SHAs. Use `--profile weekly` for
|
|
101
|
+
the expanded workload. The runner checks the exact calibrated Ruby and available
|
|
102
|
+
dependency versions before timing, then requires each worker's complete activated
|
|
103
|
+
graph to match the weekly calibration. Source and loaded-file checks and
|
|
104
|
+
response validation must also pass. An unsupported declaration or runtime change
|
|
105
|
+
requires a separately reviewed environment migration; it does not update the
|
|
106
|
+
reference or budgets.
|
|
51
107
|
|
|
52
108
|
To reclassify retained samples without making requests:
|
|
53
109
|
|
|
@@ -59,14 +115,22 @@ make performance-compare \
|
|
|
59
115
|
The commands invoke `servers/ruby/benchmarks/endpoints.rb run` and `compare`
|
|
60
116
|
directly. `run` retains `samples.json`, `report.json` and process diagnostics.
|
|
61
117
|
`compare` reads compact samples format version 2; unsupported versions are
|
|
62
|
-
rejected without conversion.
|
|
63
|
-
|
|
64
|
-
|
|
118
|
+
rejected without conversion. `report.json` retains the original measurement's
|
|
119
|
+
`harness_sha256`. Its `comparison` metadata identifies policy
|
|
120
|
+
`per-revision-response-stability-v1`, the loaded comparator file's SHA-256 and
|
|
121
|
+
the SHA-256 of the exact consumed `samples.json` bytes. Error reports retain
|
|
122
|
+
these identities when available; they omit unavailable measurement/input fields
|
|
123
|
+
instead of identifying old measurements with the current harness. The existing
|
|
124
|
+
`budget_sha256` identifies the budget. Replaying samples classifies the recorded
|
|
125
|
+
measurements under the identified comparison policy. It makes no requests and
|
|
126
|
+
leaves the original result unchanged.
|
|
127
|
+
`report.json` contains source/profile identity, overall outcome and per-workload
|
|
128
|
+
results, without raw samples or response bodies.
|
|
65
129
|
Exit codes are 0 for a pass
|
|
66
130
|
or an inconclusive timing result, 1 for a regression and 3 for errors, including
|
|
67
|
-
missing workloads,
|
|
68
|
-
remains explicitly inconclusive; a successful process exit
|
|
69
|
-
measured performance pass. Deterministic failures and regressions block CI.
|
|
131
|
+
missing workloads, invalid or unstable responses, inputs or provenance. An
|
|
132
|
+
inconclusive report remains explicitly inconclusive; a successful process exit
|
|
133
|
+
does not establish a measured performance pass. Deterministic failures and regressions block CI.
|
|
70
134
|
Keep the original evidence and diagnose failures or insufficient timing signal;
|
|
71
135
|
there is no automatic retry loop or baseline update.
|
|
72
136
|
|
|
@@ -140,10 +204,24 @@ Samples retain metrics, one uniform validated counter record with its operation
|
|
|
140
204
|
count, and distinct response examples with occurrence counts and raw bodies.
|
|
141
205
|
Replay checks those counts, counters, examples and raw/canonical identities,
|
|
142
206
|
including comparison mode, verified own version and source/environment/inputs.
|
|
143
|
-
It compares response occurrence totals across
|
|
144
|
-
|
|
207
|
+
It compares canonical response identities and occurrence totals across samples
|
|
208
|
+
from the same verified source SHA. Both columns of a baseline/self pair belong
|
|
209
|
+
to the baseline SHA. JSON map order may vary; raw byte stability is not required.
|
|
210
|
+
It does not retain request order or independently reprove each live
|
|
211
|
+
authorization/counter check.
|
|
145
212
|
The compact artifact is a summary, not an execution transcript.
|
|
146
213
|
|
|
214
|
+
Responses may differ between baseline and candidate without failing or skipping
|
|
215
|
+
the performance comparison. Each workload reports `raw_changed` and
|
|
216
|
+
`canonical_changed`, together with baseline/candidate bytes per request and
|
|
217
|
+
their byte and percentage changes. These diagnostics use the
|
|
218
|
+
occurrence-weighted output distributions from baseline/candidate pairs only;
|
|
219
|
+
baseline/self pairs do not affect the size comparison. Reports retain no raw
|
|
220
|
+
bodies. Response growth alone does not fail the comparison or change its timing
|
|
221
|
+
and allocation limits. Equal response lengths do not prove correctness. The CI
|
|
222
|
+
step summary shows these diagnostics beside wall, CPU, allocation and overall
|
|
223
|
+
outcomes, including regressions and inconclusive timing.
|
|
224
|
+
|
|
147
225
|
Allocation growth above baseline median plus the larger of 10% or 32 objects
|
|
148
226
|
per request fails. Additional fixed-reference ceilings use measured accepted
|
|
149
227
|
counts with 25% headroom plus 32 objects/request. Exact SQL query and required
|
|
@@ -153,10 +231,11 @@ a constant amount. Budget changes require measured justification and review.
|
|
|
153
231
|
Missing or uncalibrated budgets for the existing calibrated workloads fail rather
|
|
154
232
|
than disabling the gate.
|
|
155
233
|
|
|
156
|
-
Both profiles use the
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
234
|
+
Both profiles use the recorded weekly Rust dependency closure as their expected
|
|
235
|
+
runtime identity. Quick includes the Rust manifests, gemspec and loaded
|
|
236
|
+
dependencies. Both profiles retain the benchmark Gemfile, lock and shell bytes
|
|
237
|
+
and hashes. Baseline and candidate must match that environment exactly;
|
|
238
|
+
unavailable or changed identities fail. The shared
|
|
160
239
|
comparison entry point activates the recorded native JSON 2.21.2 before its
|
|
161
240
|
first require, including in unbundled workers. A newer visible or already loaded
|
|
162
241
|
JSON must not silently replace that dependency; a missing or conflicting version
|
|
@@ -171,6 +250,20 @@ the original manifest contents, hashes and declaration evidence, then compare
|
|
|
171
250
|
normalized contents. Replay verifies the transformation from the retained raw
|
|
172
251
|
inputs. Other dependency requirements and manifest contents must match.
|
|
173
252
|
|
|
253
|
+
Comparisons outside the historical 0.30 maintenance mapping retain a
|
|
254
|
+
`source_versions` receipt for each source. It validates the numeric source
|
|
255
|
+
`VERSION`, matching Ruby server/client
|
|
256
|
+
and Rust generator version constants, package identities and exact synchronized
|
|
257
|
+
internal `haveapi-client` requirements before normalizing those version fields.
|
|
258
|
+
Preparation, live workers and replay bind the raw hashes to the verified source
|
|
259
|
+
and compare the normalized manifests. Missing, duplicate, inconsistent or tampered
|
|
260
|
+
metadata fails. All other manifest bytes, external dependencies, runtime and
|
|
261
|
+
benchmark locks remain part of the strict comparison. Older raw-manifest samples
|
|
262
|
+
still replay; removing a receipt from normalized samples fails source validation.
|
|
263
|
+
This rule permits coordinated package-version changes without changing baseline
|
|
264
|
+
selection or the separate historical 0.30 maintenance-lineage proof. Response
|
|
265
|
+
changes remain diagnostics under the existing per-revision stability policy.
|
|
266
|
+
|
|
174
267
|
## GitHub workflow and baseline selection
|
|
175
268
|
|
|
176
269
|
The `servers/ruby performance` workflow runs on pushes, ordinary pull requests,
|
|
@@ -183,20 +276,23 @@ Missing event/source evidence fails even when no relevant path can be establishe
|
|
|
183
276
|
| Event | Candidate | Baseline |
|
|
184
277
|
| --- | --- | --- |
|
|
185
278
|
| Pull request | Exact event head SHA | Merge-base of head and exact event base SHA |
|
|
186
|
-
|
|
|
187
|
-
|
|
|
188
|
-
| `haveapi-0.30` push with verified maintenance lineage | Event `after` | Event `before`, covering the whole pushed batch |
|
|
189
|
-
| Feature push with verified maintenance lineage | Event `after` | Merge-base with captured `origin/haveapi-0.30` head |
|
|
279
|
+
| Existing branch push, quick | Exact event `after` | Exact nonzero event `before`, on every branch |
|
|
280
|
+
| New branch push, quick (zero `before`) | Exact event `after` | Merge-base with captured exact default-branch SHA |
|
|
190
281
|
| Weekly | Exact scheduled default-branch event SHA | Fixed accepted reference below |
|
|
191
282
|
| Manual with a complete explicit pair | Supplied full candidate SHA | Supplied full baseline SHA |
|
|
192
283
|
| Manual without either SHA | Captured default SHA | Fixed accepted reference below |
|
|
193
284
|
|
|
194
285
|
The immutable accepted reference is
|
|
195
286
|
`b85ec364057f536940d689b09e612512ffffaf82`. Required ancestry is checked. The
|
|
196
|
-
reference and budgets never move automatically. Deleted branches,
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
287
|
+
reference and budgets never move automatically. Deleted branches, malformed or
|
|
288
|
+
missing commits, unavailable new-branch fallback, partial manual pairs and equal
|
|
289
|
+
endpoints fail. PRs use the head commit, never the synthetic merge commit. An
|
|
290
|
+
existing branch push uses `before`, never `HEAD^` or a guessed fallback. A quick
|
|
291
|
+
push can compare a nonancestor before/after pair after a force push; its exact
|
|
292
|
+
event pair is carried through all ancestry guards. Direct CLI use of that
|
|
293
|
+
exception requires `--push-event selection.json` from the event resolver. Other
|
|
294
|
+
PR, manual and weekly ancestry restrictions remain. Relevance uses the direct
|
|
295
|
+
before/after tree diff, including the benchmark setup files.
|
|
200
296
|
The weekly profile always requires that fixed baseline, including an explicit
|
|
201
297
|
manual pair. An explicit quick pair may select another ancestor.
|
|
202
298
|
|
|
@@ -217,29 +313,35 @@ Diff rendering uses full blob IDs and explicit formatting options, independent
|
|
|
217
313
|
of repository-dependent abbreviation and user diff display settings. Ordinary
|
|
218
314
|
master ancestry is checked before probing the maintenance mapping, so a master
|
|
219
315
|
clone does not need maintenance objects for that path.
|
|
220
|
-
Unrelated quick baselines still fail.
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
the
|
|
226
|
-
local Ruby component VERSION assignments
|
|
227
|
-
`haveapi-client` requirements. All remaining
|
|
228
|
-
|
|
229
|
-
runtime graph and
|
|
230
|
-
|
|
316
|
+
Unrelated direct quick baselines still fail. Existing maintenance branch pushes
|
|
317
|
+
use their exact pushed pair; new branches retain the captured-default fallback
|
|
318
|
+
and reason. A `haveapi-0.30` push still requires verified maintenance lineage.
|
|
319
|
+
|
|
320
|
+
Both sources in a maintenance comparison use the prepared benchmark environment.
|
|
321
|
+
In addition to the JSON preparation rule above, these comparisons normalize the
|
|
322
|
+
shared `0.30.x` patch identity, matching local Ruby component VERSION assignments
|
|
323
|
+
and exactly synchronized internal `haveapi-client` requirements. All remaining
|
|
324
|
+
bytes in dependency manifests, gemspecs and Ruby requirements must match, as must
|
|
325
|
+
the resolved third-party runtime graph and benchmark lock bytes. Source `flake.nix` and `flake.lock` remain raw provenance
|
|
326
|
+
with source hashes; they do not supply the benchmark runtime and may differ.
|
|
327
|
+
Every other supported declaration remains strict. The consumed benchmark shell
|
|
328
|
+
and lock are retained separately and must match across workers. Raw manifest
|
|
329
|
+
contents and hashes remain bound to each verified source tree alongside their
|
|
330
|
+
normalized hashes. Replay
|
|
231
331
|
recomputes the transformation and rejects missing, inconsistent or changed inputs.
|
|
232
332
|
|
|
233
333
|
The HTML comparison may normalize the own-version segment of the documentation
|
|
234
334
|
templates' `https://webui.haveapi.org/v<own-version>/#...` anchor hrefs and
|
|
235
335
|
the version in the exact framework-generated HaveAPI footer. The footer must
|
|
236
336
|
occur once and match the verified source's own VERSION; literal examples, other
|
|
237
|
-
footer text and URLs remain exact. Validation runs after the timed loop. Complete
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
337
|
+
footer text and URLs remain exact. Validation runs after the timed loop. Complete
|
|
338
|
+
raw bodies, byte counts and hashes remain in the samples, together with the
|
|
339
|
+
normalized comparison digest and source version. Replay checks these again from
|
|
340
|
+
the raw body. This normalization applies only to the HTML documentation workloads
|
|
341
|
+
on the proven maintenance lineage. JSON/API
|
|
241
342
|
values, other URLs and hosts, example code, authorization markers, panels and all
|
|
242
|
-
other output
|
|
343
|
+
other output are not normalized. Cross-revision differences remain diagnostic;
|
|
344
|
+
within-revision stability and raw/canonical integrity still apply.
|
|
243
345
|
|
|
244
346
|
The workflow uses ephemeral GitHub-hosted Ubuntu workers, `contents: read`,
|
|
245
347
|
disabled checkout credential persistence, ordinary `pull_request` events and no
|
data/haveapi.gemspec
CHANGED
|
@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
|
|
|
15
15
|
s.required_ruby_version = ">= #{File.read('../../.ruby-version').strip}"
|
|
16
16
|
|
|
17
17
|
s.add_dependency 'activesupport', '>= 7.1'
|
|
18
|
-
s.add_dependency 'haveapi-client', '~> 0.
|
|
18
|
+
s.add_dependency 'haveapi-client', '~> 0.31.1'
|
|
19
19
|
s.add_dependency 'i18n', '>= 1.6', '< 2'
|
|
20
20
|
s.add_dependency 'json'
|
|
21
21
|
s.add_dependency 'mail'
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
require 'haveapi/client_example'
|
|
2
|
+
|
|
3
|
+
module HaveAPI::ClientExamples
|
|
4
|
+
class TsGeneratedClient < HaveAPI::ClientExample
|
|
5
|
+
id 'typescript-generated'
|
|
6
|
+
label 'TypeScript (generated)'
|
|
7
|
+
code :typescript
|
|
8
|
+
order 19
|
|
9
|
+
|
|
10
|
+
def self.available?(context)
|
|
11
|
+
context&.profile&.language == :typescript
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def init
|
|
15
|
+
render_documentation(:init)
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def auth(method, desc)
|
|
19
|
+
render_documentation(:auth, auth_method: method, auth_description: desc)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def example(sample)
|
|
23
|
+
render_documentation(:example, rendering_context: context&.with(sample:))
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
private
|
|
27
|
+
|
|
28
|
+
def render_documentation(kind, rendering_context: context, **)
|
|
29
|
+
unless self.class.available?(rendering_context)
|
|
30
|
+
return { status: :limitation, reason: 'A generated TypeScript SDK documentation profile is required.' }.freeze
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
rendering_context.profile.adapter.render(kind:, context: rendering_context, mode: :async, **)
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|