@maverickcer/env-cap 0.7.5 → 0.7.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ADOPTION.md +12 -12
- package/AGENTS.md +85 -120
- package/CHANGELOG.md +36 -0
- package/CODE_REVIEW.md +5 -5
- package/CONTRIBUTING.md +5 -5
- package/GUIDE.md +42 -24
- package/README.md +52 -34
- package/RELEASING.md +78 -64
- package/SECURITY.md +15 -15
- package/VERSIONING.md +16 -15
- package/benchmarks/READING-BENCHMARKS.md +101 -85
- package/benchmarks/README.md +12 -9
- package/benchmarks/WRITING-BENCHMARKS.md +89 -79
- package/benchmarks/performance-buildtime/BENCHMARKS.md +20 -20
- package/benchmarks/performance-runtime/BENCHMARKS.md +107 -107
- package/dist/.dts/_ipc/build-fs.d.ts +3 -3
- package/dist/.dts/_ipc/build-fs.d.ts.map +1 -1
- package/dist/.dts/_ipc/display-path.d.ts +1 -1
- package/dist/.dts/_ipc/glob.d.ts +3 -2
- package/dist/.dts/_ipc/glob.d.ts.map +1 -1
- package/dist/.dts/_ipc/no-node-fs.d.ts +2 -2
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts +16 -16
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts.map +1 -1
- package/dist/.dts/_ipc/typescript.d.ts +4 -4
- package/dist/.dts/build/assemble-project.d.ts +14 -15
- package/dist/.dts/build/assemble-project.d.ts.map +1 -1
- package/dist/.dts/build/change-model.d.ts +14 -14
- package/dist/.dts/build/change-model.d.ts.map +1 -1
- package/dist/.dts/build/check-artifacts.d.ts +4 -4
- package/dist/.dts/build/check-artifacts.d.ts.map +1 -1
- package/dist/.dts/build/citation-verification.d.ts +13 -14
- package/dist/.dts/build/citation-verification.d.ts.map +1 -1
- package/dist/.dts/build/compatibility.d.ts +24 -24
- package/dist/.dts/build/compatibility.d.ts.map +1 -1
- package/dist/.dts/build/contract-model.d.ts +10 -10
- package/dist/.dts/build/contract-model.d.ts.map +1 -1
- package/dist/.dts/build/deep-equal.d.ts +2 -2
- package/dist/.dts/build/deep-freeze.d.ts +2 -2
- package/dist/.dts/build/dependency-graph.d.ts +48 -55
- package/dist/.dts/build/dependency-graph.d.ts.map +1 -1
- package/dist/.dts/build/dependency-model.d.ts +18 -18
- package/dist/.dts/build/dependency-model.d.ts.map +1 -1
- package/dist/.dts/build/discover.d.ts +11 -11
- package/dist/.dts/build/discover.d.ts.map +1 -1
- package/dist/.dts/build/docs.d.ts +34 -38
- package/dist/.dts/build/docs.d.ts.map +1 -1
- package/dist/.dts/build/env-example.d.ts +13 -13
- package/dist/.dts/build/env-example.d.ts.map +1 -1
- package/dist/.dts/build/errors.d.ts +8 -8
- package/dist/.dts/build/evidence-cache.d.ts +23 -23
- package/dist/.dts/build/evidence-cache.d.ts.map +1 -1
- package/dist/.dts/build/evidence-model.d.ts +3 -3
- package/dist/.dts/build/evidence-reference.d.ts +15 -15
- package/dist/.dts/build/evidence-reference.d.ts.map +1 -1
- package/dist/.dts/build/evidence-snapshot.d.ts +47 -53
- package/dist/.dts/build/evidence-snapshot.d.ts.map +1 -1
- package/dist/.dts/build/exclusive-group.d.ts +4 -4
- package/dist/.dts/build/exclusive-group.d.ts.map +1 -1
- package/dist/.dts/build/finding-model.d.ts +12 -12
- package/dist/.dts/build/finding-model.d.ts.map +1 -1
- package/dist/.dts/build/generate-documentation.d.ts +19 -19
- package/dist/.dts/build/generate-documentation.d.ts.map +1 -1
- package/dist/.dts/build/generate-env-artifacts.d.ts +43 -43
- package/dist/.dts/build/generate-env-artifacts.d.ts.map +1 -1
- package/dist/.dts/build/generate-evidence.d.ts +19 -19
- package/dist/.dts/build/generate-evidence.d.ts.map +1 -1
- package/dist/.dts/build/generate-manifest.d.ts +11 -12
- package/dist/.dts/build/generate-manifest.d.ts.map +1 -1
- package/dist/.dts/build/generate-usage.d.ts +23 -24
- package/dist/.dts/build/generate-usage.d.ts.map +1 -1
- package/dist/.dts/build/generated-banner.d.ts +5 -5
- package/dist/.dts/build/governance-fields.d.ts +7 -7
- package/dist/.dts/build/humanize-key.d.ts +1 -1
- package/dist/.dts/build/humanize-key.d.ts.map +1 -1
- package/dist/.dts/build/index.d.ts +20 -36
- package/dist/.dts/build/index.d.ts.map +1 -1
- package/dist/.dts/build/lifecycle-model.d.ts +87 -85
- package/dist/.dts/build/lifecycle-model.d.ts.map +1 -1
- package/dist/.dts/build/link.d.ts +38 -39
- package/dist/.dts/build/link.d.ts.map +1 -1
- package/dist/.dts/build/literal-eval.d.ts +2 -2
- package/dist/.dts/build/literal-eval.d.ts.map +1 -1
- package/dist/.dts/build/live-expirations.d.ts +14 -13
- package/dist/.dts/build/live-expirations.d.ts.map +1 -1
- package/dist/.dts/build/manifest.d.ts +6 -6
- package/dist/.dts/build/map-utils.d.ts +4 -4
- package/dist/.dts/build/merge-documentation.d.ts +9 -8
- package/dist/.dts/build/merge-documentation.d.ts.map +1 -1
- package/dist/.dts/build/ownership-model.d.ts +14 -15
- package/dist/.dts/build/ownership-model.d.ts.map +1 -1
- package/dist/.dts/build/parse.d.ts +20 -21
- package/dist/.dts/build/parse.d.ts.map +1 -1
- package/dist/.dts/build/reference-projections.d.ts +36 -38
- package/dist/.dts/build/reference-projections.d.ts.map +1 -1
- package/dist/.dts/build/resolution/resolve-import.d.ts +18 -14
- package/dist/.dts/build/resolution/resolve-import.d.ts.map +1 -1
- package/dist/.dts/build/resolution/resolve-package-schema.d.ts +26 -26
- package/dist/.dts/build/resolution/resolve-package-schema.d.ts.map +1 -1
- package/dist/.dts/build/resolution/resolve-within-root.d.ts +9 -9
- package/dist/.dts/build/sarif.d.ts +5 -5
- package/dist/.dts/build/sarif.d.ts.map +1 -1
- package/dist/.dts/build/scan-dependencies.d.ts +17 -16
- package/dist/.dts/build/scan-dependencies.d.ts.map +1 -1
- package/dist/.dts/build/sort-by-identity.d.ts +1 -1
- package/dist/.dts/build/source-position.d.ts +13 -13
- package/dist/.dts/build/tool-version.d.ts +1 -1
- package/dist/.dts/build/usage-report.d.ts +19 -19
- package/dist/.dts/build/usage-report.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/index.d.ts +16 -5
- package/dist/.dts/eslint-plugin/index.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/no-raw-process-env.d.ts +4 -4
- package/dist/.dts/evidence/define-projection.d.ts +15 -16
- package/dist/.dts/evidence/define-projection.d.ts.map +1 -1
- package/dist/.dts/evidence/index.d.ts +11 -15
- package/dist/.dts/evidence/index.d.ts.map +1 -1
- package/dist/.dts/helpers/index.d.ts +15 -18
- package/dist/.dts/helpers/index.d.ts.map +1 -1
- package/dist/.dts/helpers/validators.d.ts +2 -2
- package/dist/.dts/helpers/validators.d.ts.map +1 -1
- package/dist/.dts/node/index.d.ts +19 -0
- package/dist/.dts/node/index.d.ts.map +1 -1
- package/dist/.dts/runtime/create.d.ts +3 -3
- package/dist/.dts/runtime/document.d.ts +90 -99
- package/dist/.dts/runtime/document.d.ts.map +1 -1
- package/dist/.dts/runtime/errors.d.ts +5 -5
- package/dist/.dts/runtime/reset.d.ts +1 -1
- package/dist/.dts/runtime/types.d.ts +23 -24
- package/dist/.dts/runtime/types.d.ts.map +1 -1
- package/dist/.dts/runtime/validate.d.ts +1 -1
- package/dist/.dts7/_ipc/build-fs.d.ts +3 -3
- package/dist/.dts7/_ipc/build-fs.d.ts.map +1 -1
- package/dist/.dts7/_ipc/display-path.d.ts +1 -1
- package/dist/.dts7/_ipc/glob.d.ts +3 -2
- package/dist/.dts7/_ipc/glob.d.ts.map +1 -1
- package/dist/.dts7/_ipc/no-node-fs.d.ts +2 -2
- package/dist/.dts7/_ipc/resolve-tsconfig-paths.d.ts +16 -16
- package/dist/.dts7/_ipc/resolve-tsconfig-paths.d.ts.map +1 -1
- package/dist/.dts7/_ipc/typescript.d.ts +4 -4
- package/dist/.dts7/build/assemble-project.d.ts +14 -15
- package/dist/.dts7/build/assemble-project.d.ts.map +1 -1
- package/dist/.dts7/build/change-model.d.ts +14 -14
- package/dist/.dts7/build/change-model.d.ts.map +1 -1
- package/dist/.dts7/build/check-artifacts.d.ts +4 -4
- package/dist/.dts7/build/check-artifacts.d.ts.map +1 -1
- package/dist/.dts7/build/citation-verification.d.ts +13 -14
- package/dist/.dts7/build/citation-verification.d.ts.map +1 -1
- package/dist/.dts7/build/compatibility.d.ts +24 -24
- package/dist/.dts7/build/compatibility.d.ts.map +1 -1
- package/dist/.dts7/build/contract-model.d.ts +10 -10
- package/dist/.dts7/build/contract-model.d.ts.map +1 -1
- package/dist/.dts7/build/deep-equal.d.ts +2 -2
- package/dist/.dts7/build/deep-freeze.d.ts +2 -2
- package/dist/.dts7/build/dependency-graph.d.ts +48 -55
- package/dist/.dts7/build/dependency-graph.d.ts.map +1 -1
- package/dist/.dts7/build/dependency-model.d.ts +18 -18
- package/dist/.dts7/build/dependency-model.d.ts.map +1 -1
- package/dist/.dts7/build/discover.d.ts +11 -11
- package/dist/.dts7/build/discover.d.ts.map +1 -1
- package/dist/.dts7/build/docs.d.ts +34 -38
- package/dist/.dts7/build/docs.d.ts.map +1 -1
- package/dist/.dts7/build/env-example.d.ts +13 -13
- package/dist/.dts7/build/env-example.d.ts.map +1 -1
- package/dist/.dts7/build/errors.d.ts +8 -8
- package/dist/.dts7/build/evidence-cache.d.ts +23 -23
- package/dist/.dts7/build/evidence-cache.d.ts.map +1 -1
- package/dist/.dts7/build/evidence-model.d.ts +3 -3
- package/dist/.dts7/build/evidence-reference.d.ts +15 -15
- package/dist/.dts7/build/evidence-reference.d.ts.map +1 -1
- package/dist/.dts7/build/evidence-snapshot.d.ts +47 -53
- package/dist/.dts7/build/evidence-snapshot.d.ts.map +1 -1
- package/dist/.dts7/build/exclusive-group.d.ts +4 -4
- package/dist/.dts7/build/exclusive-group.d.ts.map +1 -1
- package/dist/.dts7/build/finding-model.d.ts +12 -12
- package/dist/.dts7/build/finding-model.d.ts.map +1 -1
- package/dist/.dts7/build/generate-documentation.d.ts +19 -19
- package/dist/.dts7/build/generate-documentation.d.ts.map +1 -1
- package/dist/.dts7/build/generate-env-artifacts.d.ts +43 -43
- package/dist/.dts7/build/generate-env-artifacts.d.ts.map +1 -1
- package/dist/.dts7/build/generate-evidence.d.ts +19 -19
- package/dist/.dts7/build/generate-evidence.d.ts.map +1 -1
- package/dist/.dts7/build/generate-manifest.d.ts +11 -12
- package/dist/.dts7/build/generate-manifest.d.ts.map +1 -1
- package/dist/.dts7/build/generate-usage.d.ts +23 -24
- package/dist/.dts7/build/generate-usage.d.ts.map +1 -1
- package/dist/.dts7/build/generated-banner.d.ts +5 -5
- package/dist/.dts7/build/governance-fields.d.ts +7 -7
- package/dist/.dts7/build/humanize-key.d.ts +1 -1
- package/dist/.dts7/build/humanize-key.d.ts.map +1 -1
- package/dist/.dts7/build/index.d.ts +20 -36
- package/dist/.dts7/build/index.d.ts.map +1 -1
- package/dist/.dts7/build/lifecycle-model.d.ts +87 -85
- package/dist/.dts7/build/lifecycle-model.d.ts.map +1 -1
- package/dist/.dts7/build/link.d.ts +38 -39
- package/dist/.dts7/build/link.d.ts.map +1 -1
- package/dist/.dts7/build/literal-eval.d.ts +2 -2
- package/dist/.dts7/build/literal-eval.d.ts.map +1 -1
- package/dist/.dts7/build/live-expirations.d.ts +14 -13
- package/dist/.dts7/build/live-expirations.d.ts.map +1 -1
- package/dist/.dts7/build/manifest.d.ts +6 -6
- package/dist/.dts7/build/map-utils.d.ts +4 -4
- package/dist/.dts7/build/merge-documentation.d.ts +9 -8
- package/dist/.dts7/build/merge-documentation.d.ts.map +1 -1
- package/dist/.dts7/build/ownership-model.d.ts +14 -15
- package/dist/.dts7/build/ownership-model.d.ts.map +1 -1
- package/dist/.dts7/build/parse.d.ts +20 -21
- package/dist/.dts7/build/parse.d.ts.map +1 -1
- package/dist/.dts7/build/reference-projections.d.ts +36 -38
- package/dist/.dts7/build/reference-projections.d.ts.map +1 -1
- package/dist/.dts7/build/resolution/resolve-import.d.ts +18 -14
- package/dist/.dts7/build/resolution/resolve-import.d.ts.map +1 -1
- package/dist/.dts7/build/resolution/resolve-package-schema.d.ts +26 -26
- package/dist/.dts7/build/resolution/resolve-package-schema.d.ts.map +1 -1
- package/dist/.dts7/build/resolution/resolve-within-root.d.ts +9 -9
- package/dist/.dts7/build/sarif.d.ts +5 -5
- package/dist/.dts7/build/sarif.d.ts.map +1 -1
- package/dist/.dts7/build/scan-dependencies.d.ts +17 -16
- package/dist/.dts7/build/scan-dependencies.d.ts.map +1 -1
- package/dist/.dts7/build/sort-by-identity.d.ts +1 -1
- package/dist/.dts7/build/source-position.d.ts +13 -13
- package/dist/.dts7/build/tool-version.d.ts +1 -1
- package/dist/.dts7/build/usage-report.d.ts +19 -19
- package/dist/.dts7/build/usage-report.d.ts.map +1 -1
- package/dist/.dts7/eslint-plugin/index.d.ts +16 -5
- package/dist/.dts7/eslint-plugin/index.d.ts.map +1 -1
- package/dist/.dts7/eslint-plugin/no-raw-process-env.d.ts +4 -4
- package/dist/.dts7/evidence/define-projection.d.ts +15 -16
- package/dist/.dts7/evidence/define-projection.d.ts.map +1 -1
- package/dist/.dts7/evidence/index.d.ts +11 -15
- package/dist/.dts7/evidence/index.d.ts.map +1 -1
- package/dist/.dts7/helpers/index.d.ts +15 -18
- package/dist/.dts7/helpers/index.d.ts.map +1 -1
- package/dist/.dts7/helpers/validators.d.ts +2 -2
- package/dist/.dts7/helpers/validators.d.ts.map +1 -1
- package/dist/.dts7/node/index.d.ts +19 -0
- package/dist/.dts7/node/index.d.ts.map +1 -1
- package/dist/.dts7/runtime/create.d.ts +3 -3
- package/dist/.dts7/runtime/document.d.ts +90 -99
- package/dist/.dts7/runtime/document.d.ts.map +1 -1
- package/dist/.dts7/runtime/errors.d.ts +5 -5
- package/dist/.dts7/runtime/reset.d.ts +1 -1
- package/dist/.dts7/runtime/types.d.ts +23 -24
- package/dist/.dts7/runtime/types.d.ts.map +1 -1
- package/dist/.dts7/runtime/validate.d.ts +1 -1
- package/dist/{build-DNAYWD2B.js → build-33O5SDTC.js} +37 -39
- package/dist/build-33O5SDTC.js.map +1 -0
- package/dist/build.cjs +36 -38
- package/dist/build.cjs.map +1 -1
- package/dist/build.js +36 -38
- package/dist/build.js.map +1 -1
- package/dist/{chunk-HAGMVMN5.js → chunk-4D67AQFY.js} +2 -2
- package/dist/cli/index.js +29 -28
- package/dist/cli/index.js.map +1 -1
- package/dist/eslint-plugin/index.cjs.map +1 -1
- package/dist/eslint-plugin/index.js.map +1 -1
- package/dist/evidence.cjs +1 -1
- package/dist/evidence.cjs.map +1 -1
- package/dist/evidence.js +1 -1
- package/dist/evidence.js.map +1 -1
- package/dist/helpers.cjs +4 -4
- package/dist/helpers.cjs.map +1 -1
- package/dist/helpers.js +4 -4
- package/dist/helpers.js.map +1 -1
- package/dist/index.cjs.map +1 -1
- package/dist/index.js.map +1 -1
- package/examples/README.md +1 -1
- package/package.json +20 -13
- package/schemas/contract-model.schema.json +7 -7
- package/schemas/env-cap-report.schema.json +88 -88
- package/skills/env-cap/SKILL.md +12 -13
- package/skills/env-cap/references/api.md +1 -1
- package/specs/architecture.md +145 -120
- package/specs/decisions/0049-typescript-7-scanner-uses-a-bundled-typescript-6.md +9 -0
- package/specs/generated-artifacts.md +78 -70
- package/specs/migrations/README.md +6 -6
- package/specs/migrations/from-centralized-schema.md +1 -1
- package/specs/migrations/from-t3-env.md +9 -9
- package/PROMPT.md +0 -89
- package/dist/build-DNAYWD2B.js.map +0 -1
- /package/dist/{chunk-HAGMVMN5.js.map → chunk-4D67AQFY.js.map} +0 -0
|
@@ -1,40 +1,49 @@
|
|
|
1
1
|
# Reading the benchmarks
|
|
2
2
|
|
|
3
|
-
This guide is for the person **reading** `BENCHMARKS.md
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
3
|
+
This guide is for the person **reading** `BENCHMARKS.md`. That reader may be:
|
|
4
|
+
|
|
5
|
+
- a platform engineer deciding whether to run this package;
|
|
6
|
+
- a CTO estimating what it adds to the infrastructure bill;
|
|
7
|
+
- a maintainer checking a change.
|
|
8
|
+
|
|
9
|
+
It explains why the benchmarks exist, how to read each number, and, as important, what you must **not** do
|
|
10
|
+
with them.
|
|
7
11
|
|
|
8
12
|
Its companion, [WRITING-BENCHMARKS.md](WRITING-BENCHMARKS.md), explains how benchmarks are written and
|
|
9
13
|
documented.
|
|
10
14
|
|
|
11
15
|
## Why we benchmark
|
|
12
16
|
|
|
13
|
-
Every dependency has a price. Most of it is invisible until it is large:
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
+
Every dependency has a price. Most of it is invisible until it is large:
|
|
18
|
+
|
|
19
|
+
- a few milliseconds added to every request;
|
|
20
|
+
- a CPU core that is busier than it needs to be;
|
|
21
|
+
- a function that is fast on ten items and unusable on ten thousand;
|
|
22
|
+
- a cold start that grows with every module.
|
|
23
|
+
|
|
24
|
+
By then it is expensive to remove.
|
|
17
25
|
|
|
18
26
|
The benchmarks make that price visible **before** you adopt the package and **every time** it changes.
|
|
19
|
-
They lean toward costs on purpose. The rest of the repository explains the benefits
|
|
27
|
+
They lean toward costs on purpose. The rest of the repository explains the benefits. This report tells
|
|
20
28
|
you what you are paying for them, in time and in money, and how the bill grows with your workload.
|
|
21
29
|
|
|
22
30
|
## What is in the report
|
|
23
31
|
|
|
24
32
|
`BENCHMARKS.md` has four parts, in the order a decision-maker needs them:
|
|
25
33
|
|
|
26
|
-
1. **What adopting this package costs.** The headline
|
|
27
|
-
a typical workload and at the largest tested size
|
|
28
|
-
operations, and the code size every process must load. If you read nothing else, read
|
|
34
|
+
1. **What adopting this package costs.** The headline. It shows the added latency, CPU and memory per
|
|
35
|
+
operation, at a typical workload and at the largest tested size. It also shows an estimated compute
|
|
36
|
+
cost per million operations, and the code size every process must load. If you read nothing else, read
|
|
37
|
+
this.
|
|
29
38
|
2. **End-to-end.** The same operation done with no package and through the package, using empty or
|
|
30
|
-
minimal functions on both sides
|
|
31
|
-
report's environment section lists the sizes actually used
|
|
32
|
-
nothing of your own application mixed in.
|
|
33
|
-
3. **Function by function.** Every function measured alone across the same sizes
|
|
34
|
-
benchmarked, what poor performance would mean, its documented big-O
|
|
35
|
-
variable that could change its cost, and what was deliberately not covered.
|
|
39
|
+
minimal functions on both sides. It is measured at every size of the ladder: ten doubling sizes by
|
|
40
|
+
default, and the report's environment section lists the sizes actually used. The difference is the
|
|
41
|
+
package's total impact, with nothing of your own application mixed in.
|
|
42
|
+
3. **Function by function.** Every function measured alone across the same sizes. Each comes with why it
|
|
43
|
+
is benchmarked, what poor performance would mean, and its documented big-O with the reason for it. It
|
|
44
|
+
also lists every variable that could change its cost, and what was deliberately not covered.
|
|
36
45
|
4. **What makes up one operation.** Each function's cost, multiplied by how often one operation calls
|
|
37
|
-
it,
|
|
46
|
+
it, as a share of the whole operation. Nested calls are counted once. This is where the cost actually
|
|
38
47
|
lives.
|
|
39
48
|
|
|
40
49
|
## How to read a table
|
|
@@ -51,12 +60,12 @@ you what you are paying for them, in time and in money, and how the bill grows w
|
|
|
51
60
|
|
|
52
61
|
## Big-O and the growth exponent
|
|
53
62
|
|
|
54
|
-
Big-O describes how cost grows as size grows
|
|
55
|
-
you what happens at a size you did not test. The report fits a line through the log of time against
|
|
56
|
-
|
|
57
|
-
per-call overhead plus work that grows, and the overhead hides the growth at small sizes
|
|
58
|
-
|
|
59
|
-
|
|
63
|
+
Big-O describes how cost grows as size grows. That is more useful than any single timing, because it tells
|
|
64
|
+
you what happens at a size you did not test. The report fits a line through the log of time against the
|
|
65
|
+
log of size across the **largest** sizes. On a long ladder that is the top five. A real cost is a fixed
|
|
66
|
+
per-call overhead plus work that grows, and the overhead hides the growth at small sizes. The slope is the
|
|
67
|
+
**growth exponent**, snapped to the nearest class. The fit's R² is reported next to it. When the cost
|
|
68
|
+
moves but does not follow a power law (a step, a cache cliff), no class is claimed at all. The classes:
|
|
60
69
|
|
|
61
70
|
| Exponent (about) | Class | What doubling the size does to the time |
|
|
62
71
|
| ---------------- | -------------- | --------------------------------------- |
|
|
@@ -69,19 +78,19 @@ the cost moves but does not follow a power law (a step, a cache cliff) no class
|
|
|
69
78
|
|
|
70
79
|
Each function shows its **documented** class next to its **measured** class with an agreement label:
|
|
71
80
|
|
|
72
|
-
- ✅ **matches
|
|
73
|
-
- 🟡 **close
|
|
81
|
+
- ✅ **matches**: the code behaves as documented.
|
|
82
|
+
- 🟡 **close**: neighbouring classes. Constant vs logarithmic and linear vs linearithmic cannot be
|
|
74
83
|
reliably separated on a bounded ladder; treat it as a match unless the exponent has moved.
|
|
75
|
-
- ⚠️ **differs
|
|
84
|
+
- ⚠️ **differs**: two or more classes apart. Either the documentation or the code is wrong. This is
|
|
76
85
|
the most useful line in the report.
|
|
77
|
-
- ❔ **not enough data
|
|
86
|
+
- ❔ **not enough data**: fewer than two usable sizes; something failed.
|
|
78
87
|
|
|
79
88
|
## The dollar figures
|
|
80
89
|
|
|
81
90
|
Time becomes money through two bracketing price shapes, because platforms bill differently:
|
|
82
91
|
|
|
83
|
-
- **Low:** CPU-priced compute (a container or VM billed per vCPU-hour)
|
|
84
|
-
- **High:** duration-and-memory-priced functions (billed per GB-second)
|
|
92
|
+
- **Low:** CPU-priced compute (a container or VM billed per vCPU-hour): you pay for CPU seconds.
|
|
93
|
+
- **High:** duration-and-memory-priced functions (billed per GB-second): you pay for wall seconds
|
|
85
94
|
times the memory the platform reserves, never less than its minimum.
|
|
86
95
|
|
|
87
96
|
The rates are rounded published list prices and are **assumptions**; they drift with time and region.
|
|
@@ -91,8 +100,8 @@ Treat the result as an order-of-magnitude comparison ("fractions of a cent per m
|
|
|
91
100
|
|
|
92
101
|
## Results are not comparable between runs
|
|
93
102
|
|
|
94
|
-
This is the rule that matters most
|
|
95
|
-
|
|
103
|
+
This is the rule that matters most: **a benchmark number is a measurement of one run on one machine at one
|
|
104
|
+
moment**. It is not a property of the package.
|
|
96
105
|
|
|
97
106
|
- Do **not** compare numbers from different machines, operating systems, Node versions, or runners. A
|
|
98
107
|
laptop and a CI runner differ by large factors in both directions.
|
|
@@ -106,15 +115,15 @@ one moment.** It is not a property of the package.
|
|
|
106
115
|
|
|
107
116
|
What the numbers **are** good for:
|
|
108
117
|
|
|
109
|
-
- **Change over time on comparable hardware
|
|
118
|
+
- **Change over time on comparable hardware**. The same suite on the same kind of machine before and
|
|
110
119
|
after a change shows whether the change made the package slower. Look for movement larger than the
|
|
111
120
|
run-to-run noise.
|
|
112
|
-
- **Shape, not only level
|
|
113
|
-
than a 15% shift in a timing
|
|
121
|
+
- **Shape, not only level**. A change in growth class (O(n) becoming O(n²)) is a much bigger signal
|
|
122
|
+
than a 15% shift in a timing. It is also far less sensitive to the machine than a time is, because it
|
|
114
123
|
is a ratio between sizes on the same run. It still assumes the same sizes, workload and runtime.
|
|
115
|
-
- **Relative cost between functions
|
|
124
|
+
- **Relative cost between functions**. Which function dominates the overhead is usually the same on
|
|
116
125
|
any machine, even when the absolute times are not.
|
|
117
|
-
- **Order of magnitude
|
|
126
|
+
- **Order of magnitude**. Microseconds, milliseconds or seconds per operation: that distinction is
|
|
118
127
|
stable enough to plan with.
|
|
119
128
|
|
|
120
129
|
## What the numbers do not prove
|
|
@@ -126,11 +135,11 @@ What the numbers **are** good for:
|
|
|
126
135
|
power law. A high R² does not show the class is right, and a low one means no class is reported.
|
|
127
136
|
- **The overhead is a difference.** `end-to-end:overhead` is not measured; it is the with-package
|
|
128
137
|
time minus the baseline time at each size, never below zero. A difference of two noisy medians is
|
|
129
|
-
noisier than either
|
|
130
|
-
|
|
131
|
-
groups, and the "What the package adds" table, instead.
|
|
138
|
+
noisier than either. The history page therefore shows it for reading only and gives it no complexity
|
|
139
|
+
class. The pull-request summary does not report a class shift for it. Read the baseline and
|
|
140
|
+
with-package groups, and the "What the package adds" table, instead.
|
|
132
141
|
- **A relative cost is still a measurement.** Dividing by the baseline removes much of the machine,
|
|
133
|
-
not all of it
|
|
142
|
+
not all of it. Compilation, garbage collection and contention on the runner still move it. Below the
|
|
134
143
|
timer-noise floor the percentage is not shown.
|
|
135
144
|
- **One run per commit is not a repeat.** The history holds one measurement for each recorded commit.
|
|
136
145
|
A trend is more convincing than a point; a single point moving is ordinary noise.
|
|
@@ -138,66 +147,73 @@ What the numbers **are** good for:
|
|
|
138
147
|
## Noise
|
|
139
148
|
|
|
140
149
|
Even on a quiet machine, repeated runs differ. Very fast operations (microseconds and below) are the
|
|
141
|
-
noisiest in relative terms
|
|
150
|
+
noisiest in relative terms. The engine batches them and reports the per-operation value, but a 30-40%
|
|
142
151
|
swing between runs on those is ordinary. Slow operations are steadier. Read the median and p95
|
|
143
152
|
together, and trust a trend over a single run. The pull-request summary shows the change against the last
|
|
144
|
-
run, but never judges it: a raw comparison is between two runs on different machines. What can fail a pull
|
|
145
|
-
baseline-relative ratios
|
|
146
|
-
requests and the history page".
|
|
153
|
+
run, but never judges it: a raw comparison is between two runs on different machines. What can fail a pull
|
|
154
|
+
request is limited to growth classes and baseline-relative ratios. Those are far less sensitive to the
|
|
155
|
+
machine than a raw time (see "History, pull requests and the history page").
|
|
147
156
|
|
|
148
157
|
## How the documentation is done
|
|
149
158
|
|
|
150
159
|
Documentation is part of the benchmark definition, not an afterthought, and the suite will not load
|
|
151
160
|
without it. For every function the report prints, in this order:
|
|
152
161
|
|
|
153
|
-
1. **Why we benchmark it
|
|
154
|
-
2. **What poor performance would mean
|
|
155
|
-
3. **Expected growth
|
|
156
|
-
4. **Variables that could change its cost
|
|
162
|
+
1. **Why we benchmark it**: where it runs, how often, and who feels it.
|
|
163
|
+
2. **What poor performance would mean**: the consequence in latency, capacity and spend.
|
|
164
|
+
3. **Expected growth**: the big-O and the mechanism that causes it.
|
|
165
|
+
4. **Variables that could change its cost**: each one swept, measured as a variant, or fixed at a
|
|
157
166
|
stated value.
|
|
158
|
-
5. **Deliberately not covered
|
|
159
|
-
6. **The measurements
|
|
167
|
+
5. **Deliberately not covered**: the honest gaps, with reasons.
|
|
168
|
+
6. **The measurements**: every size, median, p95, CPU, memory, throughput, and the measured class next
|
|
160
169
|
to the documented one.
|
|
161
170
|
|
|
162
171
|
If a section reads as vague, that is a documentation bug: open an issue or fix it.
|
|
163
172
|
|
|
164
173
|
## History, pull requests and the history page
|
|
165
174
|
|
|
166
|
-
When a pull request touches benchmarked code, CI re-runs the suite and posts a summary
|
|
167
|
-
change first, then any function whose growth class shifted, then ordinary timing movements
|
|
168
|
-
their thresholds. Three gates can fail the run. None of them compares a raw time with another run's,
|
|
169
|
-
two runs are on different machines.
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
175
|
+
When a pull request touches benchmarked code, CI re-runs the suite and posts a summary. The summary shows
|
|
176
|
+
the end-to-end change first, then any function whose growth class shifted, then ordinary timing movements
|
|
177
|
+
beyond their thresholds. Three gates can fail the run. None of them compares a raw time with another run's,
|
|
178
|
+
because two runs are on different machines.
|
|
179
|
+
|
|
180
|
+
Two compare the measured growth class, which describes the shape of the curve and not the speed of the
|
|
181
|
+
machine:
|
|
182
|
+
|
|
183
|
+
- against the documented big-O: a function whose class **differs** fails;
|
|
184
|
+
- against the class at the last release: a function whose class got **worse** fails, unless
|
|
185
|
+
`.repo-contract/exceptions/benchmark-complexity.json` records a person's acceptance of it.
|
|
186
|
+
|
|
173
187
|
The third is a ratio: the package's overhead _relative to its own baseline_, measured in the same run,
|
|
174
|
-
growing past an allowance
|
|
175
|
-
noisier
|
|
176
|
-
difference with a 95% confidence interval
|
|
177
|
-
compared is reported as a note, never a failure. What the gate judges is main measured on the same
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
with
|
|
188
|
+
growing past an allowance. The allowance is 50% for millisecond-scale work and 70% for microsecond-scale
|
|
189
|
+
work, which is noisier. The baseline and the package are sampled interleaved, so the overhead is a
|
|
190
|
+
per-pair difference with a 95% confidence interval. A growth that sits inside the intervals of the two runs
|
|
191
|
+
compared is reported as a note, never a failure. What the gate judges is main measured on the same runner
|
|
192
|
+
in the same job. The committed results of main and of the last release are judged only when they came from
|
|
193
|
+
the same CPU model. The reason: for fast work, the ratio also depends on how much of a bare baseline is
|
|
194
|
+
process start-up.
|
|
195
|
+
|
|
196
|
+
A function whose big-O could not be determined (too few sizes, or no difference between them) is a
|
|
197
|
+
**warning**. Nothing is known to be wrong, but nothing is checked. Raw time against another run is shown,
|
|
198
|
+
never judged. A failed gate fails the benchmark job; whether a red job blocks the merge is decided by the
|
|
199
|
+
repository's ruleset, not by the gate.
|
|
200
|
+
|
|
201
|
+
After a merge, a job measures the merge commit and records it through its own small pull request. The
|
|
202
|
+
record is the results file and a history entry naming the commit, pull request and version. Each entry
|
|
203
|
+
also keeps how noisy that run was (standard deviation, p95 and iteration count for every size) and the
|
|
204
|
+
processor it ran on. A reader can then judge later whether a change is larger than the run's own spread. A
|
|
205
|
+
generated history page opens with why the benchmarks exist and what they do and do not show. It then
|
|
206
|
+
charts every function across runs, with every figure also available as a table. It shows the baseline,
|
|
207
|
+
the with-package time and the difference side by side.
|
|
192
208
|
|
|
193
209
|
## Glossary
|
|
194
210
|
|
|
195
|
-
- **Operation
|
|
196
|
-
- **Size / `n
|
|
197
|
-
- **Baseline
|
|
198
|
-
- **Overhead / added
|
|
211
|
+
- **Operation**: one unit of the work the package performs, as defined by the suite.
|
|
212
|
+
- **Size / `n`**: how much that operation processes (records, calls, files).
|
|
213
|
+
- **Baseline**: the same operation with no package, using an empty or minimal function.
|
|
214
|
+
- **Overhead / added**: the package's own cost, with-package time minus baseline time. Derived from the
|
|
199
215
|
two measured groups, never below zero, and not itself classified.
|
|
200
|
-
- **Median / p95
|
|
201
|
-
- **Heap
|
|
202
|
-
- **Batch
|
|
203
|
-
- **Cold start
|
|
216
|
+
- **Median / p95**: the middle sample, and the value 95% of samples beat.
|
|
217
|
+
- **Heap Δ**: memory the operation left allocated.
|
|
218
|
+
- **Batch**: the number of back-to-back repetitions inside one sample for very fast operations.
|
|
219
|
+
- **Cold start**: the cost of loading and initializing the package in a fresh process.
|
package/benchmarks/README.md
CHANGED
|
@@ -1,10 +1,13 @@
|
|
|
1
1
|
# Benchmarks
|
|
2
2
|
|
|
3
|
-
This folder is the performance and cost record of the package
|
|
4
|
-
money, how each function scales, and where the cost comes from. It is built entirely by
|
|
5
|
-
[`internal-package-contract`](https://github.com/MaverickCER/internal-package-contract):
|
|
6
|
-
|
|
7
|
-
|
|
3
|
+
This folder is the performance and cost record of the package. It shows what adopting it costs in
|
|
4
|
+
time and money, how each function scales, and where the cost comes from. It is built entirely by
|
|
5
|
+
[`internal-package-contract`](https://github.com/MaverickCER/internal-package-contract):
|
|
6
|
+
|
|
7
|
+
- the package supplies **what to run** (an input);
|
|
8
|
+
- the contract **stresses it through ten sizes, measures it, checks the big-O and prices it**;
|
|
9
|
+
- the contract writes a **report** and a machine-readable **result** (the output).
|
|
10
|
+
|
|
8
11
|
Nothing in this folder implements timing, statistics or reporting.
|
|
9
12
|
|
|
10
13
|
## Read first
|
|
@@ -96,8 +99,8 @@ export default defineSuite({
|
|
|
96
99
|
| `inEndToEnd` | no | How often one end-to-end operation calls the function, for the attribution section. |
|
|
97
100
|
|
|
98
101
|
`expectedComplexity` is one of `constant`, `logarithmic`, `linear`, `linearithmic`, `quadratic`,
|
|
99
|
-
`exponential-or-worse`. Return a value that depends on the work
|
|
100
|
-
nothing is optimized away
|
|
102
|
+
`exponential-or-worse`. Return a value that depends on the work; the contract keeps it alive so
|
|
103
|
+
nothing is optimized away. Return `timed(ms)` only when an operation is timed elsewhere, such as a
|
|
101
104
|
child process.
|
|
102
105
|
|
|
103
106
|
## The output: `results.json`
|
|
@@ -121,6 +124,6 @@ analysis complexity measured class, exponent and notation per group, with
|
|
|
121
124
|
cost the typical and largest sizes used for the headline numbers
|
|
122
125
|
```
|
|
123
126
|
|
|
124
|
-
|
|
125
|
-
**change over time on comparable hardware** and at **changes in growth class**, never at one absolute
|
|
127
|
+
Results are not comparable between machines or runs. History and pull-request summaries therefore look
|
|
128
|
+
at **change over time on comparable hardware** and at **changes in growth class**, never at one absolute
|
|
126
129
|
number. See [READING-BENCHMARKS.md](READING-BENCHMARKS.md).
|
|
@@ -1,26 +1,28 @@
|
|
|
1
1
|
# Writing benchmarks
|
|
2
2
|
|
|
3
3
|
This guide is for the person adding or changing a benchmark. It explains **what** to test, **how** to
|
|
4
|
-
test it, **why** each choice is made, and **how to document** the result
|
|
5
|
-
can read
|
|
6
|
-
explains how to interpret what you produce.
|
|
4
|
+
test it, **why** each choice is made, and **how to document** the result. The goal is a result a platform
|
|
5
|
+
engineer or CTO can read without knowing the code. Its companion,
|
|
6
|
+
[READING-BENCHMARKS.md](READING-BENCHMARKS.md), explains how to interpret what you produce.
|
|
7
7
|
|
|
8
8
|
The benchmarks exist to answer one question for the people who run this package in production:
|
|
9
|
-
|
|
10
|
-
|
|
9
|
+
|
|
10
|
+
**What does it cost me, in time and in money, and how does that grow as my workload grows?**
|
|
11
|
+
|
|
12
|
+
The rest of the repository makes the case for what the package gives you. The benchmarks are deliberately
|
|
11
13
|
honest about what it costs.
|
|
12
14
|
|
|
13
15
|
## The three views
|
|
14
16
|
|
|
15
17
|
Every suite measures the package from three angles, each building on the last.
|
|
16
18
|
|
|
17
|
-
1. **End-to-end: the total impact.** One operation done two ways, with empty or minimal functions
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
2. **Functions: each part on its own.** Every function the end-to-end run uses
|
|
22
|
-
|
|
23
|
-
|
|
19
|
+
1. **End-to-end: the total impact.** One operation is done two ways, with empty or minimal functions plugged
|
|
20
|
+
in. It runs once with no package (the _baseline_) and once routed through the package. The difference is
|
|
21
|
+
the package's own cost: the floor it imposes on every operation. Keep both sides trivial on purpose: if
|
|
22
|
+
you plug in real work, you are measuring that work.
|
|
23
|
+
2. **Functions: each part on its own.** Every function the end-to-end run uses is measured separately across
|
|
24
|
+
the full size ladder. So is every public function worth knowing about. That shows how each one scales and
|
|
25
|
+
whether it behaves the way we say it does.
|
|
24
26
|
3. **Contribution: where the cost lives.** Each function says how often one end-to-end operation calls
|
|
25
27
|
it (`inEndToEnd`). The report multiplies that by the function's measured cost and shows its share
|
|
26
28
|
of the overhead, so effort goes to the function that actually matters.
|
|
@@ -38,20 +40,20 @@ a function whose cost nobody can see.
|
|
|
38
40
|
serverless invocation that scales from zero.
|
|
39
41
|
- **Memory**, as well as time: what the operation allocates and retains. The report records the heap
|
|
40
42
|
delta of every sample.
|
|
41
|
-
- **Every variable that could change the cost.** This is the part that is
|
|
42
|
-
that matters most. Walk through each of these for every function and decide, explicitly,
|
|
43
|
-
is _swept_ (the size axis), a _variant_ (measured separately), or _fixed_ (held constant
|
|
44
|
-
purpose, with the value stated):
|
|
45
|
-
- **Input size
|
|
46
|
-
- **Input shape
|
|
47
|
-
|
|
48
|
-
- **State
|
|
49
|
-
- **Options and modes
|
|
50
|
-
- **Outcome
|
|
43
|
+
- **Every variable that could change the cost.** This is the part that is tempting to skip and the part
|
|
44
|
+
that matters most. Walk through each of these for every function and decide, explicitly, how it is
|
|
45
|
+
treated. Each is _swept_ (the size axis), a _variant_ (measured separately), or _fixed_ (held constant
|
|
46
|
+
on purpose, with the value stated):
|
|
47
|
+
- **Input size**: the size ladder below. Always swept.
|
|
48
|
+
- **Input shape**: flat vs deeply nested, many small items vs few large ones, wide vs narrow objects.
|
|
49
|
+
Also strings vs numbers, and sorted vs random vs adversarial order.
|
|
50
|
+
- **State**: empty vs populated, first call (cold) vs repeated call (warm), cache hit vs miss.
|
|
51
|
+
- **Options and modes**: every flag that changes the algorithm used.
|
|
52
|
+
- **Outcome**: the success path **and** the error path. Errors are often slower and often hit
|
|
51
53
|
under load, exactly when slowness hurts most.
|
|
52
|
-
- **Concurrency
|
|
54
|
+
- **Concurrency**: one caller vs many at once, if the function keeps shared state or does async
|
|
53
55
|
work.
|
|
54
|
-
- **Environment
|
|
56
|
+
- **Environment**: runtime and version, platform, available CPU and memory.
|
|
55
57
|
- **What you deliberately do not cover**, listed in `notCovered` with the reason. An honest gap is
|
|
56
58
|
documentation; a silent gap is a trap.
|
|
57
59
|
|
|
@@ -63,14 +65,14 @@ Every function is measured at **20, 40, 80, 160, 320, 640, 1280, 2560, 5120 and
|
|
|
63
65
|
The default ladder of ten doubling points is not arbitrary. Big-O is estimated by fitting a line through
|
|
64
66
|
`log(time)` against `log(size)`; its slope is the growth exponent. Three points cannot tell `O(n)`
|
|
65
67
|
from `O(n log n)` or `O(1)` from `O(log n)`. A ladder spanning more than two orders of magnitude, with
|
|
66
|
-
ten points, can. `typicalN` (default 640) marks the size the cost summary quotes
|
|
67
|
-
|
|
68
|
+
ten points, can. `typicalN` (default 640) marks the size the cost summary quotes. Choose the size a real
|
|
69
|
+
operation of this package typically handles, and say why in `workload.description`.
|
|
68
70
|
|
|
69
|
-
Never shorten the ladder for committed results, with one recorded exception
|
|
71
|
+
Never shorten the ladder for committed results, with one recorded exception. A function too slow to
|
|
70
72
|
measure at the largest sizes (it compiles a package, or spawns a process, per unit) may list its own
|
|
71
|
-
`tiers
|
|
72
|
-
prints beside its results. `--quick` exists for a smoke test and says so in its output; do not
|
|
73
|
-
what it produces.
|
|
73
|
+
`tiers`. These are at least four ascending sizes, given together with a `tiersReason` saying why, which
|
|
74
|
+
the report prints beside its results. `--quick` exists for a smoke test and says so in its output; do not
|
|
75
|
+
commit what it produces.
|
|
74
76
|
|
|
75
77
|
## How to test
|
|
76
78
|
|
|
@@ -80,52 +82,52 @@ what it produces.
|
|
|
80
82
|
whose result is thrown away can be optimized into nothing, and you will have benchmarked an empty
|
|
81
83
|
loop.
|
|
82
84
|
- **Fresh state when the work changes its own input.** If `run` mutates what `setup` built (committing
|
|
83
|
-
to a store, draining a queue), set `fresh: true
|
|
85
|
+
to a store, draining a queue), set `fresh: true`. Each sample then gets a new `setup`. Otherwise the
|
|
84
86
|
second sample measures a different, usually easier, problem.
|
|
85
87
|
- **Deterministic input.** No `Math.random()`, no `Date.now()` in input generation. Generate from the
|
|
86
88
|
index (`item i = { id: i, name: "item-" + i }`) so two runs measure the same thing. If you need
|
|
87
89
|
pseudo-randomness, use a seeded generator and record the seed in the suite.
|
|
88
90
|
- **No real I/O.** A network call or disk read measures the network or the disk. Simulate it with a
|
|
89
|
-
function that resolves from a prepared pool
|
|
91
|
+
function that resolves from a prepared pool. Keep the real code path (queues, deduplication, error
|
|
90
92
|
handling) intact, because that is what the package contributes.
|
|
91
93
|
- **Import the package the way a consumer does.** Its built entry point, never a source path. The
|
|
92
94
|
published `exports` map is part of the cost (and of the risk). A package whose costly code is internal
|
|
93
|
-
(an engine no entry point exposes on its own) may import that source instead
|
|
94
|
-
|
|
95
|
-
- **Let the engine handle fast operations.**
|
|
96
|
-
back to back inside one sample and the result divided
|
|
97
|
-
|
|
95
|
+
(an engine no entry point exposes on its own) may import that source instead. Its benchmarks README then
|
|
96
|
+
says so and why.
|
|
97
|
+
- **Let the engine handle fast operations.** Timers cannot resolve operations faster than about a
|
|
98
|
+
millisecond. Those are repeated back to back inside one sample, and the result is divided. You do
|
|
99
|
+
nothing; the report records the batch size.
|
|
98
100
|
- **Quiet machine.** Close heavy programs, plug in a laptop, and run on the same machine you compared
|
|
99
101
|
against last time. Never compare across machines (see the reading guide).
|
|
100
102
|
|
|
101
103
|
## How to document a benchmark
|
|
102
104
|
|
|
103
|
-
`defineSuite` refuses a suite with missing or placeholder documentation
|
|
104
|
-
|
|
105
|
-
|
|
105
|
+
`defineSuite` refuses a suite with missing or placeholder documentation. `npm run benchmark:check` runs
|
|
106
|
+
that validation without measuring anything, so it can run in CI on every change. Every function needs the
|
|
107
|
+
following, written for a reader who does not know the code.
|
|
106
108
|
|
|
107
|
-
### `why
|
|
109
|
+
### `why`: why we benchmark it
|
|
108
110
|
|
|
109
111
|
The reason this function earns a benchmark: where it runs, how often, and who feels it. Not what it
|
|
110
|
-
does
|
|
112
|
+
does, but why its cost matters.
|
|
111
113
|
|
|
112
|
-
> **Good:** "Runs once per getter call, on every request that reads fetched data
|
|
113
|
-
> is multiplied by the application's whole read traffic."
|
|
114
|
+
> **Good:** "Runs once per getter call, on every request that reads fetched data. Its per-call cost
|
|
115
|
+
> is therefore multiplied by the application's whole read traffic."
|
|
114
116
|
>
|
|
115
117
|
> **Bad:** "Tests the commit function."
|
|
116
118
|
|
|
117
|
-
### `poorPerformanceMeans
|
|
119
|
+
### `poorPerformanceMeans`: what slow would mean
|
|
118
120
|
|
|
119
121
|
Translate slowness into consequences for the people running the system: latency users notice, server
|
|
120
122
|
capacity, cloud spend, cold-start time, memory pressure.
|
|
121
123
|
|
|
122
|
-
> **Good:** "A slowdown adds directly to every response time and to compute cost per request
|
|
124
|
+
> **Good:** "A slowdown adds directly to every response time and to compute cost per request. At
|
|
123
125
|
> 10,000 items a quadratic regression would turn a 50 ms commit into several seconds and exhaust
|
|
124
126
|
> request timeouts."
|
|
125
127
|
>
|
|
126
128
|
> **Bad:** "Performance would be worse."
|
|
127
129
|
|
|
128
|
-
### `expectedComplexity` and `complexityReason
|
|
130
|
+
### `expectedComplexity` and `complexityReason`: the big-O and why
|
|
129
131
|
|
|
130
132
|
Pick the class you **expect from reading the code** before you run anything:
|
|
131
133
|
|
|
@@ -139,27 +141,27 @@ Pick the class you **expect from reading the code** before you run anything:
|
|
|
139
141
|
| `cubic-or-worse` | O(n³) or worse | Anything faster-growing than quadratic. |
|
|
140
142
|
|
|
141
143
|
The reason is one or two sentences that **name the mechanism**, so a reader can check it against the
|
|
142
|
-
code
|
|
143
|
-
run, the report compares the measured class with this one. **Agreement** is the goal
|
|
144
|
-
means the documentation or the code is wrong
|
|
145
|
-
(constant/logarithmic, linear/linearithmic) are reported as "close" because a bounded ladder
|
|
146
|
-
reliably separate them.
|
|
144
|
+
code. For example, "walks every newly arrived item once to freeze it" is a reason; "it is linear" is not.
|
|
145
|
+
After the run, the report compares the measured class with this one. **Agreement** is the goal. A
|
|
146
|
+
**difference** means the documentation or the code is wrong, and both are worth fixing. Neighbouring
|
|
147
|
+
classes (constant/logarithmic, linear/linearithmic) are reported as "close" because a bounded ladder
|
|
148
|
+
cannot reliably separate them.
|
|
147
149
|
|
|
148
|
-
### `variables
|
|
150
|
+
### `variables`: everything that could change the cost
|
|
149
151
|
|
|
150
152
|
One row per variable from the checklist above, each marked:
|
|
151
153
|
|
|
152
|
-
- `swept
|
|
153
|
-
- `variant
|
|
154
|
-
- `fixed
|
|
154
|
+
- `swept`: it is the size axis.
|
|
155
|
+
- `variant`: measured separately; describe each in `variants`.
|
|
156
|
+
- `fixed`: held constant on purpose; give the `value`, so a reader knows what was and was not
|
|
155
157
|
covered.
|
|
156
158
|
|
|
157
|
-
### `covers` and `notBenchmarked
|
|
159
|
+
### `covers` and `notBenchmarked`: every input of every public function
|
|
158
160
|
|
|
159
161
|
A public function with more than one input (several parameters, or an object with several keys) has
|
|
160
|
-
several things that can change its cost
|
|
161
|
-
|
|
162
|
-
|
|
162
|
+
several things that can change its cost. The `BenchmarkCoverage` check fails until each one is accounted
|
|
163
|
+
for. In the benchmark that exercises the function, list what you did with every parameter, and with every
|
|
164
|
+
key of an object parameter:
|
|
163
165
|
|
|
164
166
|
```js
|
|
165
167
|
covers: {
|
|
@@ -172,17 +174,23 @@ covers: {
|
|
|
172
174
|
},
|
|
173
175
|
```
|
|
174
176
|
|
|
175
|
-
Each is one of
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
177
|
+
Each is one of:
|
|
178
|
+
|
|
179
|
+
- `swept`: the suite's unit varies it;
|
|
180
|
+
- `sweep:<variable>`: one of the function's `sweeps`;
|
|
181
|
+
- `variant:<name>`: one of its `variants`;
|
|
182
|
+
- `fixed: <why>`: held at one value, and why;
|
|
183
|
+
- `not covered: <why>`: left out, and why.
|
|
184
|
+
|
|
185
|
+
A function that has no benchmark at all is listed in the suite's `notBenchmarked` with the reason. One
|
|
186
|
+
such reason: a benchmarked function that calls it already accounts for its cost.
|
|
179
187
|
|
|
180
|
-
### `sweeps
|
|
188
|
+
### `sweeps`: a second variable that scales
|
|
181
189
|
|
|
182
190
|
Some functions grow with more than one input: the number of records _and_ how many fields each record
|
|
183
191
|
has. One big-O for both would be wrong, because the two do not have to grow at the same rate. List each
|
|
184
|
-
other variable that scales under `sweeps
|
|
185
|
-
held at its `baseValue
|
|
192
|
+
other variable that scales under `sweeps`, and the suite measures it on its own. Every other variable is
|
|
193
|
+
held at its `baseValue`; the suite's own unit is held at `primaryValue`, by default the typical size:
|
|
186
194
|
|
|
187
195
|
```js
|
|
188
196
|
sweeps: [
|
|
@@ -197,28 +205,30 @@ sweeps: [
|
|
|
197
205
|
```
|
|
198
206
|
|
|
199
207
|
The variable must also be listed under `variables` as `swept`. `setup` (or `input`) receives a third
|
|
200
|
-
argument, `{ variable, values }
|
|
201
|
-
measurement, so one function builds its input for any of them. In a sweep of another variable, `n` is
|
|
202
|
-
|
|
203
|
-
|
|
208
|
+
argument, `{ variable, values }`. It holds the variable being swept and the value of every variable at
|
|
209
|
+
this measurement, so one function builds its input for any of them. In a sweep of another variable, `n` is
|
|
210
|
+
that variable's value, not the size of the unit. Always read every variable from `context.values`, which
|
|
211
|
+
is right for the primary sweep too. Each sweep is its own group (`fn:<id>#<variable>`), with its own
|
|
212
|
+
growth class and its own documented big-O. It also has its own row in the report, the history and the
|
|
213
|
+
pull-request summary.
|
|
204
214
|
|
|
205
|
-
### `variants
|
|
215
|
+
### `variants`: measured alternatives
|
|
206
216
|
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
chart in history, so a regression in one path cannot hide behind another.
|
|
217
|
+
Some variables change the algorithm: cache hit vs miss, flat vs nested, success vs error. Measure each
|
|
218
|
+
alternative as a named variant across the whole ladder. Each variant is its own row in the report and its
|
|
219
|
+
own chart in history, so a regression in one path cannot hide behind another.
|
|
210
220
|
|
|
211
|
-
### `notCovered
|
|
221
|
+
### `notCovered`: the honest gaps
|
|
212
222
|
|
|
213
223
|
Name what you chose not to measure and why ("concurrent callers: the function is synchronous and
|
|
214
224
|
keeps no shared state"). A reader should never wonder whether something was forgotten.
|
|
215
225
|
|
|
216
|
-
### `inEndToEnd
|
|
226
|
+
### `inEndToEnd`: where it fits in the whole
|
|
217
227
|
|
|
218
228
|
State how often one end-to-end operation calls this function (`callsPerOperation`, a number or a
|
|
219
229
|
function of `n`) and describe the call. This is what lets section 3 of the report attribute the
|
|
220
|
-
time of one operation to the functions that cause it. A function measured alone includes everything
|
|
221
|
-
|
|
230
|
+
time of one operation to the functions that cause it. A function measured alone includes everything it
|
|
231
|
+
calls. When one attributed function calls another, list the callee's id in `includes`. Its time is then
|
|
222
232
|
subtracted from the caller and counted once.
|
|
223
233
|
|
|
224
234
|
### The end-to-end block
|