@maverickcer/env-cap 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ADOPTION.md +144 -0
- package/AGENTS.md +146 -0
- package/CHANGELOG.md +438 -0
- package/CODE_OF_CONDUCT.md +124 -0
- package/CODE_REVIEW.md +200 -0
- package/CONTRIBUTING.md +131 -0
- package/GUIDE.md +1214 -0
- package/PROMPT.md +89 -0
- package/README.md +92 -200
- package/RELEASING.md +160 -0
- package/SECURITY.md +270 -0
- package/SUPPORT.md +76 -0
- package/VERSIONING.md +118 -0
- package/benchmarks/READING-BENCHMARKS.md +197 -0
- package/benchmarks/README.md +126 -0
- package/benchmarks/WRITING-BENCHMARKS.md +252 -0
- package/benchmarks/performance-buildtime/BENCHMARKS.md +889 -0
- package/benchmarks/performance-runtime/BENCHMARKS.md +1988 -0
- package/dist/.dts/_ipc/build-fs.d.ts +53 -0
- package/dist/.dts/_ipc/build-fs.d.ts.map +1 -0
- package/dist/.dts/_ipc/compiler-api.d.ts +17 -0
- package/dist/.dts/_ipc/compiler-api.d.ts.map +1 -0
- package/dist/.dts/_ipc/display-path.d.ts +27 -0
- package/dist/.dts/_ipc/display-path.d.ts.map +1 -0
- package/dist/.dts/_ipc/generated-banner.d.ts +58 -0
- package/dist/.dts/_ipc/generated-banner.d.ts.map +1 -0
- package/dist/.dts/_ipc/glob.d.ts +22 -0
- package/dist/.dts/_ipc/glob.d.ts.map +1 -0
- package/dist/.dts/_ipc/no-node-fs.d.ts +34 -0
- package/dist/.dts/_ipc/no-node-fs.d.ts.map +1 -0
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts +134 -0
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts.map +1 -0
- package/dist/.dts/_ipc/typescript.d.ts +20 -0
- package/dist/.dts/_ipc/typescript.d.ts.map +1 -0
- package/dist/.dts/build/compatibility.d.ts.map +1 -1
- package/dist/.dts/build/compiler-api.d.ts +3 -6
- package/dist/.dts/build/compiler-api.d.ts.map +1 -1
- package/dist/.dts/build/display-path.d.ts +3 -21
- package/dist/.dts/build/display-path.d.ts.map +1 -1
- package/dist/.dts/build/generated-banner.d.ts +21 -42
- package/dist/.dts/build/generated-banner.d.ts.map +1 -1
- package/dist/.dts/build/glob.d.ts +3 -15
- package/dist/.dts/build/glob.d.ts.map +1 -1
- package/dist/.dts/build/link.d.ts.map +1 -1
- package/dist/.dts/build/merge-documentation.d.ts +35 -0
- package/dist/.dts/build/merge-documentation.d.ts.map +1 -0
- package/dist/.dts/build/resolution/resolve-tsconfig-paths.d.ts +11 -112
- package/dist/.dts/build/resolution/resolve-tsconfig-paths.d.ts.map +1 -1
- package/dist/.dts/build/types.d.ts +4 -53
- package/dist/.dts/build/types.d.ts.map +1 -1
- package/dist/.dts/build/typescript.d.ts +3 -17
- package/dist/.dts/build/typescript.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/glob.d.ts +4 -15
- package/dist/.dts/eslint-plugin/glob.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/index.d.ts +1 -1
- package/dist/.dts/eslint-plugin/no-node-fs.d.ts +8 -18
- package/dist/.dts/eslint-plugin/no-node-fs.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/no-raw-process-env.d.ts.map +1 -1
- package/dist/{build-H4KFV44Z.js → build-WKDIAWF7.js} +198 -60
- package/dist/build-WKDIAWF7.js.map +1 -0
- package/dist/build.cjs +194 -56
- package/dist/build.cjs.map +1 -1
- package/dist/build.js +194 -56
- package/dist/build.js.map +1 -1
- package/dist/{chunk-7GVPOHDU.js → chunk-KEGYUHBU.js} +2 -2
- package/dist/cli/index.js +36 -29
- package/dist/cli/index.js.map +1 -1
- package/dist/eslint-plugin/index.cjs +84 -54
- package/dist/eslint-plugin/index.cjs.map +1 -1
- package/dist/eslint-plugin/index.js +83 -53
- package/dist/eslint-plugin/index.js.map +1 -1
- package/examples/README.md +95 -0
- package/package.json +48 -14
- package/skills/env-cap/SKILL.md +91 -0
- package/skills/env-cap/references/api.md +220 -0
- package/specs/architecture.md +634 -0
- package/specs/decisions/0001-runtime-documentation-separation.md +197 -0
- package/specs/decisions/0002-static-analysis-never-execution.md +233 -0
- package/specs/decisions/0003-no-global-env-object.md +172 -0
- package/specs/decisions/0004-no-client-server-package-split.md +116 -0
- package/specs/decisions/0005-warn-not-throw-by-default.md +93 -0
- package/specs/decisions/0006-self-redacting-contracts.md +124 -0
- package/specs/decisions/0007-processor-not-transformer.md +109 -0
- package/specs/decisions/0008-gzip-size-budget.md +173 -0
- package/specs/decisions/0009-exclusive-groups-are-always-errors.md +139 -0
- package/specs/decisions/0010-dependency-ownership-engine-scope-boundary.md +100 -0
- package/specs/decisions/0011-shared-discovery-compute-atomic-write-non-atomic.md +104 -0
- package/specs/decisions/0012-live-expiration-overrides-not-ast-functions.md +145 -0
- package/specs/decisions/0013-json-output-is-a-versioned-mirror.md +136 -0
- package/specs/decisions/0014-cross-package-schema-discovery.md +206 -0
- package/specs/decisions/0015-security-backport-window.md +60 -0
- package/specs/decisions/0016-check-mode-compute-before-compare-never-partial-write.md +113 -0
- package/specs/decisions/0017-eslint-plugin-entry-point.md +113 -0
- package/specs/decisions/0018-rotation-alert-issue-on-non-pr-runs.md +104 -0
- package/specs/decisions/0019-published-json-schema-generated-from-types.md +100 -0
- package/specs/decisions/0020-declaration-maps-via-separate-tsc-pass.md +101 -0
- package/specs/decisions/0021-manifest-change-report-persisted-snapshot.md +177 -0
- package/specs/decisions/0022-validation-contexts.md +267 -0
- package/specs/decisions/0023-tsconfig-path-alias-resolution.md +182 -0
- package/specs/decisions/0024-fact-model-architecture.md +144 -0
- package/specs/decisions/0025-contract-model-json-projection.md +105 -0
- package/specs/decisions/0026-finding-model-unifies-four-families.md +109 -0
- package/specs/decisions/0027-dependency-model-fact-shape-not-engine-access.md +98 -0
- package/specs/decisions/0028-ownership-model-shared-effective-owner.md +85 -0
- package/specs/decisions/0029-lifecycle-model-deprecation-rename-fields.md +98 -0
- package/specs/decisions/0030-change-model-wraps-manifest-change-report.md +97 -0
- package/specs/decisions/0031-evidence-entry-point.md +85 -0
- package/specs/decisions/0032-evidence-projection-provenance-mechanism.md +120 -0
- package/specs/decisions/0033-reference-projections-live-in-examples.md +81 -0
- package/specs/decisions/0034-three-tier-examples-restructuring.md +85 -0
- package/specs/decisions/0035-documentenv-metadata-generalization-and-compliance-fields.md +85 -0
- package/specs/decisions/0036-exact-source-position-evidence.md +122 -0
- package/specs/decisions/0037-developer-declared-dynamic-access-citations.md +114 -0
- package/specs/decisions/0038-always-computed-evidence-model-backs-generate-check-docs.md +244 -0
- package/specs/decisions/0039-scanner-local-dataflow-boundary.md +160 -0
- package/specs/decisions/0040-library-surfaces-do-not-acquire-node-fs.md +156 -0
- package/specs/decisions/0041-dual-package-hazard-checked-documented.md +75 -0
- package/specs/decisions/0042-init-cli-subcommand-scaffolds-only.md +95 -0
- package/specs/decisions/0043-env-cap-resolver-relocated-and-duplicated.md +71 -0
- package/specs/decisions/0044-strict-escalates-every-warning-family.md +88 -0
- package/specs/decisions/0045-promote-experimental-surfaces-to-stable.md +103 -0
- package/specs/decisions/0046-cli-restricted-to-runtime-and-evidence-output.md +177 -0
- package/specs/decisions/0047-supported-toolchain-node-22-and-typescript-5-6.md +30 -0
- package/specs/decisions/0048-eslint-plugin-does-not-vendor-typescript-eslint.md +25 -0
- package/specs/decisions/0049-typescript-7-scanner-uses-a-bundled-typescript-6.md +87 -0
- package/specs/decisions/0050-a-variable-declared-twice-is-documented-once.md +55 -0
- package/specs/generated-artifacts.md +104 -0
- package/specs/migrations/README.md +30 -0
- package/specs/migrations/from-centralized-schema.md +503 -0
- package/specs/migrations/from-dotenv.md +352 -0
- package/specs/migrations/from-envalid.md +482 -0
- package/specs/migrations/from-process-env.md +398 -0
- package/specs/migrations/from-t3-env.md +416 -0
- package/specs/migrations/from-zod.md +303 -0
- package/dist/build-H4KFV44Z.js.map +0 -1
- /package/dist/{chunk-7GVPOHDU.js.map → chunk-KEGYUHBU.js.map} +0 -0
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
# Benchmarks
|
|
2
|
+
|
|
3
|
+
This folder is the performance and cost record of the package: what adopting it costs in time and
|
|
4
|
+
money, how each function scales, and where the cost comes from. It is built entirely by
|
|
5
|
+
[`internal-package-contract`](https://github.com/MaverickCER/internal-package-contract): the package
|
|
6
|
+
supplies **what to run** (an input), the contract **stresses it through ten sizes, measures it, checks
|
|
7
|
+
the big-O and prices it**, and writes a **report** and a machine-readable **result** (the output).
|
|
8
|
+
Nothing in this folder implements timing, statistics or reporting.
|
|
9
|
+
|
|
10
|
+
## Read first
|
|
11
|
+
|
|
12
|
+
| File | For whom | What it is |
|
|
13
|
+
| ----------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------- |
|
|
14
|
+
| `performance-*/BENCHMARKS.md` | Everyone | The generated report: cost at a glance, end-to-end, per function, attribution. |
|
|
15
|
+
| [READING-BENCHMARKS.md](READING-BENCHMARKS.md) | Readers | Why we benchmark, how to read the numbers, and why they are not comparable between runs. |
|
|
16
|
+
| [WRITING-BENCHMARKS.md](WRITING-BENCHMARKS.md) | Contributors | What to test, how, why, and how to document it. |
|
|
17
|
+
| `performance-*/suite.mjs` (one per area, below) | Contributors | The input: what this package runs. |
|
|
18
|
+
| `results.json` | Tools | The output: every measurement and analysis, in a documented shape. |
|
|
19
|
+
|
|
20
|
+
The history page on the project website and the pull-request summaries link back here.
|
|
21
|
+
|
|
22
|
+
## Two suites
|
|
23
|
+
|
|
24
|
+
| Suite | What it measures | Unit of `n` |
|
|
25
|
+
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
|
26
|
+
| [performance-runtime](performance-runtime/BENCHMARKS.md) | What a service pays at startup: process cold start, `createEnv`, `validateEnv`, and every processor and validator on its own. | environment variables (or the size of one value, for the helpers) |
|
|
27
|
+
| [performance-buildtime](performance-buildtime/BENCHMARKS.md) | What a pipeline or developer pays to discover schemas and generate manifests, documentation, usage reports and evidence. | declared variables (ten per schema file) |
|
|
28
|
+
|
|
29
|
+
The build-time ladder stops at 5,120 variables because `discoverSchemaFiles` refuses more than 1,000
|
|
30
|
+
files; its report says so.
|
|
31
|
+
|
|
32
|
+
## Commands
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
npm run benchmark # both suites -> results.json + BENCHMARKS.md in each
|
|
36
|
+
npm run benchmark -- --quick # smoke test; never commit its output
|
|
37
|
+
npm run benchmark -- --only <id> # run a subset while developing
|
|
38
|
+
npm run benchmark:check # validate the suite's documentation; measures nothing
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## The input: `suite.mjs`
|
|
42
|
+
|
|
43
|
+
A suite is plain data plus the functions to run. Import the package's real functions and describe
|
|
44
|
+
the arguments to give them at size `n`; the contract does the rest.
|
|
45
|
+
|
|
46
|
+
```js
|
|
47
|
+
import { defineSuite } from "internal-package-contract/benchmark"
|
|
48
|
+
import { parse, validate } from "../dist/index.js"
|
|
49
|
+
|
|
50
|
+
export default defineSuite({
|
|
51
|
+
package: { name: "my-package", bundleFiles: ["dist/index.js"] },
|
|
52
|
+
workload: { unit: "record", description: "One record in the collection.", typicalN: 640 },
|
|
53
|
+
|
|
54
|
+
endToEnd: {
|
|
55
|
+
purpose: "What routing one operation through the package adds over a bare function.",
|
|
56
|
+
baseline: { description: "Empty function, no package.", call: (n) => n, input: (n) => [n] },
|
|
57
|
+
withPackage: {
|
|
58
|
+
description: "The same work through the package.",
|
|
59
|
+
call: parse,
|
|
60
|
+
input: (n) => [makeInput(n)],
|
|
61
|
+
},
|
|
62
|
+
variables: [{ name: "records", how: "swept", description: "The tier axis." }],
|
|
63
|
+
},
|
|
64
|
+
|
|
65
|
+
functions: [
|
|
66
|
+
{
|
|
67
|
+
id: "validate",
|
|
68
|
+
name: "validate",
|
|
69
|
+
why: "...",
|
|
70
|
+
poorPerformanceMeans: "...",
|
|
71
|
+
expectedComplexity: "linear",
|
|
72
|
+
complexityReason: "...",
|
|
73
|
+
variables: [{ name: "records", how: "swept", description: "The tier axis." }],
|
|
74
|
+
call: validate, // the imported function under test
|
|
75
|
+
input: (n) => [makeInput(n)], // its arguments at size n -> validate(...input(n))
|
|
76
|
+
inEndToEnd: { callsPerOperation: 1, description: "Once per operation." },
|
|
77
|
+
},
|
|
78
|
+
],
|
|
79
|
+
})
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
| Field | Required | Meaning |
|
|
83
|
+
| ----------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------- |
|
|
84
|
+
| `package.name`, `package.bundleFiles` | yes / no | The package, and the shipped files whose size is reported. |
|
|
85
|
+
| `workload.unit`, `workload.description`, `workload.typicalN` | yes | What one unit of `n` is, and the size the cost summary quotes (one of the tiers). |
|
|
86
|
+
| `tiers` | no | The size ladder; defaults to `20, 40, 80, 160, 320, 640, 1280, 2560, 5120, 10240`. |
|
|
87
|
+
| `endToEnd.purpose`, `.baseline`, `.withPackage`, `.variables` | yes | The total-impact comparison with empty or minimal functions on both sides. |
|
|
88
|
+
| `functions[]` | yes | One entry per function. Each is documented (below) and runnable. |
|
|
89
|
+
| `call` + `input(n, variantOptions)` | one form | The function and its arguments at size `n`. Or use `setup(n)` + `run(ctx)` for stateful work. |
|
|
90
|
+
| `fresh` | no | Build new state for every sample (for work that consumes its own input). |
|
|
91
|
+
| `variants[]` | no | Named alternatives (cache hit/miss, input shapes), each measured over the whole ladder. |
|
|
92
|
+
| `tiers` + `tiersReason` (per function) | no | A shorter ladder for a function that cannot take the largest sizes, with the reason. |
|
|
93
|
+
| `sampling` | no | Override warmup/iteration counts for a very slow function. |
|
|
94
|
+
| `why`, `poorPerformanceMeans`, `expectedComplexity`, `complexityReason` | yes | The documentation. `defineSuite` rejects missing or placeholder text. |
|
|
95
|
+
| `variables[]`, `notCovered[]` | yes / no | Every variable that could change the cost (swept / variant / fixed) and honest gaps. |
|
|
96
|
+
| `inEndToEnd` | no | How often one end-to-end operation calls the function, for the attribution section. |
|
|
97
|
+
|
|
98
|
+
`expectedComplexity` is one of `constant`, `logarithmic`, `linear`, `linearithmic`, `quadratic`,
|
|
99
|
+
`exponential-or-worse`. Return a value that depends on the work (the contract keeps it alive so
|
|
100
|
+
nothing is optimized away); return `timed(ms)` only when an operation is timed elsewhere, such as a
|
|
101
|
+
child process.
|
|
102
|
+
|
|
103
|
+
## The output: `results.json`
|
|
104
|
+
|
|
105
|
+
`results.json` has three parts. The shape is fixed (`schemaVersion` 3) and checked on every run by
|
|
106
|
+
`validateResults`, which `internal-package-contract/benchmark` also exports.
|
|
107
|
+
|
|
108
|
+
```text
|
|
109
|
+
metadata package, workload, tiers, environment (CPU, cores, memory, Node), git, bundleSizes,
|
|
110
|
+
costRates, timing, generatedBy
|
|
111
|
+
results one entry per measured group: "end-to-end:baseline", "end-to-end:with-package",
|
|
112
|
+
"end-to-end:overhead", "fn:<id>", "fn:<id>@<variant>"
|
|
113
|
+
tiers.n<size> = { id, status, inputs: { <unit>: size },
|
|
114
|
+
durationMs{min,median,p95,max,stdDev,iterations},
|
|
115
|
+
cpuMs{...}, heapDeltaBytes, opsPerSecond, configuration }
|
|
116
|
+
analysis complexity measured class, exponent and notation per group, with the documented
|
|
117
|
+
class and an agreement of matches / close / differs
|
|
118
|
+
endToEnd per size: baseline, with package, added time, added CPU, added memory,
|
|
119
|
+
estimated cost per million operations
|
|
120
|
+
contribution per function: calls per operation, estimated time, share of the overhead
|
|
121
|
+
cost the typical and largest sizes used for the headline numbers
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Because results are not comparable between machines or runs, history and pull-request summaries look at
|
|
125
|
+
**change over time on comparable hardware** and at **changes in growth class**, never at one absolute
|
|
126
|
+
number. See [READING-BENCHMARKS.md](READING-BENCHMARKS.md).
|
|
@@ -0,0 +1,252 @@
|
|
|
1
|
+
# Writing benchmarks
|
|
2
|
+
|
|
3
|
+
This guide is for the person adding or changing a benchmark. It explains **what** to test, **how** to
|
|
4
|
+
test it, **why** each choice is made, and **how to document** the result so a platform engineer or CTO
|
|
5
|
+
can read it without knowing the code. Its companion, [READING-BENCHMARKS.md](READING-BENCHMARKS.md),
|
|
6
|
+
explains how to interpret what you produce.
|
|
7
|
+
|
|
8
|
+
The benchmarks exist to answer one question for the people who run this package in production:
|
|
9
|
+
**what does it cost me, in time and in money, and how does that grow as my workload grows?** The rest
|
|
10
|
+
of the repository makes the case for what the package gives you; the benchmarks are deliberately
|
|
11
|
+
honest about what it costs.
|
|
12
|
+
|
|
13
|
+
## The three views
|
|
14
|
+
|
|
15
|
+
Every suite measures the package from three angles, each building on the last.
|
|
16
|
+
|
|
17
|
+
1. **End-to-end: the total impact.** One operation done two ways, with empty or minimal functions
|
|
18
|
+
plugged in: once with no package (the _baseline_) and once routed through the package. The
|
|
19
|
+
difference is the package's own cost -- the floor it imposes on every operation. Keep both sides
|
|
20
|
+
trivial on purpose: if you plug in real work, you are measuring that work.
|
|
21
|
+
2. **Functions: each part on its own.** Every function the end-to-end run uses (and every public
|
|
22
|
+
function worth knowing about) is measured separately across the full size ladder, so we can see
|
|
23
|
+
how each one scales and whether it behaves the way we say it does.
|
|
24
|
+
3. **Contribution: where the cost lives.** Each function says how often one end-to-end operation calls
|
|
25
|
+
it (`inEndToEnd`). The report multiplies that by the function's measured cost and shows its share
|
|
26
|
+
of the overhead, so effort goes to the function that actually matters.
|
|
27
|
+
|
|
28
|
+
## What to test
|
|
29
|
+
|
|
30
|
+
Use this checklist when you add a function or change an existing one. A function with no benchmark is
|
|
31
|
+
a function whose cost nobody can see.
|
|
32
|
+
|
|
33
|
+
- **Every public function** the package exports, and every internal function that runs on the hot
|
|
34
|
+
path of the end-to-end operation.
|
|
35
|
+
- **The end-to-end operation** the package exists to perform, in its smallest realistic form.
|
|
36
|
+
- **Cold start**, if the package does work at import or startup (module evaluation, building
|
|
37
|
+
registries, reading configuration). Cold start is paid by every process, every deploy and every
|
|
38
|
+
serverless invocation that scales from zero.
|
|
39
|
+
- **Memory**, not just time: what the operation allocates and retains. The report records the heap
|
|
40
|
+
delta of every sample.
|
|
41
|
+
- **Every variable that could change the cost.** This is the part that is easy to skip and the part
|
|
42
|
+
that matters most. Walk through each of these for every function and decide, explicitly, whether it
|
|
43
|
+
is _swept_ (the size axis), a _variant_ (measured separately), or _fixed_ (held constant on
|
|
44
|
+
purpose, with the value stated):
|
|
45
|
+
- **Input size** -- the size ladder below. Always swept.
|
|
46
|
+
- **Input shape** -- flat vs deeply nested, many small items vs few large ones, wide vs narrow
|
|
47
|
+
objects, strings vs numbers, sorted vs random vs adversarial order.
|
|
48
|
+
- **State** -- empty vs populated, first call (cold) vs repeated call (warm), cache hit vs miss.
|
|
49
|
+
- **Options and modes** -- every flag that changes the algorithm used.
|
|
50
|
+
- **Outcome** -- the success path **and** the error path. Errors are often slower and often hit
|
|
51
|
+
under load, exactly when slowness hurts most.
|
|
52
|
+
- **Concurrency** -- one caller vs many at once, if the function keeps shared state or does async
|
|
53
|
+
work.
|
|
54
|
+
- **Environment** -- runtime and version, platform, available CPU and memory.
|
|
55
|
+
- **What you deliberately do not cover**, listed in `notCovered` with the reason. An honest gap is
|
|
56
|
+
documentation; a silent gap is a trap.
|
|
57
|
+
|
|
58
|
+
## The size ladder
|
|
59
|
+
|
|
60
|
+
Every function is measured at **20, 40, 80, 160, 320, 640, 1280, 2560, 5120 and 10240** units of
|
|
61
|
+
`n`. `n` is whatever one suite declares as its workload unit (a record, a variable, a file, a call).
|
|
62
|
+
|
|
63
|
+
The default ladder of ten doubling points is not arbitrary. Big-O is estimated by fitting a line through
|
|
64
|
+
`log(time)` against `log(size)`; its slope is the growth exponent. Three points cannot tell `O(n)`
|
|
65
|
+
from `O(n log n)` or `O(1)` from `O(log n)`. A ladder spanning more than two orders of magnitude, with
|
|
66
|
+
ten points, can. `typicalN` (default 640) marks the size the cost summary quotes -- choose the size a
|
|
67
|
+
real operation of this package typically handles, and say why in `workload.description`.
|
|
68
|
+
|
|
69
|
+
Never shorten the ladder for committed results, with one recorded exception: a function too slow to
|
|
70
|
+
measure at the largest sizes (it compiles a package, or spawns a process, per unit) may list its own
|
|
71
|
+
`tiers` -- at least four ascending sizes -- together with a `tiersReason` saying why, which the report
|
|
72
|
+
prints beside its results. `--quick` exists for a smoke test and says so in its output; do not commit
|
|
73
|
+
what it produces.
|
|
74
|
+
|
|
75
|
+
## How to test
|
|
76
|
+
|
|
77
|
+
- **Separate setup from the measured work.** `setup(n)` builds the input and is not timed; `run(ctx)`
|
|
78
|
+
is the only thing measured. Generating a 10,000-item array inside `run` measures the generator.
|
|
79
|
+
- **Return a value that depends on the work.** The engine keeps the returned value alive. A function
|
|
80
|
+
whose result is thrown away can be optimized into nothing, and you will have benchmarked an empty
|
|
81
|
+
loop.
|
|
82
|
+
- **Fresh state when the work changes its own input.** If `run` mutates what `setup` built (committing
|
|
83
|
+
to a store, draining a queue), set `fresh: true` so each sample gets a new `setup`. Otherwise the
|
|
84
|
+
second sample measures a different, usually easier, problem.
|
|
85
|
+
- **Deterministic input.** No `Math.random()`, no `Date.now()` in input generation. Generate from the
|
|
86
|
+
index (`item i = { id: i, name: "item-" + i }`) so two runs measure the same thing. If you need
|
|
87
|
+
pseudo-randomness, use a seeded generator and record the seed in the suite.
|
|
88
|
+
- **No real I/O.** A network call or disk read measures the network or the disk. Simulate it with a
|
|
89
|
+
function that resolves from a prepared pool; keep the real code path (queues, deduplication, error
|
|
90
|
+
handling) intact, because that is what the package contributes.
|
|
91
|
+
- **Import the package the way a consumer does.** Its built entry point, never a source path. The
|
|
92
|
+
published `exports` map is part of the cost (and of the risk). A package whose costly code is internal
|
|
93
|
+
(an engine no entry point exposes on its own) may import that source instead, and its benchmarks
|
|
94
|
+
README says so and why.
|
|
95
|
+
- **Let the engine handle fast operations.** Operations faster than about a millisecond are repeated
|
|
96
|
+
back to back inside one sample and the result divided, because timers cannot resolve them. You
|
|
97
|
+
do nothing; the report records the batch size.
|
|
98
|
+
- **Quiet machine.** Close heavy programs, plug in a laptop, and run on the same machine you compared
|
|
99
|
+
against last time. Never compare across machines (see the reading guide).
|
|
100
|
+
|
|
101
|
+
## How to document a benchmark
|
|
102
|
+
|
|
103
|
+
`defineSuite` refuses a suite with missing or placeholder documentation, and `npm run benchmark:check`
|
|
104
|
+
runs that validation without measuring anything, so it can run in CI on every change. Every function
|
|
105
|
+
needs the following, written for a reader who does not know the code.
|
|
106
|
+
|
|
107
|
+
### `why` -- why we benchmark it
|
|
108
|
+
|
|
109
|
+
The reason this function earns a benchmark: where it runs, how often, and who feels it. Not what it
|
|
110
|
+
does -- why its cost matters.
|
|
111
|
+
|
|
112
|
+
> **Good:** "Runs once per getter call, on every request that reads fetched data, so its per-call cost
|
|
113
|
+
> is multiplied by the application's whole read traffic."
|
|
114
|
+
>
|
|
115
|
+
> **Bad:** "Tests the commit function."
|
|
116
|
+
|
|
117
|
+
### `poorPerformanceMeans` -- what slow would mean
|
|
118
|
+
|
|
119
|
+
Translate slowness into consequences for the people running the system: latency users notice, server
|
|
120
|
+
capacity, cloud spend, cold-start time, memory pressure.
|
|
121
|
+
|
|
122
|
+
> **Good:** "A slowdown adds directly to every response time and to compute cost per request; at
|
|
123
|
+
> 10,000 items a quadratic regression would turn a 50 ms commit into several seconds and exhaust
|
|
124
|
+
> request timeouts."
|
|
125
|
+
>
|
|
126
|
+
> **Bad:** "Performance would be worse."
|
|
127
|
+
|
|
128
|
+
### `expectedComplexity` and `complexityReason` -- the big-O and why
|
|
129
|
+
|
|
130
|
+
Pick the class you **expect from reading the code** before you run anything:
|
|
131
|
+
|
|
132
|
+
| Class | Notation | Typical cause |
|
|
133
|
+
| ---------------- | -------------- | --------------------------------------------------------------------------- |
|
|
134
|
+
| `constant` | O(1) | A lookup or a fixed amount of work regardless of size. |
|
|
135
|
+
| `logarithmic` | O(log n) | Binary search, balanced-tree operations. |
|
|
136
|
+
| `linear` | O(n) | One pass over the input. |
|
|
137
|
+
| `linearithmic` | O(n log n) | Sorting, or a pass that does a log-cost operation per item. |
|
|
138
|
+
| `quadratic` | O(n²) | A pass inside a pass (nested loops, repeated scans, repeated array copies). |
|
|
139
|
+
| `cubic-or-worse` | O(n³) or worse | Anything faster-growing than quadratic. |
|
|
140
|
+
|
|
141
|
+
The reason is one or two sentences that **name the mechanism**, so a reader can check it against the
|
|
142
|
+
code: "walks every newly arrived item once to freeze it" is a reason; "it is linear" is not. After the
|
|
143
|
+
run, the report compares the measured class with this one. **Agreement** is the goal; a **difference**
|
|
144
|
+
means the documentation or the code is wrong -- both are worth fixing. Neighbouring classes
|
|
145
|
+
(constant/logarithmic, linear/linearithmic) are reported as "close" because a bounded ladder cannot
|
|
146
|
+
reliably separate them.
|
|
147
|
+
|
|
148
|
+
### `variables` -- everything that could change the cost
|
|
149
|
+
|
|
150
|
+
One row per variable from the checklist above, each marked:
|
|
151
|
+
|
|
152
|
+
- `swept` -- it is the size axis.
|
|
153
|
+
- `variant` -- measured separately; describe each in `variants`.
|
|
154
|
+
- `fixed` -- held constant on purpose; give the `value`, so a reader knows what was and was not
|
|
155
|
+
covered.
|
|
156
|
+
|
|
157
|
+
### `covers` and `notBenchmarked` -- every input of every public function
|
|
158
|
+
|
|
159
|
+
A public function with more than one input (several parameters, or an object with several keys) has
|
|
160
|
+
several things that can change its cost, and the `BenchmarkCoverage` check fails until each one is
|
|
161
|
+
accounted for. In the benchmark that exercises the function, list what you did with every parameter, and
|
|
162
|
+
with every key of an object parameter:
|
|
163
|
+
|
|
164
|
+
```js
|
|
165
|
+
covers: {
|
|
166
|
+
validateEnv: {
|
|
167
|
+
"options.schema": "swept",
|
|
168
|
+
"options.values": "variant:invalid-values",
|
|
169
|
+
"options.cache": "fixed: reset before every sample, so each run measures a cold validation",
|
|
170
|
+
"options.strict": "not covered: strict mode only changes the error message, not the work done",
|
|
171
|
+
},
|
|
172
|
+
},
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
Each is one of `swept` (the suite's unit varies it), `sweep:<variable>` (one of the function's `sweeps`),
|
|
176
|
+
`variant:<name>` (one of its `variants`), `fixed: <why>` (held at one value, and why) or `not covered: <why>`
|
|
177
|
+
(left out, and why). A function that has no benchmark at all is listed in the suite's `notBenchmarked` with
|
|
178
|
+
the reason, for example because a benchmarked function that calls it already accounts for its cost.
|
|
179
|
+
|
|
180
|
+
### `sweeps` -- a second variable that scales
|
|
181
|
+
|
|
182
|
+
Some functions grow with more than one input: the number of records _and_ how many fields each record
|
|
183
|
+
has. One big-O for both would be wrong, because the two do not have to grow at the same rate. List each
|
|
184
|
+
other variable that scales under `sweeps` and the suite measures it on its own, with every other variable
|
|
185
|
+
held at its `baseValue` (the suite's own unit is held at `primaryValue`, by default the typical size):
|
|
186
|
+
|
|
187
|
+
```js
|
|
188
|
+
sweeps: [
|
|
189
|
+
{
|
|
190
|
+
variable: "width",
|
|
191
|
+
description: "How many fields each record has.",
|
|
192
|
+
baseValue: 4,
|
|
193
|
+
expectedComplexity: "quadratic",
|
|
194
|
+
complexityReason: "Every field is compared with every other field of the record.",
|
|
195
|
+
},
|
|
196
|
+
],
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
The variable must also be listed under `variables` as `swept`. `setup` (or `input`) receives a third
|
|
200
|
+
argument, `{ variable, values }`: the variable being swept and the value of every variable at this
|
|
201
|
+
measurement, so one function builds its input for any of them. In a sweep of another variable, `n` is that variable's value, not the size of the unit: always read every variable from `context.values`, which is right for the primary sweep too. Each sweep is its own group
|
|
202
|
+
(`fn:<id>#<variable>`), with its own growth class, its own documented big-O, and its own row in the
|
|
203
|
+
report, the history and the pull-request summary.
|
|
204
|
+
|
|
205
|
+
### `variants` -- measured alternatives
|
|
206
|
+
|
|
207
|
+
When a variable changes the algorithm (cache hit vs miss, flat vs nested, success vs error), measure
|
|
208
|
+
each as a named variant across the whole ladder. Each variant is its own row in the report and its own
|
|
209
|
+
chart in history, so a regression in one path cannot hide behind another.
|
|
210
|
+
|
|
211
|
+
### `notCovered` -- the honest gaps
|
|
212
|
+
|
|
213
|
+
Name what you chose not to measure and why ("concurrent callers: the function is synchronous and
|
|
214
|
+
keeps no shared state"). A reader should never wonder whether something was forgotten.
|
|
215
|
+
|
|
216
|
+
### `inEndToEnd` -- where it fits in the whole
|
|
217
|
+
|
|
218
|
+
State how often one end-to-end operation calls this function (`callsPerOperation`, a number or a
|
|
219
|
+
function of `n`) and describe the call. This is what lets section 3 of the report attribute the
|
|
220
|
+
time of one operation to the functions that cause it. A function measured alone includes everything
|
|
221
|
+
it calls: when one attributed function calls another, list the callee's id in `includes` so its time is
|
|
222
|
+
subtracted from the caller and counted once.
|
|
223
|
+
|
|
224
|
+
### The end-to-end block
|
|
225
|
+
|
|
226
|
+
`endToEnd.purpose` says what the comparison shows and why it matters to someone adopting the package.
|
|
227
|
+
`baseline.description` and `withPackage.description` state exactly what each side does, and must make
|
|
228
|
+
clear that both use empty or minimal functions. List the end-to-end `variables` the same way as for a
|
|
229
|
+
function.
|
|
230
|
+
|
|
231
|
+
## Reviewing a benchmark change
|
|
232
|
+
|
|
233
|
+
Before you open a pull request:
|
|
234
|
+
|
|
235
|
+
- [ ] `npm run benchmark:check` passes (the documentation is complete and not a placeholder).
|
|
236
|
+
- [ ] A **full** `npm run benchmark` ran on a quiet machine and no function reports **differs** that
|
|
237
|
+
you cannot explain.
|
|
238
|
+
- [ ] Every new public function has a benchmark, a `why`, a `poorPerformanceMeans`, a big-O with a
|
|
239
|
+
mechanism, and a variables table.
|
|
240
|
+
- [ ] Every variable on the checklist is `swept`, a `variant`, `fixed` with a value, or in
|
|
241
|
+
`notCovered` with a reason.
|
|
242
|
+
- [ ] The end-to-end baseline still uses empty or minimal functions.
|
|
243
|
+
- [ ] You did not commit a `--quick` result.
|
|
244
|
+
|
|
245
|
+
## Commands
|
|
246
|
+
|
|
247
|
+
```sh
|
|
248
|
+
npm run benchmark # full ladder -> results.json and BENCHMARKS.md in each benchmarks/performance-*/ folder
|
|
249
|
+
npm run benchmark -- --quick # smoke test; never commit its output
|
|
250
|
+
npm run benchmark -- --only package-name,end-to-end # run a subset while developing
|
|
251
|
+
npm run benchmark:check # validate documentation and shape; measures nothing
|
|
252
|
+
```
|