@maverickcer/env-cap 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ADOPTION.md +144 -0
- package/AGENTS.md +146 -0
- package/CHANGELOG.md +438 -0
- package/CODE_OF_CONDUCT.md +124 -0
- package/CODE_REVIEW.md +200 -0
- package/CONTRIBUTING.md +131 -0
- package/GUIDE.md +1214 -0
- package/PROMPT.md +89 -0
- package/README.md +92 -200
- package/RELEASING.md +160 -0
- package/SECURITY.md +270 -0
- package/SUPPORT.md +76 -0
- package/VERSIONING.md +118 -0
- package/benchmarks/READING-BENCHMARKS.md +197 -0
- package/benchmarks/README.md +126 -0
- package/benchmarks/WRITING-BENCHMARKS.md +252 -0
- package/benchmarks/performance-buildtime/BENCHMARKS.md +889 -0
- package/benchmarks/performance-runtime/BENCHMARKS.md +1988 -0
- package/dist/.dts/_ipc/build-fs.d.ts +53 -0
- package/dist/.dts/_ipc/build-fs.d.ts.map +1 -0
- package/dist/.dts/_ipc/compiler-api.d.ts +17 -0
- package/dist/.dts/_ipc/compiler-api.d.ts.map +1 -0
- package/dist/.dts/_ipc/display-path.d.ts +27 -0
- package/dist/.dts/_ipc/display-path.d.ts.map +1 -0
- package/dist/.dts/_ipc/generated-banner.d.ts +58 -0
- package/dist/.dts/_ipc/generated-banner.d.ts.map +1 -0
- package/dist/.dts/_ipc/glob.d.ts +22 -0
- package/dist/.dts/_ipc/glob.d.ts.map +1 -0
- package/dist/.dts/_ipc/no-node-fs.d.ts +34 -0
- package/dist/.dts/_ipc/no-node-fs.d.ts.map +1 -0
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts +134 -0
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts.map +1 -0
- package/dist/.dts/_ipc/typescript.d.ts +20 -0
- package/dist/.dts/_ipc/typescript.d.ts.map +1 -0
- package/dist/.dts/build/compatibility.d.ts.map +1 -1
- package/dist/.dts/build/compiler-api.d.ts +3 -6
- package/dist/.dts/build/compiler-api.d.ts.map +1 -1
- package/dist/.dts/build/display-path.d.ts +3 -21
- package/dist/.dts/build/display-path.d.ts.map +1 -1
- package/dist/.dts/build/generated-banner.d.ts +21 -42
- package/dist/.dts/build/generated-banner.d.ts.map +1 -1
- package/dist/.dts/build/glob.d.ts +3 -15
- package/dist/.dts/build/glob.d.ts.map +1 -1
- package/dist/.dts/build/link.d.ts.map +1 -1
- package/dist/.dts/build/merge-documentation.d.ts +35 -0
- package/dist/.dts/build/merge-documentation.d.ts.map +1 -0
- package/dist/.dts/build/resolution/resolve-tsconfig-paths.d.ts +11 -112
- package/dist/.dts/build/resolution/resolve-tsconfig-paths.d.ts.map +1 -1
- package/dist/.dts/build/types.d.ts +4 -53
- package/dist/.dts/build/types.d.ts.map +1 -1
- package/dist/.dts/build/typescript.d.ts +3 -17
- package/dist/.dts/build/typescript.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/glob.d.ts +4 -15
- package/dist/.dts/eslint-plugin/glob.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/index.d.ts +1 -1
- package/dist/.dts/eslint-plugin/no-node-fs.d.ts +8 -18
- package/dist/.dts/eslint-plugin/no-node-fs.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/no-raw-process-env.d.ts.map +1 -1
- package/dist/{build-H4KFV44Z.js → build-WKDIAWF7.js} +198 -60
- package/dist/build-WKDIAWF7.js.map +1 -0
- package/dist/build.cjs +194 -56
- package/dist/build.cjs.map +1 -1
- package/dist/build.js +194 -56
- package/dist/build.js.map +1 -1
- package/dist/{chunk-7GVPOHDU.js → chunk-KEGYUHBU.js} +2 -2
- package/dist/cli/index.js +36 -29
- package/dist/cli/index.js.map +1 -1
- package/dist/eslint-plugin/index.cjs +84 -54
- package/dist/eslint-plugin/index.cjs.map +1 -1
- package/dist/eslint-plugin/index.js +83 -53
- package/dist/eslint-plugin/index.js.map +1 -1
- package/examples/README.md +95 -0
- package/package.json +48 -14
- package/skills/env-cap/SKILL.md +91 -0
- package/skills/env-cap/references/api.md +220 -0
- package/specs/architecture.md +634 -0
- package/specs/decisions/0001-runtime-documentation-separation.md +197 -0
- package/specs/decisions/0002-static-analysis-never-execution.md +233 -0
- package/specs/decisions/0003-no-global-env-object.md +172 -0
- package/specs/decisions/0004-no-client-server-package-split.md +116 -0
- package/specs/decisions/0005-warn-not-throw-by-default.md +93 -0
- package/specs/decisions/0006-self-redacting-contracts.md +124 -0
- package/specs/decisions/0007-processor-not-transformer.md +109 -0
- package/specs/decisions/0008-gzip-size-budget.md +173 -0
- package/specs/decisions/0009-exclusive-groups-are-always-errors.md +139 -0
- package/specs/decisions/0010-dependency-ownership-engine-scope-boundary.md +100 -0
- package/specs/decisions/0011-shared-discovery-compute-atomic-write-non-atomic.md +104 -0
- package/specs/decisions/0012-live-expiration-overrides-not-ast-functions.md +145 -0
- package/specs/decisions/0013-json-output-is-a-versioned-mirror.md +136 -0
- package/specs/decisions/0014-cross-package-schema-discovery.md +206 -0
- package/specs/decisions/0015-security-backport-window.md +60 -0
- package/specs/decisions/0016-check-mode-compute-before-compare-never-partial-write.md +113 -0
- package/specs/decisions/0017-eslint-plugin-entry-point.md +113 -0
- package/specs/decisions/0018-rotation-alert-issue-on-non-pr-runs.md +104 -0
- package/specs/decisions/0019-published-json-schema-generated-from-types.md +100 -0
- package/specs/decisions/0020-declaration-maps-via-separate-tsc-pass.md +101 -0
- package/specs/decisions/0021-manifest-change-report-persisted-snapshot.md +177 -0
- package/specs/decisions/0022-validation-contexts.md +267 -0
- package/specs/decisions/0023-tsconfig-path-alias-resolution.md +182 -0
- package/specs/decisions/0024-fact-model-architecture.md +144 -0
- package/specs/decisions/0025-contract-model-json-projection.md +105 -0
- package/specs/decisions/0026-finding-model-unifies-four-families.md +109 -0
- package/specs/decisions/0027-dependency-model-fact-shape-not-engine-access.md +98 -0
- package/specs/decisions/0028-ownership-model-shared-effective-owner.md +85 -0
- package/specs/decisions/0029-lifecycle-model-deprecation-rename-fields.md +98 -0
- package/specs/decisions/0030-change-model-wraps-manifest-change-report.md +97 -0
- package/specs/decisions/0031-evidence-entry-point.md +85 -0
- package/specs/decisions/0032-evidence-projection-provenance-mechanism.md +120 -0
- package/specs/decisions/0033-reference-projections-live-in-examples.md +81 -0
- package/specs/decisions/0034-three-tier-examples-restructuring.md +85 -0
- package/specs/decisions/0035-documentenv-metadata-generalization-and-compliance-fields.md +85 -0
- package/specs/decisions/0036-exact-source-position-evidence.md +122 -0
- package/specs/decisions/0037-developer-declared-dynamic-access-citations.md +114 -0
- package/specs/decisions/0038-always-computed-evidence-model-backs-generate-check-docs.md +244 -0
- package/specs/decisions/0039-scanner-local-dataflow-boundary.md +160 -0
- package/specs/decisions/0040-library-surfaces-do-not-acquire-node-fs.md +156 -0
- package/specs/decisions/0041-dual-package-hazard-checked-documented.md +75 -0
- package/specs/decisions/0042-init-cli-subcommand-scaffolds-only.md +95 -0
- package/specs/decisions/0043-env-cap-resolver-relocated-and-duplicated.md +71 -0
- package/specs/decisions/0044-strict-escalates-every-warning-family.md +88 -0
- package/specs/decisions/0045-promote-experimental-surfaces-to-stable.md +103 -0
- package/specs/decisions/0046-cli-restricted-to-runtime-and-evidence-output.md +177 -0
- package/specs/decisions/0047-supported-toolchain-node-22-and-typescript-5-6.md +30 -0
- package/specs/decisions/0048-eslint-plugin-does-not-vendor-typescript-eslint.md +25 -0
- package/specs/decisions/0049-typescript-7-scanner-uses-a-bundled-typescript-6.md +87 -0
- package/specs/decisions/0050-a-variable-declared-twice-is-documented-once.md +55 -0
- package/specs/generated-artifacts.md +104 -0
- package/specs/migrations/README.md +30 -0
- package/specs/migrations/from-centralized-schema.md +503 -0
- package/specs/migrations/from-dotenv.md +352 -0
- package/specs/migrations/from-envalid.md +482 -0
- package/specs/migrations/from-process-env.md +398 -0
- package/specs/migrations/from-t3-env.md +416 -0
- package/specs/migrations/from-zod.md +303 -0
- package/dist/build-H4KFV44Z.js.map +0 -1
- /package/dist/{chunk-7GVPOHDU.js.map → chunk-KEGYUHBU.js.map} +0 -0
package/SECURITY.md
ADDED
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
This document defines the security boundaries and threat model for
|
|
4
|
+
`env-cap`: what protections the package intentionally provides, what
|
|
5
|
+
responsibilities remain with the application, and why those boundaries exist.
|
|
6
|
+
|
|
7
|
+
For implementation details such as contract redaction, error handling,
|
|
8
|
+
`.env.example` generation, and documentation handling, see the [Security
|
|
9
|
+
section of the Guide](GUIDE.md#security-model). This document describes the policy
|
|
10
|
+
those mechanisms enforce rather than duplicating their implementation details.
|
|
11
|
+
|
|
12
|
+
## Reporting a vulnerability
|
|
13
|
+
|
|
14
|
+
Report suspected security vulnerabilities privately through a
|
|
15
|
+
[GitHub security advisory](https://github.com/maverickcer/env-cap/security/advisories/new).
|
|
16
|
+
|
|
17
|
+
Alternatively, contact the maintainer through the contact information
|
|
18
|
+
provided on the npm package page.
|
|
19
|
+
|
|
20
|
+
Please do not create a public issue for a suspected vulnerability before it
|
|
21
|
+
has been reviewed and triaged.
|
|
22
|
+
|
|
23
|
+
## Disclosure
|
|
24
|
+
|
|
25
|
+
Reports are handled in private until they can be disclosed without helping an attacker. A report is
|
|
26
|
+
acknowledged, assessed and prioritised by severity, exploitability and impact; acknowledgement does not
|
|
27
|
+
promise that it is accepted as a vulnerability or that a fix can be provided.
|
|
28
|
+
|
|
29
|
+
When a fix is available, a security advisory is published on this repository naming the affected versions,
|
|
30
|
+
the patched version, the severity, what a user can do if they cannot upgrade at once, and whether
|
|
31
|
+
exploitation is known to have occurred. It does not publish a working exploit or details that expose other
|
|
32
|
+
systems before their owners have had time to update. If a vulnerability is being exploited, or a fix cannot
|
|
33
|
+
be released promptly, users are warned and given mitigations before the fix is complete. Resolved
|
|
34
|
+
advisories stay public: a record of vulnerabilities handled openly is part of what this package asks you to
|
|
35
|
+
trust.
|
|
36
|
+
|
|
37
|
+
## Runtime security model
|
|
38
|
+
|
|
39
|
+
The runtime package (`createEnv`, `validateEnv`, and imports from the package
|
|
40
|
+
root) is intentionally limited to environment processing and validation. It
|
|
41
|
+
does not attempt to become a secret-management platform.
|
|
42
|
+
|
|
43
|
+
The runtime does not:
|
|
44
|
+
|
|
45
|
+
- **Persist secrets.**
|
|
46
|
+
Resolved environment values exist only in memory. After successful
|
|
47
|
+
validation, values are frozen and retained for the lifetime of the process.
|
|
48
|
+
The runtime never writes secrets to disk, databases, external services, or
|
|
49
|
+
persistent storage.
|
|
50
|
+
|
|
51
|
+
- **Retrieve secrets.**
|
|
52
|
+
The package has no integrations with secret-management systems such as
|
|
53
|
+
HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Secret
|
|
54
|
+
Manager. Applications are responsible for obtaining secrets through their
|
|
55
|
+
existing infrastructure and passing resolved values into:
|
|
56
|
+
|
|
57
|
+
```ts
|
|
58
|
+
validateEnv({ values })
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
* **Manage secret lifecycle operations.**
|
|
62
|
+
There is no rotation engine, scheduler, command execution, or automated
|
|
63
|
+
refresh process. Fields such as `expiresAt` and refresh instructions from
|
|
64
|
+
`documentEnv()` are documentation metadata only. They are consumed by
|
|
65
|
+
generated artifacts and never executed by the runtime.
|
|
66
|
+
|
|
67
|
+
* **Send telemetry.**
|
|
68
|
+
No package entry point (`.`, `./build`, or `./helpers`) performs network
|
|
69
|
+
requests or sends usage information.
|
|
70
|
+
|
|
71
|
+
* **Log secret values.**
|
|
72
|
+
The library does not write logs or emit environment values. Validation
|
|
73
|
+
errors contain contract information, variable names, and messages produced
|
|
74
|
+
by user-provided processors or validators, but the library itself never
|
|
75
|
+
inserts raw or processed secret values into errors.
|
|
76
|
+
|
|
77
|
+
* **Embed documentation metadata into runtime state.**
|
|
78
|
+
`documentEnv()` is intentionally inert during execution. Documentation
|
|
79
|
+
fields are only consumed by build-time static analysis and are not retained
|
|
80
|
+
by the runtime validation system.
|
|
81
|
+
|
|
82
|
+
Runtime contracts intentionally contain only:
|
|
83
|
+
|
|
84
|
+
- variable definitions required for processing (`default`, `processor`,
|
|
85
|
+
`validator`)
|
|
86
|
+
- resolved, frozen values accessible through explicit property access
|
|
87
|
+
- minimal identity information required for runtime errors
|
|
88
|
+
|
|
89
|
+
Information that exists only for humans, operators, or generated artifacts
|
|
90
|
+
belongs in `documentEnv()`. Responsibilities such as secret storage, rotation,
|
|
91
|
+
telemetry, and access auditing belong to infrastructure outside this package.
|
|
92
|
+
|
|
93
|
+
## Dual-package hazard
|
|
94
|
+
|
|
95
|
+
The runtime keeps identity-sensitive state in module scope: the private
|
|
96
|
+
`WeakMap` that associates a contract object with its schema
|
|
97
|
+
(`src/runtime/registry.ts`), and the validation cache
|
|
98
|
+
(`src/runtime/cache.ts`). If a consumer's dependency tree resolves
|
|
99
|
+
`env-cap` through two different specifiers — one importer
|
|
100
|
+
getting the ESM build, one `require()`r getting the CJS build, via a mixed
|
|
101
|
+
ESM/CJS dependency graph or a re-exporting intermediate package — Node
|
|
102
|
+
loads two entirely separate module instances, each with its own registry
|
|
103
|
+
and cache. No npm package that ships both ESM and CJS builds can prevent
|
|
104
|
+
this; it is a property of how Node's two module systems resolve
|
|
105
|
+
independently.
|
|
106
|
+
|
|
107
|
+
`env-cap` treats this as a checked, documented risk class rather than an
|
|
108
|
+
unstated unknown
|
|
109
|
+
([ADR 0041](specs/decisions/0041-dual-package-hazard-checked-documented.md));
|
|
110
|
+
`test/runtime/dual-package-hazard.test.ts` pins the exact consequence in
|
|
111
|
+
CI. That consequence is **fail-fast, not silent corruption**: a contract
|
|
112
|
+
created by `createEnv()` in one instance and passed to `validateEnv()`,
|
|
113
|
+
`resetEnvCache()`, or `isEnvContract()` resolved from the other instance is
|
|
114
|
+
not recognized — `validateEnv()`/`resetEnvCache()` throw a `TypeError`
|
|
115
|
+
("this value was not created by createEnv()") and `isEnvContract()` returns
|
|
116
|
+
`false`, immediately and visibly, rather than validating against the wrong
|
|
117
|
+
schema or reading a stale cache. Keep each capability's `createEnv()` call
|
|
118
|
+
and the `validateEnv()` call that consumes its manifest resolving through
|
|
119
|
+
the same module specifier (the normal case for an application that imports
|
|
120
|
+
`env-cap` one way throughout).
|
|
121
|
+
|
|
122
|
+
## Build-time security model
|
|
123
|
+
|
|
124
|
+
`generateEnvManifest()` (`./build`) is a static-analysis tool, not an
|
|
125
|
+
execution engine.
|
|
126
|
+
|
|
127
|
+
The build system:
|
|
128
|
+
|
|
129
|
+
- **Never imports or executes schema files.**
|
|
130
|
+
Schema files are analyzed as TypeScript ASTs. Only safe literal
|
|
131
|
+
expressions are evaluated, such as strings, numbers, booleans, arrays, and
|
|
132
|
+
object literals.
|
|
133
|
+
|
|
134
|
+
Dynamic expressions such as:
|
|
135
|
+
|
|
136
|
+
- function calls
|
|
137
|
+
- variables
|
|
138
|
+
- computed values
|
|
139
|
+
- interpolated template strings
|
|
140
|
+
|
|
141
|
+
are treated as unresolved information and reported rather than executed or
|
|
142
|
+
guessed.
|
|
143
|
+
|
|
144
|
+
- **Never writes outside the configured project root.**
|
|
145
|
+
All generated output paths (`location`, `docs.location`,
|
|
146
|
+
`envExample.location`) are resolved and verified against `root` before any
|
|
147
|
+
file is written.
|
|
148
|
+
|
|
149
|
+
Paths that escape the configured root through absolute paths or traversal
|
|
150
|
+
segments fail before any file is written. A direct `generateEnvManifest()`/
|
|
151
|
+
`generateDocumentation()`/`generateUsageReport()` call throws that pass's
|
|
152
|
+
own error type (`EnvManifestGenerationError`/`EnvDocumentationGenerationError`/
|
|
153
|
+
`EnvUsageAnalysisError`); through `generateEnvArtifacts()` or the CLI (the
|
|
154
|
+
primary documented workflow), the same escape is aggregated into
|
|
155
|
+
`EnvProjectGenerationError` instead.
|
|
156
|
+
|
|
157
|
+
- **Never overwrites existing hand-managed `.env.example` files.**
|
|
158
|
+
If a generated example would conflict with an existing file, the generator
|
|
159
|
+
preserves the original and writes generated output separately.
|
|
160
|
+
|
|
161
|
+
- **Avoids unnecessary traversal of dependency directories.**
|
|
162
|
+
The general file-discovery walk that powers `include`/`exclude` glob
|
|
163
|
+
matching excludes directories such as `node_modules` and `.git` during
|
|
164
|
+
traversal rather than scanning and filtering afterward -- this walk never
|
|
165
|
+
descends into `node_modules` at all, regardless of `include`/`exclude`.
|
|
166
|
+
|
|
167
|
+
A separate, narrow mechanism exists for reading a schema
|
|
168
|
+
that lives only inside an installed dependency: the `packages` option
|
|
169
|
+
(see [ADR 0014](specs/decisions/0014-cross-package-schema-discovery.md)).
|
|
170
|
+
It is opt-in -- no package is ever considered unless its exact name
|
|
171
|
+
appears in a `packages` array the consuming project supplies -- and
|
|
172
|
+
bounded by construction, not by a traversal exclusion: it reads exactly
|
|
173
|
+
one `package.json` per named package to find a declared entry point, then
|
|
174
|
+
reads exactly one declared file, verified to be a real file (not a
|
|
175
|
+
symlink escaping the package directory) under a fixed size cap, before it
|
|
176
|
+
is parsed. It never calls `readdir` on any directory, named package or
|
|
177
|
+
not, and a package's own transitive dependencies are never considered.
|
|
178
|
+
|
|
179
|
+
A second, separate mechanism resolves TypeScript path
|
|
180
|
+
aliases (`compilerOptions.paths`/`baseUrl` in `tsconfig.json`) during
|
|
181
|
+
static analysis, on by default (see
|
|
182
|
+
[ADR 0023](specs/decisions/0023-tsconfig-path-alias-resolution.md)).
|
|
183
|
+
Unlike `packages`, it never crosses into `node_modules`: any resolution
|
|
184
|
+
landing on a path containing a `node_modules` segment is discarded, so a
|
|
185
|
+
bare specifier can only ever resolve into `node_modules` through the
|
|
186
|
+
`packages` mechanism above, never through alias resolution. It only ever
|
|
187
|
+
resolves to a real `.ts`/`.tsx` file already inside the scanned project.
|
|
188
|
+
|
|
189
|
+
## Application responsibilities
|
|
190
|
+
|
|
191
|
+
`env-cap` intentionally does not attempt to solve problems outside the
|
|
192
|
+
environment-contract lifecycle.
|
|
193
|
+
|
|
194
|
+
Applications remain responsible for:
|
|
195
|
+
|
|
196
|
+
- **Protecting documentation metadata.**
|
|
197
|
+
Values passed to `documentEnv()` are ordinary source-code data. If an
|
|
198
|
+
`env.schema.ts` file is imported into a browser bundle, documentation fields
|
|
199
|
+
inside that file may be included in the client output.
|
|
200
|
+
|
|
201
|
+
This means:
|
|
202
|
+
|
|
203
|
+
- descriptions are public
|
|
204
|
+
- runbook links are public
|
|
205
|
+
- operational notes are public
|
|
206
|
+
- lifecycle instructions are public
|
|
207
|
+
|
|
208
|
+
Documentation metadata must never contain credentials, private keys, or
|
|
209
|
+
information that should not appear in a client bundle.
|
|
210
|
+
|
|
211
|
+
- **Writing safe processors and validators.**
|
|
212
|
+
Custom validation logic controls its own errors.
|
|
213
|
+
|
|
214
|
+
For example:
|
|
215
|
+
|
|
216
|
+
```ts
|
|
217
|
+
throw new Error(`Invalid value: ${value}`)
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
may expose a secret if the thrown message is later logged. The package
|
|
221
|
+
protects values passing through its own runtime boundaries but cannot
|
|
222
|
+
sanitize arbitrary application-generated error messages.
|
|
223
|
+
|
|
224
|
+
- **Managing secret storage and rotation.**
|
|
225
|
+
Obtaining, storing, rotating, and revoking secrets are infrastructure
|
|
226
|
+
responsibilities. `env-cap` validates configuration after it has been
|
|
227
|
+
supplied; it does not replace a secret-management system.
|
|
228
|
+
|
|
229
|
+
## Supply-chain posture
|
|
230
|
+
|
|
231
|
+
For a reviewer checking this project's publish/build pipeline rather than
|
|
232
|
+
its runtime API: releases are published via npm's OIDC trusted publishing
|
|
233
|
+
(`.github/workflows/release.yml`) — there is no long-lived `NPM_TOKEN`
|
|
234
|
+
secret in this repository to leak, rotate, or scope. The workflow's
|
|
235
|
+
`id-token: write` permission is exchanged for a short-lived publish
|
|
236
|
+
credential per run, tied to this exact repository and workflow file, and
|
|
237
|
+
the same OIDC flow attaches npm provenance attestations to each published
|
|
238
|
+
version. The runtime and helpers entry points are verified against real
|
|
239
|
+
Node, Bun, and Deno engines in CI (`test/cross-runtime/`), not merely
|
|
240
|
+
asserted to be isomorphic by inspection. Test coverage is enforced
|
|
241
|
+
ratchet-up-only (see `vitest.config.ts`; [`CONTRIBUTING.md`](CONTRIBUTING.md)
|
|
242
|
+
forbids lowering a threshold to accommodate a drop). See
|
|
243
|
+
[`RELEASING.md`](RELEASING.md#first-time-setup) for the one-time
|
|
244
|
+
trusted-publisher setup this depends on.
|
|
245
|
+
|
|
246
|
+
## Supported versions
|
|
247
|
+
|
|
248
|
+
This project has not reached a `1.0` release.
|
|
249
|
+
|
|
250
|
+
**Today, before `1.0`:** security fixes are provided on the latest published
|
|
251
|
+
`0.x` version only. There is no separate long-term-support branch and no
|
|
252
|
+
extended security-support policy yet -- an application pinned to an older
|
|
253
|
+
`0.x` release does not receive backported fixes; upgrading to the latest
|
|
254
|
+
`0.x` is the only supported remediation path.
|
|
255
|
+
|
|
256
|
+
**Starting at `1.0`:** once this project ships a `1.0` release, security
|
|
257
|
+
fixes will be backported to the latest minor release of the previous major
|
|
258
|
+
version for a minimum of six months after a new major version ships. This
|
|
259
|
+
window may be extended at the maintainer's discretion -- for example, to
|
|
260
|
+
give large downstream consumers more migration time -- but once a minimum
|
|
261
|
+
end date has been stated for a given major version's backport window, it
|
|
262
|
+
will never be shortened. Fixes for the current major version continue to
|
|
263
|
+
target its latest published minor/patch release, exactly as fixes do today
|
|
264
|
+
for `0.x`; the backport commitment only extends that same treatment one
|
|
265
|
+
major version back, for a bounded time.
|
|
266
|
+
|
|
267
|
+
This is a forward commitment about a future release line, not a claim that
|
|
268
|
+
backporting has already been exercised -- see
|
|
269
|
+
[`ADOPTION.md`](ADOPTION.md#versioning-stability-and-long-term-support) for
|
|
270
|
+
how to weigh that distinction today.
|
package/SUPPORT.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
<!-- Managed by internal-package-contract. Do not edit this file: change template/package/SUPPORT.md in the standard and run `internal-package-contract sync`. -->
|
|
2
|
+
|
|
3
|
+
# Support
|
|
4
|
+
|
|
5
|
+
Thank you for using `@maverickcer/env-cap`.
|
|
6
|
+
|
|
7
|
+
This document explains how to get help, report issues, and request improvements.
|
|
8
|
+
|
|
9
|
+
## Before Opening a Discussion or Issue
|
|
10
|
+
|
|
11
|
+
Please take a moment to:
|
|
12
|
+
|
|
13
|
+
- Read the project documentation and README.
|
|
14
|
+
- Search existing GitHub Discussions.
|
|
15
|
+
- Search existing GitHub Issues.
|
|
16
|
+
|
|
17
|
+
This helps reduce duplicate reports and may provide an immediate answer.
|
|
18
|
+
|
|
19
|
+
## Questions and Usage Help
|
|
20
|
+
|
|
21
|
+
If you have questions about using `@maverickcer/env-cap`, need help understanding its behavior, or would like guidance on best practices, please start a **GitHub Discussion**.
|
|
22
|
+
|
|
23
|
+
Discussions are the best place for:
|
|
24
|
+
|
|
25
|
+
- Usage questions
|
|
26
|
+
- Design discussions
|
|
27
|
+
- Architecture advice
|
|
28
|
+
- Integration guidance
|
|
29
|
+
- General feedback
|
|
30
|
+
|
|
31
|
+
## Bug Reports
|
|
32
|
+
|
|
33
|
+
If you believe you have found a bug, please open a **GitHub Issue**.
|
|
34
|
+
|
|
35
|
+
When reporting a bug, include:
|
|
36
|
+
|
|
37
|
+
- The version of `@maverickcer/env-cap`
|
|
38
|
+
- Your Node.js version
|
|
39
|
+
- Your operating system
|
|
40
|
+
- Steps to reproduce the issue
|
|
41
|
+
- Expected behavior
|
|
42
|
+
- Actual behavior
|
|
43
|
+
- Any relevant code snippets, configuration, or logs
|
|
44
|
+
|
|
45
|
+
Whenever possible, provide a minimal reproducible example.
|
|
46
|
+
|
|
47
|
+
## Feature Requests
|
|
48
|
+
|
|
49
|
+
Feature requests are welcome.
|
|
50
|
+
|
|
51
|
+
Before opening a new request, please search existing Issues and Discussions to avoid duplicates.
|
|
52
|
+
|
|
53
|
+
A strong feature request explains:
|
|
54
|
+
|
|
55
|
+
- The problem being solved
|
|
56
|
+
- Why the current behavior is insufficient
|
|
57
|
+
- The proposed API or workflow
|
|
58
|
+
- Alternative solutions that were considered
|
|
59
|
+
|
|
60
|
+
## Security Vulnerabilities
|
|
61
|
+
|
|
62
|
+
Please **do not** report security vulnerabilities through public Issues or Discussions.
|
|
63
|
+
|
|
64
|
+
Instead, follow the responsible disclosure process described in `SECURITY.md`.
|
|
65
|
+
|
|
66
|
+
## Supported Versions
|
|
67
|
+
|
|
68
|
+
Only the latest released version is actively supported.
|
|
69
|
+
|
|
70
|
+
Older releases may receive limited support or no updates.
|
|
71
|
+
|
|
72
|
+
## Response Expectations
|
|
73
|
+
|
|
74
|
+
`@maverickcer/env-cap` is maintained as an open source project.
|
|
75
|
+
|
|
76
|
+
While all Issues and Discussions are appreciated, response times cannot be guaranteed. Community participation is encouraged, and well-documented bug reports and feature proposals help improve the project for everyone.
|
package/VERSIONING.md
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# Versioning and API stability
|
|
2
|
+
|
|
3
|
+
`env-cap` follows [Semantic Versioning](https://semver.org/). This document
|
|
4
|
+
defines what that promise actually covers, since "semver" alone doesn't say
|
|
5
|
+
which surface it applies to. Three tiers exist:
|
|
6
|
+
|
|
7
|
+
## Stable
|
|
8
|
+
|
|
9
|
+
Covered by semver. A breaking change to any of the following requires a
|
|
10
|
+
major version bump (once the package reaches 1.0 -- see
|
|
11
|
+
[Pre-1.0 status](#pre-10-status) below):
|
|
12
|
+
|
|
13
|
+
- **Public runtime APIs**: `createEnv`, `validateEnv`, `documentEnv`,
|
|
14
|
+
`resetEnvCache`, `isEnvContract`, and the exported error classes
|
|
15
|
+
(`EnvValidationError`, `EnvNotReadyError`) from the package root.
|
|
16
|
+
- **Documented build APIs**: `generateEnvManifest`, `generateDocumentation`,
|
|
17
|
+
`generateUsageReport`, `generateEnvArtifacts` and their documented
|
|
18
|
+
options, from `@maverickcer/env-cap/build`.
|
|
19
|
+
- **`@maverickcer/env-cap/node`** (`nodeBuildFileSystem`): the only sanctioned way to give
|
|
20
|
+
`./build` a filesystem, and required by every generator script `init` scaffolds.
|
|
21
|
+
- **CLI flags**: every flag listed in `env-cap --help`, and the CLI's exit codes.
|
|
22
|
+
- **`--json` output**: versioned independently via its own `schemaVersion`
|
|
23
|
+
field -- see [ADR 0013](specs/decisions/0013-json-output-is-a-versioned-mirror.md).
|
|
24
|
+
A purely additive field never requires a `schemaVersion` bump; a changed
|
|
25
|
+
or removed field does.
|
|
26
|
+
- **The published JSON Schema** (`schemas/env-cap-report.schema.json`, the
|
|
27
|
+
`@maverickcer/env-cap/schema` export), kept in lockstep with `--json`'s
|
|
28
|
+
documented shape and generated directly from it -- see
|
|
29
|
+
[ADR 0019](specs/decisions/0019-published-json-schema-generated-from-types.md).
|
|
30
|
+
Follows the same additive-only rule as `--json` itself.
|
|
31
|
+
- **The generated manifest schema**: the shape of the file
|
|
32
|
+
`generateEnvManifest()`/`generateEnvArtifacts()` writes.
|
|
33
|
+
- **Documented configuration conventions**: for example, the `envCap.schema`
|
|
34
|
+
package.json field (see [ADR 0014](specs/decisions/0014-cross-package-schema-discovery.md)).
|
|
35
|
+
- **The ESLint plugin**: `@maverickcer/env-cap/eslint-plugin`'s exported rule
|
|
36
|
+
name(s), each rule's `RuleOptions` shape, and the default `env.schema.ts`/
|
|
37
|
+
`.tsx` allowlist -- see [ADR 0017](specs/decisions/0017-eslint-plugin-entry-point.md).
|
|
38
|
+
- **The `packages` option** (cross-package schema discovery, ADR 0014) and
|
|
39
|
+
**`tsconfig` option** (TypeScript path-alias resolution, ADR 0023) on every
|
|
40
|
+
generator.
|
|
41
|
+
- **`@maverickcer/env-cap/build`'s lower-level discovery/linking primitives** --
|
|
42
|
+
everything `./build` exports beyond the four generator orchestrators,
|
|
43
|
+
e.g. `discoverSchemaFiles`, `linkFiles`, `parseSchemaFile`, `renderManifest`,
|
|
44
|
+
`renderDocs`, `detectCompatibilityIssues`, `detectExclusiveGroupIssues`,
|
|
45
|
+
`generateEvidenceModel`, `getEvidenceModel`/`computeSourceFingerprint`,
|
|
46
|
+
`renderUsageReport`, and their accompanying types.
|
|
47
|
+
- **`@maverickcer/env-cap/evidence`** (`defineEvidenceProjection`, and the `EvidenceModel`
|
|
48
|
+
shape it projects over) -- see [ADR 0031](specs/decisions/0031-evidence-entry-point.md)
|
|
49
|
+
and [ADR 0032](specs/decisions/0032-evidence-projection-provenance-mechanism.md).
|
|
50
|
+
- **The persisted evidence artifact** (`docs/env.evidence.json` or wherever a
|
|
51
|
+
project's `evidence.location`/`--evidence <path>` points) -- a direct,
|
|
52
|
+
literal serialization of `EvidenceModel`.
|
|
53
|
+
- **The `init` CLI subcommand**'s scaffolded file set and template contents
|
|
54
|
+
-- see [ADR 0042](specs/decisions/0042-init-cli-subcommand-scaffolds-only.md).
|
|
55
|
+
- **The composite GitHub Action** (`action.yml`): its input and output names and meanings, and
|
|
56
|
+
the PR comment's sections. The major tag it is used through (`uses: MaverickCER/env-cap@v0`
|
|
57
|
+
while the package is 0.x, `@v1` from 1.0.0) moves only within a major version.
|
|
58
|
+
- **Supported toolchain**: Node.js `>=22`, `moduleResolution` `node16`/`nodenext`/`bundler` (not the legacy
|
|
59
|
+
`node10`, which cannot resolve subpath exports), and the TypeScript versions named in the
|
|
60
|
+
`typescript` peer range (`^5 || ^6 || ^7`). The build-time scanner needs TypeScript's classic
|
|
61
|
+
compiler API, which TypeScript 7 does not ship, so it uses the consumer's `typescript` when that has
|
|
62
|
+
the API and otherwise the bundled `@typescript/typescript6` dependency -- no configuration, and `tsc`
|
|
63
|
+
stays TypeScript 7 ([ADR 0049](specs/decisions/0049-typescript-7-scanner-uses-a-bundled-typescript-6.md)).
|
|
64
|
+
Under TypeScript 7 the `./build` and `./evidence` declarations need `skipLibCheck`; the runtime,
|
|
65
|
+
helpers and node entry points type-check strictly.
|
|
66
|
+
|
|
67
|
+
All promoted from Experimental per [ADR 0045](specs/decisions/0045-promote-experimental-surfaces-to-stable.md)
|
|
68
|
+
-- see that ADR for why each one had already cleared this tier's own
|
|
69
|
+
graduation criterion (below).
|
|
70
|
+
|
|
71
|
+
## Experimental
|
|
72
|
+
|
|
73
|
+
Explicitly labeled as such, in the README, the relevant option's own JSDoc,
|
|
74
|
+
and the ADR that introduces it. An Experimental surface may change shape --
|
|
75
|
+
including in a breaking way -- in a minor or patch release, without that
|
|
76
|
+
being a semver violation. This is not a loophole for casual churn: a feature
|
|
77
|
+
ships Experimental because it is genuinely new enough that real-world
|
|
78
|
+
feedback is likely to reveal a better shape, not because we're unwilling to
|
|
79
|
+
commit to _something_. Once a feature has been through at least one real
|
|
80
|
+
feedback cycle without a need to break it, its ADR's Status moves from
|
|
81
|
+
Proposed/Experimental to Accepted, and it becomes Stable.
|
|
82
|
+
|
|
83
|
+
Nothing currently ships Experimental -- see [ADR 0045](specs/decisions/0045-promote-experimental-surfaces-to-stable.md)
|
|
84
|
+
for the most recent round of promotions. A future feature genuinely new
|
|
85
|
+
enough to warrant it will be added here, explicitly, with its own ADR.
|
|
86
|
+
|
|
87
|
+
## Private
|
|
88
|
+
|
|
89
|
+
Never covered by semver, may change at any time without notice:
|
|
90
|
+
|
|
91
|
+
- Internal modules and functions not re-exported from `.`, `./build`, or
|
|
92
|
+
`./helpers`.
|
|
93
|
+
- Parsing/AST-traversal implementation details (e.g. exactly how
|
|
94
|
+
`src/build/parse.ts` walks a source file).
|
|
95
|
+
- The dependency-ownership engine's internals (`scan-dependencies.ts`,
|
|
96
|
+
`dependency-graph.ts`) -- see [ADR 0010](specs/decisions/0010-dependency-ownership-engine-scope-boundary.md).
|
|
97
|
+
- Any behavior not documented in the README, a JSDoc comment on a public
|
|
98
|
+
export, or an ADR.
|
|
99
|
+
|
|
100
|
+
## Pre-1.0 status
|
|
101
|
+
|
|
102
|
+
`env-cap` has not yet reached a `1.0` release. Per [Keep a
|
|
103
|
+
Changelog](https://keepachangelog.com/en/1.1.0/)/semver convention, **minor
|
|
104
|
+
versions may include breaking changes to the Stable tier before 1.0** --
|
|
105
|
+
this document defines _scope_ (what would eventually be covered), not a
|
|
106
|
+
promise that it is already fully locked in at `0.x`.
|
|
107
|
+
|
|
108
|
+
How a `0.x` bump is chosen: the shared release tooling from
|
|
109
|
+
`internal-package-contract` deflates one level below 1.0.0 -- a breaking change
|
|
110
|
+
releases a minor, a feature a patch, and the API-contract gate requires only a
|
|
111
|
+
minor for a breaking API diff -- so **nothing automated can publish `1.0.0`**.
|
|
112
|
+
Crossing to 1.0.0 takes a human-authored `major` changeset, and that release
|
|
113
|
+
pull request is not auto-merged. The Experimental and Private tiers behave the
|
|
114
|
+
same before and after 1.0: Experimental surfaces may change at any version;
|
|
115
|
+
Private internals always may.
|
|
116
|
+
|
|
117
|
+
See [`SECURITY.md`](SECURITY.md#supported-versions) for the related, and
|
|
118
|
+
separate, question of which versions receive security fixes.
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
# Reading the benchmarks
|
|
2
|
+
|
|
3
|
+
This guide is for the person **reading** `BENCHMARKS.md`: a platform engineer deciding whether to run
|
|
4
|
+
this package, a CTO estimating what it adds to the infrastructure bill, or a maintainer checking a
|
|
5
|
+
change. It explains why the benchmarks exist, how to read each number, and -- just as important --
|
|
6
|
+
what you must **not** do with them.
|
|
7
|
+
|
|
8
|
+
Its companion, [WRITING-BENCHMARKS.md](WRITING-BENCHMARKS.md), explains how benchmarks are written and
|
|
9
|
+
documented.
|
|
10
|
+
|
|
11
|
+
## Why we benchmark
|
|
12
|
+
|
|
13
|
+
Every dependency has a price. Most of it is invisible until it is large: a few milliseconds added to
|
|
14
|
+
every request, a CPU core that is busier than it needs to be, a function that is fast on ten items and
|
|
15
|
+
unusable on ten thousand, a cold start that grows with every module. By then it is expensive to
|
|
16
|
+
remove.
|
|
17
|
+
|
|
18
|
+
The benchmarks make that price visible **before** you adopt the package and **every time** it changes.
|
|
19
|
+
They lean toward costs on purpose. The rest of the repository explains the benefits; this report tells
|
|
20
|
+
you what you are paying for them, in time and in money, and how the bill grows with your workload.
|
|
21
|
+
|
|
22
|
+
## What is in the report
|
|
23
|
+
|
|
24
|
+
`BENCHMARKS.md` has four parts, in the order a decision-maker needs them:
|
|
25
|
+
|
|
26
|
+
1. **What adopting this package costs.** The headline: added latency, CPU and memory per operation at
|
|
27
|
+
a typical workload and at the largest tested size, an estimated compute cost per million
|
|
28
|
+
operations, and the code size every process must load. If you read nothing else, read this.
|
|
29
|
+
2. **End-to-end.** The same operation done with no package and through the package, using empty or
|
|
30
|
+
minimal functions on both sides, at every size of the ladder (ten doubling sizes by default; the
|
|
31
|
+
report's environment section lists the sizes actually used). The difference is the package's total impact, with
|
|
32
|
+
nothing of your own application mixed in.
|
|
33
|
+
3. **Function by function.** Every function measured alone across the same sizes, with: why it is
|
|
34
|
+
benchmarked, what poor performance would mean, its documented big-O and the reason for it, every
|
|
35
|
+
variable that could change its cost, and what was deliberately not covered.
|
|
36
|
+
4. **What makes up one operation.** Each function's cost, multiplied by how often one operation calls
|
|
37
|
+
it, with nested calls counted once, as a share of the whole operation -- where the cost actually
|
|
38
|
+
lives.
|
|
39
|
+
|
|
40
|
+
## How to read a table
|
|
41
|
+
|
|
42
|
+
| Column | Meaning |
|
|
43
|
+
| --------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
44
|
+
| size (`n`) | The workload size for that row. The unit (record, call, file...) is stated at the top of the section. |
|
|
45
|
+
| Median | The middle sample. Robust against the occasional slow outlier; the number to quote. |
|
|
46
|
+
| p95 | 95% of samples were this fast or faster. The gap between median and p95 is the variability you should plan for. |
|
|
47
|
+
| CPU (median) | CPU time actually spent. Can exceed wall time slightly because of background compilation and garbage collection threads. |
|
|
48
|
+
| Heap Δ | Memory the operation left allocated (can be negative when garbage was collected during the sample). |
|
|
49
|
+
| Ops/s | Operations per second on one core at that size: the throughput ceiling of the operation alone. |
|
|
50
|
+
| Added / Added % | End-to-end only: time added over the baseline, absolute and as a percentage. |
|
|
51
|
+
|
|
52
|
+
## Big-O and the growth exponent
|
|
53
|
+
|
|
54
|
+
Big-O describes how cost grows as size grows -- more useful than any single timing, because it tells
|
|
55
|
+
you what happens at a size you did not test. The report fits a line through the log of time against
|
|
56
|
+
the log of size across the **largest** sizes (the top five of a long ladder: a real cost is a fixed
|
|
57
|
+
per-call overhead plus work that grows, and the overhead hides the growth at small sizes); the slope
|
|
58
|
+
is the **growth exponent**, snapped to the nearest class. The fit's R² is reported next to it, and when
|
|
59
|
+
the cost moves but does not follow a power law (a step, a cache cliff) no class is claimed at all:
|
|
60
|
+
|
|
61
|
+
| Exponent (about) | Class | What doubling the size does to the time |
|
|
62
|
+
| ---------------- | -------------- | --------------------------------------- |
|
|
63
|
+
| 0 | O(1) | Nothing. |
|
|
64
|
+
| 0.3 | O(log n) | Adds a small constant. |
|
|
65
|
+
| 1 | O(n) | Doubles it. |
|
|
66
|
+
| 1.2 | O(n log n) | A little more than doubles it. |
|
|
67
|
+
| 2 | O(n²) | Quadruples it. Dangerous at scale. |
|
|
68
|
+
| 3 or more | O(n³) or worse | Eight times or more. Unusable at scale. |
|
|
69
|
+
|
|
70
|
+
Each function shows its **documented** class next to its **measured** class with an agreement label:
|
|
71
|
+
|
|
72
|
+
- ✅ **matches** -- the code behaves as documented.
|
|
73
|
+
- 🟡 **close** -- neighbouring classes. Constant vs logarithmic and linear vs linearithmic cannot be
|
|
74
|
+
reliably separated on a bounded ladder; treat it as a match unless the exponent has moved.
|
|
75
|
+
- ⚠️ **differs** -- two or more classes apart. Either the documentation or the code is wrong. This is
|
|
76
|
+
the most useful line in the report.
|
|
77
|
+
- ❔ **not enough data** -- fewer than two usable sizes; something failed.
|
|
78
|
+
|
|
79
|
+
## The dollar figures
|
|
80
|
+
|
|
81
|
+
Time becomes money through two bracketing price shapes, because platforms bill differently:
|
|
82
|
+
|
|
83
|
+
- **Low:** CPU-priced compute (a container or VM billed per vCPU-hour) -- you pay for CPU seconds.
|
|
84
|
+
- **High:** duration-and-memory-priced functions (billed per GB-second) -- you pay for wall seconds
|
|
85
|
+
times the memory the platform reserves, never less than its minimum.
|
|
86
|
+
|
|
87
|
+
The rates are rounded published list prices and are **assumptions**; they drift with time and region.
|
|
88
|
+
Override them in `benchmark.config.json` → `costRates` to match your platform and negotiated pricing.
|
|
89
|
+
Treat the result as an order-of-magnitude comparison ("fractions of a cent per million operations" vs
|
|
90
|
+
"dollars per million operations"), not a quote.
|
|
91
|
+
|
|
92
|
+
## Results are not comparable between runs
|
|
93
|
+
|
|
94
|
+
This is the rule that matters most. **A benchmark number is a measurement of one run on one machine at
|
|
95
|
+
one moment.** It is not a property of the package.
|
|
96
|
+
|
|
97
|
+
- Do **not** compare numbers from different machines, operating systems, Node versions, or runners. A
|
|
98
|
+
laptop and a CI runner differ by large factors in both directions.
|
|
99
|
+
- Do **not** compare today's number with last month's if anything about the machine or load changed.
|
|
100
|
+
Background work, thermal throttling, other tenants on a shared runner and power settings all move
|
|
101
|
+
timings by 10-40%.
|
|
102
|
+
- Do **not** compare one package's numbers with another package's. The workloads differ; the
|
|
103
|
+
comparison means nothing.
|
|
104
|
+
- Do **not** treat any single number as a promise. It is not a guarantee, a service-level agreement or
|
|
105
|
+
a competitive claim.
|
|
106
|
+
|
|
107
|
+
What the numbers **are** good for:
|
|
108
|
+
|
|
109
|
+
- **Change over time on comparable hardware.** The same suite on the same kind of machine before and
|
|
110
|
+
after a change shows whether the change made the package slower. Look for movement larger than the
|
|
111
|
+
run-to-run noise.
|
|
112
|
+
- **Shape, not just level.** A change in growth class (O(n) becoming O(n²)) is a much bigger signal
|
|
113
|
+
than a 15% shift in a timing, and it is far less sensitive to the machine than a time is, because it
|
|
114
|
+
is a ratio between sizes on the same run. It still assumes the same sizes, workload and runtime.
|
|
115
|
+
- **Relative cost between functions.** Which function dominates the overhead is usually the same on
|
|
116
|
+
any machine, even when the absolute times are not.
|
|
117
|
+
- **Order of magnitude.** Microseconds, milliseconds or seconds per operation: that distinction is
|
|
118
|
+
stable enough to plan with.
|
|
119
|
+
|
|
120
|
+
## What the numbers do not prove
|
|
121
|
+
|
|
122
|
+
- **A complexity class is empirical.** It describes how measured cost grew across the sizes that were
|
|
123
|
+
tested. It is not a proof of asymptotic complexity, and it can be wrong when the tested range is
|
|
124
|
+
narrow or a fixed cost dominates.
|
|
125
|
+
- **R² is fit quality, not confidence.** It says how closely the measurements follow the fitted
|
|
126
|
+
power law. A high R² does not show the class is right, and a low one means no class is reported.
|
|
127
|
+
- **The overhead is a difference.** `end-to-end:overhead` is not measured; it is the with-package
|
|
128
|
+
time minus the baseline time at each size, never below zero. A difference of two noisy medians is
|
|
129
|
+
noisier than either, so the history page shows it for reading only and gives it no complexity class,
|
|
130
|
+
and the pull-request summary does not report a class shift for it. Read the baseline and with-package
|
|
131
|
+
groups, and the "What the package adds" table, instead.
|
|
132
|
+
- **A relative cost is still a measurement.** Dividing by the baseline removes much of the machine,
|
|
133
|
+
not all of it: compilation, garbage collection and contention on the runner still move it. Below the
|
|
134
|
+
timer-noise floor the percentage is not shown.
|
|
135
|
+
- **One run per commit is not a repeat.** The history holds one measurement for each recorded commit.
|
|
136
|
+
A trend is more convincing than a point; a single point moving is ordinary noise.
|
|
137
|
+
|
|
138
|
+
## Noise
|
|
139
|
+
|
|
140
|
+
Even on a quiet machine, repeated runs differ. Very fast operations (microseconds and below) are the
|
|
141
|
+
noisiest in relative terms; the engine batches them and reports the per-operation value, but a 30-40%
|
|
142
|
+
swing between runs on those is ordinary. Slow operations are steadier. Read the median and p95
|
|
143
|
+
together, and trust a trend over a single run. The pull-request summary shows the change against the last
|
|
144
|
+
run, but never judges it: a raw comparison is between two runs on different machines. What can fail a pull request is limited to growth classes and
|
|
145
|
+
baseline-relative ratios, which are far less sensitive to the machine than a raw time: see "History, pull
|
|
146
|
+
requests and the history page".
|
|
147
|
+
|
|
148
|
+
## How the documentation is done
|
|
149
|
+
|
|
150
|
+
Documentation is part of the benchmark definition, not an afterthought, and the suite will not load
|
|
151
|
+
without it. For every function the report prints, in this order:
|
|
152
|
+
|
|
153
|
+
1. **Why we benchmark it** -- where it runs, how often, and who feels it.
|
|
154
|
+
2. **What poor performance would mean** -- the consequence in latency, capacity and spend.
|
|
155
|
+
3. **Expected growth** -- the big-O and the mechanism that causes it.
|
|
156
|
+
4. **Variables that could change its cost** -- each one swept, measured as a variant, or fixed at a
|
|
157
|
+
stated value.
|
|
158
|
+
5. **Deliberately not covered** -- the honest gaps, with reasons.
|
|
159
|
+
6. **The measurements** -- every size, median, p95, CPU, memory, throughput, and the measured class next
|
|
160
|
+
to the documented one.
|
|
161
|
+
|
|
162
|
+
If a section reads as vague, that is a documentation bug: open an issue or fix it.
|
|
163
|
+
|
|
164
|
+
## History, pull requests and the history page
|
|
165
|
+
|
|
166
|
+
When a pull request touches benchmarked code, CI re-runs the suite and posts a summary: the end-to-end
|
|
167
|
+
change first, then any function whose growth class shifted, then ordinary timing movements beyond
|
|
168
|
+
their thresholds. Three gates can fail the run. None of them compares a raw time with another run's, because
|
|
169
|
+
two runs are on different machines. Two compare growth classes, which describe the shape of the curve and
|
|
170
|
+
not the speed of the machine: the measured class against the documented big-O (a function whose class
|
|
171
|
+
**differs** fails), and against the class at the last release (a function whose class has gotten **worse**
|
|
172
|
+
fails, unless `.repo-contract/exceptions/benchmark-complexity.json` records a person's acceptance of it).
|
|
173
|
+
The third is a ratio: the package's overhead _relative to its own baseline_, measured in the same run,
|
|
174
|
+
growing past an allowance (50% for millisecond-scale work, 70% for microsecond-scale work, which is
|
|
175
|
+
noisier) against both the last run on main and the last release. A function whose big-O could not be
|
|
176
|
+
determined (too few sizes, or no difference between them) is a **warning**: nothing is known to be wrong, but
|
|
177
|
+
nothing is checked. Raw time against another run is shown, never judged.
|
|
178
|
+
|
|
179
|
+
After a merge, a job measures the merge commit and records it -- the results file and a history entry
|
|
180
|
+
naming the commit, pull request and version -- through its own small pull request. Each entry also keeps
|
|
181
|
+
how noisy that run was (standard deviation, p95 and iteration count for every size) and the processor it
|
|
182
|
+
ran on, so a reader can judge later whether a change is larger than the run's own spread. A generated
|
|
183
|
+
history page opens with why the benchmarks exist and what they do and do not show, then charts every
|
|
184
|
+
function across runs, with every figure also available as a table, and shows the baseline, the
|
|
185
|
+
with-package time and the difference side by side.
|
|
186
|
+
|
|
187
|
+
## Glossary
|
|
188
|
+
|
|
189
|
+
- **Operation** -- one unit of the work the package performs, as defined by the suite.
|
|
190
|
+
- **Size / `n`** -- how much that operation processes (records, calls, files).
|
|
191
|
+
- **Baseline** -- the same operation with no package, using an empty or minimal function.
|
|
192
|
+
- **Overhead / added** -- with-package time minus baseline time: the package's own cost. Derived from the
|
|
193
|
+
two measured groups, never below zero, and not itself classified.
|
|
194
|
+
- **Median / p95** -- the middle sample, and the value 95% of samples beat.
|
|
195
|
+
- **Heap Δ** -- memory the operation left allocated.
|
|
196
|
+
- **Batch** -- the number of back-to-back repetitions inside one sample for very fast operations.
|
|
197
|
+
- **Cold start** -- the cost of loading and initializing the package in a fresh process.
|