@maverickcer/env-cap 0.7.4 → 0.7.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ADOPTION.md +106 -124
- package/AGENTS.md +94 -128
- package/CHANGELOG.md +42 -0
- package/CODE_REVIEW.md +69 -173
- package/CONTRIBUTING.md +57 -74
- package/GUIDE.md +405 -866
- package/README.md +112 -89
- package/RELEASING.md +81 -65
- package/SECURITY.md +76 -227
- package/SUPPORT.md +21 -45
- package/VERSIONING.md +69 -105
- package/benchmarks/READING-BENCHMARKS.md +103 -81
- package/benchmarks/README.md +12 -9
- package/benchmarks/WRITING-BENCHMARKS.md +90 -80
- package/benchmarks/performance-buildtime/BENCHMARKS.md +20 -20
- package/benchmarks/performance-runtime/BENCHMARKS.md +107 -107
- package/dist/.dts/_ipc/build-fs.d.ts +12 -17
- package/dist/.dts/_ipc/build-fs.d.ts.map +1 -1
- package/dist/.dts/_ipc/compiler-api.d.ts +6 -6
- package/dist/.dts/_ipc/display-path.d.ts +1 -1
- package/dist/.dts/_ipc/display-path.d.ts.map +1 -1
- package/dist/.dts/_ipc/generated-banner.d.ts +8 -8
- package/dist/.dts/_ipc/glob.d.ts +9 -8
- package/dist/.dts/_ipc/glob.d.ts.map +1 -1
- package/dist/.dts/_ipc/no-node-fs.d.ts +2 -2
- package/dist/.dts/_ipc/no-node-fs.d.ts.map +1 -1
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts +16 -16
- package/dist/.dts/_ipc/resolve-tsconfig-paths.d.ts.map +1 -1
- package/dist/.dts/_ipc/typescript.d.ts +4 -4
- package/dist/.dts/build/assemble-project.d.ts +14 -15
- package/dist/.dts/build/assemble-project.d.ts.map +1 -1
- package/dist/.dts/build/change-model.d.ts +14 -14
- package/dist/.dts/build/change-model.d.ts.map +1 -1
- package/dist/.dts/build/check-artifacts.d.ts +4 -4
- package/dist/.dts/build/check-artifacts.d.ts.map +1 -1
- package/dist/.dts/build/citation-verification.d.ts +13 -14
- package/dist/.dts/build/citation-verification.d.ts.map +1 -1
- package/dist/.dts/build/compatibility.d.ts +24 -24
- package/dist/.dts/build/compatibility.d.ts.map +1 -1
- package/dist/.dts/build/contract-model.d.ts +10 -10
- package/dist/.dts/build/contract-model.d.ts.map +1 -1
- package/dist/.dts/build/deep-equal.d.ts +2 -2
- package/dist/.dts/build/deep-freeze.d.ts +2 -2
- package/dist/.dts/build/dependency-graph.d.ts +48 -55
- package/dist/.dts/build/dependency-graph.d.ts.map +1 -1
- package/dist/.dts/build/dependency-model.d.ts +18 -18
- package/dist/.dts/build/dependency-model.d.ts.map +1 -1
- package/dist/.dts/build/discover.d.ts +11 -11
- package/dist/.dts/build/discover.d.ts.map +1 -1
- package/dist/.dts/build/docs.d.ts +34 -38
- package/dist/.dts/build/docs.d.ts.map +1 -1
- package/dist/.dts/build/env-example.d.ts +13 -13
- package/dist/.dts/build/env-example.d.ts.map +1 -1
- package/dist/.dts/build/errors.d.ts +8 -8
- package/dist/.dts/build/evidence-cache.d.ts +23 -23
- package/dist/.dts/build/evidence-cache.d.ts.map +1 -1
- package/dist/.dts/build/evidence-model.d.ts +3 -3
- package/dist/.dts/build/evidence-reference.d.ts +15 -15
- package/dist/.dts/build/evidence-reference.d.ts.map +1 -1
- package/dist/.dts/build/evidence-snapshot.d.ts +47 -53
- package/dist/.dts/build/evidence-snapshot.d.ts.map +1 -1
- package/dist/.dts/build/exclusive-group.d.ts +4 -4
- package/dist/.dts/build/exclusive-group.d.ts.map +1 -1
- package/dist/.dts/build/finding-model.d.ts +12 -12
- package/dist/.dts/build/finding-model.d.ts.map +1 -1
- package/dist/.dts/build/generate-documentation.d.ts +19 -19
- package/dist/.dts/build/generate-documentation.d.ts.map +1 -1
- package/dist/.dts/build/generate-env-artifacts.d.ts +43 -43
- package/dist/.dts/build/generate-env-artifacts.d.ts.map +1 -1
- package/dist/.dts/build/generate-evidence.d.ts +19 -19
- package/dist/.dts/build/generate-evidence.d.ts.map +1 -1
- package/dist/.dts/build/generate-manifest.d.ts +11 -12
- package/dist/.dts/build/generate-manifest.d.ts.map +1 -1
- package/dist/.dts/build/generate-usage.d.ts +23 -24
- package/dist/.dts/build/generate-usage.d.ts.map +1 -1
- package/dist/.dts/build/generated-banner.d.ts +5 -5
- package/dist/.dts/build/governance-fields.d.ts +7 -7
- package/dist/.dts/build/humanize-key.d.ts +1 -1
- package/dist/.dts/build/humanize-key.d.ts.map +1 -1
- package/dist/.dts/build/index.d.ts +20 -36
- package/dist/.dts/build/index.d.ts.map +1 -1
- package/dist/.dts/build/lifecycle-model.d.ts +90 -83
- package/dist/.dts/build/lifecycle-model.d.ts.map +1 -1
- package/dist/.dts/build/link.d.ts +38 -39
- package/dist/.dts/build/link.d.ts.map +1 -1
- package/dist/.dts/build/literal-eval.d.ts +2 -2
- package/dist/.dts/build/literal-eval.d.ts.map +1 -1
- package/dist/.dts/build/live-expirations.d.ts +14 -13
- package/dist/.dts/build/live-expirations.d.ts.map +1 -1
- package/dist/.dts/build/manifest.d.ts +6 -6
- package/dist/.dts/build/map-utils.d.ts +4 -4
- package/dist/.dts/build/merge-documentation.d.ts +9 -8
- package/dist/.dts/build/merge-documentation.d.ts.map +1 -1
- package/dist/.dts/build/ownership-model.d.ts +14 -15
- package/dist/.dts/build/ownership-model.d.ts.map +1 -1
- package/dist/.dts/build/parse.d.ts +20 -21
- package/dist/.dts/build/parse.d.ts.map +1 -1
- package/dist/.dts/build/reference-projections.d.ts +36 -38
- package/dist/.dts/build/reference-projections.d.ts.map +1 -1
- package/dist/.dts/build/resolution/resolve-import.d.ts +18 -14
- package/dist/.dts/build/resolution/resolve-import.d.ts.map +1 -1
- package/dist/.dts/build/resolution/resolve-package-schema.d.ts +26 -26
- package/dist/.dts/build/resolution/resolve-package-schema.d.ts.map +1 -1
- package/dist/.dts/build/resolution/resolve-within-root.d.ts +9 -9
- package/dist/.dts/build/sarif.d.ts +5 -5
- package/dist/.dts/build/sarif.d.ts.map +1 -1
- package/dist/.dts/build/scan-dependencies.d.ts +17 -16
- package/dist/.dts/build/scan-dependencies.d.ts.map +1 -1
- package/dist/.dts/build/sort-by-identity.d.ts +1 -1
- package/dist/.dts/build/source-position.d.ts +13 -13
- package/dist/.dts/build/tool-version.d.ts +1 -1
- package/dist/.dts/build/usage-report.d.ts +19 -19
- package/dist/.dts/build/usage-report.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/index.d.ts +16 -5
- package/dist/.dts/eslint-plugin/index.d.ts.map +1 -1
- package/dist/.dts/eslint-plugin/no-raw-process-env.d.ts +4 -4
- package/dist/.dts/evidence/define-projection.d.ts +15 -16
- package/dist/.dts/evidence/define-projection.d.ts.map +1 -1
- package/dist/.dts/evidence/index.d.ts +11 -15
- package/dist/.dts/evidence/index.d.ts.map +1 -1
- package/dist/.dts/helpers/index.d.ts +15 -18
- package/dist/.dts/helpers/index.d.ts.map +1 -1
- package/dist/.dts/helpers/validators.d.ts +2 -2
- package/dist/.dts/helpers/validators.d.ts.map +1 -1
- package/dist/.dts/node/index.d.ts +19 -0
- package/dist/.dts/node/index.d.ts.map +1 -1
- package/dist/.dts/runtime/create.d.ts +6 -3
- package/dist/.dts/runtime/create.d.ts.map +1 -1
- package/dist/.dts/runtime/document.d.ts +91 -97
- package/dist/.dts/runtime/document.d.ts.map +1 -1
- package/dist/.dts/runtime/errors.d.ts +5 -5
- package/dist/.dts/runtime/index.d.ts +11 -10
- package/dist/.dts/runtime/index.d.ts.map +1 -1
- package/dist/.dts/runtime/reset.d.ts +1 -1
- package/dist/.dts/runtime/types.d.ts +39 -24
- package/dist/.dts/runtime/types.d.ts.map +1 -1
- package/dist/.dts/runtime/validate.d.ts +15 -2
- package/dist/.dts/runtime/validate.d.ts.map +1 -1
- package/dist/.dts7/_ipc/build-fs.d.ts +48 -0
- package/dist/.dts7/_ipc/build-fs.d.ts.map +1 -0
- package/dist/.dts7/_ipc/compiler-api.d.ts +17 -0
- package/dist/.dts7/_ipc/compiler-api.d.ts.map +1 -0
- package/dist/.dts7/_ipc/display-path.d.ts +27 -0
- package/dist/.dts7/_ipc/display-path.d.ts.map +1 -0
- package/dist/.dts7/_ipc/generated-banner.d.ts +58 -0
- package/dist/.dts7/_ipc/generated-banner.d.ts.map +1 -0
- package/dist/.dts7/_ipc/glob.d.ts +23 -0
- package/dist/.dts7/_ipc/glob.d.ts.map +1 -0
- package/dist/.dts7/_ipc/no-node-fs.d.ts +21 -0
- package/dist/.dts7/_ipc/no-node-fs.d.ts.map +1 -0
- package/dist/.dts7/_ipc/resolve-tsconfig-paths.d.ts +134 -0
- package/dist/.dts7/_ipc/resolve-tsconfig-paths.d.ts.map +1 -0
- package/dist/.dts7/_ipc/typescript.d.ts +20 -0
- package/dist/.dts7/_ipc/typescript.d.ts.map +1 -0
- package/dist/.dts7/build/assemble-project.d.ts +46 -0
- package/dist/.dts7/build/assemble-project.d.ts.map +1 -0
- package/dist/.dts7/build/change-model.d.ts +63 -0
- package/dist/.dts7/build/change-model.d.ts.map +1 -0
- package/dist/.dts7/build/check-artifacts.d.ts +48 -0
- package/dist/.dts7/build/check-artifacts.d.ts.map +1 -0
- package/dist/.dts7/build/citation-verification.d.ts +61 -0
- package/dist/.dts7/build/citation-verification.d.ts.map +1 -0
- package/dist/.dts7/build/compatibility.d.ts +83 -0
- package/dist/.dts7/build/compatibility.d.ts.map +1 -0
- package/dist/.dts7/build/compiler-api.d.ts +12 -0
- package/dist/.dts7/build/compiler-api.d.ts.map +1 -0
- package/dist/.dts7/build/contract-model.d.ts +93 -0
- package/dist/.dts7/build/contract-model.d.ts.map +1 -0
- package/dist/.dts7/build/deep-equal.d.ts +9 -0
- package/dist/.dts7/build/deep-equal.d.ts.map +1 -0
- package/dist/.dts7/build/deep-freeze.d.ts +20 -0
- package/dist/.dts7/build/deep-freeze.d.ts.map +1 -0
- package/dist/.dts7/build/dependency-graph.d.ts +163 -0
- package/dist/.dts7/build/dependency-graph.d.ts.map +1 -0
- package/dist/.dts7/build/dependency-model.d.ts +81 -0
- package/dist/.dts7/build/dependency-model.d.ts.map +1 -0
- package/dist/.dts7/build/discover.d.ts +41 -0
- package/dist/.dts7/build/discover.d.ts.map +1 -0
- package/dist/.dts7/build/display-path.d.ts +6 -0
- package/dist/.dts7/build/display-path.d.ts.map +1 -0
- package/dist/.dts7/build/docs.d.ts +172 -0
- package/dist/.dts7/build/docs.d.ts.map +1 -0
- package/dist/.dts7/build/env-example.d.ts +85 -0
- package/dist/.dts7/build/env-example.d.ts.map +1 -0
- package/dist/.dts7/build/errors.d.ts +78 -0
- package/dist/.dts7/build/errors.d.ts.map +1 -0
- package/dist/.dts7/build/evidence-cache.d.ts +74 -0
- package/dist/.dts7/build/evidence-cache.d.ts.map +1 -0
- package/dist/.dts7/build/evidence-model.d.ts +50 -0
- package/dist/.dts7/build/evidence-model.d.ts.map +1 -0
- package/dist/.dts7/build/evidence-reference.d.ts +67 -0
- package/dist/.dts7/build/evidence-reference.d.ts.map +1 -0
- package/dist/.dts7/build/evidence-snapshot.d.ts +142 -0
- package/dist/.dts7/build/evidence-snapshot.d.ts.map +1 -0
- package/dist/.dts7/build/exclusive-group.d.ts +19 -0
- package/dist/.dts7/build/exclusive-group.d.ts.map +1 -0
- package/dist/.dts7/build/finding-model.d.ts +80 -0
- package/dist/.dts7/build/finding-model.d.ts.map +1 -0
- package/dist/.dts7/build/generate-documentation.d.ts +176 -0
- package/dist/.dts7/build/generate-documentation.d.ts.map +1 -0
- package/dist/.dts7/build/generate-env-artifacts.d.ts +191 -0
- package/dist/.dts7/build/generate-env-artifacts.d.ts.map +1 -0
- package/dist/.dts7/build/generate-evidence.d.ts +77 -0
- package/dist/.dts7/build/generate-evidence.d.ts.map +1 -0
- package/dist/.dts7/build/generate-manifest.d.ts +100 -0
- package/dist/.dts7/build/generate-manifest.d.ts.map +1 -0
- package/dist/.dts7/build/generate-usage.d.ts +94 -0
- package/dist/.dts7/build/generate-usage.d.ts.map +1 -0
- package/dist/.dts7/build/generated-banner.d.ts +43 -0
- package/dist/.dts7/build/generated-banner.d.ts.map +1 -0
- package/dist/.dts7/build/glob.d.ts +6 -0
- package/dist/.dts7/build/glob.d.ts.map +1 -0
- package/dist/.dts7/build/governance-fields.d.ts +34 -0
- package/dist/.dts7/build/governance-fields.d.ts.map +1 -0
- package/dist/.dts7/build/humanize-key.d.ts +14 -0
- package/dist/.dts7/build/humanize-key.d.ts.map +1 -0
- package/dist/.dts7/build/index.d.ts +89 -0
- package/dist/.dts7/build/index.d.ts.map +1 -0
- package/dist/.dts7/build/lifecycle-model.d.ts +181 -0
- package/dist/.dts7/build/lifecycle-model.d.ts.map +1 -0
- package/dist/.dts7/build/link.d.ts +222 -0
- package/dist/.dts7/build/link.d.ts.map +1 -0
- package/dist/.dts7/build/literal-eval.d.ts +24 -0
- package/dist/.dts7/build/literal-eval.d.ts.map +1 -0
- package/dist/.dts7/build/live-expirations.d.ts +48 -0
- package/dist/.dts7/build/live-expirations.d.ts.map +1 -0
- package/dist/.dts7/build/manifest.d.ts +31 -0
- package/dist/.dts7/build/manifest.d.ts.map +1 -0
- package/dist/.dts7/build/map-utils.d.ts +8 -0
- package/dist/.dts7/build/map-utils.d.ts.map +1 -0
- package/dist/.dts7/build/merge-documentation.d.ts +36 -0
- package/dist/.dts7/build/merge-documentation.d.ts.map +1 -0
- package/dist/.dts7/build/ownership-model.d.ts +58 -0
- package/dist/.dts7/build/ownership-model.d.ts.map +1 -0
- package/dist/.dts7/build/parse.d.ts +206 -0
- package/dist/.dts7/build/parse.d.ts.map +1 -0
- package/dist/.dts7/build/reference-projections.d.ts +168 -0
- package/dist/.dts7/build/reference-projections.d.ts.map +1 -0
- package/dist/.dts7/build/resolution/resolve-import.d.ts +62 -0
- package/dist/.dts7/build/resolution/resolve-import.d.ts.map +1 -0
- package/dist/.dts7/build/resolution/resolve-package-schema.d.ts +83 -0
- package/dist/.dts7/build/resolution/resolve-package-schema.d.ts.map +1 -0
- package/dist/.dts7/build/resolution/resolve-tsconfig-paths.d.ts +18 -0
- package/dist/.dts7/build/resolution/resolve-tsconfig-paths.d.ts.map +1 -0
- package/dist/.dts7/build/resolution/resolve-within-root.d.ts +35 -0
- package/dist/.dts7/build/resolution/resolve-within-root.d.ts.map +1 -0
- package/dist/.dts7/build/sarif.d.ts +66 -0
- package/dist/.dts7/build/sarif.d.ts.map +1 -0
- package/dist/.dts7/build/scan-dependencies.d.ts +71 -0
- package/dist/.dts7/build/scan-dependencies.d.ts.map +1 -0
- package/dist/.dts7/build/sort-by-identity.d.ts +14 -0
- package/dist/.dts7/build/sort-by-identity.d.ts.map +1 -0
- package/dist/.dts7/build/source-position.d.ts +69 -0
- package/dist/.dts7/build/source-position.d.ts.map +1 -0
- package/dist/.dts7/build/tool-version.d.ts +8 -0
- package/dist/.dts7/build/tool-version.d.ts.map +1 -0
- package/dist/.dts7/build/types.d.ts +7 -0
- package/dist/.dts7/build/types.d.ts.map +1 -0
- package/dist/.dts7/build/typescript.d.ts +6 -0
- package/dist/.dts7/build/typescript.d.ts.map +1 -0
- package/dist/.dts7/build/usage-report.d.ts +124 -0
- package/dist/.dts7/build/usage-report.d.ts.map +1 -0
- package/dist/.dts7/cli/filesystem.d.ts +7 -0
- package/dist/.dts7/cli/filesystem.d.ts.map +1 -0
- package/dist/.dts7/eslint-plugin/glob.d.ts +7 -0
- package/dist/.dts7/eslint-plugin/glob.d.ts.map +1 -0
- package/dist/.dts7/eslint-plugin/index.d.ts +38 -0
- package/dist/.dts7/eslint-plugin/index.d.ts.map +1 -0
- package/dist/.dts7/eslint-plugin/no-node-fs.d.ts +15 -0
- package/dist/.dts7/eslint-plugin/no-node-fs.d.ts.map +1 -0
- package/dist/.dts7/eslint-plugin/no-raw-process-env.d.ts +23 -0
- package/dist/.dts7/eslint-plugin/no-raw-process-env.d.ts.map +1 -0
- package/dist/.dts7/evidence/define-projection.d.ts +51 -0
- package/dist/.dts7/evidence/define-projection.d.ts.map +1 -0
- package/dist/.dts7/evidence/index.d.ts +19 -0
- package/dist/.dts7/evidence/index.d.ts.map +1 -0
- package/dist/.dts7/helpers/index.d.ts +132 -0
- package/dist/.dts7/helpers/index.d.ts.map +1 -0
- package/dist/.dts7/helpers/processors.d.ts +32 -0
- package/dist/.dts7/helpers/processors.d.ts.map +1 -0
- package/dist/.dts7/helpers/validators.d.ts +68 -0
- package/dist/.dts7/helpers/validators.d.ts.map +1 -0
- package/dist/.dts7/node/index.d.ts +21 -0
- package/dist/.dts7/node/index.d.ts.map +1 -0
- package/dist/.dts7/runtime/cache.d.ts +17 -0
- package/dist/.dts7/runtime/cache.d.ts.map +1 -0
- package/dist/.dts7/runtime/create.d.ts +23 -0
- package/dist/.dts7/runtime/create.d.ts.map +1 -0
- package/dist/.dts7/runtime/document.d.ts +214 -0
- package/dist/.dts7/runtime/document.d.ts.map +1 -0
- package/dist/.dts7/runtime/errors.d.ts +55 -0
- package/dist/.dts7/runtime/errors.d.ts.map +1 -0
- package/dist/.dts7/runtime/index.d.ts +25 -0
- package/dist/.dts7/runtime/index.d.ts.map +1 -0
- package/dist/.dts7/runtime/registry.d.ts +12 -0
- package/dist/.dts7/runtime/registry.d.ts.map +1 -0
- package/dist/.dts7/runtime/reset.d.ts +8 -0
- package/dist/.dts7/runtime/reset.d.ts.map +1 -0
- package/dist/.dts7/runtime/types.d.ts +190 -0
- package/dist/.dts7/runtime/types.d.ts.map +1 -0
- package/dist/.dts7/runtime/validate.d.ts +27 -0
- package/dist/.dts7/runtime/validate.d.ts.map +1 -0
- package/dist/{build-6XFHJFQK.js → build-UEENZEUX.js} +43 -42
- package/dist/build-UEENZEUX.js.map +1 -0
- package/dist/build.cjs +42 -41
- package/dist/build.cjs.map +1 -1
- package/dist/build.js +42 -41
- package/dist/build.js.map +1 -1
- package/dist/build.ts7.d.cts +1 -0
- package/dist/build.ts7.d.ts +1 -0
- package/dist/{chunk-RL5KFVUT.js → chunk-TBL3O7IG.js} +2 -2
- package/dist/cli/index.js +29 -28
- package/dist/cli/index.js.map +1 -1
- package/dist/eslint-plugin/index.cjs.map +1 -1
- package/dist/eslint-plugin/index.js.map +1 -1
- package/dist/eslint-plugin/index.ts7.d.cts +1 -0
- package/dist/eslint-plugin/index.ts7.d.ts +1 -0
- package/dist/evidence.cjs +1 -1
- package/dist/evidence.cjs.map +1 -1
- package/dist/evidence.js +1 -1
- package/dist/evidence.js.map +1 -1
- package/dist/evidence.ts7.d.cts +1 -0
- package/dist/evidence.ts7.d.ts +1 -0
- package/dist/helpers.cjs +4 -4
- package/dist/helpers.cjs.map +1 -1
- package/dist/helpers.js +4 -4
- package/dist/helpers.js.map +1 -1
- package/dist/helpers.ts7.d.cts +1 -0
- package/dist/helpers.ts7.d.ts +1 -0
- package/dist/index.cjs.map +1 -1
- package/dist/index.js.map +1 -1
- package/dist/index.ts7.d.cts +1 -0
- package/dist/index.ts7.d.ts +1 -0
- package/dist/node.ts7.d.cts +1 -0
- package/dist/node.ts7.d.ts +1 -0
- package/examples/README.md +46 -80
- package/package.json +43 -12
- package/schemas/contract-model.schema.json +7 -7
- package/schemas/env-cap-report.schema.json +88 -88
- package/skills/env-cap/SKILL.md +12 -13
- package/skills/env-cap/references/api.md +1 -1
- package/specs/architecture.md +147 -122
- package/specs/decisions/0001-runtime-documentation-separation.md +1 -1
- package/specs/decisions/0014-cross-package-schema-discovery.md +4 -1
- package/specs/decisions/0023-tsconfig-path-alias-resolution.md +4 -1
- package/specs/decisions/0049-typescript-7-scanner-uses-a-bundled-typescript-6.md +13 -8
- package/specs/generated-artifacts.md +91 -71
- package/specs/migrations/README.md +6 -6
- package/specs/migrations/from-centralized-schema.md +1 -1
- package/specs/migrations/from-t3-env.md +9 -9
- package/PROMPT.md +0 -89
- package/dist/build-6XFHJFQK.js.map +0 -1
- /package/dist/{chunk-RL5KFVUT.js.map → chunk-TBL3O7IG.js.map} +0 -0
package/VERSIONING.md
CHANGED
|
@@ -1,118 +1,82 @@
|
|
|
1
1
|
# Versioning and API stability
|
|
2
2
|
|
|
3
|
-
`env-cap` follows [Semantic Versioning](https://semver.org/).
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
-
|
|
22
|
-
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
-- see [ADR 0042](specs/decisions/0042-init-cli-subcommand-scaffolds-only.md).
|
|
55
|
-
- **The composite GitHub Action** (`action.yml`): its input and output names and meanings, and
|
|
56
|
-
the PR comment's sections. The major tag it is used through (`uses: MaverickCER/env-cap@v0`
|
|
57
|
-
while the package is 0.x, `@v1` from 1.0.0) moves only within a major version.
|
|
58
|
-
- **Supported toolchain**: Node.js `>=22`, `moduleResolution` `node16`/`nodenext`/`bundler` (not the legacy
|
|
59
|
-
`node10`, which cannot resolve subpath exports), and the TypeScript versions named in the
|
|
60
|
-
`typescript` peer range (`^5 || ^6 || ^7`). The build-time scanner needs TypeScript's classic
|
|
61
|
-
compiler API, which TypeScript 7 does not ship, so it uses the consumer's `typescript` when that has
|
|
62
|
-
the API and otherwise the bundled `@typescript/typescript6` dependency -- no configuration, and `tsc`
|
|
63
|
-
stays TypeScript 7 ([ADR 0049](specs/decisions/0049-typescript-7-scanner-uses-a-bundled-typescript-6.md)).
|
|
64
|
-
Under TypeScript 7 the `./build` and `./evidence` declarations need `skipLibCheck`; the runtime,
|
|
65
|
-
helpers and node entry points type-check strictly.
|
|
66
|
-
|
|
67
|
-
All promoted from Experimental per [ADR 0045](specs/decisions/0045-promote-experimental-surfaces-to-stable.md)
|
|
68
|
-
-- see that ADR for why each one had already cleared this tier's own
|
|
69
|
-
graduation criterion (below).
|
|
3
|
+
Every entry point is Stable, and two baseline checks enforce the release level a change needs. Before 1.0, a breaking change needs only a minor release. `env-cap` follows [Semantic Versioning](https://semver.org/).
|
|
4
|
+
|
|
5
|
+
## Tiers
|
|
6
|
+
|
|
7
|
+
| Tier | What a release may do | Enforced by |
|
|
8
|
+
| ------------ | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
|
|
9
|
+
| Stable | A breaking change needs a major release from 1.0 (a minor before 1.0) | `ApiContract` and `SurfaceContract` diff against committed baselines and require a matching changeset |
|
|
10
|
+
| Experimental | May change shape in any release, including a patch | Labelled in this document, the option's TSDoc and its ADR. Nothing ships Experimental today |
|
|
11
|
+
| Private | May change at any time without notice | Not exported from `package.json#exports`; no check needed |
|
|
12
|
+
|
|
13
|
+
## Every entry point and its tier
|
|
14
|
+
|
|
15
|
+
Generated from `package.json#exports` and `docs/surface.json` by `internal-package-contract generate`; the `GeneratedRegions` check fails if it is stale.
|
|
16
|
+
|
|
17
|
+
<!-- ipc:begin entry-points -->
|
|
18
|
+
|
|
19
|
+
| Export | Source | Environment | Tier | Purpose |
|
|
20
|
+
| ------------------------------------ | -------------------- | ------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
21
|
+
| `@maverickcer/env-cap` | `src/runtime/` | No Node module or Node global | Stable | `createEnv`, `documentEnv`, `validateEnv`, `resetEnvCache`, `isEnvContract` and the error types |
|
|
22
|
+
| `@maverickcer/env-cap/build` | `src/build/` | Node only, in development and CI | Stable | Discovery, AST analysis and artifact generation. The four `generate*()` orchestrators and the lower-level primitives are Stable (ADR 0045, `VERSIONING.md`). |
|
|
23
|
+
| `@maverickcer/env-cap/helpers` | `src/helpers/` | No Node module; `processors.base64()` returns a Node `Buffer` | Stable | `processors` and `validators` |
|
|
24
|
+
| `@maverickcer/env-cap/node` | `src/node/` | Node only, in development and CI | Stable | `nodeBuildFileSystem`, the `BuildFileSystem` adapter a build script hands to `./build` |
|
|
25
|
+
| `@maverickcer/env-cap/evidence` | `src/evidence/` | No Node module or Node global | Stable | `defineEvidenceProjection`, pure transforms over the Evidence Model (Stable, see `VERSIONING.md`) |
|
|
26
|
+
| `@maverickcer/env-cap/eslint-plugin` | `src/eslint-plugin/` | Node only, inside the project's ESLint | Stable | The `no-raw-process-env` and `no-node-fs` lint rules |
|
|
27
|
+
| `@maverickcer/env-cap/schema` | `schemas/` | Static JSON | Stable | The JSON Schema for `env-cap --json` output (`env-cap-report.schema.json`) and the model schemas under `./schema/*` |
|
|
28
|
+
|
|
29
|
+
<!-- ipc:end entry-points -->
|
|
30
|
+
|
|
31
|
+
## What the Stable tier covers
|
|
32
|
+
|
|
33
|
+
| Surface | Checked by | Limit |
|
|
34
|
+
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
35
|
+
| Runtime APIs: `createEnv`, `validateEnv`, `documentEnv`, `resetEnvCache`, `isEnvContract`, the error classes | `ApiContract` (runtime baseline) | Signatures, not behaviour; tests cover behaviour |
|
|
36
|
+
| The four `generate*()` functions, their options, and the lower-level `./build` primitives such as `discoverSchemaFiles`, `parseSchemaFile`, `generateEvidenceModel` and `getEvidenceModel` | `ApiContract` (build baseline) | The dependency-ownership engine's internals are Private ([ADR 0010](specs/decisions/0010-dependency-ownership-engine-scope-boundary.md)) |
|
|
37
|
+
| `@maverickcer/env-cap/helpers`, `/evidence`, `/node` and `/eslint-plugin` exports, including rule names, options and the default allow list | `ApiContract` (one baseline per entry) | None found |
|
|
38
|
+
| CLI flags in `env-cap --help` and `env-cap init --help`, the exit codes of those two, and the exit codes `env-cap --help` lists | `SurfaceContract` | The baseline records the listed codes; `test/cli/main.test.ts` and `test/cli/direct-run.test.ts` check that the CLI returns them. `init`'s other exit codes are pinned by `test/cli/init.test.ts` only |
|
|
39
|
+
| The `--json` report and the published JSON Schema (`@maverickcer/env-cap/schema`) | `SurfaceContract` diffs the schema; the `Schema` check fails if it is not regenerated from `src/cli/json.ts` | `test/build/schema-version-bump.test.ts` fails when a field is removed, retyped or made optional at an unchanged `schemaVersion`. A change of meaning with the same shape is not detected |
|
|
40
|
+
| The persisted evidence file, a serialization of `EvidenceModel` | `ApiContract` (evidence baseline) for the type | The committed evidence of two examples validates against the schema; a freshly written file is not checked |
|
|
41
|
+
| The generated manifest's shape | Tests in `test/build/manifest.test.ts` | No baseline diff |
|
|
42
|
+
| The section headings of the generated Markdown (`ENVIRONMENT.md`, the ownership and usage reports), the comment headers of `.env.example`, and the Action summary's sections | `SurfaceContract` (renderer files listed under `internal-package-contract.surfaceRenderers` in `package.json`) | The wording inside a section, and a heading built from a variable or feature name, are not baselined |
|
|
43
|
+
| The `envCap.schema` field in a package's `package.json`, and the `packages` and `tsconfig` options | Tests in `test/build/resolution/` | No baseline diff for the field |
|
|
44
|
+
| The files and templates `env-cap init` writes | `test/cli/init.test.ts` pins the exact templates | None found |
|
|
45
|
+
| The GitHub Action's input names and defaults | `SurfaceContract` | Input descriptions are not baselined; the summary's section headings are, through the row above |
|
|
46
|
+
| The supported toolchain: Node.js `>=22` and `moduleResolution` `node16`, `nodenext` or `bundler` | `EngineMatrix` (Node 22 and 24) and `TypeResolution` | Node 26 runs as an informational job |
|
|
47
|
+
| TypeScript 5, 6 and 7 for type-checking every entry point except `./eslint-plugin`, under `strict` and full library checking | `scripts/typescript-compat.mjs` in CI | CI installs the newest 5.x, 6.x and 7.x only, in `typescript-5` and `typescript-compat`, which `required` waits for |
|
|
48
|
+
|
|
49
|
+
The Action is used through a major tag: `uses: maverickcer/env-cap@v0` while the package is `0.x`, `@v1` from 1.0.0. That tag moves only within a major version.
|
|
50
|
+
|
|
51
|
+
The scanner needs TypeScript's classic compiler API, which TypeScript 7 does not ship. It uses your `typescript` when that has the API, and the bundled TypeScript 6 otherwise. Your `tsc` stays TypeScript 7 ([ADR 0049](specs/decisions/0049-typescript-7-scanner-uses-a-bundled-typescript-6.md)).
|
|
52
|
+
|
|
53
|
+
Everything above was promoted from Experimental in [ADR 0045](specs/decisions/0045-promote-experimental-surfaces-to-stable.md).
|
|
70
54
|
|
|
71
55
|
## Experimental
|
|
72
56
|
|
|
73
|
-
|
|
74
|
-
and the ADR that introduces it. An Experimental surface may change shape --
|
|
75
|
-
including in a breaking way -- in a minor or patch release, without that
|
|
76
|
-
being a semver violation. This is not a loophole for casual churn: a feature
|
|
77
|
-
ships Experimental because it is genuinely new enough that real-world
|
|
78
|
-
feedback is likely to reveal a better shape, not because we're unwilling to
|
|
79
|
-
commit to _something_. Once a feature has been through at least one real
|
|
80
|
-
feedback cycle without a need to break it, its ADR's Status moves from
|
|
81
|
-
Proposed/Experimental to Accepted, and it becomes Stable.
|
|
57
|
+
A surface ships Experimental when it is new enough that real use may show a better shape. It is labelled here, in the option's TSDoc, and in the ADR that introduces it. After one feedback cycle without a break, its ADR moves to Accepted and it becomes Stable.
|
|
82
58
|
|
|
83
|
-
Nothing
|
|
84
|
-
for the most recent round of promotions. A future feature genuinely new
|
|
85
|
-
enough to warrant it will be added here, explicitly, with its own ADR.
|
|
59
|
+
Nothing ships Experimental today.
|
|
86
60
|
|
|
87
61
|
## Private
|
|
88
62
|
|
|
89
|
-
Never covered by semver
|
|
63
|
+
Never covered by semver:
|
|
90
64
|
|
|
91
|
-
-
|
|
92
|
-
|
|
93
|
-
-
|
|
94
|
-
|
|
95
|
-
- The dependency-ownership engine's internals (`scan-dependencies.ts`,
|
|
96
|
-
`dependency-graph.ts`) -- see [ADR 0010](specs/decisions/0010-dependency-ownership-engine-scope-boundary.md).
|
|
97
|
-
- Any behavior not documented in the README, a JSDoc comment on a public
|
|
98
|
-
export, or an ADR.
|
|
65
|
+
- Modules and functions not re-exported from an entry point in `package.json#exports`.
|
|
66
|
+
- How `src/build/parse.ts` walks a source file.
|
|
67
|
+
- The dependency-ownership engine (`scan-dependencies.ts`, `dependency-graph.ts`).
|
|
68
|
+
- Behaviour not documented in the README, the Guide, a TSDoc comment on a public export, or an ADR.
|
|
99
69
|
|
|
100
70
|
## Pre-1.0 status
|
|
101
71
|
|
|
102
|
-
`env-cap` has not
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
pull request is not auto-merged. The Experimental and Private tiers behave the
|
|
114
|
-
same before and after 1.0: Experimental surfaces may change at any version;
|
|
115
|
-
Private internals always may.
|
|
116
|
-
|
|
117
|
-
See [`SECURITY.md`](SECURITY.md#supported-versions) for the related, and
|
|
118
|
-
separate, question of which versions receive security fixes.
|
|
72
|
+
`env-cap` has not reached 1.0. Before 1.0, **a minor release may include a breaking change to the Stable tier.** This document defines the scope the 1.0 promise will cover.
|
|
73
|
+
|
|
74
|
+
How a `0.x` release level is chosen:
|
|
75
|
+
|
|
76
|
+
1. For changesets it generates from commit messages, the release tooling lowers each level by one below 1.0. A breaking commit releases a minor, and a `feat` commit a patch. A hand-written changeset is released at the level it names.
|
|
77
|
+
2. The `ApiContract` check requires only a minor for a breaking API diff.
|
|
78
|
+
3. Commit-generated bumps stop below 1.0.0. Crossing it takes a hand-written `major` changeset, and the workflow only posts a notice when its release pull request is opened. A person merges every release pull request.
|
|
79
|
+
|
|
80
|
+
Experimental and Private tiers behave the same before and after 1.0.
|
|
81
|
+
|
|
82
|
+
[`SECURITY.md`](SECURITY.md#supported-versions) covers the separate question of which versions get security fixes.
|
|
@@ -1,40 +1,49 @@
|
|
|
1
1
|
# Reading the benchmarks
|
|
2
2
|
|
|
3
|
-
This guide is for the person **reading** `BENCHMARKS.md
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
what
|
|
3
|
+
This guide is for the person **reading** `BENCHMARKS.md`. That reader may be:
|
|
4
|
+
|
|
5
|
+
- a platform engineer deciding whether to run this package;
|
|
6
|
+
- a CTO estimating what it adds to the infrastructure bill;
|
|
7
|
+
- a maintainer checking a change.
|
|
8
|
+
|
|
9
|
+
It explains why the benchmarks exist, how to read each number, and, as important, what you must **not** do
|
|
10
|
+
with them.
|
|
7
11
|
|
|
8
12
|
Its companion, [WRITING-BENCHMARKS.md](WRITING-BENCHMARKS.md), explains how benchmarks are written and
|
|
9
13
|
documented.
|
|
10
14
|
|
|
11
15
|
## Why we benchmark
|
|
12
16
|
|
|
13
|
-
Every dependency has a price. Most of it is invisible until it is large:
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
+
Every dependency has a price. Most of it is invisible until it is large:
|
|
18
|
+
|
|
19
|
+
- a few milliseconds added to every request;
|
|
20
|
+
- a CPU core that is busier than it needs to be;
|
|
21
|
+
- a function that is fast on ten items and unusable on ten thousand;
|
|
22
|
+
- a cold start that grows with every module.
|
|
23
|
+
|
|
24
|
+
By then it is expensive to remove.
|
|
17
25
|
|
|
18
26
|
The benchmarks make that price visible **before** you adopt the package and **every time** it changes.
|
|
19
|
-
They lean toward costs on purpose. The rest of the repository explains the benefits
|
|
27
|
+
They lean toward costs on purpose. The rest of the repository explains the benefits. This report tells
|
|
20
28
|
you what you are paying for them, in time and in money, and how the bill grows with your workload.
|
|
21
29
|
|
|
22
30
|
## What is in the report
|
|
23
31
|
|
|
24
32
|
`BENCHMARKS.md` has four parts, in the order a decision-maker needs them:
|
|
25
33
|
|
|
26
|
-
1. **What adopting this package costs.** The headline
|
|
27
|
-
a typical workload and at the largest tested size
|
|
28
|
-
operations, and the code size every process must load. If you read nothing else, read
|
|
34
|
+
1. **What adopting this package costs.** The headline. It shows the added latency, CPU and memory per
|
|
35
|
+
operation, at a typical workload and at the largest tested size. It also shows an estimated compute
|
|
36
|
+
cost per million operations, and the code size every process must load. If you read nothing else, read
|
|
37
|
+
this.
|
|
29
38
|
2. **End-to-end.** The same operation done with no package and through the package, using empty or
|
|
30
|
-
minimal functions on both sides
|
|
31
|
-
report's environment section lists the sizes actually used
|
|
32
|
-
nothing of your own application mixed in.
|
|
33
|
-
3. **Function by function.** Every function measured alone across the same sizes
|
|
34
|
-
benchmarked, what poor performance would mean, its documented big-O
|
|
35
|
-
variable that could change its cost, and what was deliberately not covered.
|
|
39
|
+
minimal functions on both sides. It is measured at every size of the ladder: ten doubling sizes by
|
|
40
|
+
default, and the report's environment section lists the sizes actually used. The difference is the
|
|
41
|
+
package's total impact, with nothing of your own application mixed in.
|
|
42
|
+
3. **Function by function.** Every function measured alone across the same sizes. Each comes with why it
|
|
43
|
+
is benchmarked, what poor performance would mean, and its documented big-O with the reason for it. It
|
|
44
|
+
also lists every variable that could change its cost, and what was deliberately not covered.
|
|
36
45
|
4. **What makes up one operation.** Each function's cost, multiplied by how often one operation calls
|
|
37
|
-
it,
|
|
46
|
+
it, as a share of the whole operation. Nested calls are counted once. This is where the cost actually
|
|
38
47
|
lives.
|
|
39
48
|
|
|
40
49
|
## How to read a table
|
|
@@ -42,7 +51,7 @@ you what you are paying for them, in time and in money, and how the bill grows w
|
|
|
42
51
|
| Column | Meaning |
|
|
43
52
|
| --------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
44
53
|
| size (`n`) | The workload size for that row. The unit (record, call, file...) is stated at the top of the section. |
|
|
45
|
-
| Median | The middle sample.
|
|
54
|
+
| Median | The middle sample. Unmoved by the occasional slow outlier; the number to quote. |
|
|
46
55
|
| p95 | 95% of samples were this fast or faster. The gap between median and p95 is the variability you should plan for. |
|
|
47
56
|
| CPU (median) | CPU time actually spent. Can exceed wall time slightly because of background compilation and garbage collection threads. |
|
|
48
57
|
| Heap Δ | Memory the operation left allocated (can be negative when garbage was collected during the sample). |
|
|
@@ -51,12 +60,12 @@ you what you are paying for them, in time and in money, and how the bill grows w
|
|
|
51
60
|
|
|
52
61
|
## Big-O and the growth exponent
|
|
53
62
|
|
|
54
|
-
Big-O describes how cost grows as size grows
|
|
55
|
-
you what happens at a size you did not test. The report fits a line through the log of time against
|
|
56
|
-
|
|
57
|
-
per-call overhead plus work that grows, and the overhead hides the growth at small sizes
|
|
58
|
-
|
|
59
|
-
|
|
63
|
+
Big-O describes how cost grows as size grows. That is more useful than any single timing, because it tells
|
|
64
|
+
you what happens at a size you did not test. The report fits a line through the log of time against the
|
|
65
|
+
log of size across the **largest** sizes. On a long ladder that is the top five. A real cost is a fixed
|
|
66
|
+
per-call overhead plus work that grows, and the overhead hides the growth at small sizes. The slope is the
|
|
67
|
+
**growth exponent**, snapped to the nearest class. The fit's R² is reported next to it. When the cost
|
|
68
|
+
moves but does not follow a power law (a step, a cache cliff), no class is claimed at all. The classes:
|
|
60
69
|
|
|
61
70
|
| Exponent (about) | Class | What doubling the size does to the time |
|
|
62
71
|
| ---------------- | -------------- | --------------------------------------- |
|
|
@@ -69,19 +78,19 @@ the cost moves but does not follow a power law (a step, a cache cliff) no class
|
|
|
69
78
|
|
|
70
79
|
Each function shows its **documented** class next to its **measured** class with an agreement label:
|
|
71
80
|
|
|
72
|
-
- ✅ **matches
|
|
73
|
-
- 🟡 **close
|
|
81
|
+
- ✅ **matches**: the code behaves as documented.
|
|
82
|
+
- 🟡 **close**: neighbouring classes. Constant vs logarithmic and linear vs linearithmic cannot be
|
|
74
83
|
reliably separated on a bounded ladder; treat it as a match unless the exponent has moved.
|
|
75
|
-
- ⚠️ **differs
|
|
84
|
+
- ⚠️ **differs**: two or more classes apart. Either the documentation or the code is wrong. This is
|
|
76
85
|
the most useful line in the report.
|
|
77
|
-
- ❔ **not enough data
|
|
86
|
+
- ❔ **not enough data**: fewer than two usable sizes; something failed.
|
|
78
87
|
|
|
79
88
|
## The dollar figures
|
|
80
89
|
|
|
81
90
|
Time becomes money through two bracketing price shapes, because platforms bill differently:
|
|
82
91
|
|
|
83
|
-
- **Low:** CPU-priced compute (a container or VM billed per vCPU-hour)
|
|
84
|
-
- **High:** duration-and-memory-priced functions (billed per GB-second)
|
|
92
|
+
- **Low:** CPU-priced compute (a container or VM billed per vCPU-hour): you pay for CPU seconds.
|
|
93
|
+
- **High:** duration-and-memory-priced functions (billed per GB-second): you pay for wall seconds
|
|
85
94
|
times the memory the platform reserves, never less than its minimum.
|
|
86
95
|
|
|
87
96
|
The rates are rounded published list prices and are **assumptions**; they drift with time and region.
|
|
@@ -91,8 +100,8 @@ Treat the result as an order-of-magnitude comparison ("fractions of a cent per m
|
|
|
91
100
|
|
|
92
101
|
## Results are not comparable between runs
|
|
93
102
|
|
|
94
|
-
This is the rule that matters most
|
|
95
|
-
|
|
103
|
+
This is the rule that matters most: **a benchmark number is a measurement of one run on one machine at one
|
|
104
|
+
moment**. It is not a property of the package.
|
|
96
105
|
|
|
97
106
|
- Do **not** compare numbers from different machines, operating systems, Node versions, or runners. A
|
|
98
107
|
laptop and a CI runner differ by large factors in both directions.
|
|
@@ -101,20 +110,20 @@ one moment.** It is not a property of the package.
|
|
|
101
110
|
timings by 10-40%.
|
|
102
111
|
- Do **not** compare one package's numbers with another package's. The workloads differ; the
|
|
103
112
|
comparison means nothing.
|
|
104
|
-
- Do **not** treat any single number as a promise. It is not a
|
|
113
|
+
- Do **not** treat any single number as a promise. It is not a commitment, a service-level agreement or
|
|
105
114
|
a competitive claim.
|
|
106
115
|
|
|
107
116
|
What the numbers **are** good for:
|
|
108
117
|
|
|
109
|
-
- **Change over time on comparable hardware
|
|
118
|
+
- **Change over time on comparable hardware**. The same suite on the same kind of machine before and
|
|
110
119
|
after a change shows whether the change made the package slower. Look for movement larger than the
|
|
111
120
|
run-to-run noise.
|
|
112
|
-
- **Shape, not
|
|
113
|
-
than a 15% shift in a timing
|
|
121
|
+
- **Shape, not only level**. A change in growth class (O(n) becoming O(n²)) is a much bigger signal
|
|
122
|
+
than a 15% shift in a timing. It is also far less sensitive to the machine than a time is, because it
|
|
114
123
|
is a ratio between sizes on the same run. It still assumes the same sizes, workload and runtime.
|
|
115
|
-
- **Relative cost between functions
|
|
124
|
+
- **Relative cost between functions**. Which function dominates the overhead is usually the same on
|
|
116
125
|
any machine, even when the absolute times are not.
|
|
117
|
-
- **Order of magnitude
|
|
126
|
+
- **Order of magnitude**. Microseconds, milliseconds or seconds per operation: that distinction is
|
|
118
127
|
stable enough to plan with.
|
|
119
128
|
|
|
120
129
|
## What the numbers do not prove
|
|
@@ -126,11 +135,11 @@ What the numbers **are** good for:
|
|
|
126
135
|
power law. A high R² does not show the class is right, and a low one means no class is reported.
|
|
127
136
|
- **The overhead is a difference.** `end-to-end:overhead` is not measured; it is the with-package
|
|
128
137
|
time minus the baseline time at each size, never below zero. A difference of two noisy medians is
|
|
129
|
-
noisier than either
|
|
130
|
-
|
|
131
|
-
groups, and the "What the package adds" table, instead.
|
|
138
|
+
noisier than either. The history page therefore shows it for reading only and gives it no complexity
|
|
139
|
+
class. The pull-request summary does not report a class shift for it. Read the baseline and
|
|
140
|
+
with-package groups, and the "What the package adds" table, instead.
|
|
132
141
|
- **A relative cost is still a measurement.** Dividing by the baseline removes much of the machine,
|
|
133
|
-
not all of it
|
|
142
|
+
not all of it. Compilation, garbage collection and contention on the runner still move it. Below the
|
|
134
143
|
timer-noise floor the percentage is not shown.
|
|
135
144
|
- **One run per commit is not a repeat.** The history holds one measurement for each recorded commit.
|
|
136
145
|
A trend is more convincing than a point; a single point moving is ordinary noise.
|
|
@@ -138,60 +147,73 @@ What the numbers **are** good for:
|
|
|
138
147
|
## Noise
|
|
139
148
|
|
|
140
149
|
Even on a quiet machine, repeated runs differ. Very fast operations (microseconds and below) are the
|
|
141
|
-
noisiest in relative terms
|
|
150
|
+
noisiest in relative terms. The engine batches them and reports the per-operation value, but a 30-40%
|
|
142
151
|
swing between runs on those is ordinary. Slow operations are steadier. Read the median and p95
|
|
143
152
|
together, and trust a trend over a single run. The pull-request summary shows the change against the last
|
|
144
|
-
run, but never judges it: a raw comparison is between two runs on different machines. What can fail a pull
|
|
145
|
-
baseline-relative ratios
|
|
146
|
-
requests and the history page".
|
|
153
|
+
run, but never judges it: a raw comparison is between two runs on different machines. What can fail a pull
|
|
154
|
+
request is limited to growth classes and baseline-relative ratios. Those are far less sensitive to the
|
|
155
|
+
machine than a raw time (see "History, pull requests and the history page").
|
|
147
156
|
|
|
148
157
|
## How the documentation is done
|
|
149
158
|
|
|
150
159
|
Documentation is part of the benchmark definition, not an afterthought, and the suite will not load
|
|
151
160
|
without it. For every function the report prints, in this order:
|
|
152
161
|
|
|
153
|
-
1. **Why we benchmark it
|
|
154
|
-
2. **What poor performance would mean
|
|
155
|
-
3. **Expected growth
|
|
156
|
-
4. **Variables that could change its cost
|
|
162
|
+
1. **Why we benchmark it**: where it runs, how often, and who feels it.
|
|
163
|
+
2. **What poor performance would mean**: the consequence in latency, capacity and spend.
|
|
164
|
+
3. **Expected growth**: the big-O and the mechanism that causes it.
|
|
165
|
+
4. **Variables that could change its cost**: each one swept, measured as a variant, or fixed at a
|
|
157
166
|
stated value.
|
|
158
|
-
5. **Deliberately not covered
|
|
159
|
-
6. **The measurements
|
|
167
|
+
5. **Deliberately not covered**: the honest gaps, with reasons.
|
|
168
|
+
6. **The measurements**: every size, median, p95, CPU, memory, throughput, and the measured class next
|
|
160
169
|
to the documented one.
|
|
161
170
|
|
|
162
171
|
If a section reads as vague, that is a documentation bug: open an issue or fix it.
|
|
163
172
|
|
|
164
173
|
## History, pull requests and the history page
|
|
165
174
|
|
|
166
|
-
When a pull request touches benchmarked code, CI re-runs the suite and posts a summary
|
|
167
|
-
change first, then any function whose growth class shifted, then ordinary timing movements
|
|
168
|
-
their thresholds. Three gates can fail the run. None of them compares a raw time with another run's,
|
|
169
|
-
two runs are on different machines.
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
175
|
+
When a pull request touches benchmarked code, CI re-runs the suite and posts a summary. The summary shows
|
|
176
|
+
the end-to-end change first, then any function whose growth class shifted, then ordinary timing movements
|
|
177
|
+
beyond their thresholds. Three gates can fail the run. None of them compares a raw time with another run's,
|
|
178
|
+
because two runs are on different machines.
|
|
179
|
+
|
|
180
|
+
Two compare the measured growth class, which describes the shape of the curve and not the speed of the
|
|
181
|
+
machine:
|
|
182
|
+
|
|
183
|
+
- against the documented big-O: a function whose class **differs** fails;
|
|
184
|
+
- against the class at the last release: a function whose class got **worse** fails, unless
|
|
185
|
+
`.repo-contract/exceptions/benchmark-complexity.json` records a person's acceptance of it.
|
|
186
|
+
|
|
173
187
|
The third is a ratio: the package's overhead _relative to its own baseline_, measured in the same run,
|
|
174
|
-
growing past an allowance
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
188
|
+
growing past an allowance. The allowance is 50% for millisecond-scale work and 70% for microsecond-scale
|
|
189
|
+
work, which is noisier. The baseline and the package are sampled interleaved, so the overhead is a
|
|
190
|
+
per-pair difference with a 95% confidence interval. A growth that sits inside the intervals of the two runs
|
|
191
|
+
compared is reported as a note, never a failure. What the gate judges is main measured on the same runner
|
|
192
|
+
in the same job. The committed results of main and of the last release are judged only when they came from
|
|
193
|
+
the same CPU model. The reason: for fast work, the ratio also depends on how much of a bare baseline is
|
|
194
|
+
process start-up.
|
|
195
|
+
|
|
196
|
+
A function whose big-O could not be determined (too few sizes, or no difference between them) is a
|
|
197
|
+
**warning**. Nothing is known to be wrong, but nothing is checked. Raw time against another run is shown,
|
|
198
|
+
never judged. A failed gate fails the benchmark job; whether a red job blocks the merge is decided by the
|
|
199
|
+
repository's ruleset, not by the gate.
|
|
200
|
+
|
|
201
|
+
After a merge, a job measures the merge commit and records it through its own small pull request. The
|
|
202
|
+
record is the results file and a history entry naming the commit, pull request and version. Each entry
|
|
203
|
+
also keeps how noisy that run was (standard deviation, p95 and iteration count for every size) and the
|
|
204
|
+
processor it ran on. A reader can then judge later whether a change is larger than the run's own spread. A
|
|
205
|
+
generated history page opens with why the benchmarks exist and what they do and do not show. It then
|
|
206
|
+
charts every function across runs, with every figure also available as a table. It shows the baseline,
|
|
207
|
+
the with-package time and the difference side by side.
|
|
186
208
|
|
|
187
209
|
## Glossary
|
|
188
210
|
|
|
189
|
-
- **Operation
|
|
190
|
-
- **Size / `n
|
|
191
|
-
- **Baseline
|
|
192
|
-
- **Overhead / added
|
|
211
|
+
- **Operation**: one unit of the work the package performs, as defined by the suite.
|
|
212
|
+
- **Size / `n`**: how much that operation processes (records, calls, files).
|
|
213
|
+
- **Baseline**: the same operation with no package, using an empty or minimal function.
|
|
214
|
+
- **Overhead / added**: the package's own cost, with-package time minus baseline time. Derived from the
|
|
193
215
|
two measured groups, never below zero, and not itself classified.
|
|
194
|
-
- **Median / p95
|
|
195
|
-
- **Heap
|
|
196
|
-
- **Batch
|
|
197
|
-
- **Cold start
|
|
216
|
+
- **Median / p95**: the middle sample, and the value 95% of samples beat.
|
|
217
|
+
- **Heap Δ**: memory the operation left allocated.
|
|
218
|
+
- **Batch**: the number of back-to-back repetitions inside one sample for very fast operations.
|
|
219
|
+
- **Cold start**: the cost of loading and initializing the package in a fresh process.
|
package/benchmarks/README.md
CHANGED
|
@@ -1,10 +1,13 @@
|
|
|
1
1
|
# Benchmarks
|
|
2
2
|
|
|
3
|
-
This folder is the performance and cost record of the package
|
|
4
|
-
money, how each function scales, and where the cost comes from. It is built entirely by
|
|
5
|
-
[`internal-package-contract`](https://github.com/MaverickCER/internal-package-contract):
|
|
6
|
-
|
|
7
|
-
|
|
3
|
+
This folder is the performance and cost record of the package. It shows what adopting it costs in
|
|
4
|
+
time and money, how each function scales, and where the cost comes from. It is built entirely by
|
|
5
|
+
[`internal-package-contract`](https://github.com/MaverickCER/internal-package-contract):
|
|
6
|
+
|
|
7
|
+
- the package supplies **what to run** (an input);
|
|
8
|
+
- the contract **stresses it through ten sizes, measures it, checks the big-O and prices it**;
|
|
9
|
+
- the contract writes a **report** and a machine-readable **result** (the output).
|
|
10
|
+
|
|
8
11
|
Nothing in this folder implements timing, statistics or reporting.
|
|
9
12
|
|
|
10
13
|
## Read first
|
|
@@ -96,8 +99,8 @@ export default defineSuite({
|
|
|
96
99
|
| `inEndToEnd` | no | How often one end-to-end operation calls the function, for the attribution section. |
|
|
97
100
|
|
|
98
101
|
`expectedComplexity` is one of `constant`, `logarithmic`, `linear`, `linearithmic`, `quadratic`,
|
|
99
|
-
`exponential-or-worse`. Return a value that depends on the work
|
|
100
|
-
nothing is optimized away
|
|
102
|
+
`exponential-or-worse`. Return a value that depends on the work; the contract keeps it alive so
|
|
103
|
+
nothing is optimized away. Return `timed(ms)` only when an operation is timed elsewhere, such as a
|
|
101
104
|
child process.
|
|
102
105
|
|
|
103
106
|
## The output: `results.json`
|
|
@@ -121,6 +124,6 @@ analysis complexity measured class, exponent and notation per group, with
|
|
|
121
124
|
cost the typical and largest sizes used for the headline numbers
|
|
122
125
|
```
|
|
123
126
|
|
|
124
|
-
|
|
125
|
-
**change over time on comparable hardware** and at **changes in growth class**, never at one absolute
|
|
127
|
+
Results are not comparable between machines or runs. History and pull-request summaries therefore look
|
|
128
|
+
at **change over time on comparable hardware** and at **changes in growth class**, never at one absolute
|
|
126
129
|
number. See [READING-BENCHMARKS.md](READING-BENCHMARKS.md).
|