cdk-preflight 0.0.174 → 0.0.176

package/AGENTS.md

@@ -10,7 +10,7 @@ Scope: **every deploy-time failure that no existing CDK mechanism already catche
 
 Security and compliance policy (cdk-nag, Control Tower proactive controls) stays out under the same rule rather than as a special exception: those mechanisms exist, so that gap is not ours to fill.
 
-Growth is the normal state. The upstream engine ships slowly and carefully by design; this pack does not have to, so rules land here continuously and are deleted without ceremony once the engine covers them (see [Rule lifecycle](#rule-lifecycle-add-fast-retire-without-ceremony)). A shrinking rule count is neither a goal nor a problem.
+Growth is the normal state. The upstream engine ships slowly and carefully by design; this pack does not have to, so rules land here continuously, stop being evaluated on engines that cover them, and are deleted without ceremony once no supported `aws-cdk-lib` bundles an engine that lacks them (see [Rule lifecycle](#rule-lifecycle-add-fast-retire-without-ceremony)). A shrinking rule count is neither a goal nor a problem.
 
 ## Design principles (do not violate)
 
@@ -18,7 +18,7 @@ Growth is the normal state. The upstream engine ships slowly and carefully by de
 2. **Every rule ships with proof.** `templates/fail.template.json` must violate exactly this rule; `templates/pass.template.json` must be clean. `meta.yaml#repro` records how the deploy-time failure was verified (`real-deploy` / `research-case` / `doc-only` — the last one requires an explanation in `evidence`).
 
    **Both fixtures sit on the boundary.** For an ordered constraint (a number, a length, a count) the fail template carries the *tightest violating* value and the pass template the *exact limit* — a 20-character minimum gets a 19-character fail and a 20-character pass, not 5 and 26. A slack pair proves only the direction of the rule: `count(v) < 20` still fires on a 5-character fail template when the constant is wrong (`< 10`), and a 26-character pass template stays silent even at `< 25`, so neither the constant nor the comparison operator is pinned by the fixtures. It also weakens the real-deploy gate, which is the only thing that establishes where AWS's limit actually is — a 19-character key that fails CREATE proves the documented 20, a 5-character one proves "somewhere above 5". A range the rule checks at both ends gets a resource at each end (the fixture assertion is "at least one diagnostic for this rule", so several violating resources in one fail template are fine, and a branch a single fixture cannot reach gets a targeted case in `test/rules.test.ts`). An end the engine's schema already enforces stays out of the fail template — the rule does not check it (principle 1) and the duplication guard would flag it. **The threshold does not have to be a literal.** When the limit is another property's value — `MinValue` against `MaxValue`, a container's `Memory` against the task's — the same rule applies: the fail template carries the two values one apart in the violating direction and the pass template makes them equal. For a comparison that already violates at equality (`>=`, `<=`) the two swap: the fail sits at equality and the pass is one apart. `MinValue 10 / MaxValue 1` pins nothing — the rule could read `mn > mx + 5` and still fire, and a pass of `1 / 10` stays silent whether the comparison is `>` or `>=`. Constraints with no ordering — a missing property, a wrong enum member, a cross-resource mismatch — have no boundary to sit on. The evidence line records the values that were actually deployed, so it pins the boundary too: `bench 2026-09-13 us-east-1: 19-char Value -> "API Key value should be at least 20 characters" (ROLLBACK_COMPLETE); pass 20-char -> CREATE_COMPLETE`.
-3. **Upstream contribution is welcome, never a gate.** Constraints expressible in schemas or generic engine rules make good PRs to [cloudformation-validate](https://github.com/aws-cloudformation/cloudformation-validate) (open an issue first) — but a rule ships here first and never waits on one. Track status in `meta.yaml#upstream` (`none` / `pending-engine` / `engine-pr` / `retired`); the field exists so that retirement is bookkeeping rather than archaeology.
+3. **Upstream contribution is welcome, never a gate.** Constraints expressible in schemas or generic engine rules make good PRs to [cloudformation-validate](https://github.com/aws-cloudformation/cloudformation-validate) (open an issue first) — but a rule ships here first and never waits on one. Track status in `meta.yaml#upstream` (`none` / `pending-engine` / `cfn-schema` / `engine-pr` / `retired`); the field exists so that retirement is bookkeeping rather than archaeology. `retired` always comes with `meta.yaml#supersededBy` (see [Rule lifecycle](#rule-lifecycle-add-fast-retire-without-ceremony)).
 4. **Tests are the contract.** Never merge with a red test; never weaken an assertion to make it pass. New behavior needs a new test first.
 5. **The boundary is the engine, not the CDK L2 layer.** Rules validate synthesized templates, so an L2 construct that validates (or structurally prevents) the same mistake does not make a rule redundant — L1 usage, escape hatches, `addPropertyOverride`, and externally generated templates all bypass L2. Overlapping an L2 guard is fine and expected; overlapping the bundled engine is forbidden (principle 1). L2 coverage is not a retirement trigger either — deleting a rule because an L2 construct checks it would strip the coverage from exactly the users this pack exists for. Only a layer that sees the same synthesized template retires a rule: the bundled engine, or CloudFormation's server-side validation. Noting an L2 overlap in the PR is useful context, not something to justify. See "Where this pack sits among validation layers" below.
 
@@ -32,7 +32,7 @@ Five other validation layers sit around a CDK app. Exactly one of them is a boun
 |---|---|---|---|
 | CDK L2 construct validation | synth, only on that construct's prop path | per-construct guards (e.g. `Volume` iops checks, SNS auto-`.fifo`, Fargate hardcoding `awsvpc`) | **Neither a boundary nor a retirement trigger** (principle 5). Bypassed by L1 / escape hatches / external templates, so its coverage is not our coverage. Overlap allowed and unremarkable |
 | CDK L1 generated validators (`CfnXxx`) | synth, every CDK app | type and required-property checks only — never value ranges, patterns, or cross-field rules | **No practical overlap**: pack rules are value / cross-field constraints by construction |
-| Bundled engine default rules (`@aws/cloudformation-validate`: SCHEMA / CFN_LINT / ENGINE) | synth, via this plugin | patched registry-schema ranges/patterns/enums + cfn-lint rules | **The hard boundary** (principle 1) **and the only retirement trigger**. If the bare engine reports ERROR/FATAL on the minimal violating template, the rule must not exist — enforced by the jest duplication guard, which turns red by itself when the engine catches up |
+| Bundled engine default rules (`@aws/cloudformation-validate`: SCHEMA / CFN_LINT / ENGINE) | synth, via this plugin | patched registry-schema ranges/patterns/enums + cfn-lint rules | **The hard boundary** (principle 1) **and the only retirement trigger**. If the bare engine reports ERROR/FATAL on the minimal violating template, a *new* rule must not exist — enforced by the jest duplication guard. When a later engine catches up with an *existing* rule, the same guard turns red until the rule records `supersededBy` (it stays for users on older `aws-cdk-lib`) |
 | cfn-lint (external CLI) | outside the synth path (a separate CI step, if the user runs one) | near-parity with the bundled engine | **Not a boundary** — never disqualifies a rule. A check cfn-lint has but the engine lacks is an upstream-PR candidate *for the engine* (project policy: contribute to cloudformation-validate, not to cfn-lint) |
 | CloudFormation server-side pre-deploy validation | CreateStack / UpdateStack / CreateChangeSet, before resources are touched | property syntax against RAW registry schemas for non-excluded types; resource name conflicts | **Almost no overlap** with this pack's domain: ~420 resource types are excluded (IAM, EC2 SG, ECS, RDS, SFN, …) and raw schemas carry none of the doc-only limits — 0/16 real-world cases detected (measured 2026-09, cdk-validation-gap-research). **Not a retirement trigger** — this is the one place where a layer that *blocks the deploy* still does not make a rule redundant, because what it returns is not equivalent. A raw-schema violation comes back as a stack-level `Validation failed with 1 error(s). Call DescribeEvents to retrieve the full list of issues…`, which **names no property**: the developer has to make a second API call to find out what was wrong. A rule here fails `cdk synth` instead, before any AWS call, naming the property path. Audited 2026-09-17: 36 shipped rules sit in this band and 30+ of their `repro.evidence` lines record that sentence verbatim. They carry `upstream: cfn-schema` |
 | CloudFormation + service APIs | deploy (CREATE/UPDATE) | everything else: cross-field rules, service-side business rules, quotas | **The target, not a duplicate to avoid.** A rule exists *iff* it front-runs a real deploy-time failure here; `meta.yaml#repro` (real-deploy gate) proves that equivalence |
@@ -72,18 +72,27 @@ The two failure modes are watched separately: **staleness** (the constraint disa
 
 **Adding.** Two gates, both mechanical, and nothing else can reject a candidate: the duplication guard (`test/rules.shard*.test.ts`) and the real-deploy gate (`bench/verify-rule.sh`). The real-deploy gate is the one thing that did *not* get looser with the widened scope, and deliberately so — `enforce` is the default mode, so a false positive is a hard synth failure in someone's CI. The CloudFront survey is the standing evidence: 3 of 9 documented constraints deployed clean (see "A doc sentence is a hypothesis" above). Volume comes from running the bench over more services, never from lowering that bar.
 
-**Retiring.** The trigger is the bundled engine starting to block the constraint — never CloudFormation's server-side validation (see the layer table: it blocks the deploy but names no property), never an L2 construct, never cfn-lint. You do not have to watch for it; two mechanisms report it:
+**Retiring.** The trigger is the bundled engine starting to block the constraint — never CloudFormation's server-side validation (see the layer table: it blocks the deploy but names no property), never an L2 construct, never cfn-lint. "Block" means the engine reports ERROR/FATAL **at every location the rule reports** on its fail template (same logical id, same or nested property path). An engine error elsewhere in the template does not count: a fail template that lists several cases can have one of them caught by an unrelated engine rule (it happened: E9002 caught the FromPort > ToPort case of `pf-ec2-sg-port-range` and none of its out-of-range cases). You do not have to watch for it; two mechanisms report it:
 
-- The jest duplication guard turns red on its own the first time an `aws-cdk-lib` bump brings an engine that reports ERROR/FATAL on a rule's fail template, and its failure message names the rule to delete.
-- `npx projen redundancy-scan` lists every such rule at once (no AWS account, no cost — it just re-runs each fail template through the bare engine). The monthly `monthly-verify` workflow runs it and files the results under a **REDUNDANT** heading in the same issue that reports drift, so obsolescence and staleness arrive on one page.
+- The jest duplication guard turns red on its own the first time an `aws-cdk-lib` bump brings an engine that reports ERROR/FATAL on a rule's fail template, and its failure message names the rule.
+- `npx projen redundancy-scan` lists every such rule at once (no AWS account, no cost — it re-runs each fail template through the engine). The monthly `monthly-verify` workflow runs it and files the results under **REDUNDANT** headings in the same issue that reports drift, so obsolescence and staleness arrive on one page.
 
-Then:
+A retired rule is **not deleted while any supported `aws-cdk-lib` still bundles an engine that lets the mistake through**. The peerDependency floor (`cdkVersion` in `.projenrc.ts`, currently older than the `aws-cdk-lib` used for development) decides who is supported. So the steps are:
 
-1. `rm -rf rules/<service>/<rule-id>/` — delete outright, no tombstone. `upstream: retired` exists for a rule mid-flight, not for a corpse.
-2. `npx projen bundle-rules` (regenerates `src/rules.generated.ts`; the file is gitignored, so nothing to commit).
-3. One line in the PR body naming the engine rule id that replaced it.
+1. `npx projen redundancy-scan --bisect --write` — for every rule the current engine fully covers, installs each published `@aws/cloudformation-validate` newer than the floor's engine, finds the first version from which the engine covers it for good, and writes it to `meta.yaml`:
 
-Users pinned to an older `aws-cdk-lib` keep the old behavior by staying on the older cdk-preflight, so a deletion never leaves them uncovered.
+   ```yaml
+   upstream: retired
+   supersededBy:
+     engine: 1.8.0-beta   # first engine version that covers the fail template
+     cdk: 2.268.0         # first aws-cdk-lib that bundles that engine or a later one
+     engineRules: [E3510] # the engine rules that replace this one
+   ```
+
+   A rule the scan lists as **partial** (the engine blocks some of the fail template's cases, not all) is not retired: move the covered cases out of the fail template and keep the rule, noting in `rule.rego` which branch newer engines duplicate.
+2. At synth time, `Preflight.apply` drops a retired rule when the running engine is at least `supersededBy.engine` **and** dropping it does not change whether synth fails: in observe mode (both findings are warnings), with `strict: true` (the engine's ERROR/FATAL fails synth), or when the app sets the context `@aws-cdk/core:validateAgainstDefaultRules` (the CDK then keeps built-in errors as errors) without `CDK_VALIDATION=false`. In the default enforce mode the CDK downgrades built-in ERROR/FATAL to warnings, so the rule keeps running there — the user sees the engine's warning next to the rule's error, which is the price of keeping the build red. An unreadable engine version drops nothing.
+3. The duplication guard keeps checking a retired rule: the current engine must still cover every location the rule reports (a regression or a wrong record turns it red), and the floor must still be below `supersededBy.cdk`.
+4. When the floor is raised to or past `supersededBy.cdk`, that guard turns red and the scan lists the rule as **deletable**: `rm -rf rules/<service>/<rule-id>/` — delete outright, no tombstone — and `npx projen bundle-rules`. Name the engine rule ids from `engineRules` in the PR body.
 
 **Release contract.** New rules land in **minor** releases. Because `enforce` is the default, a minor upgrade can newly fail a `cdk synth` that passed yesterday — that is the intended behavior of a pack whose whole purpose is to grow, not a regression to be apologised for. The documented escape hatches are: pin the version for a frozen rule set, `exclude: ['<rule-id>']` for one rule, `enforce: false` for warn-only. Keep that sentence in the README whenever it is reworded.
 
@@ -92,7 +101,8 @@ Users pinned to an older `aws-cdk-lib` keep the old behavior by staying on the o
 ```
 rules/<service>/<rule-id>/
   rule.rego                 # package cdk_preflight / import rego.v1 / violation contains make_diag_full(...)
-  meta.yaml                 # id, service, resourceTypes, severity, title, constraintSource, upstream, repro
+  meta.yaml                 # id, service, resourceTypes, severity, title, constraintSource, upstream, repro,
+                            # supersededBy (retired rules only)
   templates/fail.template.json
   templates/pass.template.json
 rules/_lib/<name>.rego      # shared helpers (never emit violations); loaded before every rule, in every mode,

package/API.md

@@ -92,6 +92,11 @@ public readonly strict: boolean;
 
 In enforce mode, additionally fail synthesis on error-class findings (severity ERROR/FATAL) of the built-in validation engine itself, e.g. schema violations like `F3034`. This is the workaround for the CDK behavior where all built-in findings are downgraded to warnings.
 
+Since the engine's own errors then fail synthesis, bundled rules that the
+running engine already covers are skipped instead of being reported twice.
+The same happens with `enforce: false`, and when the app sets the context
+`@aws-cdk/core:validateAgainstDefaultRules: true`.
+
 Only effective in enforce mode (the default).
 
 ---

package/README.md

@@ -19,7 +19,7 @@ Some CloudFormation constraints are not expressed in resource provider schemas
 
 cdk-preflight is a curated [Rego rule pack](docs/rules.md) for exactly those constraints, evaluated with the CloudFormation validation engine that ships inside `aws-cdk-lib` (>= 2.267.0). By default a violation **fails `cdk synth`** — a template that is known to fail at deploy time never leaves your machine.
 
-The pack aims at **every deploy-time failure that no existing CDK mechanism already catches** — nothing narrower. Every bundled rule is backed by a `fail`/`pass` template pair, and the failure has been reproduced against real AWS. The handful of rules that could not be reproduced are marked `doc-only` and report as **warnings**, as does a rule whose remaining false positives cannot be told from the template (a cross-account Lambda layer the owner may have shared): they show up in the validation report but never fail synth. Rules that the built-in validation engine already covers are deliberately **not** duplicated — a test suite enforces this.
+The pack aims at **every deploy-time failure that no existing CDK mechanism already catches** — nothing narrower. Every bundled rule is backed by a `fail`/`pass` template pair, and the failure has been reproduced against real AWS. The handful of rules that could not be reproduced are marked `doc-only` and report as **warnings**, as does a rule whose remaining false positives cannot be told from the template (a cross-account Lambda layer the owner may have shared): they show up in the validation report but never fail synth. Rules that the built-in validation engine already covers are deliberately **not** duplicated — a test suite enforces this, and a rule the engine catches up with later is skipped on engines that cover it (see [Scope and rule lifecycle](#scope-and-rule-lifecycle)).
 
 > **Requires `aws-cdk-lib` >= 2.267.0** (released 2026-08-27) — the first release that bundles
 > the CloudFormation validation engine. On older versions the rules cannot run at all.
@@ -288,7 +288,7 @@ CDK L2 construct validation is deliberately **not** one of those layers. `CfnXxx
 
 That makes growth the normal state, and it has a consequence worth knowing before you upgrade: **new rules land in minor releases, so a minor upgrade can newly fail a `cdk synth` that passed yesterday.** That is intended, not a regression. If you need a frozen rule set, pin the version; to drop a single rule, `exclude: ['<rule-id>']`; to see everything without failing the build, `enforce: false`.
 
-Rules move the other way too. Once the validation engine bundled in `aws-cdk-lib` (or CloudFormation's own pre-deploy validation) starts blocking a constraint, the rule is deleted rather than kept as a duplicate — staying on an older `aws-cdk-lib` and an older cdk-preflight keeps the old behavior.
+Rules move the other way too. Once the validation engine bundled in `aws-cdk-lib` starts blocking a constraint, the rule records the engine version that covers it and stops being evaluated on that engine or later — but only where that changes nothing about whether synth fails: with `enforce: false`, with `strict: true`, or when your app sets the context `@aws-cdk/core:validateAgainstDefaultRules: true`. In the default enforce mode the CDK reports its own engine's errors as warnings, so the rule keeps running and you may see both. Users on an older `aws-cdk-lib` keep the rule either way; it is deleted only once the minimum supported `aws-cdk-lib` bundles an engine that covers it.
 
 ## How it works