cdk-preflight 0.0.168 → 0.0.169

package/.claude/skills/add-preflight-rule/SKILL.md

@@ -29,6 +29,7 @@ cdk-preflight のルール追加パイプライン。AGENTS.md の設計原則
 4. **ローカルゲート**: まず `npx ts-node --transpile-only --project test/tsconfig.json scripts/rule-check.ts check <service|rule-id>...` を回す。`rules/` を直接読んで 1 エンジンに全ルールを載せ、fail が自分のルールで鳴るか / pass が全ルール無音か / どちらも組み込みエンジンに止められないかを返す（`bundle-rules` も meta.yaml の evidence も要らないので、実機ゲート前の直しはここで回す。80 本で数秒）。全部 `ok` になってから `npx projen bundle-rules && npx jest test/rules test/structure.test.ts`（`test/rules` は `rules.test.ts` と、ルール表を回す `rules.shard*.test.ts` の両方に当たる）（`bundle-rules` は `src/rules.generated.ts` / `docs/rules.md` に加えて **README のルール数バッジと対応リソースタイプ一覧も書き換える**。手で直さない。古いままなら `structure.test.ts` が落ちる）。**`rules/<service>/` を新設しても `npx projen` は要らない** — `.github/workflows/monthly-verify.yml` のサービス行列は実行時に `scripts/plan-services.py` が `rules/` を読んで組むので workflow ファイル自体は変わらない（#247 で実行時計算へ移行済み。2026-09-26 に `rules/verifiedpermissions/` を新設した PR #286 / #287 でも self-mutation は skipping のままだった）。2026-09-13 に Athena 追加で踏んだのは移行前の話。`structure.test.ts` は手順 3 の境界値を機械で見る（rego のしきい値ごとに fail へ「限界に最も近い違反値」、pass へ「限界そのもの」が現れるか）。しきい値の検出が意味を持たないルールだけ `rules/_boundary-exceptions.txt` に理由付きで逃がす。**jest は `-t` で対象を絞る**。フルスイートは PR 直前の 1 回だけでよく、実測では 401 回中 73 回がフル実行で合計 5.8 時間を溶かしている。
 5. **実機再現ゲート**: `bash bench/verify-rule.sh <rule-id>`（要 AWS 認証）。観測したエラーメッセージと日付を `meta.yaml#repro.evidence` に記録。
    - **evidence には実際にデプロイした値を書く**（`bench 2026-09-13 us-east-1: 19-char Value -> "...at least 20 characters" (ROLLBACK_COMPLETE); pass 20-char -> CREATE_COMPLETE`）。境界ちょうどのフィクスチャと組で、限界の位置そのものが meta.yaml から読める
+   - **evidence の拒否文から、制約を名指ししている部分を `repro.expect` に写す**（`expect: "must not be longer than 256 characters"`）。1 行のダブルクォート文字列で、中に `"` と `\` を入れない（月次ランナーは YAML パーサ無しに sed で読む）。evidence に逐語で含まれていないと `bundle-rules` が落ちる。月次はこの文面で照合し、別の理由で倒れたら INCONCLUSIVE にする（#264）。**2026-09-27 以降に足す real-deploy ルールでは必須**。サービスが汎用文しか返さない場合（`GeneralServiceException` など）は、その汎用文を写す
    - fail テンプレートがデプロイに**成功**したら、それはドキュメント側の誤り（BROKEN-EXPECTATION）。ルールを削除し、証拠を issue に残して終了する。CloudFront では明文化された制約 9 件中 3 件がこれだった（2026-09-02）
    - **予想と違う理由**で失敗した場合（他アカウントの ARN、ドメイン所有権の検証など）は証拠にならない。サービスエラーが対象の制約そのものを名指しするまでテンプレートを作り直すか、除去できない交絡は `evidence` に明記する
    - `doc-only` は「再現に安価に作れないリソース（検証済み ACM 証明書、所有ドメイン等）が要る」場合に限る最終手段であって、まだ試していない制約への近道ではない。詳細は AGENTS.md の "A doc sentence is a hypothesis, not evidence" に従う

package/.jsii

@@ -9600,6 +9600,6 @@
       "symbolId": "src/index:PreflightOptions"
     }
   },
-  "version": "0.0.168",
-  "fingerprint": "G3aMkBNCkKzel6i+CuxZmGgbZ9hnfizvksJTGdXg26M="
+  "version": "0.0.169",
+  "fingerprint": "IE1+ptdzBXn/2xOMLLAWbC+hDC40fi4VT/ojoRRLku4="
 }

package/AGENTS.md

@@ -163,7 +163,7 @@ If you are a subagent running one of these phases, do not spawn further agents.
    - A helper several rules of one service need (parsing an opaque DSL, enumerating nested scopes) goes in `rules/_lib/<service>.rego` with a `_pf_<service>lib_` prefix (`rules/_lib/sfn.rego` is the reference: ASL from `DefinitionString` *or* the L1 `Definition` object, nested Parallel/Map scopes to depth 3, effective QueryLanguage). Rules never reference another rule's helpers — `exclude` can unload any rule, a lib is always loaded.
    - **Cheap screen for definition-level constraints**: some services expose their create-time validator as a free API (`aws stepfunctions validate-state-machine-definition --type STANDARD|EXPRESS` is the same validator CreateStateMachine runs). Use it to triage doc hypotheses before spending a CloudFormation deploy on each — 100 Step Functions hypotheses took minutes (2026-09-05). The real-deploy gate stays.
 3. `npx ts-node --transpile-only --project test/tsconfig.json scripts/rule-check.ts check <service|rule-id>` while iterating: it reads `rules/` straight from disk into a single engine (no bundle, no meta validation, no jest) and reports, per rule, whether the fail template fires its own rule, the pass template is silent for every rule, and neither trips a built-in ERROR/FATAL. Then `npx projen bundle-rules` and `npx jest test/rules test/structure.test.ts` (the `test/rules` pattern picks up both `rules.test.ts` and the `rules.shard*.test.ts` files that carry the per-rule table) — the duplication guard and fixture checks run there for real. Prefer `jest -t` while iterating; the full suite is ~11,400 tests / ~2.5 min on 8 cores (measured 2026-09-14), so keep it for the pre-PR run.
-4. **Real-deploy gate**: `bash bench/verify-rule.sh <rule-id>` deploys the fail template (expects CREATE to fail; records the service error message) and, where cheap, the pass template (expects success, then deletes). Paste the observed error into `meta.yaml#repro.evidence` with the date. Only `doc-only` rules may skip this, with justification.
+4. **Real-deploy gate**: `bash bench/verify-rule.sh <rule-id>` deploys the fail template (expects CREATE to fail; records the service error message) and, where cheap, the pass template (expects success, then deletes). Paste the observed error into `meta.yaml#repro.evidence` with the date, then copy the part of it that names the constraint into `meta.yaml#repro.expect` — one line, `expect: "<fragment>"`, no `"` or `\` inside (the monthly runner reads it with `sed`), verbatim from the evidence (`bundle-rules` checks). Every monthly run matches the fail stack's reason against it, so a stack that falls over for anything else is INCONCLUSIVE rather than verified. Required for `real-deploy` rules added on or after 2026-09-27; when the service only ever says something generic (`GeneralServiceException`), quote that. Only `doc-only` rules may skip this, with justification.
 5. Update nothing else by hand, and **do not commit the generated artifacts** — a rule PR touches `rules/**` and
    nothing else. `src/rules.generated.ts` is gitignored and written by `npx projen bundle-rules`, which `preCompile`
    spawns on every build. `docs/rules.md` and the parts of `README.md` between the `<!-- supported-resources:start -->`
@@ -246,6 +246,7 @@ If you are a subagent running one of these phases, do not spawn further agents.
 - **Fixture name tagging must skip reference properties.** The generator that suffixes fixture names with the rule id has to key off the resource type: `ACLName` is the name on `AWS::MemoryDB::ACL` but a *reference* on `AWS::MemoryDB::Cluster`, and tagging it made every MemoryDB pass template fail with "ACL open-access-port not found" (measured 2026-09-06). Same for `ParameterGroupName` / `CacheParameterGroupName`.
 - **`bench/verify-rule.sh` polls for 60 minutes** (15 → 30 on 2026-09-06, → 60 on 2026-09-13): ElastiCache and MemoryDB clusters take 10-25 minutes to create and about as long to roll back, and an MSK Provisioned cluster took 31m34s to reach CREATE_COMPLETE at its minimum size; a shorter poll reported TIMEOUT / INCONCLUSIVE on rules whose constraint had already fired.
 - **A non-zero `aws` exit is not evidence of anything** (measured 2026-09-22, four parallel `verify-rule.sh` in us-east-1). `describe-stacks … 2>/dev/null || echo GONE` collapsed throttling, blips and expired credentials into `GONE`; `cleanup()` then deleted two `pf-servicediscovery-*` stacks mid-create, and in one of them `reason_of` captured the real failure a moment later — proof from the run's own log that the stack still existed. The same collapse on `describe-stack-events` is worse than losing a run: an unreadable reason leaves `FTYPE`/`REASON` empty, the scaffolding guard silently no-ops, and a VPC-quota rollback reports `OK: verified`. Only `ValidationError … does not exist` means gone. Everything else is retried inside the poll budget and then reported under its own reason (`could not read the … status` / `… events`), never as `GONE` and never as a dirty fixture; a `LEFTOVER` line likewise needs a confirmed-present stack, not just a `wait` that failed. `bench/verify-rule.test.sh` pins all of it with a stubbed `aws` on `PATH` — 13 of its assertions fail against the pre-fix script.
+- **A fail stack falling over is not evidence that the constraint fired** (measured 2026-09-25, #264). `verify-rule.sh` verified any rollback that did not look like a quota, and in the cloudformation monthly run only 9 of the 21 OKs named their constraint. Three holes. `reason_of` dropped the stack's own rows with `ResourceType!='AWS::CloudFormation::Stack'`, which also dropped a *nested* stack's real failure and picked up a sibling's collateral `Resource creation cancelled` instead; the stack's own rows are the ones whose `LogicalResourceId` is the stack name. A failure that belongs to no resource (an Output's Export) puts its reason only on the stack's `ROLLBACK_IN_PROGRESS` row under the default rollback. And a synchronous `ValidationError` from `create-stack` — the strongest evidence there is, with no scaffolding to collapse — was always INCONCLUSIVE, which pinned the cloudformation monthly at `overall=4`. Now an unreadable reason, a collateral cancellation or the `The following resource(s) failed to create` summary is INCONCLUSIVE, and with `repro.expect` set the reason, or the synchronous rejection, has to contain it. Replayed over the dev account's 90-day event history (3,110 stacks, 2026-09-26), the new reader changes the reason of those 5 cloudformation rules and of no other rule. The event reads also moved to `--output json`: with `--output text` the CLI applies `--query` page by page (100 events a page), so on a stack of more than 100 events `[-1]` answered once per page, and "no reason on either page" came back as two `None` lines, which is not `None` (measured 2026-09-27: `length(StackEvents)` is `100` and `16` in text, `116` in json).
 - **Probing the raw engine with custom rules**: the rule set must be named `violation` (`data.<pkg>.violation` is looked up; a `diagnostics` set fails with "not a valid rule path"), and hand-built diagnostic objects need lowercase severities (`error`, not `ERROR`/`CUSTOM` — those are display labels added by the wrapper). Pack rules never see this because `make_diag_full` handles the shape.
 - **`as` is a reserved keyword in the engine's Rego build** (measured 2026-09-06): `acts := object.get(s, "Action", null)` is fine but `as := ...` fails to parse ("unexpected keyword `as`") and, like every parse error, takes the whole custom package down — every rule stops firing, and the jest failure names an unrelated rule.
 - **A Managed Flink application deploys with no `ApplicationConfiguration` at all** (measured 2026-09-06), which is what makes `AWS::KinesisAnalyticsV2::Application` fixtures cheap: no S3 bucket, no JAR, no Glue database, just an IAM role the service can assume. Adding a "minimal" code stanza is worse than none — CreateApplication really fetches the S3 object (`We are unable to get the specified fileKey`) and rejects `PLAINTEXT` for a Flink runtime.

package/lib/index.js

@@ -17,7 +17,7 @@ const rules_generated_1 = require("./rules.generated");
  * Preflight.apply(app);
  */
 class Preflight {
-    static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-preflight.Preflight", version: "0.0.168" };
+    static [JSII_RTTI_SYMBOL_1] = { fqn: "cdk-preflight.Preflight", version: "0.0.169" };
     /**
      * Register the cdk-preflight rules on an App or Stage.
      */

package/package.json

@@ -89,7 +89,7 @@
   "publishConfig": {
     "access": "public"
   },
-  "version": "0.0.168",
+  "version": "0.0.169",
   "jest": {
     "coverageProvider": "v8",
     "coveragePathIgnorePatterns": [