toga-ai 1.0.830 → 1.0.831
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -6,13 +6,15 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: workflow
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-17
|
|
10
10
|
owners: ["rgirish"]
|
|
11
11
|
files: []
|
|
12
12
|
related:
|
|
13
13
|
- 2.0/standards/ssl-certificate-trust-model.md
|
|
14
14
|
- 2.0/apps/saml/workflows/rotating-the-saml-sp-certificate.md
|
|
15
15
|
- 1.0/apps/tools/features/cloudfront-client-setup.md
|
|
16
|
+
- 1.0/apps/togaview/workflows/route53-cross-account-zone-migration.md
|
|
17
|
+
- 2.0/apps/worker2/features/cross-account-aws-access.md
|
|
16
18
|
---
|
|
17
19
|
|
|
18
20
|
Replace the browser-facing public TLS certs (ACM/CloudFront/ALB/Elastic Beanstalk) for TOGa front-end domains before expiry; open when rotating certs or moving off an imported cert.
|
|
@@ -21,12 +23,14 @@ Replace the browser-facing public TLS certs (ACM/CloudFront/ALB/Elastic Beanstal
|
|
|
21
23
|
|
|
22
24
|
Replace the public TLS certs that terminate HTTPS for TOGa front-end domains (togahub, togacommerce, togadesk, togaretail, togasupply, togaview, togaiq) before they expire. This is the browser-facing cert on CloudFront and ALBs — **not** the SAML SP credential (a pinned bilateral cert; see [Rotating the SAML SP Certificate](../../saml/workflows/rotating-the-saml-sp-certificate.md)).
|
|
23
25
|
|
|
24
|
-
Written from the 2026-09-16 rotation
|
|
26
|
+
Written from the 2026-09-16/17 rotation across **three** AWS accounts (starting with **654654170868**), where one IMPORTED multi-domain cert (14 SANs, copied into us-east-1, us-west-2 and eu-west-1) served 4 CloudFront distributions and 12 ALB listeners and expired the same day — and the **same** 14-SAN cert also sat on 5 load balancers in a second account, with no valid cert alongside. Final sweep: 84 ALB listeners and 82 CloudFront distributions.
|
|
25
27
|
|
|
26
|
-
The
|
|
27
|
-
1. **Elastic Beanstalk keeps its own saved cert setting**, separate from the ALB listener. Fixing the listener is not enough — the next deploy puts the old cert straight back.
|
|
28
|
+
The things that bite:
|
|
29
|
+
1. **Elastic Beanstalk keeps its own saved cert setting**, separate from the ALB listener. Fixing the listener is not enough — the next deploy puts the old cert straight back. Confirmed identical in both accounts, so treat it as a rule, not a one-off.
|
|
28
30
|
2. **CloudFront holds exactly ONE cert per distribution.** No staging. The swap *is* the cutover, 5-15 minutes.
|
|
29
31
|
3. **An ACM wildcard matches exactly one label.** `*.togasupply.com` covers `elite.togasupply.com` but **not** `compass.beta.togasupply.com`.
|
|
32
|
+
4. **Certs cannot cross AWS accounts.** Each account needs its own certs (step 3b).
|
|
33
|
+
5. **An audit script that swallows errors lies to you.** An expired SSO token looks exactly like "all clear" (step 1).
|
|
30
34
|
|
|
31
35
|
**Root cause to avoid repeating:** an **IMPORTED** ACM cert never auto-renews. Always replace with **Amazon-issued, DNS-validated** ACM certs, which renew themselves.
|
|
32
36
|
|
|
@@ -34,38 +38,78 @@ The three things that bite:
|
|
|
34
38
|
|
|
35
39
|
## Steps
|
|
36
40
|
|
|
37
|
-
1. **Audit EVERYTHING,
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
41
|
+
1. **Audit EVERYTHING, in EVERY account — with an error-aware script.**
|
|
42
|
+
|
|
43
|
+
**Verify the login first, per profile:** `aws sts get-caller-identity`. Then enumerate **all** CloudFront distributions and **all** ALB listeners in **every** region, resolve each attached cert's `NotAfter`, flag anything expiring before your cutoff. Do **not** just follow the expiring cert's `InUseBy` list — that is how the rotation found a **second** cert already expired six months earlier (2026-03-24), still the only cert on 2 ALB listeners and 2 CloudFront distributions, serving cert errors since March unnoticed.
|
|
44
|
+
|
|
45
|
+
**The script must fail loudly.** The day-1 script wrapped every AWS call in try/except and returned `{}` on failure. When the SSO token expired mid-run it printed "DOWN NOW: 0" and "LB GONE" for healthy load balancers — an auth failure was indistinguishable from a clean result, and it cost real time. Required:
|
|
46
|
+
- check each subprocess return code, collect the errors,
|
|
47
|
+
- print `N API ERRORS — RESULT IS NOT TRUSTWORTHY` instead of an empty list,
|
|
48
|
+
- print **how many** listeners and distributions were actually checked. A count of `0` with a clean result is the tell.
|
|
49
|
+
|
|
50
|
+
Also audit **every Elastic Beanstalk environment's saved config**, including suspended ones (a suspended env with no live listener can still hold a stale cert ARN that returns on rebuild).
|
|
51
|
+
|
|
52
|
+
**Check each CloudFront distribution's ORIGINS too** — one distribution had a real alias and live DNS but all 3 origins pointed at load balancers that no longer existed (dead leftover; don't spend change-window time on it).
|
|
53
|
+
|
|
54
|
+
2. **Resolve each hostname in DNS — the alias list is not the source of truth.** A CloudFront distribution can carry an alias it does **not** actually serve. `nychh.togacommerce.com`, `prudential.togacommerce.com` and `spglobal.togacommerce.com` were listed as aliases on a distribution, but their Route 53 records CNAME to an Elastic Beanstalk environment in a **different AWS account**. Reading the alias list alone gave the wrong answer and nearly left them unprotected. **Confirm every hostname against its Route 53 record** before deciding which resource to fix.
|
|
55
|
+
|
|
56
|
+
3. **Decide the cert shape: one cert per domain.** Replace a bundled multi-domain cert with **one cert per domain** (root + wildcard, e.g. `togahub.com` + `*.togahub.com`). Reasons: a 14-SAN request exceeds ACM's default 10-domain quota; one domain's problem no longer takes down all seven; each distribution only needs the domain it serves. All new certs **Amazon-issued and DNS-validated** so they auto-renew. This is case 2 of the [SSL Certificate Trust-Model Strategy](../../../standards/ssl-certificate-trust-model.md): AWS terminates the TLS, so use ACM public, non-exportable, auto-renewing.
|
|
57
|
+
|
|
58
|
+
3b. **Each account needs its OWN certs — they cannot be shared or copied.** A cert ARN contains the account ID, and an ALB can only use a cert in **its own account and region**. There is **no RAM sharing** for ACM certs. Export is also not an option for Amazon-issued certs — verified both ways: `describe-certificate` reports `Options.Export = DISABLED`, and `aws acm export-certificate` fails with `ValidationException: ... is not an export enabled certificate`.
|
|
59
|
+
|
|
60
|
+
The cross-account split is cheap and normal: **request the cert in the account that owns the load balancer, and add the DNS validation CNAMEs in whichever account hosts the Route 53 zone.** ACM only checks that the record resolves — it does not care which account hosts the zone. Worked first try; 3 certs went requested → `ISSUED` in under a minute. **A second cert for the same domain in a different account gets a DIFFERENT validation CNAME name and value**, so it does not collide with or overwrite the first account's record — both coexist in one zone.
|
|
61
|
+
|
|
62
|
+
4. **Request the certs in the right regions.** **CloudFront only reads certs from `us-east-1`** (no exceptions). **An ALB reads certs from its own region** — a togahub ALB in us-west-2 needs a us-west-2 cert; the same domain on an eu-west-1 ALB needs a separate eu-west-1 cert. Check ACM first — unused per-domain certs may already exist (3 did in this rotation); request only what's missing.
|
|
63
|
+
|
|
64
|
+
5. **Validate by DNS in Route 53.** Add the validation CNAMEs. Certs for the **same domain in the same account, different regions, share ONE validation record** — name and value identical, so 6 certs across 3 regions needed only 5 CNAMEs. (Different **account** = different record; see step 3b.) Validation completes ~2 minutes once records are live. Wait for `ISSUED` before touching anything.
|
|
65
|
+
|
|
66
|
+
6. **Stage the ALB listeners first (safe, additive).** `aws elbv2 add-listener-certificates` is **purely additive**. A listener holds many certs and picks by SNI, so adding the new cert alongside the old changes nothing for live traffic. Do this ahead of the change window for every listener.
|
|
67
|
+
|
|
68
|
+
7. **Promote the new cert to DEFAULT with `modify-listener` — you cannot remove a default cert.** `aws elbv2 remove-listener-certificates` on the listener's default cert fails with `OperationNotPermitted: Default certificate cannot be removed`. Use:
|
|
69
|
+
```
|
|
70
|
+
aws elbv2 modify-listener --listener-arn <arn> --certificates CertificateArn=<new-arn>
|
|
71
|
+
```
|
|
72
|
+
That single call promotes the valid cert **and** drops the old expired cert from the listener — no separate remove call is needed.
|
|
73
|
+
|
|
74
|
+
8. **Before removing any expired cert, check whether another cert on the same listener covers its SANs.** One load balancer carried 9 certs and 15 host-header rules; two certs were expired, but each was the **only** cert on that listener covering its hostnames (one for `alpha.togacommerce.com`, one for `walmarttechservices.com` plus 7 related domains). Removing them would have deleted the only record that those hostnames are meant to work at all. If nothing else covers the SANs, the host is **already broken** and the choice is replace-or-retire — a product decision, not cleanup. (Here the developer decided those hosts were not needed and left them as-is.)
|
|
75
|
+
|
|
76
|
+
9. **THE ONE PEOPLE MISS: update the Elastic Beanstalk saved config.** Adding a cert to an ALB listener does **not** update the EB environment's saved setting `aws:elbv2:listener:443 / SSLCertificateArns`. After all 12 listeners were fixed and verified healthy, 12 EB environments still named the **old** cert; the second account repeated it exactly — all 9 of its environments still named the old cert after their listeners were fixed. Nothing breaks right away — but the next **deploy, restart, rebuild or scaling event** re-applies saved config and puts the expired cert back. A listener-level audit does not catch this.
|
|
43
77
|
```
|
|
44
78
|
aws elasticbeanstalk update-environment \
|
|
45
79
|
--environment-name <env> \
|
|
46
80
|
--option-settings "Namespace=aws:elbv2:listener:443,OptionName=SSLCertificateArns,Value=<new-arn>"
|
|
47
81
|
```
|
|
48
|
-
Observed: this cleanly **replaced** the old cert on the listener (leaving only the new one), and every environment stayed Green/Ok through the update — no disruption. **Gotcha:** an EB environment whose CloudFormation stack is in `DELETE_FAILED` cannot be updated at all — `update-environment` is refused; that needs separate stack cleanup.
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
82
|
+
Observed in **both** accounts: this cleanly **replaced** the old cert on the listener (leaving only the new one), and every environment stayed Green/Ok through the update — no disruption. **Gotcha:** an EB environment whose CloudFormation stack is in `DELETE_FAILED` cannot be updated at all — `update-environment` is refused; that needs separate stack cleanup.
|
|
83
|
+
|
|
84
|
+
10. **Check alias coverage before any CloudFront swap.** CloudFront **rejects** an update if any alias on the distribution is not covered by the new cert. Check every alias against the new cert's SANs **programmatically** — this is where the one-label wildcard rule bites (`*.togasupply.com` does not cover `compass.beta.togasupply.com`). Beta and gamma environments generally need their own certs.
|
|
85
|
+
|
|
86
|
+
11. **Swap CloudFront, smallest blast radius first.** `ViewerCertificate.ACMCertificateArn` is a **single string**, not a list — no "add new alongside old"; the swap is the cutover, deploys in 5-15 minutes. Order used, lowest risk first: (1) distributions **already broken** (nothing to lose); (2) single-alias; (3) mid-size (9 aliases); (4) the largest last (the 21-alias distribution fronting 21 named enterprise clients). Back up each distribution's config before every swap — `aws cloudfront get-distribution-config`, and keep the **ETag** (needed to update). **Hard edge:** once the old cert's expiry moment has passed there is no useful CloudFront rollback (rollback restores an expired cert). Only forward.
|
|
87
|
+
|
|
88
|
+
12. **Verify on the live hosts.**
|
|
89
|
+
```
|
|
90
|
+
echo | openssl s_client -servername HOST -connect HOST:443 2>/dev/null \
|
|
91
|
+
| openssl x509 -noout -issuer -enddate
|
|
92
|
+
```
|
|
93
|
+
Also run it **without** `-servername` to check the no-SNI / default-cert path — that path can still serve the old cert after everything else looks fine. Finish by re-running the step-1 audit **in every account**: every CloudFront distribution, every ALB listener, and every EB saved config should name a cert with a future `NotAfter`.
|
|
57
94
|
|
|
58
95
|
## Gotchas
|
|
59
96
|
|
|
60
97
|
- **IMPORTED certs never auto-renew.** If the audit shows `Type: IMPORTED`, it is a future outage.
|
|
61
98
|
- **A cert copied into 3 regions is 3 separate certs** — all 3 expire at once, all 3 need replacing.
|
|
62
|
-
- **
|
|
99
|
+
- **Certs never cross accounts.** No RAM sharing; Amazon-issued certs are `Export = DISABLED`. Request per account, validate in whichever account holds the zone. See step 3b.
|
|
100
|
+
- **An audit that catches exceptions and returns empty prints a false all-clear.** Check return codes, count what you checked, and shout on API errors. See step 1.
|
|
101
|
+
- **DNS decides which resource serves a host, not the CloudFront alias list.** Always confirm with the Route 53 record. See step 2.
|
|
102
|
+
- **A listener's default cert cannot be removed** — `modify-listener` promotes and cleans up in one call. See step 7.
|
|
103
|
+
- **Don't remove an expired cert that is the only coverage for its hostnames** — that is a retire-or-replace decision. See step 8.
|
|
104
|
+
- **EB saved config is the silent landmine.** Repeats in every account. See step 9.
|
|
63
105
|
- **CloudFront takes one cert; ALBs take many.** Two different risk profiles in the same rotation — plan them separately.
|
|
64
106
|
- **Wildcards match one label only.** Verify alias coverage with code, not by eye.
|
|
65
|
-
- **Expired certs hide.** Sweep
|
|
107
|
+
- **Expired certs hide.** Sweep every account; do not trust `InUseBy`.
|
|
66
108
|
|
|
67
109
|
## Related
|
|
68
110
|
|
|
69
111
|
- [SSL Certificate Trust-Model Strategy](../../../standards/ssl-certificate-trust-model.md)
|
|
70
112
|
- [Rotating the SAML SP Certificate](../../saml/workflows/rotating-the-saml-sp-certificate.md)
|
|
71
113
|
- [CloudFront client setup (1.0 tools)](../../../../1.0/apps/tools/features/cloudfront-client-setup.md)
|
|
114
|
+
- [Route 53 Hosted Zone Migration Between AWS Accounts](../../../../1.0/apps/togaview/workflows/route53-cross-account-zone-migration.md)
|
|
115
|
+
- [Cross-account AWS access for worker2 crons](../../worker2/features/cross-account-aws-access.md)
|
package/package.json
CHANGED