toga-ai 1.0.833 → 1.0.834

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -61,5 +61,5 @@
 | [USPS DPV Deliverability Verdict (is this address actually insurable/shippable?)](features/usps-dpv-deliverability.md) | **USPS returning HTTP 200 with a populated address is NOT evidence that the address is deliverable.** The authoritative signal is USPS's **DPV (Delivery Point V |
 | [Refreshing a Local Dev Database from Beta (dev-sandbox)](workflows/local-db-refresh-from-beta.md) | How to reset a local 2.0 dev database from the **beta / dev-sandbox** environment: dump each schema (`Core`, `Client_<Id>`, `Logs_<Id>`, …) from the beta host, |
 | [Deleting a shared branch does not remove bad commits — a stale local clone merges them back](workflows/recreated-shared-branch-stale-local-remerge.md) | **Deleting and recreating a shared environment branch removes only the *ref*.** Every teammate who still has that branch checked out locally keeps the full pre- |
-| [Rotating Public TLS Certificates (ACM, CloudFront, ALB, Elastic Beanstalk)](workflows/rotating-public-tls-certificates.md) | How to replace the public TLS certificates that terminate HTTPS for TOGa front-end domains (togahub, togacommerce, togadesk, togaretail, togasupply, togaview, t |
+| [Rotating Public TLS Certificates (ACM, CloudFront, ALB, Elastic Beanstalk, API Gateway)](workflows/rotating-public-tls-certificates.md) | How to replace the public TLS certificates that terminate HTTPS for TOGa front-end domains (togahub, togacommerce, togadesk, togaretail, togasupply, togaview, t |
 | [Running a 2.0 App Locally (browser, end-to-end via api2)](workflows/running-a-2.0-app-locally.md) | The full dependency chain required to run a 2.0 client app **through the browser**, end-to-end, against a **local `api2`** (e.g. |

package/knowledge/2.0/apps/_underscore/workflows/rotating-public-tls-certificates.md

@@ -1,12 +1,12 @@
 ---
-title: Rotating Public TLS Certificates (ACM, CloudFront, ALB, Elastic Beanstalk)
+title: Rotating Public TLS Certificates (ACM, CloudFront, ALB, Elastic Beanstalk, API Gateway)
 framework: "2.0"
 repo: _underscore
 project: _Underscore
 client: shared
 type: workflow
 status: active
-updated: 2026-09-16
+updated: 2026-09-17
 owners: ["rgirish"]
 files: []
 related:
@@ -27,13 +27,15 @@ This runbook was written from the 2026-09-16 rotation in AWS account **654654170
 one IMPORTED multi-domain cert (14 SANs, copied into us-east-1, us-west-2 and eu-west-1) served
 4 CloudFront distributions and 12 ALB listeners and expired the same day.
 
-**The three things that bite you:**
+**The four things that bite you:**
 1. **Elastic Beanstalk keeps its own saved cert setting**, separate from the ALB listener. Fixing
    the listener is not enough — the next deploy puts the old cert straight back.
 2. **CloudFront holds exactly ONE cert per distribution.** There is no staging. The swap *is* the
    cutover, and it takes 5-15 minutes.
 3. **An ACM wildcard matches exactly one label.** `*.togasupply.com` covers
    `elite.togasupply.com` but **not** `compass.beta.togasupply.com`.
+4. **API Gateway custom domains hold their own cert** and are invisible to a CloudFront/ALB/EB
+   sweep. This is what the 2026-09-16 rotation missed — see step 6b.
 
 **Root cause to avoid repeating:** an **IMPORTED** ACM cert never auto-renews. Always replace with
 **Amazon-issued, DNS-validated** ACM certs, which renew themselves.
@@ -123,6 +125,60 @@ one), and every environment stayed Green/Ok through the update — no traffic di
 **Gotcha:** an EB environment whose CloudFormation stack is in `DELETE_FAILED` cannot be updated at
 all — `update-environment` is refused. That needs separate stack cleanup.
 
+## Step 6b — Audit API Gateway custom domains (the 2026-09-16 miss)
+
+**API Gateway is a FOURTH resource type.** The 2026-09-16 rotation swept CloudFront, ALB listeners
+and EB saved configs, and reported "0 endpoints at risk". The next morning `webhook.togahub.com`
+was hard-down — it is an API Gateway custom domain, so none of those three sweeps could ever have
+found it. Proof of the failure, not just a stale cert:
+
+```
+$ curl -sS -o /dev/null -w "http=%{http_code} ssl=%{ssl_verify_result}\n" https://webhook.togahub.com/
+curl: (60) SSL certificate problem: certificate has expired
+http=000 ssl=10
+```
+
+**Find them from DNS — an API Gateway domain CNAMEs to `*.execute-api.<region>.amazonaws.com`:**
+
+```
+aws route53 list-resource-record-sets --hosted-zone-id <zid> --max-items 400 \
+  --query "ResourceRecordSets[?Type=='CNAME'].[Name,ResourceRecords[0].Value]" --output text \
+  | grep -i execute-api
+```
+
+Across all seven TOGa zones this found exactly one: `webhook.togahub.com` →
+`d-yiy4od2ade.execute-api.us-east-1.amazonaws.com`.
+
+**An unknown account ID in `InUseBy` can be AWS itself.** The expired cert's `InUseBy` listed 3
+ALBs in account **250044486744**, which no one has a profile for. That is AWS-managed edge
+infrastructure — an edge-optimized API Gateway domain terminates TLS on AWS's own load balancers.
+Treat it as a pointer to API Gateway, **not** as a rogue account to go hunt credentials for.
+
+**Permissions block this from the CLI.** The `GoAgilant-Developers` SSO role has **no**
+`apigateway:GET` in any of the three accounts (654654170868, 502614707982, 975050298201) —
+`get-domain-names`, `get-domain-name` and the `apigatewayv2` equivalents all return
+`AccessDeniedException`. The domain cannot be listed, read or fixed with the standard developer
+role. Use the Console, or get `apigateway:GET` + `apigateway:PATCH` added first.
+
+**Check the endpoint type FIRST — it decides which field you patch.** `get-domain-name` returns
+`endpointConfiguration.types`. An **EDGE** domain reads its cert from `us-east-1` and uses
+`/certificateArn`; a **REGIONAL** domain reads from its own region and uses
+`/regionalCertificateArn`. Patching the wrong field silently does nothing useful.
+`webhook.togahub.com` turned out to be **REGIONAL** (us-east-1), despite `InUseBy` showing
+AWS-managed edge ALBs — so do not infer the type from `InUseBy`.
+
+```
+aws apigateway get-domain-name --domain-name webhook.togahub.com \
+  --query 'endpointConfiguration.types'
+
+aws apigateway update-domain-name --domain-name webhook.togahub.com \
+  --patch-operations op=replace,path=/regionalCertificateArn,value=<new-arn>
+```
+
+The domain goes `UPDATING` and takes a few minutes to return to `AVAILABLE` (~4 min observed).
+Poll `domainNameStatus` before verifying — checking too early shows the old cert and looks like a
+failed patch. No downtime beyond the already-broken TLS; base path mappings are untouched.
+
 ## Step 7 — Check alias coverage before any CloudFront swap
 
 CloudFront **rejects** an update if any alias on the distribution is not covered by the new cert.
@@ -155,11 +211,22 @@ echo | openssl s_client -servername HOST -connect HOST:443 2>/dev/null \
   | openssl x509 -noout -issuer -enddate
 ```
 
+`openssl` shows you the cert, but it does **not** prove a client can connect. Add a real verify —
+and treat a known-bad host as the control that proves your check actually fails:
+
+```
+curl -sS -o /dev/null -w "http=%{http_code} ssl=%{ssl_verify_result}\n" https://HOST/
+```
+
+A live expired cert returns `http=000` with `SSL certificate problem: certificate has expired`.
+
 Also run it **without** `-servername` to check the no-SNI / default-cert path — that path can still
 be serving the old cert after everything else looks fine.
 
-Finish by re-running the step 1 audit: every CloudFront distribution, every ALB listener, and every
-EB saved config should name a cert with a future `NotAfter`.
+Finish by re-running the step 1 audit: every CloudFront distribution, every ALB listener, every
+EB saved config **and every API Gateway custom domain** should name a cert with a future
+`NotAfter`. A sweep that omits any one of those four resource types is not a clean result — it is
+an untested one.
 
 ## Gotchas
 
@@ -170,9 +237,28 @@ EB saved config should name a cert with a future `NotAfter`.
 - **CloudFront takes one cert; ALBs take many.** Two completely different risk profiles in the same
   rotation — plan them separately.
 - **Wildcards match one label only.** Verify alias coverage with code, not by eye.
+- **API Gateway custom domains are a fourth resource type.** A CloudFront + ALB + EB sweep misses
+  them completely. Find them by `execute-api` CNAMEs in Route 53. See step 6b.
+- **An unknown account ID in `InUseBy` may be AWS itself.** `250044486744` is AWS-managed edge
+  infrastructure for API Gateway, not a rogue account. See step 6b.
+- **The `GoAgilant-Developers` role needs `apigateway:GET` + `apigateway:PATCH`** on
+  `arn:aws:apigateway:*::/domainnames*`. Added to the permission set 2026-09-17; it was missing
+  during the rotation. Granted via IAM Identity Center, then **provisioned** to the account — the
+  provision step is what actually applies it. See step 6b.
+- **Patch the field that matches the endpoint type.** REGIONAL uses `/regionalCertificateArn`,
+  EDGE uses `/certificateArn`. See step 6b.
+- **`openssl` is not proof of health.** It prints the cert even when clients cannot connect. Use
+  `curl` and check for `http=000`. See step 9.
 - **Expired certs hide.** Sweep the whole account; do not trust `InUseBy`.
 
 ## Change history
+- 2026-09-17 — Added step 6b (API Gateway custom domains) after `webhook.togahub.com` was found
+  serving an expired cert and failing TLS the day after the rotation reported all-clear; added the
+  `curl` proof step, the AWS-owned-account `InUseBy` tell, and the missing `apigateway` permission.
+  **Fixed the same day:** `apigateway:GET`/`PATCH` added to the permission set, domain confirmed
+  REGIONAL, `regionalCertificateArn` swapped to the Amazon-issued `*.togahub.com` cert
+  (`a53d1ff6-...`, expires 2027-03-31). Verified `http=404 ssl=0` (was `http=000 ssl=10`), with
+  `expired.badssl.com` as the control that still fails. (rgirish)
 - 2026-09-16 — Created from the account 654654170868 rotation: replaced an expiring 14-SAN IMPORTED
   cert with 6 per-domain Amazon-issued DNS-validated certs, relinked 12 ALB listeners, 6 CloudFront
   distributions and 11 EB environment configs; documented the EB saved-config trap. (rgirish)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.833",
+  "version": "1.0.834",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",