toga-ai 1.0.829 → 1.0.831

package/knowledge/1.0/apps/library/architecture.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: architecture
 status: active
-updated: 2026-09-16
+updated: 2026-09-17
 owners: [jcardinal]
 files:
   - library/_.php
@@ -85,3 +85,4 @@ Consuming apps use **no Composer** — this library is the "vendor" directory. P
 - **`debugVar()` can halt execution** (its `$stopExecution` arg) — remove all calls before merging.
 - **app/ classes silently shadow library/ classes.** The autoloader checks both `__APPROOT__` and `__LIBROOT__`, so an app-level file with the same class path overrides the core one with no warning.
 - **`__APPROOT__` resolution needs the `_` folder.** It is found by locating an ancestor folder named `_`; each app must have that `_` folder at its root or the constant won't resolve.
+- **⚠ SECURITY — OPEN, SYSTEMIC, not fixed (2026-09-17): TLS peer verification is OFF by default for EVERY `App_ApiTransaction` call.** `library/app/apitransaction.php:363` computes `CURLOPT_SSL_VERIFYPEER` from `$sslVerifyPeer` (default `null`, line 77) as `(((is_null($sslVerifyPeer) && isHTTPS()) || !$sslVerifyPeer) ? false : true)` — which resolves to **`false` on HTTPS**. So every outbound HTTPS call through the shared transport (NetSuite OAuth token exchange, every REST read, the RESTlet call, and any other `App_Api_*` integration) runs with peer verification disabled — a MITM can capture the bearer token or inject responses. Pre-existing core defect, **not** caused by the NetSuite retry; affects all callers. Fix: `CURLOPT_SSL_VERIFYPEER = is_null($sslVerifyPeer) ? isHTTPS() : (bool)$sslVerifyPeer;` and have `rest.php` pass `setSSLVerifyPeer(true)` explicitly. No fix chosen yet. The paired token-logging defect (`send()` writes the bearer token to the request log) is on the [SuiteQL/REST shim doc](features/netsuite-suiteql-rest-shim.md).

package/knowledge/1.0/apps/library/features/netsuite-suiteql-rest-shim.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-17
 owners: [dfranks, jcardinal, kyalamarthi, rgirish]
 files:
   - library/app/api/netsuite/rest.php
@@ -39,6 +39,8 @@ Header SuiteQL → per-record detail → line SuiteQL, assembled into SOAP-shape
 
 **Data model:** reads NetSuite `transaction`, `transactionline`, `previoustransactionlinelink` via SuiteQL and `/record/v1/{type}/{id}` via REST. Writes nothing. **Client variations:** none — uniform across clients (NetSuite is a single shared account).
 
+**Transient-read retry in `send()`.** `send()` retries a transient failure before failing loud, but **only for read-only calls** — a `GET`, or a `POST` to `/query/v1/suiteql` (the read-only SuiteQL endpoint); a mutating write is never retried. This absorbs a single NetSuite blip in a large read burst (hundreds of reads per sync run) that would otherwise abort a whole sync section. Detail + scope in Gotchas.
+
 ### Transaction status sourcing (1.0) — and the 2026.2 REST `status.id` change
 
 1.0 reaches NetSuite over **two transports, and neither reads status off a REST record GET body**:
@@ -85,6 +87,8 @@ Field semantics (verified on the live account):
 
   **Consequence: two same-day records disagree about whether the SELECT is present, so verify the branch you are on** — `git grep -n "tl.location" library/app/api/netsuite/rest.php` — rather than trusting either doc. And note what restoring it *re-animates*: the leaf→root location rollup at `worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php:1636-1934`, which **auto-inserts and renames `Forecast.Locations` rows**, has been dead code the whole time `$line->location` was null. It stays dormant only because `import_open_orders.php` is `"active": 0` — re-enable that cron on a branch that selects `tl.location` and a dimension table starts being written by a code path nobody has exercised since June. See [open-orders sync](../../../../2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md).
 - **A merge resolved "in favour of `_production`" is a silent feature revert — re-grep for your own additions afterwards.** The c3caa11a case is the pattern: it removed two SELECT columns while leaving their consumers in place. It also broke `listLocations()`'s closing brace, and *that* half **was** repaired later (**5c1c194e**, "TRUE-80090: fixing syntax issue") — which is precisely what makes the whole breakage read as already-fixed to the next person who looks.
+- **`send()` retries a TRANSIENT read failure before failing loud — READ-ONLY scope only (2026-09-17).** There is no per-record isolation in the sync engine, so ONE transient NetSuite blip in a large read burst used to abort the whole section and collapse its window (see [per-client sync](../../worker/features/netsuite-togasupply-per-client-sync.md)). `App_Api_Netsuite_Rest::send()` now retries only when `method === 'GET'` **or** a `POST` to a route starting `/query/v1/suiteql` (the read-only SuiteQL endpoint) — **never a mutating write** (a POST/PUT that got no response could double-create). Transient set = HTTP status `0` (curl no-response/timeout; `App_ApiTransaction::execute` leaves `responseCode` null → cast 0), `429`, `502/503/504` — **NOT** bare `500`, not other `4xx`. 3 attempts total (1 + 2 retries); sleep 1s then 3s + `random_int(0,1)` jitter; one `error_log` per retry. After the last attempt the ORIGINAL throw-or-return runs unchanged, so **fail-loud is preserved** (cursor still cannot advance past an unimported record). Consts `TRANSIENT_RETRY_MAX_ATTEMPTS=3` / `TRANSIENT_RETRY_FIRST_DELAY_SECONDS=1` / `TRANSIENT_RETRY_SECOND_DELAY_SECONDS=3`. The retry wraps `execute` regardless of the throw flag, so it **also** covers the non-throwing GET reads (`send(...,false)`, e.g. `fetchItemFulfillmentById`'s header GET) where a transient blip would otherwise read as a false "record not found" and silently advance the cursor. Ships via a **worker** deploy (worker box clones library at deploy).
+- **⚠ SECURITY — OPEN, not fixed (2026-09-17): `send()` writes the live `Authorization: Bearer <token>` to the API request log.** `library/app/apitransaction.php:393` (Toga2 logs) / `:403` (Toga1 logs) save `requestHeaders` when `$this->log && self::$isLoggingEnabled`. `authenticate()` calls `setLogging(false)` (`rest.php:187`) for exactly this reason; `send()` does **not**, so every logged call stores the bearer token (the transient-retry above multiplies it to up to 3 token writes per fully-failed call). Fix options — **redact the `Authorization` header in the logger** (fixes ALL `App_ApiTransaction` callers, keeps logging — preferred), or `setLogging(false)` in `send()`/`callRestlet()`. No fix chosen yet; record the LOCATION only, never a token value. A separate, systemic transport defect (TLS peer verification off for every `App_ApiTransaction` call) is tracked on [library architecture](../architecture.md).
 - The shim must stay PHP 7.2-compatible (library is 7.2 prod): no arrow functions, typed props, `??=`, or `match`. Lint with `C:\xampp7\php\php.exe -l` before deploying.
 - Field availability varies by NetSuite account **and** record type — probe the live account before assuming a column/relationship exists.
 - **`App_NetSuite::getCustomFieldValue()` (`library/app/netsuite.php`) must be warning-safe.** 1.0 runs on PHP 7.2 in an unguarded path, so any PHP warning → `exit()` (kills the cron). It was hardened 2026-08-27 with `?? []` / `?? null` guards on the custom-field-list traversal. Keep that discipline when touching it.

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-17
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -101,6 +101,18 @@ its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in
 in prod* (see below), never *tune the window size down*. (Corollary: a window that has grown back to
 the cap is positive evidence the section is completing.)
 
+**One cause of a throw is now retried away: a single TRANSIENT NetSuite blip in a large read burst.**
+Because there is no per-record isolation, draining a burst of hundreds of NetSuite reads per run means
+ONE transient blip (empty HTTP status, `429`, `502/503/504`) at any read call site aborted the whole
+section → RUNNING → window ÷3, even though nothing was wrong with the data. `App_Api_Netsuite_Rest::send()`
+now retries transient **read-only** calls (GET / SuiteQL POST) before failing loud, so a lone blip no
+longer collapses the window; a real poison record still throws and freezes as before. Scope + limits on
+the [SuiteQL/REST shim doc](../../library/features/netsuite-suiteql-rest-shim.md). Live example: NYCHH
+ITEM_FULFILLMENTS was collapsing (mode `96000-RUNNING`, cursor frozen `2026-09-15 16:42:05`) draining a
+~255-fulfillment burst — a different call site failed each run (`fetchItemFulfillmentById` /
+`fetchItemFlagsByIds` / `fetchSalesOrderByIdFull`), which is the signature of a transient, not a poison
+record.
+
 **Where the error is: prod `Logs.Issue` / `Logs.Event`, NOT Sentry.** Read worker errors from the
 production log DB by default — you do not need a developer to point you there. The base `Logs`
 schema (prod-logs cluster) holds `Issue` (deduped, with `errorMessage` + `trace`) and `Event` (each

package/knowledge/2.0/apps/_underscore/workflows/rotating-public-tls-certificates.md

@@ -6,13 +6,15 @@ project: _Underscore
 client: shared
 type: workflow
 status: active
-updated: 2026-09-16
+updated: 2026-09-17
 owners: ["rgirish"]
 files: []
 related:
   - 2.0/standards/ssl-certificate-trust-model.md
   - 2.0/apps/saml/workflows/rotating-the-saml-sp-certificate.md
   - 1.0/apps/tools/features/cloudfront-client-setup.md
+  - 1.0/apps/togaview/workflows/route53-cross-account-zone-migration.md
+  - 2.0/apps/worker2/features/cross-account-aws-access.md
 ---
 
 Replace the browser-facing public TLS certs (ACM/CloudFront/ALB/Elastic Beanstalk) for TOGa front-end domains before expiry; open when rotating certs or moving off an imported cert.
@@ -21,12 +23,14 @@ Replace the browser-facing public TLS certs (ACM/CloudFront/ALB/Elastic Beanstal
 
 Replace the public TLS certs that terminate HTTPS for TOGa front-end domains (togahub, togacommerce, togadesk, togaretail, togasupply, togaview, togaiq) before they expire. This is the browser-facing cert on CloudFront and ALBs — **not** the SAML SP credential (a pinned bilateral cert; see [Rotating the SAML SP Certificate](../../saml/workflows/rotating-the-saml-sp-certificate.md)).
 
-Written from the 2026-09-16 rotation in AWS account **654654170868**, where one IMPORTED multi-domain cert (14 SANs, copied into us-east-1, us-west-2 and eu-west-1) served 4 CloudFront distributions and 12 ALB listeners and expired the same day.
+Written from the 2026-09-16/17 rotation across **three** AWS accounts (starting with **654654170868**), where one IMPORTED multi-domain cert (14 SANs, copied into us-east-1, us-west-2 and eu-west-1) served 4 CloudFront distributions and 12 ALB listeners and expired the same day — and the **same** 14-SAN cert also sat on 5 load balancers in a second account, with no valid cert alongside. Final sweep: 84 ALB listeners and 82 CloudFront distributions.
 
-The three things that bite:
-1. **Elastic Beanstalk keeps its own saved cert setting**, separate from the ALB listener. Fixing the listener is not enough — the next deploy puts the old cert straight back.
+The things that bite:
+1. **Elastic Beanstalk keeps its own saved cert setting**, separate from the ALB listener. Fixing the listener is not enough — the next deploy puts the old cert straight back. Confirmed identical in both accounts, so treat it as a rule, not a one-off.
 2. **CloudFront holds exactly ONE cert per distribution.** No staging. The swap *is* the cutover, 5-15 minutes.
 3. **An ACM wildcard matches exactly one label.** `*.togasupply.com` covers `elite.togasupply.com` but **not** `compass.beta.togasupply.com`.
+4. **Certs cannot cross AWS accounts.** Each account needs its own certs (step 3b).
+5. **An audit script that swallows errors lies to you.** An expired SSO token looks exactly like "all clear" (step 1).
 
 **Root cause to avoid repeating:** an **IMPORTED** ACM cert never auto-renews. Always replace with **Amazon-issued, DNS-validated** ACM certs, which renew themselves.
 
@@ -34,38 +38,78 @@ The three things that bite:
 
 ## Steps
 
-1. **Audit EVERYTHING, not just the cert you know about.** Do **not** just follow the expiring cert's `InUseBy` list. Enumerate **all** CloudFront distributions and **all** ALB listeners in **every** region, resolve each attached cert's `NotAfter`, flag anything expiring before your cutoff. This is how the 2026-09-16 rotation found a **second** cert already expired six months earlier (2026-03-24), still the only cert on 2 ALB listeners and 2 CloudFront distributions — 4 live togadesk hosts serving cert errors since March, unnoticed. Also audit **every Elastic Beanstalk environment's saved config**, including suspended ones (a suspended env with no live listener can still hold a stale cert ARN that returns on rebuild). **Check each CloudFront distribution's origins first** — one distribution had a real alias and live DNS but all 3 origins pointed at load balancers that no longer existed (dead leftover; don't spend change-window time on it).
-2. **Decide the cert shape: one cert per domain.** Replace a bundled multi-domain cert with **one cert per domain** (root + wildcard, e.g. `togahub.com` + `*.togahub.com`). Reasons: a 14-SAN request exceeds ACM's default 10-domain quota; one domain's problem no longer takes down all seven; each distribution only needs the domain it serves. All new certs **Amazon-issued and DNS-validated** so they auto-renew. This is case 2 of the [SSL Certificate Trust-Model Strategy](../../../standards/ssl-certificate-trust-model.md): AWS terminates the TLS, so use ACM public, non-exportable, auto-renewing.
-3. **Request the certs in the right regions.** **CloudFront only reads certs from `us-east-1`** (no exceptions). **An ALB reads certs from its own region** — a togahub ALB in us-west-2 needs a us-west-2 cert; the same domain on an eu-west-1 ALB needs a separate eu-west-1 cert. Check ACM first — unused per-domain certs may already exist (3 did in this rotation); request only what's missing.
-4. **Validate by DNS in Route 53.** Add the validation CNAMEs. Certs for the **same domain in different regions share ONE validation record** — CNAME name and value are identical, so 6 certs across 3 regions needed only 5 CNAMEs. Validation completes ~2 minutes once records are live. Wait for `ISSUED` before touching anything.
-5. **Stage the ALB listeners first (safe, additive).** `aws elbv2 add-listener-certificates` is **purely additive**. A listener holds many certs and picks by SNI, so adding the new cert alongside the old changes nothing for live traffic. Do this ahead of the change window for every listener.
-6. **THE ONE PEOPLE MISS: update the Elastic Beanstalk saved config.** Adding a cert to an ALB listener does **not** update the EB environment's saved setting `aws:elbv2:listener:443 / SSLCertificateArns`. After all 12 listeners were fixed and verified healthy, 12 EB environments still named the **old** cert in saved config. Nothing breaks right away — but the next **deploy, restart, rebuild or scaling event** re-applies saved config and puts the expired cert back. A listener-level audit does not catch this.
+1. **Audit EVERYTHING, in EVERY account — with an error-aware script.**
+
+   **Verify the login first, per profile:** `aws sts get-caller-identity`. Then enumerate **all** CloudFront distributions and **all** ALB listeners in **every** region, resolve each attached cert's `NotAfter`, flag anything expiring before your cutoff. Do **not** just follow the expiring cert's `InUseBy` list — that is how the rotation found a **second** cert already expired six months earlier (2026-03-24), still the only cert on 2 ALB listeners and 2 CloudFront distributions, serving cert errors since March unnoticed.
+
+   **The script must fail loudly.** The day-1 script wrapped every AWS call in try/except and returned `{}` on failure. When the SSO token expired mid-run it printed "DOWN NOW: 0" and "LB GONE" for healthy load balancers — an auth failure was indistinguishable from a clean result, and it cost real time. Required:
+   - check each subprocess return code, collect the errors,
+   - print `N API ERRORS — RESULT IS NOT TRUSTWORTHY` instead of an empty list,
+   - print **how many** listeners and distributions were actually checked. A count of `0` with a clean result is the tell.
+
+   Also audit **every Elastic Beanstalk environment's saved config**, including suspended ones (a suspended env with no live listener can still hold a stale cert ARN that returns on rebuild).
+
+   **Check each CloudFront distribution's ORIGINS too** — one distribution had a real alias and live DNS but all 3 origins pointed at load balancers that no longer existed (dead leftover; don't spend change-window time on it).
+
+2. **Resolve each hostname in DNS — the alias list is not the source of truth.** A CloudFront distribution can carry an alias it does **not** actually serve. `nychh.togacommerce.com`, `prudential.togacommerce.com` and `spglobal.togacommerce.com` were listed as aliases on a distribution, but their Route 53 records CNAME to an Elastic Beanstalk environment in a **different AWS account**. Reading the alias list alone gave the wrong answer and nearly left them unprotected. **Confirm every hostname against its Route 53 record** before deciding which resource to fix.
+
+3. **Decide the cert shape: one cert per domain.** Replace a bundled multi-domain cert with **one cert per domain** (root + wildcard, e.g. `togahub.com` + `*.togahub.com`). Reasons: a 14-SAN request exceeds ACM's default 10-domain quota; one domain's problem no longer takes down all seven; each distribution only needs the domain it serves. All new certs **Amazon-issued and DNS-validated** so they auto-renew. This is case 2 of the [SSL Certificate Trust-Model Strategy](../../../standards/ssl-certificate-trust-model.md): AWS terminates the TLS, so use ACM public, non-exportable, auto-renewing.
+
+3b. **Each account needs its OWN certs — they cannot be shared or copied.** A cert ARN contains the account ID, and an ALB can only use a cert in **its own account and region**. There is **no RAM sharing** for ACM certs. Export is also not an option for Amazon-issued certs — verified both ways: `describe-certificate` reports `Options.Export = DISABLED`, and `aws acm export-certificate` fails with `ValidationException: ... is not an export enabled certificate`.
+
+   The cross-account split is cheap and normal: **request the cert in the account that owns the load balancer, and add the DNS validation CNAMEs in whichever account hosts the Route 53 zone.** ACM only checks that the record resolves — it does not care which account hosts the zone. Worked first try; 3 certs went requested → `ISSUED` in under a minute. **A second cert for the same domain in a different account gets a DIFFERENT validation CNAME name and value**, so it does not collide with or overwrite the first account's record — both coexist in one zone.
+
+4. **Request the certs in the right regions.** **CloudFront only reads certs from `us-east-1`** (no exceptions). **An ALB reads certs from its own region** — a togahub ALB in us-west-2 needs a us-west-2 cert; the same domain on an eu-west-1 ALB needs a separate eu-west-1 cert. Check ACM first — unused per-domain certs may already exist (3 did in this rotation); request only what's missing.
+
+5. **Validate by DNS in Route 53.** Add the validation CNAMEs. Certs for the **same domain in the same account, different regions, share ONE validation record** — name and value identical, so 6 certs across 3 regions needed only 5 CNAMEs. (Different **account** = different record; see step 3b.) Validation completes ~2 minutes once records are live. Wait for `ISSUED` before touching anything.
+
+6. **Stage the ALB listeners first (safe, additive).** `aws elbv2 add-listener-certificates` is **purely additive**. A listener holds many certs and picks by SNI, so adding the new cert alongside the old changes nothing for live traffic. Do this ahead of the change window for every listener.
+
+7. **Promote the new cert to DEFAULT with `modify-listener` — you cannot remove a default cert.** `aws elbv2 remove-listener-certificates` on the listener's default cert fails with `OperationNotPermitted: Default certificate cannot be removed`. Use:
+   ```
+   aws elbv2 modify-listener --listener-arn <arn> --certificates CertificateArn=<new-arn>
+   ```
+   That single call promotes the valid cert **and** drops the old expired cert from the listener — no separate remove call is needed.
+
+8. **Before removing any expired cert, check whether another cert on the same listener covers its SANs.** One load balancer carried 9 certs and 15 host-header rules; two certs were expired, but each was the **only** cert on that listener covering its hostnames (one for `alpha.togacommerce.com`, one for `walmarttechservices.com` plus 7 related domains). Removing them would have deleted the only record that those hostnames are meant to work at all. If nothing else covers the SANs, the host is **already broken** and the choice is replace-or-retire — a product decision, not cleanup. (Here the developer decided those hosts were not needed and left them as-is.)
+
+9. **THE ONE PEOPLE MISS: update the Elastic Beanstalk saved config.** Adding a cert to an ALB listener does **not** update the EB environment's saved setting `aws:elbv2:listener:443 / SSLCertificateArns`. After all 12 listeners were fixed and verified healthy, 12 EB environments still named the **old** cert; the second account repeated it exactly — all 9 of its environments still named the old cert after their listeners were fixed. Nothing breaks right away — but the next **deploy, restart, rebuild or scaling event** re-applies saved config and puts the expired cert back. A listener-level audit does not catch this.
    ```
    aws elasticbeanstalk update-environment \
      --environment-name <env> \
      --option-settings "Namespace=aws:elbv2:listener:443,OptionName=SSLCertificateArns,Value=<new-arn>"
    ```
-   Observed: this cleanly **replaced** the old cert on the listener (leaving only the new one), and every environment stayed Green/Ok through the update — no disruption. **Gotcha:** an EB environment whose CloudFormation stack is in `DELETE_FAILED` cannot be updated at all — `update-environment` is refused; that needs separate stack cleanup.
-7. **Check alias coverage before any CloudFront swap.** CloudFront **rejects** an update if any alias on the distribution is not covered by the new cert. Check every alias against the new cert's SANs **programmatically** — this is where the one-label wildcard rule bites (`*.togasupply.com` does not cover `compass.beta.togasupply.com`). Beta and gamma environments generally need their own certs.
-8. **Swap CloudFront, smallest blast radius first.** `ViewerCertificate.ACMCertificateArn` is a **single string**, not a list — no "add new alongside old"; the swap is the cutover, deploys in 5-15 minutes. Order used, lowest risk first: (1) distributions **already broken** (nothing to lose); (2) single-alias; (3) mid-size (9 aliases); (4) the largest last (the 21-alias distribution fronting 21 named enterprise clients). Back up each distribution's config before every swap — `aws cloudfront get-distribution-config`, and keep the **ETag** (needed to update). **Hard edge:** once the old cert's expiry moment has passed there is no useful CloudFront rollback (rollback restores an expired cert). Only forward.
-9. **Verify on the live hosts.**
-   ```
-   echo | openssl s_client -servername HOST -connect HOST:443 2>/dev/null \
-     | openssl x509 -noout -issuer -enddate
-   ```
-   Also run it **without** `-servername` to check the no-SNI / default-cert path — that path can still serve the old cert after everything else looks fine. Finish by re-running the step-1 audit: every CloudFront distribution, every ALB listener, and every EB saved config should name a cert with a future `NotAfter`.
+   Observed in **both** accounts: this cleanly **replaced** the old cert on the listener (leaving only the new one), and every environment stayed Green/Ok through the update — no disruption. **Gotcha:** an EB environment whose CloudFormation stack is in `DELETE_FAILED` cannot be updated at all — `update-environment` is refused; that needs separate stack cleanup.
+
+10. **Check alias coverage before any CloudFront swap.** CloudFront **rejects** an update if any alias on the distribution is not covered by the new cert. Check every alias against the new cert's SANs **programmatically** — this is where the one-label wildcard rule bites (`*.togasupply.com` does not cover `compass.beta.togasupply.com`). Beta and gamma environments generally need their own certs.
+
+11. **Swap CloudFront, smallest blast radius first.** `ViewerCertificate.ACMCertificateArn` is a **single string**, not a list — no "add new alongside old"; the swap is the cutover, deploys in 5-15 minutes. Order used, lowest risk first: (1) distributions **already broken** (nothing to lose); (2) single-alias; (3) mid-size (9 aliases); (4) the largest last (the 21-alias distribution fronting 21 named enterprise clients). Back up each distribution's config before every swap — `aws cloudfront get-distribution-config`, and keep the **ETag** (needed to update). **Hard edge:** once the old cert's expiry moment has passed there is no useful CloudFront rollback (rollback restores an expired cert). Only forward.
+
+12. **Verify on the live hosts.**
+    ```
+    echo | openssl s_client -servername HOST -connect HOST:443 2>/dev/null \
+      | openssl x509 -noout -issuer -enddate
+    ```
+    Also run it **without** `-servername` to check the no-SNI / default-cert path — that path can still serve the old cert after everything else looks fine. Finish by re-running the step-1 audit **in every account**: every CloudFront distribution, every ALB listener, and every EB saved config should name a cert with a future `NotAfter`.
 
 ## Gotchas
 
 - **IMPORTED certs never auto-renew.** If the audit shows `Type: IMPORTED`, it is a future outage.
 - **A cert copied into 3 regions is 3 separate certs** — all 3 expire at once, all 3 need replacing.
-- **EB saved config is the silent landmine.** See step 6.
+- **Certs never cross accounts.** No RAM sharing; Amazon-issued certs are `Export = DISABLED`. Request per account, validate in whichever account holds the zone. See step 3b.
+- **An audit that catches exceptions and returns empty prints a false all-clear.** Check return codes, count what you checked, and shout on API errors. See step 1.
+- **DNS decides which resource serves a host, not the CloudFront alias list.** Always confirm with the Route 53 record. See step 2.
+- **A listener's default cert cannot be removed** — `modify-listener` promotes and cleans up in one call. See step 7.
+- **Don't remove an expired cert that is the only coverage for its hostnames** — that is a retire-or-replace decision. See step 8.
+- **EB saved config is the silent landmine.** Repeats in every account. See step 9.
 - **CloudFront takes one cert; ALBs take many.** Two different risk profiles in the same rotation — plan them separately.
 - **Wildcards match one label only.** Verify alias coverage with code, not by eye.
-- **Expired certs hide.** Sweep the whole account; do not trust `InUseBy`.
+- **Expired certs hide.** Sweep every account; do not trust `InUseBy`.
 
 ## Related
 
 - [SSL Certificate Trust-Model Strategy](../../../standards/ssl-certificate-trust-model.md)
 - [Rotating the SAML SP Certificate](../../saml/workflows/rotating-the-saml-sp-certificate.md)
 - [CloudFront client setup (1.0 tools)](../../../../1.0/apps/tools/features/cloudfront-client-setup.md)
+- [Route 53 Hosted Zone Migration Between AWS Accounts](../../../../1.0/apps/togaview/workflows/route53-cross-account-zone-migration.md)
+- [Cross-account AWS access for worker2 crons](../../worker2/features/cross-account-aws-access.md)

package/knowledge/clients/nychh/profile.md

@@ -18,7 +18,7 @@ project: _Underscore
 client: nychh
 type: profile
 status: active
-updated: 2026-09-16
+updated: 2026-09-17
 owners: ["jcardinal", "apeterson", "bala", "akhokhani"]
 files:
   - dbchanges2/Client_Nychh/2026-09-02a - TransferOrderNetsuitePushInterceptor.sql
@@ -73,6 +73,7 @@ TOGA 2.0 client on `_underscore`, prod schema `Client_Nychh`. Supply-chain / ful
 - **Missing NetSuite records (item fulfillments + others) root-caused and fixed (2026-09-09).** Two separate bugs in the 1.0 NetSuite→TOGa Supply sync starved NYCHH imports, both deployed and verified in prod: (1) the resume cursor was **read on the wrong clock** (`strtotime()` in Chicago vs the Eastern-stored value), shifting the fetch window +1h and **silently skipping** any record in that 1-hour shadow — hit all 6 sections and all clients; and (2) the fulfillment `GET /sales-orders` ran at `depth=5` and **timed out at 900s**, freezing the section before the fulfillment row was written. Proven on IF `6138135` (Eastern 2025-09-04 17:26:58). Fixes + the standing "cursor is always Eastern" rule live in the [per-client sync feature doc](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
 - **ITEM_RECEIPTS froze on a case-mismatched serial → duplicate-`Units` EV-10/1062, root-caused + fixed (2026-09-11, fix NOT deployed).** The receipts feed threw before its cursor PUT (freezing it and ITEM_FULFILLMENTS behind it) because `syncItemReceiptFromNetsuite` keyed its existing-`Units` PHP lookup on the DB's stored serial casing (`'na100'`) but read it with the uppercased serial (`'NA100'`); the missed row took the create path and collided on the case-insensitive `Units.itemId_serialNumber` unique key → MySQL 1062 / api2 EV-10. Proven on `Client_Nychh.Units` 23400-23409 (serials `'na100'..'na109'`, `c_netsuiteInternalInventoryAssignmentId` NULL). Library-only fix (`strtoupper()` on both the key and the reads). This is a **different** duplicate from the open sales-order `Items` duplicate below (658-BFZH). Mechanism + the EV-10 → `Logs_Nychh.Api` diagnosis traps: [per-client sync doc](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
 - **ITEM_FULFILLMENTS freeze RESOLVED — five stacked reconcile bugs fixed, deployed + verified (2026-09-13).** The frozen section (prod: 25,766 unit links for 1,192 real units on one fulfillment) is cleared: (1) unit dedup now reads existing units **flat + fully paginated** instead of from a truncated nested GET; (2) an unmatched fulfillment line **fails loud** instead of creating a never-matchable NULL-parent orphan (self-heal deletes existing orphans); (3) the SO-items fetch is flat + paginated (api2 truncated the nested list at 25); (4) fulfillment→SO line matching prefers the stable `orderLineUniqueKey` over the drifting line number; (5) a part-number last-resort match for lines whose NetSuite link points at a TAX line. **Package tracking numbers now import to all three bridges (317/318/319)** — the shim fetches `/package?expandSubResources=true` (a bare sub-resource GET returned no fields), so one NetSuite tracking number fans to the header, item, and every serial unit. Engine + shim detail: [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md), [SuiteQL/REST shim](../../1.0/apps/library/features/netsuite-suiteql-rest-shim.md).
+- **ITEM_FULFILLMENTS collapsing on a transient NetSuite blip — fixed in the shared client, NOT deployed (2026-09-17).** The section was collapsing again (mode `96000-RUNNING`, cursor frozen `2026-09-15 16:42:05`) while draining a ~255-fulfillment burst; a **different** read call site failed each run — the signature of a transient blip, not a poison record. Root cause is engine-wide (no per-record isolation), fixed by a transient-read retry in `App_Api_Netsuite_Rest::send()`. After the **worker** deploy, reset `Client_Nychh.Parameters` `NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS` to `864000-IDLE` (leave the cursor; it drains the ~261 waiting). Mechanism: [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md), [SuiteQL/REST shim](../../1.0/apps/library/features/netsuite-suiteql-rest-shim.md). **⚠ Two OPEN security defects surfaced in the same layer** (not fixed): `send()` logs the live `Authorization: Bearer` token to the API request log, and TLS peer verification is off for every `App_ApiTransaction` call — see the shim + library architecture docs.
 - **RESOLVED (2026-09-14): INVENTORY_ADJUSTMENTS destination wedge.** `POST /v2/transfer-orders` for **TO 277871** (SO 7102750) failed **EV-10 "destinationLocationId cannot be null"** every run because its customer **3997 "1896 Laurel Hill Management, LLC"** is not a NYCHH customer and has **no `Client_Nychh.Locations` row** (though it IS in `Customers`, id 19 — "maps to a Location" is the scope signal, not "is in Customers"). It reached NYCHH because the adjustment cascade scopes by the shared HQ warehouse then imports a fulfillment for a different customer. Fix: the importer now SKIPS out-of-scope transfer orders (loud log) instead of forcing a null destination; library deployed + verified. A follow-on regression from the added param froze SALES_ORDERS (a 3rd, direct cron call site, issue 754 "Too few arguments") — fixed; **redeploy worker**. See [NetSuite → TransferOrders import](./features/netsuite-transfer-order-import.md).
 - **NYCHH sync scope + data-gap scale + backfill approach (2026-09-09).**
   - **Scope = 22 NetSuite customers:** parent **28908** "NYC Health + Hospitals" + 21 hospital child customers (`24145, 29273, 29276, 31584, 31910-31916, 31918-31925, 32229, 33674`).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.829",
+  "version": "1.0.831",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",