toga-ai 1.0.816 → 1.0.818

package/knowledge/1.0/apps/store/INDEX.md

@@ -0,0 +1,6 @@
+# store (Store) — 1.0 knowledge
+
+| Doc | Summary |
+|-----|---------|
+| [Store (1.0 Storefront / SAML ACS) Architecture](architecture.md) | `store` is a framework **1.0** (`App_`) storefront app. |
+| [Store SAML SP Certificate & Private Keys (togahub.com)](features/saml-sso-certificate-and-keys.md) | How the `store` repo's SAML Service Provider credential — the `togahub.com` X.509 cert and its RSA private key — is stored, loaded and rotated. |

package/knowledge/1.0/apps/store/architecture.md

@@ -0,0 +1,72 @@
+---
+title: Store (1.0 Storefront / SAML ACS) Architecture
+framework: "1.0"
+repo: store
+project: Store
+client: shared
+type: architecture
+status: active
+updated: 2026-09-15
+owners: ["rgirish"]
+files:
+  - store/_/app/framework.php
+  - store/config.production.ini
+related:
+  - ../library/architecture.md
+  - ./features/saml-sso-certificate-and-keys.md
+  - ../../../clients/prudential/workflows/store-saml-deploy-topology.md
+---
+
+## Summary
+
+`store` is a framework **1.0** (`App_`) storefront app. It also hosts the **SAML Assertion
+Consumer Service (ACS)** for several clients on the `*.togahub.com` hostnames — the inbound
+IdP assertion is decrypted **here**, not in the 2.0 `saml` gateway. SAML lives inline in
+`store/_/app/framework.php` as a large `switch ($returnTo)` on the RelayState value, with one
+inline `$settingsInfo` block per client and a `case 'meta' / case 'acs'` pair inside each.
+
+The SP cert/key material lives in the sibling `library` repo under `library/ssl/toga/` and is
+read from disk at request time via `__LIBROOT__`.
+
+**Critical rules:** never paste private-key material into `framework.php` — read it from
+`library/ssl/toga/*.key` with `file_get_contents`. The SAML ACS host and the storefront host for
+a client can be **different EB environments**; check DNS before assuming a deploy target
+(see the Prudential store SAML deploy-topology workflow). Certificate and private key must be
+rotated **together** — the three inline SP blocks each carry an `x509cert` that has to match the
+key file.
+
+## SAML entry points
+
+Routing is by the RelayState value (`$_POST['RelayState']`, read as `$returnTo`), not by path:
+
+| Sub-route | Purpose |
+|---|---|
+| `?meta` | SP metadata XML from `OneLogin_Saml2_Settings::getSPMetadata()` |
+| `?acs`  | Assertion Consumer Service — decrypts the inbound `SAMLResponse` |
+
+Two SAML stacks are in play in the same file:
+- **php-saml (`OneLogin_Saml2_*`)** — builds the SP settings + `?meta` metadata.
+- **LightSAML + xmlseclibs** — does the actual `?acs` assertion decryption
+  (`decryptMultiAssertion`). See the SAML SP certificate & keys feature doc.
+
+The RUMCSI branch (`rumcsi.togaview.agilantsolutions.com`) skips decryption entirely and parses
+the raw assertion XML with `SimpleXMLElement` + XPath.
+
+## Handoff to the storefront
+
+After a successful assertion, the Prudential-style branches base64-encode a plain
+`xid:first:last:email` string and redirect to `<returnTo>?hash=<base64>`. This is **not
+encrypted and not signed** — unlike the 2.0 gateway's `?saml=` encrypted + uuid-validated
+handoff. See the "Client_True migration (deferred)" section of the SAML SP certificate & keys
+feature doc.
+
+## Known issues
+
+- `case 'meta'` sets the `Content-Type` header and `exit`s **without ever echoing
+  `$metadata`** — the `?meta` endpoint returns an empty body. Present in both `meta` blocks.
+- `'debug' => true` is set on all three SAML `$settingsInfo` blocks, and a `debugVar()` call
+  dumps SAML attributes (XID, name, email) to output.
+- `'strict'` is never set, so php-saml defaults it to `false` and skips condition, timestamp and
+  audience validation. Neither `wantAssertionsSigned` nor `wantMessagesSigned` is set.
+- Prudential's SAML config is ~700 lines of inline `switch` in one file with no
+  `_Model_<Client>_ClientAuthentication` equivalent.

package/knowledge/1.0/apps/store/features/saml-sso-certificate-and-keys.md

@@ -0,0 +1,188 @@
+---
+title: Store SAML SP Certificate & Private Keys (togahub.com)
+framework: "1.0"
+repo: store
+project: Store
+client: shared
+type: feature
+status: active
+updated: 2026-09-15
+owners: ["rgirish"]
+files:
+  - store/_/app/framework.php
+  - library/ssl/toga/togahub_com_2027-03-31.crt
+  - library/ssl/toga/toga_private_key_2027-03-31.key
+related:
+  - ../architecture.md
+  - ../../library/features/app-sso-initiation.md
+  - ../../../2.0/apps/saml/workflows/rotating-the-saml-sp-certificate.md
+  - ../../../2.0/standards/ssl-certificate-trust-model.md
+  - ../../../clients/prudential/workflows/store-saml-deploy-topology.md
+---
+
+## Summary
+
+How the `store` repo's SAML Service Provider credential — the `togahub.com` X.509 cert and its
+RSA private key — is stored, loaded and rotated. `store` decrypts inbound IdP assertions itself
+(LightSAML), so this credential is what makes Prudential/RUMCSI SSO work. It is **separate from**
+the 2.0 `saml` gateway's SP credential — rotating one does not rotate the other. (As of
+2026-09-15 the gateway's `/acs` loads its decrypt certs from AWS Secrets Manager, not from disk;
+`store` still reads files. See the SP cert-rotation workflow.)
+
+The cert and key files live in the sibling `library` repo at `library/ssl/toga/` and are read at
+request time through `__LIBROOT__`.
+
+## How it works
+
+### Where the material lives (never paste values)
+
+- **AWS Secrets Manager**, account **654654170868**, region **us-east-1**.
+- Secret: **`toga/ssl/togahub_com-casigned-2027-03-31`**.
+- JSON keys: `certificate`, `privateKey`, `certificateChain`, `passphrase`.
+- On disk (deployed with the `library` repo):
+  - `library/ssl/toga/togahub_com_2027-03-31.crt` — active cert
+  - `library/ssl/toga/togahub_com_2027-03-31-chain.crt` — chain
+  - `library/ssl/toga/toga_private_key_2027-03-31.key` — active key, **stored decrypted**, mode `600`
+  - `library/ssl/toga/togahub_com_2026-02-27.crt` + `toga_private_key.key` — previous pair,
+    kept only as the rotation fallback
+
+### Current certificate (rotated 2026-09-15)
+
+| | Old | New |
+|---|---|---|
+| File | `togahub_com_2026-02-27.crt` | `togahub_com_2027-03-31.crt` |
+| Issuer | DigiCert / Thawte | Amazon (ACM, `Amazon RSA 2048 M01`) |
+| Validity | expired **2026-09-16** | **2026-09-15 → 2027-03-31** |
+| SANs | 14 | **2** — `togahub.com`, `*.togahub.com` |
+
+**The new cert covers only `togahub.com`.** The six other domains the old cert carried
+(`togacommerce`, `togadesk`, `togaview`, `togasupply`, `togaretail`, `togaiq`, plus their
+wildcards) are **not** on it — each of those needs its own ACM cert on its own listener.
+
+### Dual-keypair decrypt (rotation overlap)
+
+`case 'acs'` in `store/_/app/framework.php` builds a `$samlKeyPairs` array — **active keypair
+first, previous keypair second** — then one `LightSaml\Credential\X509Credential` per readable
+pair, and passes the whole list to
+`$reader->decryptMultiAssertion($credentials, $decryptDeserializeContext)`.
+
+LightSAML's `decryptMulti()` loops the credentials, swallows each individual failure and returns
+on the first success, re-throwing only when all fail. So **no custom retry logic is needed** and
+it fails closed. An IdP still encrypting to the old cert and one already on the new cert both
+work during the overlap. Unreadable keypairs are skipped with `error_log`; an empty credential
+list throws `RuntimeException`.
+
+This mirrors the overlap the 2.0 `saml` gateway provides, though the gateway now does it with
+a multi-cert list out of Secrets Manager rather than two files on disk — see the SP
+cert-rotation workflow.
+
+### Two ACS handlers, two SAML libraries — tell them apart from the stack trace
+
+`store/_/app/framework.php` has **two** separate `case 'acs'` handlers that use **different SAML
+libraries**. They are easy to confuse when reading a production stack trace, and they run on
+**different servers**:
+
+| Handler | Library | Host / EB environment |
+|---|---|---|
+| The `togahub` host block's `case 'acs'` (the `$samlKeyPairs` dual-keypair decrypt) | **LightSAML** (`decryptMultiAssertion`) | `prudential.togahub.com` → `prudential-store-prod` |
+| The `prudential` host block's `case 'acs'` | **OneLogin_Saml2_Auth** (`isAuthenticated()`, `getNameId()`, `getAttribute('givenname'/'surname'/'emailaddress')`) | `prudential.togacommerce.com` → `true-togacommerce-prod` |
+
+**Diagnostic rule:** a trace with `LightSaml\Model\Assertion\*` frames is the **togahub** block
+(`prudential-store-prod`). A trace with `OneLogin` frames is the **prudential** block
+(`true-togacommerce-prod`). This is how a production decrypt error gets pinned to the right
+server — and the two are deployed by different pipeline actions, so fixing the wrong one changes
+nothing. See the Prudential store SAML deploy-topology workflow.
+
+### Private key loading
+
+The three inline `$settingsInfo` SP blocks read the key from disk:
+
+```php
+'privateKey' => trim(file_get_contents(__LIBROOT__ . '/ssl/toga/toga_private_key_2027-03-31.key')),
+```
+
+The key file must be **decrypted on disk** because `framework.php` calls
+`KeyHelper::createPrivateKey($path, '', true)` with an empty password — an encrypted PEM will not
+load.
+
+**Deploy model:** the `ssl/**/*.key` ignore rule was deliberately removed from
+`library/.gitignore` so the key ships with the repo. That is the team's current model for this
+key; it is a record of what we do, not a recommendation.
+
+## Gotchas
+
+- **PEM newlines are stored as SPACES in Secrets Manager.** The `certificate` / `privateKey`
+  values in `toga/ssl/togahub_com-casigned-2027-03-31` have their line breaks replaced by spaces.
+  `openssl` rejects the value straight out of the secret. Re-wrap the base64 body at 64 chars
+  before writing it to disk.
+- **The date in a secret name is the EXPIRY date, not the issue date.** The secret
+  `toga/ssl/togahub_com-casigned-2026-09-16` is **not** a newer cert — it holds the cert that was
+  already expiring. Read the dates out of the cert itself, never off the secret name.
+- **Rotate the cert and the key together.** Before 2026-09-15 the three SP blocks had the raw
+  private key pasted inline as a base64 string, and the `x509cert` values had been rotated while
+  those inline keys had not — cert and key no longer matched, which would have broken SSO on the
+  next deploy. Keys are now read from a file, so they cannot drift apart silently.
+- **Don't assume where the ACS runs.** The host that terminates `?acs` for a client may be a
+  different EB environment from the storefront. See the Prudential store SAML deploy-topology
+  workflow before deploying a cert change.
+- **A decrypt stack trace CANNOT tell you how many keys were tried.** In
+  `library/LightSaml/Model/Assertion/EncryptedElementReader.php`, `decryptMulti()` returns
+  `$this->decrypt($key)` from *inside* the `foreach`, and re-throws `$lastException` after the
+  loop. The re-thrown exception keeps the stack from where it was first thrown, so the captured
+  trace looks **identical** whether one key was tried or several all failed. Seeing
+  `decryptMultiAssertion` in a trace therefore does **not** prove the dual-keypair code is
+  deployed — the old single-key code called the same method with a one-item array. To tell them
+  apart, check the deployed EB `Deployment.VersionLabel`, or look for the new code's distinct log
+  line `SAML keypair not readable, skipping:`.
+- **A mismatched cert/key pair fails with an OAEP padding error, not a clear message.** The
+  symptom is `Failure decrypting Data (openssl private) -
+  RSA_padding_check_PKCS1_OAEP_mgf1: oaep decoding error` (also seen as
+  `RSA_EAY_PRIVATE_DECRYPT: padding check failed`) thrown out of the LightSAML
+  `EncryptedElementReader`. Confirm a suspected mismatch by comparing the **modulus** of the cert
+  and the key with `openssl x509 -noout -modulus` / `openssl rsa -noout -modulus` — identical
+  modulus means the pair matches. Compare the **committed git blobs**, not just the local working
+  files, or you will verify a pair that was never deployed.
+
+## Client_True migration (deferred 2026-09-15)
+
+Moving Prudential SAML onto the `Client_True` pattern (config `[saml]` section +
+`Core.ClientAuthentications` / `ClientAuthentications_Sso` rows + `App_Sso` + a
+`_Model_<Client>_ClientAuthentication`) was reviewed and **deferred to its own ticket** — the cert
+expired the next day and the migration is multi-day work.
+
+Reference implementation to copy when it is picked up: `tools/config.production.ini` `[saml]`
+section, `library/app/sso.php` (`App_Sso::initiate`), `tools/mvc/sso/initiate/get.php`,
+`tools/mvc/sso/get.php`.
+
+What the migration must handle:
+- Prudential today has ~700 lines of inline `switch` keyed on `$returnTo`, with certs pasted into
+  PHP, and no `_Model_Prudential_ClientAuthentication`.
+- Its handoff is a plain-base64 `?hash=` of `xid:first:last:email` — unencrypted and forgeable —
+  versus the encrypted, uuid-validated `?saml=` handoff `Client_True` uses.
+- Prudential **auto-creates users on first login**. `Client_True`'s `getAuthenticatedSsoUser()`
+  does not, so that behavior must be written in or SSO logins will start failing for new users.
+
+## Open security items (not fixed 2026-09-15)
+
+- The old private key is still in git history — `library` commits `6addc58f7` and `29d55bb97`,
+  plus the `store` `framework.php` history. Tracked as a separate compromised-credential ticket.
+- `'debug' => true`, missing `'strict'`, and the attribute-dumping `debugVar()` — see the store
+  architecture doc's Known issues.
+
+## Change history
+- 2026-09-15 — Documented that `framework.php` has two `case 'acs'` handlers on two different
+  SAML libraries (LightSAML for `togahub` vs OneLogin for `prudential`) running on two different
+  EB environments, and how to read a stack trace to tell which one failed. Recorded that a
+  LightSAML `decryptMulti()` trace cannot reveal how many keys were tried (re-thrown exception
+  keeps the original stack), so `decryptMultiAssertion` in a trace does not prove the
+  dual-keypair build is deployed — check the EB VersionLabel or the
+  `SAML keypair not readable, skipping:` log line. Added the OAEP-padding-error signature of a
+  mismatched cert/key pair and the openssl-modulus check against committed blobs. (rgirish)
+- 2026-09-15 — Rotated the `togahub.com` SP cert to the ACM-issued `togahub_com_2027-03-31.crt`
+  and added a dual-keypair fallback in `?acs` so both the old and new certs decrypt during the
+  overlap (`decryptMultiAssertion`). Removed three hardcoded RSA private-key blobs from
+  `store/_/app/framework.php` and replaced them with `file_get_contents` on
+  `library/ssl/toga/toga_private_key_2027-03-31.key`; chmod 600 the old world-readable key.
+  Recorded the Secrets Manager location, the space-instead-of-newline PEM gotcha, and that the
+  new cert carries only 2 SANs. Deferred the `Client_True` SAML migration. Commit `3839e49a`.
+  (rgirish)

package/knowledge/INDEX.md

@@ -14,6 +14,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
 - **test** (Test) — 16 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
 - **toga** (TOGa) — 3 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
+- **store** (Store) — 2 doc(s) → [1.0/apps/store/INDEX.md](1.0/apps/store/INDEX.md)
 - **tools** (Tools) — 18 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
 
 ## 2.0 framework

package/knowledge/clients/prudential/INDEX.md

@@ -16,3 +16,4 @@
 | [Prudential Financial](profile.md) | 2.0 | Prudential is a TOGA client whose device-fulfillment flow is driven by **Dell** via the Dell API (`Client_Prudential.Apis.id = 2`). |
 | [Prudential: Dell ASN failed POST backfill replay](workflows/dell-asn-backfill-replay.md) | 2.0 | When Dell ASN POSTs fail in bulk (e.g. |
 | [Prudential — HTTP/2 disabled on the Commerce ALB (proxy workaround) + open re-enable action](workflows/http2-alb-workaround.md) | 2.0 | Prudential users hit **`ERR_HTTP2_PROTOCOL_ERROR`** in Chrome and Edge (both Chromium) loading `https://prudential.togacommerce.com`, while the site worked fine |
+| [Prudential store/SAML Deploy Topology & the Wedged-Agent Deploy Timeout](workflows/store-saml-deploy-topology.md) | 1.0 | Where Prudential's SAML ACS actually runs versus where its storefront runs — they are **two different Elastic Beanstalk environments**, and the CodePipeline act |

package/knowledge/clients/prudential/profile.md

@@ -12,6 +12,7 @@ apps:
   - toga2-commerce
   - toga25-supply
   - saml
+  - store
 project: _Underscore
 client: prudential
 type: profile
@@ -33,6 +34,7 @@ related:
 - features/supply-orders-csv-export.md
   - features/order-email-pdf-attachments.md
   - features/itemclasses-device-tier-data.md
+  - workflows/store-saml-deploy-topology.md
 ---
 
 ## Summary
@@ -80,6 +82,13 @@ The order-created cron branches on this id; the shipped/delivered crons branch o
   breaks HTTP/2, causing `ERR_HTTP2_PROTOCOL_ERROR` in Chrome/Edge. HTTP/2 is currently disabled
   globally on the `true-togacommerce-prod` ALB as a workaround — **open action to re-enable once
   their IT fixes the proxy.** See workflows/http2-alb-workaround.md.
+- **SAML ACS and the storefront are DIFFERENT EB environments.** `prudential.togahub.com` (where
+  the SAML assertion is decrypted, in the 1.0 `store` repo) points at `prudential-store-prod`
+  (us-west-2, account 502614707982), deployed by the pipeline action **`PrudentialStore`**.
+  `prudential.togacommerce.com` (the storefront) points at `true-togacommerce-prod` and is
+  deployed by **`TrueProductionDeployment`** — so the action named "PrudentialStore" does *not*
+  deploy the storefront. `rumcsi.togahub.com` shares `prudential-store-prod`. See
+  workflows/store-saml-deploy-topology.md.
 - Dell will not change their payload shape — see the Dell ASN units interceptor feature doc for the
   PRE/POST translation that keeps their feed working after the tracking-number bridge migration.
 - Service request region is resolved from `Customers.name` string (`'USA'`/`'India'`/`'Ireland'`).
@@ -117,6 +126,10 @@ The order-created cron branches on this id; the shipped/delivered crons branch o
   linking, the "null null" contact-export incident + dedup/backfill remediation.
 - features/order-email-pdf-attachments.md — which PDF guide is attached to which customer
   order email at which stage, the S3 asset list, and the four crons that send them.
+- workflows/store-saml-deploy-topology.md — where Prudential's SAML ACS actually runs vs
+  the storefront, the split AWS accounts, and the wedged-agent EB deploy timeout.
+- ../../1.0/apps/store/features/saml-sso-certificate-and-keys.md — the togahub.com SP cert/key
+  that decrypts Prudential's assertions.
 - features/supply-orders-csv-export.md — the Prudential-only Sales Orders CSV export in TOGa 2.5
   Supply (`toga25-supply`): its two queries, why contacts join `SalesOrders.contactId` (not
   `Units.contactId`, NULL until Dell assigns the asset), and the `ServiceRequests_Units` fan-out.

package/knowledge/clients/prudential/workflows/store-saml-deploy-topology.md

@@ -0,0 +1,157 @@
+---
+title: Prudential store/SAML Deploy Topology & the Wedged-Agent Deploy Timeout
+framework: "1.0"
+project: Store
+client: prudential
+type: workflow
+status: active
+updated: 2026-09-15
+owners: ["rgirish"]
+files:
+  - store/_/app/framework.php
+related:
+  - ../profile.md
+  - ../../../1.0/apps/store/architecture.md
+  - ../../../1.0/apps/store/features/saml-sso-certificate-and-keys.md
+---
+
+## Summary
+
+Where Prudential's SAML ACS actually runs versus where its storefront runs — they are **two
+different Elastic Beanstalk environments**, and the CodePipeline action names point at the
+opposite one from what you would guess. Read this before deploying anything SAML-related for
+Prudential. It also records the EB deploy failure mode we hit on 2026-09-15 (a wedged command
+agent that times out after 14 minutes while the environment still shows Green).
+
+## The full login flow — the two legs run on two different servers
+
+The outbound (first) leg does **not** hop through togahub. It redirects the browser straight to
+Microsoft Entra:
+
+1. User hits `prudential.togacommerce.com` (**`true-togacommerce-prod`**). The `prudential`
+   host block in `store/_/app/framework.php` fires when the session host is `prudential`, the app
+   is not in test mode, and the session has no `accountId`.
+2. With no recognized `?ssoCommand`, it falls to the `default:` case, which calls
+   `$auth->login($originUrl)` on `OneLogin_Saml2_Auth`.
+3. That redirects to Prudential's Entra tenant:
+   `https://login.microsoftonline.com/d8fde2f5-9392-4260-8a03-0ad01f4746e9/saml2`.
+4. The SP block advertises entityId `https://prudential.togahub.com/?meta` and ACS
+   `https://prudential.togahub.com/?acs` — so **Entra POSTs the response to togahub**, i.e. the
+   inbound leg lands on **`prudential-store-prod`**, a different EB environment.
+5. `prudential-store-prod` decrypts the assertion (LightSAML), then redirects back to
+   `prudential.togacommerce.com/?hash=<base64 of xid:first:last:email>`.
+
+**Why this matters:** the outbound and inbound legs are different servers with different
+deploys. A deploy that succeeds on one does **not** fix the other. Decrypt/cert changes belong on
+`prudential-store-prod`.
+
+## The routing — the pipeline action names are misleading
+
+| Hostname | CNAME → EB environment | Deployed by pipeline action |
+|---|---|---|
+| `prudential.togacommerce.com` (storefront) | `true-togacommerce-prod.us-west-2` | **`TrueProductionDeployment`** |
+| `prudential.togahub.com` (**SAML ACS**) | `prudential-store-prod.us-west-2` | **`PrudentialStore`** |
+| `rumcsi.togahub.com` | `prudential-store-prod.us-west-2` | `PrudentialStore` |
+
+**The action named `PrudentialStore` does NOT deploy the Prudential storefront.**
+`TrueProductionDeployment` does. `PrudentialStore` deploys the host that decrypts the SAML
+assertion — so **cert / decrypt code changes must land on `prudential-store-prod`.**
+
+`rumcsi.togahub.com` shares that same environment, so a bad SAML deploy there breaks RUMCSI
+login too.
+
+### Accounts and regions (they are split)
+
+- **Pipeline:** `Store-Production`, AWS account **502614707982**, region **us-west-2**.
+  Not `us-east-1`, and not account `654654170868` where most other TOGA pipelines live.
+- **Route53** zones for `togacommerce.com` and `togahub.com` are in account **654654170868**,
+  while the EB environments they point at are in **502614707982**. You need both accounts to
+  trace a hostname end-to-end.
+
+## Failure mode — 14-minute command timeout with a Green environment
+
+Symptoms in the EB event log:
+
+```
+The following instances have not responded in the allowed command timeout time
+Command execution completed on all instances. Summary: [Successful: 0, TimedOut: 1]
+Failed to deploy application.
+```
+
+The environment still reports **Green / Ok**, which is misleading — the deploy never ran, but
+the old version is still serving traffic fine.
+
+### How to confirm it is a wedged command agent
+
+1. Check what the instance actually thinks it is running:
+   ```
+   aws elasticbeanstalk describe-instances-health --attribute-names All \
+     --environment-name prudential-store-prod --region us-west-2
+   ```
+   A wedged agent shows `Deployment.VersionLabel` still on the **old** build with
+   `Deployment.Status: Deployed`, while `CPU Idle` is ~99% and load average ~0.0. That means the
+   instance never picked up the deploy command at all — it is not busy, it is deaf.
+2. Check whether SSM sees the instance:
+   ```
+   aws ssm describe-instance-information --region us-west-2
+   ```
+   An **empty** result for the instance confirms the agent is not reporting.
+
+Observed 2026-09-15: `prudential-store-prod` on `i-07ae1b81ee48df938` (t3.micro, running since
+2026-07-17), stuck on an Aug 13 build. `spglobal-store-prod` failed identically on
+`i-04a6a50cffd37457e` — that environment has **no instances and no DNS pointing at it**.
+
+## What you need to do
+
+1. Confirm the diagnosis with the two AWS commands above before touching anything.
+2. Reboot the EC2 instance (EC2 console or `aws ec2 reboot-instances`).
+3. Watch the EB event log for `Command is executing on all instances` — that means the EB command
+   channel is back.
+4. Re-run the failed pipeline deploy.
+5. If it times out again, do **not** reboot a second time. Use an EB **Instance refresh** /
+   instance replacement instead.
+
+A reboot restored the EB command channel on 2026-09-15, but SSM still did not report after 3
+minutes — EB working again is the signal to retry, not SSM.
+
+## Incident 2026-09-15 — OAEP decrypt error caused by the stuck deploy
+
+Production Sentry showed, from the togahub (LightSAML) ACS handler:
+
+```
+Failure decrypting Data (openssl private) - RSA_padding_check_PKCS1_OAEP_mgf1: oaep decoding error
+```
+(also seen as `RSA_EAY_PRIVATE_DECRYPT: padding check failed`)
+
+**Root cause: not a bad key — a deploy that never landed.** `prudential-store-prod` was still
+running the pre-rotation code (commit `275d16a9`, an Aug 13 build) because its deploy kept timing
+out with the wedged-agent failure above. That old code loaded a **mismatched cert/key pair**: the
+`togahub_com_2025-03-25.crt` cert with `toga_private_key.key`, which belongs to the 2026-02-27
+cert. Confirmed by comparing the two openssl moduli — they differ. The new dual-keypair fallback
+fixes this, but it was not on the box.
+
+The `library` repo was verified **clean and not the cause**: all three new files committed and
+pushed on `_production`, both keypairs (2027 active and 2026-02-27 fallback) verified as matching
+pairs against the **committed git blobs** (not just local files), both keys unencrypted (no
+`Proc-Type` / `DEK-Info`), and both loading via the exact call the code makes —
+`openssl_pkey_get_private($pem, "")`.
+
+**Lesson:** when a SAML decrypt error appears right after a cert rotation, check *what build is
+actually deployed* before you suspect the key material. Verify against committed blobs, and note
+that the stack trace alone cannot tell you which build is running (see the cert & keys feature
+doc's gotcha on `decryptMulti`).
+
+## Change history
+- 2026-09-15 — Traced the outbound SAML leg: `prudential.togacommerce.com` redirects the browser
+  straight to Prudential's Entra tenant (no togahub hop), and Entra POSTs back to
+  `prudential.togahub.com/?acs` on a different EB environment — so a deploy to one leg does not
+  fix the other. Recorded the OAEP decrypt incident: the error came from `prudential-store-prod`
+  still serving the pre-rotation Aug 13 build (`275d16a9`) with a mismatched cert/key pair,
+  because its deploy kept timing out; the `library` repo was verified clean against committed
+  git blobs. (rgirish)
+- 2026-09-15 — Created while rotating the `togahub.com` SAML cert. Documented that
+  `prudential.togahub.com` (SAML ACS) and `prudential.togacommerce.com` (storefront) are separate
+  EB environments with swapped-sounding pipeline action names, the split AWS accounts/regions,
+  and the wedged-command-agent deploy timeout with its `describe-instances-health` /
+  `describe-instance-information` diagnosis and reboot-then-instance-refresh remediation.
+  (rgirish)

package/knowledge/clients/rumcsi/profile.md

@@ -5,12 +5,13 @@ apps:
   - togadesk
   - togaview
   - library
+  - store
 project: TOGa Desk
 client: rumcsi
 type: profile
 status: active
-updated: 2026-08-21
-owners: ["mhammontree"]
+updated: 2026-09-15
+owners: ["mhammontree", "rgirish"]
 files: []
 related:
   - ../../1.0/apps/togadesk/features/per-client-host-restriction.md
@@ -19,6 +20,7 @@ related:
   - ../../1.0/apps/togadesk/features/email-to-ticket-intake.md
   - ../../1.0/apps/togadesk/architecture.md
   - ../../1.0/apps/togaview/architecture.md
+  - ../../1.0/apps/store/features/saml-sso-certificate-and-keys.md
 ---
 
 ## Summary
@@ -124,6 +126,12 @@ hardcoded to `clientId = 2`; no RUMC row has `adPeopleId`/`ldap_user`), and
 - **No SAML auto-provisioning.** Unlike towfoundation and newcenturyholdingsllc, the rumcsi
   branch only `SELECT`s `people`; it never creates them. That is precisely why the bulk load was
   necessary.
+- **`rumcsi.togahub.com` runs on the `prudential-store-prod` EB environment** (1.0 `store` repo,
+  us-west-2, account 502614707982) — the same host as Prudential's SAML ACS. A bad SAML/cert
+  deploy there breaks RUMCSI login too. RUMCSI's ACS branch skips assertion decryption (raw
+  `SimpleXMLElement` + XPath), so the SP cert rotation does not affect it directly. See
+  ../../1.0/apps/store/features/saml-sso-certificate-and-keys.md and
+  ../prudential/workflows/store-saml-deploy-topology.md.
 
 ## Change history
 

package/knowledge/registry.json

@@ -211,6 +211,13 @@
     "role": "app",
     "dependsOn": []
   },
+  {
+    "repo": "store",
+    "project": "Store",
+    "framework": "1.0",
+    "role": "app",
+    "dependsOn": []
+  },
   {
     "repo": "toga-blox",
     "project": "TOGa Blox",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.816",
+  "version": "1.0.818",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",