toga-ai 1.0.817 → 1.0.818

package/knowledge/1.0/apps/store/features/saml-sso-certificate-and-keys.md

@@ -76,6 +76,23 @@ This mirrors the overlap the 2.0 `saml` gateway provides, though the gateway now
 a multi-cert list out of Secrets Manager rather than two files on disk — see the SP
 cert-rotation workflow.
 
+### Two ACS handlers, two SAML libraries — tell them apart from the stack trace
+
+`store/_/app/framework.php` has **two** separate `case 'acs'` handlers that use **different SAML
+libraries**. They are easy to confuse when reading a production stack trace, and they run on
+**different servers**:
+
+| Handler | Library | Host / EB environment |
+|---|---|---|
+| The `togahub` host block's `case 'acs'` (the `$samlKeyPairs` dual-keypair decrypt) | **LightSAML** (`decryptMultiAssertion`) | `prudential.togahub.com` → `prudential-store-prod` |
+| The `prudential` host block's `case 'acs'` | **OneLogin_Saml2_Auth** (`isAuthenticated()`, `getNameId()`, `getAttribute('givenname'/'surname'/'emailaddress')`) | `prudential.togacommerce.com` → `true-togacommerce-prod` |
+
+**Diagnostic rule:** a trace with `LightSaml\Model\Assertion\*` frames is the **togahub** block
+(`prudential-store-prod`). A trace with `OneLogin` frames is the **prudential** block
+(`true-togacommerce-prod`). This is how a production decrypt error gets pinned to the right
+server — and the two are deployed by different pipeline actions, so fixing the wrong one changes
+nothing. See the Prudential store SAML deploy-topology workflow.
+
 ### Private key loading
 
 The three inline `$settingsInfo` SP blocks read the key from disk:
@@ -108,6 +125,23 @@ key; it is a record of what we do, not a recommendation.
 - **Don't assume where the ACS runs.** The host that terminates `?acs` for a client may be a
   different EB environment from the storefront. See the Prudential store SAML deploy-topology
   workflow before deploying a cert change.
+- **A decrypt stack trace CANNOT tell you how many keys were tried.** In
+  `library/LightSaml/Model/Assertion/EncryptedElementReader.php`, `decryptMulti()` returns
+  `$this->decrypt($key)` from *inside* the `foreach`, and re-throws `$lastException` after the
+  loop. The re-thrown exception keeps the stack from where it was first thrown, so the captured
+  trace looks **identical** whether one key was tried or several all failed. Seeing
+  `decryptMultiAssertion` in a trace therefore does **not** prove the dual-keypair code is
+  deployed — the old single-key code called the same method with a one-item array. To tell them
+  apart, check the deployed EB `Deployment.VersionLabel`, or look for the new code's distinct log
+  line `SAML keypair not readable, skipping:`.
+- **A mismatched cert/key pair fails with an OAEP padding error, not a clear message.** The
+  symptom is `Failure decrypting Data (openssl private) -
+  RSA_padding_check_PKCS1_OAEP_mgf1: oaep decoding error` (also seen as
+  `RSA_EAY_PRIVATE_DECRYPT: padding check failed`) thrown out of the LightSAML
+  `EncryptedElementReader`. Confirm a suspected mismatch by comparing the **modulus** of the cert
+  and the key with `openssl x509 -noout -modulus` / `openssl rsa -noout -modulus` — identical
+  modulus means the pair matches. Compare the **committed git blobs**, not just the local working
+  files, or you will verify a pair that was never deployed.
 
 ## Client_True migration (deferred 2026-09-15)
 
@@ -136,6 +170,14 @@ What the migration must handle:
   architecture doc's Known issues.
 
 ## Change history
+- 2026-09-15 — Documented that `framework.php` has two `case 'acs'` handlers on two different
+  SAML libraries (LightSAML for `togahub` vs OneLogin for `prudential`) running on two different
+  EB environments, and how to read a stack trace to tell which one failed. Recorded that a
+  LightSAML `decryptMulti()` trace cannot reveal how many keys were tried (re-thrown exception
+  keeps the original stack), so `decryptMultiAssertion` in a trace does not prove the
+  dual-keypair build is deployed — check the EB VersionLabel or the
+  `SAML keypair not readable, skipping:` log line. Added the OAEP-padding-error signature of a
+  mismatched cert/key pair and the openssl-modulus check against committed blobs. (rgirish)
 - 2026-09-15 — Rotated the `togahub.com` SP cert to the ACM-issued `togahub_com_2027-03-31.crt`
   and added a dual-keypair fallback in `?acs` so both the old and new certs decrypt during the
   overlap (`decryptMultiAssertion`). Removed three hardcoded RSA private-key blobs from

package/knowledge/clients/prudential/workflows/store-saml-deploy-topology.md

@@ -23,6 +23,28 @@ opposite one from what you would guess. Read this before deploying anything SAML
 Prudential. It also records the EB deploy failure mode we hit on 2026-09-15 (a wedged command
 agent that times out after 14 minutes while the environment still shows Green).
 
+## The full login flow — the two legs run on two different servers
+
+The outbound (first) leg does **not** hop through togahub. It redirects the browser straight to
+Microsoft Entra:
+
+1. User hits `prudential.togacommerce.com` (**`true-togacommerce-prod`**). The `prudential`
+   host block in `store/_/app/framework.php` fires when the session host is `prudential`, the app
+   is not in test mode, and the session has no `accountId`.
+2. With no recognized `?ssoCommand`, it falls to the `default:` case, which calls
+   `$auth->login($originUrl)` on `OneLogin_Saml2_Auth`.
+3. That redirects to Prudential's Entra tenant:
+   `https://login.microsoftonline.com/d8fde2f5-9392-4260-8a03-0ad01f4746e9/saml2`.
+4. The SP block advertises entityId `https://prudential.togahub.com/?meta` and ACS
+   `https://prudential.togahub.com/?acs` — so **Entra POSTs the response to togahub**, i.e. the
+   inbound leg lands on **`prudential-store-prod`**, a different EB environment.
+5. `prudential-store-prod` decrypts the assertion (LightSAML), then redirects back to
+   `prudential.togacommerce.com/?hash=<base64 of xid:first:last:email>`.
+
+**Why this matters:** the outbound and inbound legs are different servers with different
+deploys. A deploy that succeeds on one does **not** fix the other. Decrypt/cert changes belong on
+`prudential-store-prod`.
+
 ## The routing — the pipeline action names are misleading
 
 | Hostname | CNAME → EB environment | Deployed by pipeline action |
@@ -92,7 +114,41 @@ Observed 2026-09-15: `prudential-store-prod` on `i-07ae1b81ee48df938` (t3.micro,
 A reboot restored the EB command channel on 2026-09-15, but SSM still did not report after 3
 minutes — EB working again is the signal to retry, not SSM.
 
+## Incident 2026-09-15 — OAEP decrypt error caused by the stuck deploy
+
+Production Sentry showed, from the togahub (LightSAML) ACS handler:
+
+```
+Failure decrypting Data (openssl private) - RSA_padding_check_PKCS1_OAEP_mgf1: oaep decoding error
+```
+(also seen as `RSA_EAY_PRIVATE_DECRYPT: padding check failed`)
+
+**Root cause: not a bad key — a deploy that never landed.** `prudential-store-prod` was still
+running the pre-rotation code (commit `275d16a9`, an Aug 13 build) because its deploy kept timing
+out with the wedged-agent failure above. That old code loaded a **mismatched cert/key pair**: the
+`togahub_com_2025-03-25.crt` cert with `toga_private_key.key`, which belongs to the 2026-02-27
+cert. Confirmed by comparing the two openssl moduli — they differ. The new dual-keypair fallback
+fixes this, but it was not on the box.
+
+The `library` repo was verified **clean and not the cause**: all three new files committed and
+pushed on `_production`, both keypairs (2027 active and 2026-02-27 fallback) verified as matching
+pairs against the **committed git blobs** (not just local files), both keys unencrypted (no
+`Proc-Type` / `DEK-Info`), and both loading via the exact call the code makes —
+`openssl_pkey_get_private($pem, "")`.
+
+**Lesson:** when a SAML decrypt error appears right after a cert rotation, check *what build is
+actually deployed* before you suspect the key material. Verify against committed blobs, and note
+that the stack trace alone cannot tell you which build is running (see the cert & keys feature
+doc's gotcha on `decryptMulti`).
+
 ## Change history
+- 2026-09-15 — Traced the outbound SAML leg: `prudential.togacommerce.com` redirects the browser
+  straight to Prudential's Entra tenant (no togahub hop), and Entra POSTs back to
+  `prudential.togahub.com/?acs` on a different EB environment — so a deploy to one leg does not
+  fix the other. Recorded the OAEP decrypt incident: the error came from `prudential-store-prod`
+  still serving the pre-rotation Aug 13 build (`275d16a9`) with a mismatched cert/key pair,
+  because its deploy kept timing out; the `library` repo was verified clean against committed
+  git blobs. (rgirish)
 - 2026-09-15 — Created while rotating the `togahub.com` SAML cert. Documented that
   `prudential.togahub.com` (SAML ACS) and `prudential.togacommerce.com` (storefront) are separate
   EB environments with swapped-sounding pipeline action names, the split AWS accounts/regions,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.817",
+  "version": "1.0.818",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",