toga-ai 1.0.887 → 1.0.889

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-22
+updated: 2026-09-24
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -96,7 +96,7 @@ The single most expensive misreading of this engine. `startModeIteration()` writ
 `<seconds>-RUNNING` at the **start** of every section and `finishModeIteration()` writes `-IDLE` at
 the end; the ÷3 `REDUCE_INTERVAL_FACTOR` shrink fires only because the *previous* run never reached
 its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in the script** — so
-"didn't finish" can only mean **it threw**. A collapsing window therefore means *go read the error
+"didn't finish" can only mean **it threw**. **Exception (pre-2026-09-24 fix): INVOICES collapses with NO throw whenever its cursor is within 24h of now** — see Gotchas. A collapsing window therefore means *go read the error
 in prod* (see below), never *tune the window size down*. (Corollary: a window that has grown back to
 the cap is positive evidence the section is completing.)
 

package/knowledge/2.0/apps/saml/architecture.md

@@ -6,7 +6,7 @@ project: SAML SSO Gateway
 client: shared
 type: architecture
 status: active
-updated: 2026-09-22
+updated: 2026-09-24
 owners: ["rgirish", "jcardinal"]
 files:
   - saml/index.php
@@ -79,6 +79,7 @@ Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentica
 - **CodePipeline:** Source (GitHub `_production`) → ManualApproval → Deploy (EB).
 - **Composer deps** installed at postdeploy via `015_install_composer.sh`.
 - **Secrets Manager read access.** Shared `ElasticBeanstalk-EC2-Instance-Profile` carries an inline `SecretsManagerReadAll` policy (`secretsmanager:GetSecretValue` + `DescribeSecret`, `Resource: "*"`) so `/acs` can read the SP encryption secrets at runtime. Account-wide `Resource: "*"` was chosen deliberately (shared profile across EB apps); verified effective access with the IAM policy simulator.
+- **Non-prod reads prod secrets cross-account (2026-09-24).** SP secrets live only in prod acct 654654170868, encrypted with customer-managed KMS `alias/toga-secrets`; non-prod EB roles read them cross-account (key policy + secret resource policy + role inline policy `SecretsManagerCrossAccountRead`). Sandbox 975050298201 done; beta 502614707982 not yet. Procedure + add-an-account steps: SP cert-rotation workflow.
 - **Non-prod shared-ALB self-registration (2026-08-27).** Postdeploy hook `060_register_instance_to_shared_application_load_balancer.sh` + `ebs/register_instance_to_shared_application_load_balancer.php` register a non-production `saml` instance into the target group named exactly its EB env name; **skips `production`** (so `saml-production` is unaffected) and always `exit 0`. Requires `aws/aws-sdk-php` (now in `composer.json`, lock/vendor to be regenerated) and two IAM actions on the instance profile (`elasticloadbalancing:DescribeTargetGroups` region-scoped + `RegisterTargets` scoped to the non-prod target-group ARN). See [ALB target-group auto-registration](../worker2/features/alb-target-group-auto-registration.md).
 
 ## Dependencies
@@ -95,6 +96,7 @@ Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentica
 - **RelayState time check bounds the future, not the past.** `MAX_TIME_TO_LIVE` (300s) sanity-checks how far in the *future* a RelayState timestamp may be. It does not clearly bound replay of an *old* captured RelayState — review before relying on it as replay protection.
 - **`/meta` cert lines must be `trim()`-ed (CRLF).** The `.crt` files in `_underscore/Assets/ssl` are CRLF-encoded; cert-format code does `explode("\n", ...)` and indents each line, so without a per-line `trim()` a stray `\r` ends up on every base64 line inside `<ds:X509Certificate>`. Trim each line in both signing and encryption blocks.
 - **SP encryption certs load from Secrets Manager at request time.** `/acs` holds no encryption key material on disk (`_underscore/Assets/ssl/` no longer carries the SP encryption key). Rotation = edit the `ENCRYPTION_SECRET_NAMES` list + the secrets themselves, not a redeploy of on-disk `.crt`/`.key` files.
+- **Non-prod SAML depends on prod KMS/Secrets Manager; one SP key serves every env** — a non-prod leak is a prod key leak. `ENCRYPTION_SECRET_NAMES` must hold full ARNs (short names resolve only in the caller's account).
 - **Committed AWS key must be rotated.** A hardcoded AWS key was removed from `_underscore/Cloud.php`; because it was committed to git history it is compromised — rotate it. Still open: plaintext creds in `production.ini` and the hardcoded Sentry DSN.
 - **SLS not implemented.** `/sls` falls through to the default banner.
 - **`set_exception_handler(null)` at top of `saml()`.** Unhandled exceptions output raw PHP errors. All exceptions from `getAuthenticatedSsoUser()` are caught and sent to Sentry.

package/knowledge/2.0/apps/saml/workflows/rotating-the-saml-sp-certificate.md

@@ -6,8 +6,8 @@ project: SAML SSO Gateway
 client: shared
 type: workflow
 status: active
-updated: 2026-09-16
-owners: ["jcardinal"]
+updated: 2026-09-24
+owners: ["jcardinal", "rgirish"]
 files:
   - saml/Controller/Index.php
   - _underscore/Cloud.php
@@ -46,7 +46,7 @@ Because a cert change is not picked up by every IdP at once, safe rotation split
 
 ### Multi-cert decryption from Secrets Manager (2026-09-15)
 `/acs` no longer reads SP encryption keys from disk. It sources **N** SP certs from AWS Secrets Manager and decrypts against all at once, so a partner can move to a different cert while other IdPs stay on the existing one — a superset of the earlier new/old overlap.
-- **Two class constants in `saml/Controller/Index.php` drive it:** `ENCRYPTION_SECRET_NAMES` — an **ordered list of Secrets Manager secret NAMES** (not values); `ENCRYPTION_SECRET_REGION` — `'us-east-1'`.
+- **Two class constants in `saml/Controller/Index.php` drive it:** `ENCRYPTION_SECRET_NAMES` — an **ordered list of Secrets Manager secret full ARNs** (not values; full ARN incl. random suffix required since 2026-09-24 — see Cross-account access); `ENCRYPTION_SECRET_REGION` — `'us-east-1'`.
 - **How `/acs` builds credentials:** for each name it calls `_Cloud::getSecret($name, $region)` (see [_Cloud Secrets Manager helper](../../_underscore/features/cloud-secrets-manager.md)), builds an `X509Credential`, then calls `decryptMultiAssertion($credentials, ...)`. An assertion encrypted to ANY listed cert decrypts.
 - **Fails closed with coded responses:** `Invalid SAMLResponse (Code = 5)` if no usable credential can be built at all; `Code = 6` if credentials build but every decrypt fails. Each per-secret build is wrapped in try/catch — a bad/unreadable secret is skipped and logged, so one broken secret does not 500 every login.
 - **Passphrase-aware, format-tolerant key loading:** PEM stored with escaped `\n` / `\r\n` is normalized to real newlines before loading. The secret's `passphrase` is passed to `KeyHelper::createPrivateKey` **only** when the private-key PEM header is the `ENCRYPTED PRIVATE KEY` variant (ACM public-cert exports are always encrypted); a plaintext key loads with no passphrase. A large "how this works" comment sits above `ENCRYPTION_SECRET_NAMES` in the controller.
@@ -60,6 +60,16 @@ AWS Secrets Manager, account **654654170868**, region **us-east-1**. Each secret
 - `toga/ssl/togahub_com-casigned-2027-03-31` — publicly-trusted (ACM exportable) cert.
 - `toga/ssl/togahub_com-casigned-2026-09-16` — publicly-trusted (ACM exportable) cert.
 
+Full ARNs (suffixes): `...:secret:toga/ssl/togahub_com-selfsigned-2036-S2r3SR`, `...-casigned-2027-03-31-7CgTyg`, `...-casigned-2026-09-16-7FW9ng`.
+
+### Cross-account access for non-prod (2026-09-24)
+Decision: ONE copy of each SP secret, in prod acct **654654170868**; non-prod accounts read it cross-account (no copies). Fixed non-prod `Invalid SAMLResponse (Code = 5)` (secrets did not exist in sandbox acct).
+- **KMS:** secrets re-encrypted from `aws/secretsmanager` (AWS-managed key — cannot be shared cross-account) to prod customer-managed key **`alias/toga-secrets`** (us-east-1, key id `bb201536-5e61-4c53-bff2-63cb2bff7b86`). Key policy: prod root `kms:*`; each non-prod account root `kms:Decrypt` + `kms:DescribeKey` with condition `kms:ViaService = secretsmanager.us-east-1.amazonaws.com`.
+- **Secret resource policy** on each `toga/ssl/*` secret: allow the non-prod role `GetSecretValue` + `DescribeSecret`.
+- **Non-prod role inline policy** `SecretsManagerCrossAccountRead`: secretsmanager on `arn:aws:secretsmanager:us-east-1:654654170868:secret:toga/ssl/*` + kms `Decrypt`/`DescribeKey` on the key ARN.
+- **Accounts granted:** sandbox-client **975050298201**, role `aws-elasticbeanstalk-ec2-role` — done. Beta **502614707982** (`true-saml-beta`) — **NOT done**; repeat all 3 steps with that account id.
+- **Add a non-prod account** = key policy + every secret resource policy + inline policy on that account's EB instance role. New secret = new resource policy + encrypt with `alias/toga-secrets` (not the default key).
+
 ### Why a public cert was added (Prudential)
 Prudential's IdP **refuses a self-signed cert, pins our exact leaf, and cannot auto-refresh our metadata by URL** — that combination forces a publicly-trusted cert. The correct source is an **ACM exportable public cert** (gives cert + private key), not ACM Private CA. Prudential moves to the public cert while other IdPs stay on the self-signed one; all coexist in the `/acs` multi-cert array. Full rationale: [SSL certificate trust-model strategy](../../standards/ssl-certificate-trust-model.md).
 
@@ -101,6 +111,10 @@ Remaining follow-ups (not yet done):
 
 - **Trim each cert line (CRLF).** The `.crt` files in `_underscore/Assets/ssl` are **CRLF-encoded**. The `/meta` cert-format code does `explode("\n", ...)` then indents each line. Without a per-line `trim()`, a stray carriage return (`\r`) is embedded on every base64 line inside `<ds:X509Certificate>` in the metadata XML. Fix: `trim()` each line in the cert-format code — applied to both the signing and encryption blocks. (Latent bug that pre-dated this change in the single original block.)
 - **Signing cannot flip until every IdP loads the new metadata** (IdPs cache up to 7 days) — flipping early makes signature-verifying IdPs reject our login requests.
+- **Use full secret ARNs in `ENCRYPTION_SECRET_NAMES`.** A short name resolves only in the caller's own account; full ARNs work from prod and cross-account.
+- **Non-prod SAML depends on prod KMS + Secrets Manager.** One SP key serves every env — a sandbox leak is a prod key leak. Separate non-prod cert = open decision.
+- **New secrets must use `alias/toga-secrets`**, not `aws/secretsmanager` — the AWS-managed key cannot be shared, so non-prod gets Code 5.
+- **Claude Code auto mode blocks cross-account grants** ("Permission Grant") even with `Bash(aws:*)` allowed — run those aws commands with auto mode off.
 - **The private key is only in Secrets Manager** (`toga/ssl/togahub_com-selfsigned-2036`) — retrieve at cutover, never commit or paste.
 
 ## Related

package/knowledge/clients/nychh/profile.md

@@ -70,6 +70,7 @@ TOGA 2.0 client on `_underscore`, prod schema `Client_Nychh`. Supply-chain / ful
 - 🚨 **sandbox-client (`Core` + `Client_Nychh`) was reset from PRODUCTION again on 2026-09-04** — second reset in two days (first, 2026-09-03, over the stale `purchase-orders` TableView). Every "verified on <date>" sandbox-client claim in this profile dated **before 2026-09-04 now describes whatever prod had**. The 2026-09-01 → 2026-09-03 `dbchanges2` migrations may need re-running on sandbox-client. Re-read the environment before trusting any sandbox state; procedure + re-run order in [Targeting the right database](../../2.0/apps/dbchanges2/workflows/targeting-the-right-database.md).
 - **Transfer orders push OUT to NetSuite as $0 sales orders - LIVE and verified 2026-09-08.** NYCHH is the first and only client on that path; switched on purely by its `transfer-orders` (record 312) `POST`/`POST` interceptor row. NetSuite SO **289187** (id 7463537) was created from it. **The NetSuite customer is the CLIENT - 4490 / 28908 - on every transfer order, never the destination location** (a first cut sent the destination and put SO 7462031 on sub-customer 33674 "CHS"; `Locations` 1-26 are "Shipping" type mapping to sub-customers under 4490, and the bug hid because almost every order picks location 2, whose id *is* 28908). Origin location is still the NetSuite **warehouse**, and `TransferOrders.description` now goes out as the NetSuite `memo`, shown as **ORDER DESCRIPTION**. See [TransferOrder → NetSuite push](./features/transfer-order-netsuite-push.md).
 - **Transfer orders closed in NetSuite never reached the Closed stage - 18 stuck on `Pending`, oldest 2022 (fix written 2026-09-08, NOT deployed).** The 1.0 sync skips a `Closed` order before the transfer-order branch, so the importer's `case 'Closed'` was dead code. Two NYCHH gates make this confusing to verify: the transfer-order status refresh is nested inside `IS_ENABLED_INTEGRATION_SALES_ORDERS` (Jeff enabled the feeds 2026-08-28 `b063f05f` and disabled sales orders, POs, invoices and item receipts on 2026-09-03 `2b321e9d` - where every NYCHH cursor stops), and the seek cursor never re-reads a passed order. 🚨 **Their `NETSUITE_LAST_SYNC_CURSOR_ITEM_RECEIPTS` is stale at 2024-08-06**, so re-enabling that feed replays ~2 years. See [NetSuite → TransferOrders import](./features/netsuite-transfer-order-import.md).
+- **INVOICES mode stuck at `5-RUNNING` with no error (2026-09-24, worker fix NOT deployed).** Cursors were reset to 2026-09-24; INVOICES has a 24h hold-back and its skip path never wrote IDLE, so the window collapsed silently. After deploy set `NETSUITE_EXECUTION_MODE_INVOICES` to `864000-IDLE`; invoices modified before the reset need a rewind/re-import. See [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
 - **Missing NetSuite records (item fulfillments + others) root-caused and fixed (2026-09-09).** Two separate bugs in the 1.0 NetSuite→TOGa Supply sync starved NYCHH imports, both deployed and verified in prod: (1) the resume cursor was **read on the wrong clock** (`strtotime()` in Chicago vs the Eastern-stored value), shifting the fetch window +1h and **silently skipping** any record in that 1-hour shadow — hit all 6 sections and all clients; and (2) the fulfillment `GET /sales-orders` ran at `depth=5` and **timed out at 900s**, freezing the section before the fulfillment row was written. Proven on IF `6138135` (Eastern 2025-09-04 17:26:58). Fixes + the standing "cursor is always Eastern" rule live in the [per-client sync feature doc](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
 - **ITEM_RECEIPTS froze on a case-mismatched serial → duplicate-`Units` EV-10/1062, root-caused + fixed (2026-09-11, fix NOT deployed).** The receipts feed threw before its cursor PUT (freezing it and ITEM_FULFILLMENTS behind it) because `syncItemReceiptFromNetsuite` keyed its existing-`Units` PHP lookup on the DB's stored serial casing (`'na100'`) but read it with the uppercased serial (`'NA100'`); the missed row took the create path and collided on the case-insensitive `Units.itemId_serialNumber` unique key → MySQL 1062 / api2 EV-10. Proven on `Client_Nychh.Units` 23400-23409 (serials `'na100'..'na109'`, `c_netsuiteInternalInventoryAssignmentId` NULL). Library-only fix (`strtoupper()` on both the key and the reads). This is a **different** duplicate from the open sales-order `Items` duplicate below (658-BFZH). Mechanism + the EV-10 → `Logs_Nychh.Api` diagnosis traps: [per-client sync doc](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
 - **ITEM_FULFILLMENTS freeze RESOLVED — five stacked reconcile bugs fixed, deployed + verified (2026-09-13).** The frozen section (prod: 25,766 unit links for 1,192 real units on one fulfillment) is cleared: (1) unit dedup now reads existing units **flat + fully paginated** instead of from a truncated nested GET; (2) an unmatched fulfillment line **fails loud** instead of creating a never-matchable NULL-parent orphan (self-heal deletes existing orphans); (3) the SO-items fetch is flat + paginated (api2 truncated the nested list at 25); (4) fulfillment→SO line matching prefers the stable `orderLineUniqueKey` over the drifting line number; (5) a part-number last-resort match for lines whose NetSuite link points at a TAX line. **Package tracking numbers now import to all three bridges (317/318/319)** — the shim fetches `/package?expandSubResources=true` (a bare sub-resource GET returned no fields), so one NetSuite tracking number fans to the header, item, and every serial unit. Engine + shim detail: [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md), [SuiteQL/REST shim](../../1.0/apps/library/features/netsuite-suiteql-rest-shim.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.887",
+  "version": "1.0.889",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",