toga-ai 1.0.1189 → 1.0.1191

package/knowledge/1.0/apps/library/features/address-validation-gateway.md

@@ -6,16 +6,17 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: [jcardinal]
+updated: 2026-10-29
+owners: [jcardinal, rgirish]
 files:
   - library/app/api/officedepot.php
   - library/app/api/usps.php
 related:
   - ../../toga/features/odp-customer-search-loyalty-persistence.md
+  - ../../../../clients/office-depot/profile.md
 ---
 
-The shared 1.0 address-validation gateway (`App_Api_OfficeDepot::validateAddress` → USPS fallback); open when standardizing/verifying a typed address before save, or debugging a saved-raw-address bug.
+The shared 1.0 address-validation gateway (`App_Api_OfficeDepot::validateAddress` → USPS fallback); open when standardizing/verifying a typed address before save, debugging a saved-raw-address bug, or an address step that hangs.
 
 ## Summary
 
@@ -27,17 +28,23 @@ Entry points:
 - `library/app/api/officedepot.php` — `App_Api_OfficeDepot::validateAddress()`, the gateway.
 - `library/app/api/usps.php` — `App_Api_USPS::validateAddress()`, the fallback carrier.
 
-1. **Endpoint selection.** Calls the ODP **sandbox** endpoint when `App_Registry::inTestMode()` is true (`inTestMode` = `config['internal']['test_mode'] > 0`), and the **live** ODP endpoint otherwise. Production is effectively hard-pinned to live (comment "2024-01-22 JC - Always using production...").
-2. **On a real ODP answer** (HTTP 200, accept or reject) returns ODP's standardized result.
-3. **On any non-200 ODP response** (sandbox down/expired, etc.) falls back to `App_Api_USPS::validateAddress()` and returns the USPS result — a genuine standardization **or** a genuine bad-address rejection both count as a real answer and are returned.
-4. **Only when BOTH ODP and USPS are unavailable** returns `array('success' => true)` with no standardized fields (fail-open, so associates are never hard-blocked by an outage).
+1. **Endpoint selection.** Calls the ODP **sandbox** endpoint when `App_Registry::inTestMode()` is true (`inTestMode` = `config['internal']['test_mode'] > 0`), and the **live** ODP endpoint otherwise. Production is effectively hard-pinned to live (comment "2024-01-22 JC - Always using production..."). ODP is called directly (no proxy) and only answers allowlisted source IPs — see [Office Depot profile](../../../../clients/office-depot/profile.md) gotchas.
+2. **On a real ODP answer** (HTTP 200 with `validationResult`, accept or reject) returns ODP's standardized result.
+3. **On any ODP failure** — non-200, or `$api->execute()` throwing (it throws on a non-JSON body, e.g. a 403 HTML page) — logs `ODP call failed (HTTP <code>)` and falls back to `App_Api_USPS::validateAddress()`. A USPS standardization **or** bad-address rejection is a real answer and is returned.
+4. **USPS down** = result has `serviceUnavailable => true` (USPS 401/403) or one of the two service-down error strings. Then the gateway fails open: `success => true` + a "could not be validated... click OK to continue" message, so associates are never hard-blocked.
 
 ## Gotchas
 
-- **Beta-bypass class of bug (fixed 2026-08-13).** Previously the ODP-failure `else` branch returned `array('success' => true)` with **no** standardized address fields and the USPS fallback was commented out. Beta/test uses the ODP **sandbox**; when the sandbox was down/expired, callers saw `success:true` and saved the **raw typed address** (wrong city, wrong casing) with no validation. Production was unaffected (working live ODP endpoint). The fix restores the USPS fallback so validation only truly fails open when **both** carriers are down.
-- **"Both unavailable" must check BOTH USPS service-down strings.** USPS signals unavailability two ways: `'Address validation service is temporarily unavailable'` **and** `'USPS OAuth authentication failed'`. The fail-open condition must match either. Checking only the first wrongly blocks associates when ODP is down *and* USPS OAuth also fails (a php-reviewer pass caught this).
-- **Front-end write-back anti-pattern (widespread, mostly UNFIXED).** Many 1.0 front-end callbacks use the gateway response only as a pass/fail **gate**, then rebuild the save payload from the raw DOM inputs — discarding the standardized `address1`/`city`/`state`/`zipCode` the gateway returned. So even a successful validation never persists the corrected address. Known flows still with this anti-pattern: `toga/app/togarefresh2026/servicerequests/finalizequote.php`, `toga/app/togarefresh2026/appointment/select.php`, `toga/app/servicerequests/builder.php`, `toga/app/loaners/checkout.php`, `toga/app/jobjackets/view.php`, and the `togarefresh2023` variants. The 2026-08-13 gateway fix protects all from the beta bypass, but the write-back was only corrected in the `togarefresh2026` customersearch flow (see related doc). When touching any of these, write the standardized address back into the inputs before building the save payload.
+- **`usps.php` is shared — changes must be backward compatible.** Callers: `officedepot.php`, togadesk `desk/includes/classes/class.repair.php` (~L648/711, reads only `success` + address fields), store `mvc/account/address/shipping/verify/post.php` (echoes the JSON). Add new keys; never change existing values or meanings.
+- **USPS fallback cannot validate since 2026-08-01 (OPEN).** USPS added Addresses API access controls: our account gets 401/403 "The requested information is not authorized for access to Addresses API". Fix = an Addresses API license in the USPS Business Portal (cop.usps.com, My Account > API Licenses). Until then, when ODP fails every address fails open unvalidated.
+- **ODP failure used to hang the page (fixed 2026-10-29).** An ODP 403 HTML body made `App_ApiTransaction::execute()` throw → `ajax.php` 500 → browser `Unexpected token '<'` in `jsonDecode` → endless spinner (seen on the `togarefresh2026_customersearch` Next button). Keep `execute()` inside the try/catch and keep the 200-path guards (`isset($responseBody->validationResult)`, `(string)($errorDesc ?? '')`).
+- **"Both unavailable" must check every USPS down signal.** `serviceUnavailable`, `'Address validation service is temporarily unavailable'`, and `'USPS OAuth authentication failed'`. Missing one wrongly blocks associates when ODP and USPS are both down.
+- **Front-end write-back anti-pattern (widespread, mostly UNFIXED).** Many 1.0 front-end callbacks use the gateway response only as a pass/fail **gate**, then rebuild the save payload from the raw DOM inputs — discarding the standardized `address1`/`city`/`state`/`zipCode`. Known flows still with it: `toga/app/togarefresh2026/servicerequests/finalizequote.php`, `toga/app/togarefresh2026/appointment/select.php`, `toga/app/servicerequests/builder.php`, `toga/app/loaners/checkout.php`, `toga/app/jobjackets/view.php`, and the `togarefresh2023` variants. Only the `togarefresh2026` customersearch flow is fixed (see related doc). When touching any of these, write the standardized address back into the inputs before building the save payload.
 
 ## Related
 
 - `../../toga/features/odp-customer-search-loyalty-persistence.md` — the customersearch save flow, where the write-back fix was applied.
+
+## Change history
+- 2026-10-29 — Catch ODP `execute()` throw (403 HTML → endless spinner) and fall back to USPS; USPS 401/403 adds `serviceUnavailable` and the gateway fails open; USPS license still open (TRUE-83387, commit 0d7aaf08) (rgirish)
+- 2026-08-13 — Restored USPS fallback so beta (ODP sandbox down) no longer saves the raw typed address; fail open only when both carriers are down (jcardinal)

package/knowledge/1.0/apps/test/workflows/legacy-1-0-web-apps-in-toga-prod.md

@@ -6,7 +6,7 @@ project: Test
 client: shared
 type: workflow
 status: active
-updated: 2026-10-28
+updated: 2026-10-29
 owners: ["rgirish"]
 files:
   - test/team/aws/create_elastic_beanstalk.php
@@ -60,6 +60,7 @@ Shared by both:
 - DNS: EB alias zone us-west-2 `Z38NKT9BP95V3O`.
 - `ebs/register_instance_to_shared_application_load_balancer.php` hardcodes us-west-2 but skips any env name containing `-prod` — it does nothing here.
 - The 2.0 DBs (`writer.core` / `writer.client.database.togahub.com`, us-east-1, 10.201.x) are now cross-region for these apps — a small part of their traffic.
+- Outbound IP: all 3 private web subnets route through NAT `nat-0dc3b341a685b0bac` = `54.203.171.146` (VPC owner account 339712893798, not open to developers). This is the IP an IP-allowlisting partner (ODP) must allow.
 
 TOGa Retail: 2 instances. SNI: `togahub.com` / `*.togahub.com`, `*.toga.agilantsolutions.com` / `*.toga.togatechnology.com`. DNS: toga-prod `toga.togatech.com`, `odp.toga.togatechnology.com` (A alias), `chat.togahub.com` (CNAME, TTL 300); legacy `agilantsolutions.com` zone: `odp-toga`, `odp.toga`, `staples.toga`, `synnex.toga` (A alias). ODP hosts 302 to the Azure SAML login.
 
@@ -101,15 +102,16 @@ ODP calls it "sandbox"; internally it **is beta** (replaces the old shared `agil
 Done for TOGa Retail (TRUE-83361) and TOGa Desk (TRUE-83360), us-east-1 → us-west-2. EB app/env/pipeline names are per region, so the new region reuses them. New region = new Redis, so every user is logged out once.
 
 1. Create the env in the new region (steps above) on the target VPC; check it reaches the DB network and the DB SG allows 3306.
-2. Request ACM certs in the new region. If the DNS validation CNAMEs already exist in the same account, they issue in minutes (validation records are per account + domain, shared across regions).
-3. Point `SESSION_REDIS_SAVE_PATH` at the Redis cluster in the **same** region; add the env's instance SG to that Redis SG on 6379.
-4. Create the pipeline in the new region with that region's artifact bucket.
-5. Test each host against the new env CNAME (`curl --resolve` or Host header): same status and page size as the old env.
-6. UPSERT every DNS record (both accounts) to the new env's alias zone.
-7. Wait until the old ALB `RequestCount` is 0 (staff browsers keep old keep-alive connections ~10 min after the DNS change).
-8. Save the old env config to a file (EB saved configs die with the EB app — see Gotchas).
-9. Remove only the old env's instance SG rule from the old-region Redis SG ingress (us-east-1 `sg-03e79bcb9c105e100`). Never delete the us-east-1 `togadesk-sessions` cluster — Vision, View, Store, Walmart and Tools still use it (TRUE-83179).
-10. Delete the old pipeline, terminate the old env, delete the old EB app.
+2. Find the new outbound NAT IP. If the app calls a partner that allowlists IPs (ODP for TOGa Retail), get that IP allowlisted **before** the DNS cutover. TRUE-83361 missed this: ODP returned 403 from the cutover on (TRUE-83387).
+3. Request ACM certs in the new region. If the DNS validation CNAMEs already exist in the same account, they issue in minutes (validation records are per account + domain, shared across regions).
+4. Point `SESSION_REDIS_SAVE_PATH` at the Redis cluster in the **same** region; add the env's instance SG to that Redis SG on 6379.
+5. Create the pipeline in the new region with that region's artifact bucket.
+6. Test each host against the new env CNAME (`curl --resolve` or Host header): same status and page size as the old env.
+7. UPSERT every DNS record (both accounts) to the new env's alias zone.
+8. Wait until the old ALB `RequestCount` is 0 (staff browsers keep old keep-alive connections ~10 min after the DNS change).
+9. Save the old env config to a file (EB saved configs die with the EB app — see Gotchas).
+10. Remove only the old env's instance SG rule from the old-region Redis SG ingress (us-east-1 `sg-03e79bcb9c105e100`). Never delete the us-east-1 `togadesk-sessions` cluster — Vision, View, Store, Walmart and Tools still use it (TRUE-83179).
+11. Delete the old pipeline, terminate the old env, delete the old EB app.
 
 ### Where DNS lives
 
@@ -141,6 +143,7 @@ SAML ACS URLs are hard-coded (store: `prudential.togahub.com/?acs`, then redirec
 - **Old env names in other docs are history.** `prudential-store-prod`, `true-togacommerce-prod`, `spglobal-store-prod`, `*-togadesk-prod` and pipeline `Store-Production` no longer exist.
 
 ## Change history
+- 2026-10-29 — us-west-2 outbound NAT IP; region move must get partner IP allowlists (ODP) before DNS cutover (TRUE-83387) (rgirish)
 - 2026-10-28 — TOGa Desk prod moved to us-west-2 (3 x t4g.large, 4 certs, 7 DNS records), wait-for-drain step, keep us-east-1 Redis (TRUE-83360) (rgirish)
 - 2026-10-28 — TOGa Retail prod moved to us-west-2 (next to the 1.0 DBs), region-move steps, EB app/SG/cert gotchas (TRUE-83361) (rgirish)
 - 2026-10-27 — TOGa Retail sandbox for ODP (= beta internally), `-sandbox` host strip, nonprod S3 grants, `_beta` drift (TRUE-83279) (rgirish)

package/knowledge/1.0/apps/toga/features/odp-customer-search-loyalty-persistence.md

@@ -6,11 +6,12 @@ project: TOGa
 client: office-depot
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: [jcardinal]
+updated: 2026-10-29
+owners: [jcardinal, rgirish]
 files:
   - toga/app/togarefresh2026/customersearch/view.php
   - library/app/model/customer.php
+  - library/app/model/servicerequest.php
 related:
   - ./bundleconfirmation-cart-preservation.md
   - ../../library/features/address-validation-gateway.md
@@ -43,5 +44,11 @@ The "Existing PC Services" flow — `content=togarefresh2026_customersearch`, Ne
 - **Backfill deferred.** Records that went null between 2023-11-28 and this fix are NOT backfilled — a separate one-time backfill is still owed.
 - **CRITICAL pre-existing SQL-injection risk (unfixed — needs its own ticket).** The address-lookup query in `saveNewCustomer` (`library/app/model/customer.php`, ~lines 538–543) interpolates `streetAddress`/`line2`/`city`/`state`/`zip` into `$sqlCheckAddress` with **no `App_Database::sqlEscape()` and no casting**. This was not introduced by this session and was left unfixed per scope; it must be escaped per the SQL-injection rules before this method is touched again.
 
+- **Loyalty lookup survives an ODP outage, but only partly.** `library/app/model/servicerequest.php` `ajaxRefreshCustomerSearch` / `ajaxCustomerSearch` wrap ODP `execute()` in try/catch and fall through to a local TOGa `Customers` search (phone/email/name/stored `merchantId`). During an ODP block (e.g. IP not allowlisted, TRUE-83387) associates see only customers already in TOGa. The catch blocks are empty — an ODP failure leaves no log line.
+
 ## Related
 - `./bundleconfirmation-cart-preservation.md`
+
+## Change history
+- 2026-10-29 — Noted: loyalty lookup falls back to local TOGa Customers search when ODP fails (TRUE-83387) (rgirish)
+- 2026-09-16 — merchantId persisted through togarefresh2026 save; standardized address written back (jcardinal)

package/knowledge/1.0/apps/vision/workflows/deploy-to-elastic-beanstalk-al2023.md

@@ -6,8 +6,8 @@ project: Vision
 client: shared
 type: workflow
 status: active
-updated: 2026-10-24
-owners: ["jcardinal"]
+updated: 2026-10-29
+owners: ["jcardinal", "rgirish"]
 files:
   - vision/.ebextensions/009_setup_phpini.config
   - vision/.ebextensions/020_setup_git_libraries.config
@@ -74,6 +74,8 @@ Vision serves every `.php` file directly, so deny rules are the only guard:
 5. ~100 `mysql_*` calls (dead since PHP 7.0).
 6. php-fpm warns "Module redis/igbinary/msgpack already loaded" — `02-install-php-redis.sh` loads the extension ini twice (same in tools).
 7. `config.prod.ini` still has old us-west-2 RDS hosts + unmounted catalog paths.
+8. **Copyright watchdog is off since the AL2023 port (fe8940ce, 2026-10-23).** `cr.php` (called from the shared library JS on every 1.0 page) sends `Access-Control-Allow-Origin` twice: `.htaccess` `Header set` + PHP `header()` no longer merge under php-fpm, so the browser gets `*, *`, rejects it, and the check fails quietly (console CORS error only). Fix = drop one of the two headers. Before that, check `Vision_Log.CopyrightHosts`: once the check works, any calling host not listed there is redirected to `unauthorized.html` (`toga.togatech.com` etc. are listed with `redirectTo` NULL). Left as is on purpose in TRUE-83387.
 
 ## Change history
+- 2026-10-29 — Found: double CORS header disables the `cr.php` copyright watchdog since the AL2023 port; not changed (TRUE-83387) (rgirish)
 - 2026-10-23 — Created: Vision revived on PHP 8.5 / AL2023 in the prod account; folder hook, Redis sessions, s3fs `asifiles`, deny rules, open security list (TRUE-83056) (jcardinal)

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,10 +6,11 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-26
+updated: 2026-10-29
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
+  - worker1.5/crons/toga2/netsuite/common_sync_togasupply.php
   - worker/crons/toga2/netsuite/sync_togasupply_canon.php
   - worker/crons/toga2/netsuite/sync_togasupply_elite.php
   - worker/crons/toga2/netsuite/sync_togasupply_hh.php
@@ -150,8 +151,12 @@ plain thrown warning instead.
 
 **Fastest path to the real error (2026-09-29):**
 
-- `N-RUNNING` with N shrinking run after run = that section throws every run. SALES_ORDERS aborts the whole
-  script, so the sections after it freeze too — fix SO first.
+- `N-RUNNING` with N shrinking run after run = that section throws every run. One run does the sections in
+  order SALES_ORDERS → PURCHASE_ORDERS → INVOICES → ITEM_RECEIPTS → ITEM_FULFILLMENTS →
+  INVENTORY_ADJUSTMENTS, and a throw aborts the run, so every section **after** the thrower freezes too
+  (cursors stop; earlier sections stay current). Fix the first frozen section first.
+- Poison record: in `Logs_<Client>.Api`, find the PUT of `NETSUITE_EXECUTION_MODE_<SECTION>` = `<n>-RUNNING`;
+  the last GETs after it name the record the run died on.
 - Read the api2 rejection in `Logs_<Client>.Api`: `responseCode >= 400 AND responsePayload LIKE '%"api":"Agilant"%'`,
   newest first, `LIMIT 1` + tight date filter.
 - **`Logs.Issue` keeps only the FIRST trace per fingerprint** — its trace can name the wrong client and line.
@@ -2088,15 +2093,20 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
   `customerId=3`; Prudential adds the `Customers` join + `CLIENT_UUID_PRUDENTIAL_CUSTOMER`; no other client
   changed, no else/default, and the guard still throws for a genuinely-missing SO. PHP 7.2-safe
   (`$options['join'] ?? []`), signature unchanged.
+  - **S&P Global branch added 2026-10-29 (TRUE-83380, library `1bea35a4`, PR #917).** S&P's extra row is a
+    customerless copy (`customerId` NULL) from the 2025-09-10 data load, never touched by the sync. Scope:
+    `Customers` join + `Customers.c_netsuiteInternalCustomerId = (int)($nsInvoice->entity->internalId ?? 0)`
+    (the invoice's NetSuite customer, same idea as `syncSalesOrderFromNetsuite`). Fires only when NetSuite
+    changes an invoice on one of those old orders. Details: [S&P Global profile](../../../clients/spglobal/profile.md).
+  - `syncItemFulfillmentFromNetsuite`'s SO lookup is also unscoped, but on 2 rows it falls back to
+    `syncSalesOrderFromNetsuite` (scoped), so it does not throw.
   - Same trap hit the closed-order delete 2026-10-05 (SALES_ORDERS freeze, Issue 940) — 4th path; see [closed-order rule](../../library/features/netsuite-closed-order-rule.md). Every SO-by-NS-id lookup needs this scope.
-  - **⚠ OPEN follow-up (own ticket): six OTHER clients also keep two SalesOrders per NetSuite id, and the
-    fix does NOT scope them.** A read-only sweep found the same dual-SO pattern in **Growrk (2789), Canon
-    (2890), Endeavorhealth (4828), Spglobal (2893), Browardsheriff (160), Masonite (17)** (Nychh/Elite/Quad/Sbasite
-    = 0; Compass 28377). Dormant today — only Compass is actually throwing the invoice guard (0 "refusing to
-    import invoice" events for any other client in the last 5 days) — but if any of those six ever bills an
-    invoice against a duplicated SO it will false-throw the same way, and **their duplicates have a DIFFERENT
-    cause than Compass's Office-Depot dual-source.** Also a data-quality question: why do six non-Compass
-    clients have duplicate SalesOrders rows at all.
+  - **⚠ OPEN follow-up (own ticket): five OTHER clients also keep two SalesOrders per NetSuite id, and the
+    fix does NOT scope them.** A read-only sweep (2026-09-17) found the same dual-SO pattern in **Growrk (2789),
+    Canon (2890), Endeavorhealth (4828), Browardsheriff (160), Masonite (17)** (Nychh/Elite/Quad/Sbasite
+    = 0; Compass 28377). Dormant until NetSuite changes an invoice on a duplicated SO; then it false-throws
+    the same way (S&P hit exactly this on 2026-10-28, now scoped above). **Their duplicates have a DIFFERENT
+    cause than Compass's Office-Depot dual-source.**
 - **⚠ "refusing to import invoice … billed sales order not imported" can mean the SO is OLDER than the
   SO cursor (Adyen, 2026-10-01, issue 910).** Invoices from 2026-07-02 on billed 10 SOs last-modified
   2025-11-04 → 2026-06-15, but `NETSUITE_LAST_SYNC_CURSOR_SALES_ORDERS` was 2026-07-31, so the SO sync
@@ -2201,6 +2211,7 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
   [NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
 
 ## Change history
+- 2026-10-29 — S&P Global INVOICES froze at `1-RUNNING` (from 2026-10-28) on an invoice billing an SO with a customerless duplicate row; billed-SO lookup now scoped by the invoice's NetSuite customer for S&P (library `1bea35a4`, prod-verified, all 6 S&P sections IDLE). Section run order + abort cascade, poison-record tip. TRUE-83380. (bala)
 - 2026-10-26 — Prudential INVOICES froze on 9 old invoices whose Jan/Feb 2025 SOs were never imported; unblocked by a NetSuite `memo` bump on each SO (no code). Gotcha + blocker query added. TRUE-83152. (bala)
 - 2026-10-19 — Compass ITEM_FULFILLMENTS froze on a serial moved between IFs in NetSuite (1062, Issue 468); update branch now repoints within the item + prunes stale assignment links; SuiteQL paging throws on bad pages + unique ORDER BY. Fix rule 1 widened (no hand cleanup). Library only, not deployed. TRUE-82938. (jcardinal)
 - 2026-10-18 — NYCHH SALES_ORDERS froze on unknown warehouse NS location 1 (Issue 1041) after the 10-14 cursor rewind; fixed by adding the Location row, not the skip flag. Data only. TRUE-82859. (jcardinal)

package/knowledge/2.0/apps/api2/features/v2-rest-query-contract.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-10-28
+updated: 2026-10-29
 owners: [tcox, bala, apeterson, jcardinal, rgirish]
 files:
   - api2/Component/Api/V2/V2.php
@@ -422,6 +422,10 @@ where=(field:op:value,LOGIC,field:op:value,(nested,OR,nested))
   through a join alias: `ojoin=Cases:Cases.id=Tickets.caseId&where=(Cases.uuid:eq:<uuid>)` (or a
   `TableViewJoins` row that gives the alias). **Self-join (FK to the same table, e.g.
   `parentLocation.uuid`) needs an alias:** `ojoin=Locations@p:p.id=Locations.parentLocationId&where=(p.uuid:eq:<uuid>)`.
+  **Any plain field of a joined table works, not only `uuid` (incl. `c_` fields):**
+  `join=Customers:Customers.id=SalesOrders.customerId&where=(Customers.c_netsuiteInternalCustomerId:eq:27446)`
+  on `/v2/sales-orders` (verified prod 2026-10-29). Qualified names are used as is (~L6998); only `._`
+  routes to HAVING (~L3571/8648); `join` emits an INNER JOIN (~L3613).
   A path through an **inherent child**
   (`ticketUsers.user.uuid`) does work — but only when the **first** segment is an inherent child
   of the route (`buildChildPathWhereSql`). A path that starts with a forward FK hop fails
@@ -698,6 +702,7 @@ If you need the behaviour, loop uuids client-side. Point 3 is the durable, non-o
   [V2 request logging](request-logging.md).
 
 ## Change history
+- 2026-10-29 — `where` on a joined table's plain/`c_` field works (`Customers.c_netsuiteInternalCustomerId`), verified prod (TRUE-83380). (bala)
 - 2026-10-28 — `sort` orders only the main record query; nested child sets come back unordered, sort them client-side (TRUE-81848). (bala)
 - 2026-10-27 — `where` unsupported on PUT/PATCH; uuid-less PUT/PATCH with no identifying field now 400 `EV-1` instead of a full-table load + `EV-14` (TRUE-83276). (jcardinal)
 - 2026-10-24 — blox JS `null` in `where` → `= ''` → 1525 500, send `"null"`; dotted FK path fails in `fields` too; cross-cluster FK → filter client-side (TRUE-82916). (jcardinal)

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-27
+updated: 2026-10-29
 owners: ["jcardinal", "mhammontree", "bala", "rgirish", "snaredla", "amadda"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -640,6 +640,13 @@ production deploy.**
 - `_Query::fetchRow()` returns an object, not an array — access `$row->field`.
 - Keep the metric POST non-fatal (`throwExceptionsOnFailure` off) so a monitoring outage never breaks the
   worker job it rides in.
+- **"Stopped reporting / is late" can be a false alarm: the push to OneUptime timed out.** Signal:
+  `Core.WorkerJobs.output` ends with `OneUptime push FAILED.` while the check itself says `alarm=OK`, and
+  `executionTime` is ~10s higher (`$PUSH_TIMEOUT_SECONDS` in `Monitor/Compass.php` / `Operations.php`).
+  To tell a OneUptime problem from a real outage, count `ok` vs `FAILED` per minute with a `LIKE` on
+  `output` for `action LIKE 'Monitor/%'`: many monitors failing together with `alarm=OK`, while non-monitor
+  jobs run normally, = OneUptime side. Seen 2026-10-29 from 09:42 CT: about half of all pushes failed for
+  30+ minutes. No code change.
 - **Two branches adding a monitor to the same `Monitor/<Client>.php` conflict if both insert at the
   same spot.** Blank-line padding does not help — git merges by surrounding context. Put the new
   method somewhere else (e.g. after the last helper) so the hunks sit far apart. Check with
@@ -658,6 +665,7 @@ production deploy.**
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
 
 ## Change history
+- 2026-10-29 — Gotcha: "stopped reporting" incidents from failed OneUptime pushes (`OneUptime push FAILED.` + `alarm=OK` in `WorkerJobs.output`) and how to tell them from a real outage (TRUE-83380). (bala)
 - 2026-10-27 — `EmailedAsnImport` moved off php-imap2 to `_Component_Email_Inbound_Imap` (webklex), TLS now validated, no ext-imap (TRUE-82933). (jcardinal)
 - 2026-10-27 — Added `OfficeDepotLostOrders` (TRUE-82736); legacy `Logs`/`Common` via `DB_VISION_LOGS`; per-run `resetQueryCache`; worker2 project status/severity ids + export fields. (amadda)
 - 2026-10-27 — SFTP monitors: reuse `Sftp/Import` config + login when it already reads the SFTP; linked NYCDOE `StuckAsnFiles` (TRUE-82941). (mhammontree)

package/knowledge/clients/office-depot/profile.md

@@ -18,7 +18,7 @@ project: TOGa Voice
 client: office-depot
 type: profile
 status: active
-updated: 2026-10-27
+updated: 2026-10-29
 owners: [akhokhani, mhammontree, jcardinal, snaredla, rgirish]
 files: []
 related:
@@ -29,6 +29,7 @@ related:
   - ../../1.0/apps/togadesk/architecture.md
   - ../../1.0/apps/toga/features/odp-esd-mcafee-descriptions.md
   - ../../1.0/apps/toga/features/odp-tech-services-job-jacket-answers.md
+  - ../../1.0/apps/library/features/address-validation-gateway.md
 ---
 
 Office Depot uses TOGa Voice for inbound store-associate tech support and Desk ticketing; open for client scope and links to its voice configuration.
@@ -51,3 +52,7 @@ Office Depot is the production support client of the standalone Python/LiveKit T
 ## Gotchas
 
 The audience is store staff. Preserve the retail-misroute branch and verify store/work order before linking a repeat caller from a shared phone. Call artifacts contain personal information; retain only aggregate diagnostics in shared knowledge. A created ticket does not prove post-call attachments succeeded.
+
+- **ODP APIs accept only allowlisted source IPs.** `inpr.odazcloud.com` (Akamai) returns 403 HTML to any other IP. There is no proxy any more: `proxy.agilantsolutions.com` (EB `agilant-proxy-prod`) was terminated 2026-10-23 and library calls ODP directly (commit 75aac84a). So the IP ODP must allowlist is the outbound NAT IP of the env that calls it. TOGa Retail prod (us-west-2) = `54.203.171.146` (NAT `nat-0dc3b341a685b0bac`, shared VPC `true-production-vpc-2`, owner account 339712893798); allowlist request sent 2026-10-29, pending (TRUE-83387). Any region/VPC move: get the new NAT IP allowlisted before the DNS cutover — see [legacy 1.0 web apps in toga-prod](../../1.0/apps/test/workflows/legacy-1-0-web-apps-in-toga-prod.md).
+- **While ODP is blocked:** address validation falls back to USPS ([gateway](../../1.0/apps/library/features/address-validation-gateway.md)); loyalty lookup shows only customers already in TOGa ([customer search](../../1.0/apps/toga/features/odp-customer-search-loyalty-persistence.md)).
+- **Dead proxy still called:** `toga/_/browser/form/input/autocomplete/customerlookup.php` (~L35) still calls `proxy.agilantsolutions.com`; only used by the DOA page `toga/app/jobjackets/viewDOA.php`. Unfixed.

package/knowledge/clients/spglobal/profile.md

@@ -11,11 +11,12 @@ project: Worker
 client: spglobal
 type: profile
 status: active
-updated: 2026-10-21
-owners: ["jcardinal", "mhammontree", "snaredla", "rgirish"]
+updated: 2026-10-29
+owners: ["jcardinal", "mhammontree", "snaredla", "rgirish", "bala"]
 files:
   - store/_/app/framework.php
   - store/common/spglobal/welcome/get.php
+  - library/app/api/toga2.php
 related:
   - ../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
   - ../../1.0/apps/library/features/app-cloud-aws-credentials.md
@@ -25,11 +26,16 @@ S&P Global TOGa-Supply NetSuite integration client on the shared 1.0 worker+libr
 
 ## Summary
 
-- TOGa-Supply NetSuite integration client on the shared 1.0 NetSuite → TOGa Supply per-client sync (`worker` + `library`).
-- NetSuite transactions sync into its 2.0 `Client_<Id>` tenant via the TOGa2 API through a thin `worker/crons/toga2/netsuite/sync_togasupply_*.php` wrapper.
+- TOGa-Supply NetSuite integration client on the shared 1.0 NetSuite → TOGa Supply per-client sync (`worker` + `library`). 2.0 DB `Client_Spglobal`; library client uuid const `App_Api_Toga2::CLIENT_UUID_SPGLOBAL`.
+- NetSuite transactions sync into its 2.0 tenant via the TOGa2 API through a thin `crons/toga2/netsuite/sync_togasupply_*.php` wrapper.
 - Profile seeded 2026-08-29 during root-cause work on stuck togasupply integrations: S&P Global was exposed to the item-fulfillment "Undefined index" on an empty `orderLine` (`syncItemFulfillmentFromNetsuite`), fixed in shared code. Mechanics + diagnosis playbook: [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
 
 ## Integration touchpoints
 
 - **`worker` / `library` (1.0)** — shared togasupply sync engine + `App_Api_Toga2` bridge.
 - **`store` (1.0) — PunchOut from SAP Ariba** (Classic Buyer + Guided Buying) lands on `spglobal.togacommerce.com`: spglobal host block in `store/_/app/framework.php`, client code in `store/common/spglobal/`. Not vision (that is Canon's flow). A store HTTP 500 shows in Guided Buying as "Unable to connect to supplier's website". Outage 2026-10-20 (TRUE-82959): store missing `aws/aws-sdk-php`, see [1.0 AWS credentials](../../1.0/apps/library/features/app-cloud-aws-credentials.md#gotchas).
+
+## Gotchas
+
+- **Customerless duplicate SalesOrders.** Many NetSuite SO ids have 2 rows in `Client_Spglobal.SalesOrders`: a copy with `customerId` NULL from the 2025-09-10 data load (the sync never touches it) and the synced row (`customerId` 1 = S&P GLOBAL, NetSuite customer 27446). Older invoices link to the customerless copies. Any SO lookup by `c_netsuiteInternalSalesOrderId` alone returns 2 rows for those orders.
+- **INVOICES froze on this 2026-10-28 (TRUE-83380, fixed + prod-verified 2026-10-29).** NetSuite changed an invoice on one of those old orders; `syncInvoiceFromNetsuite`'s billed-SO lookup got 2 rows and threw "refusing to import invoice ... is not imported in toga" every run, freezing INVOICES at `1-RUNNING` and the three sections after it. Fix: an S&P branch scopes the lookup by the invoice's NetSuite customer (`Customers.c_netsuiteInternalCustomerId`). See [per-client sync, billed-SO scope gotcha](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1189",
+  "version": "1.0.1191",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",