toga-ai 1.0.1190 → 1.0.1192

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,10 +6,11 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-26
+updated: 2026-10-29
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
+  - worker1.5/crons/toga2/netsuite/common_sync_togasupply.php
   - worker/crons/toga2/netsuite/sync_togasupply_canon.php
   - worker/crons/toga2/netsuite/sync_togasupply_elite.php
   - worker/crons/toga2/netsuite/sync_togasupply_hh.php
@@ -150,8 +151,12 @@ plain thrown warning instead.
 
 **Fastest path to the real error (2026-09-29):**
 
-- `N-RUNNING` with N shrinking run after run = that section throws every run. SALES_ORDERS aborts the whole
-  script, so the sections after it freeze too — fix SO first.
+- `N-RUNNING` with N shrinking run after run = that section throws every run. One run does the sections in
+  order SALES_ORDERS → PURCHASE_ORDERS → INVOICES → ITEM_RECEIPTS → ITEM_FULFILLMENTS →
+  INVENTORY_ADJUSTMENTS, and a throw aborts the run, so every section **after** the thrower freezes too
+  (cursors stop; earlier sections stay current). Fix the first frozen section first.
+- Poison record: in `Logs_<Client>.Api`, find the PUT of `NETSUITE_EXECUTION_MODE_<SECTION>` = `<n>-RUNNING`;
+  the last GETs after it name the record the run died on.
 - Read the api2 rejection in `Logs_<Client>.Api`: `responseCode >= 400 AND responsePayload LIKE '%"api":"Agilant"%'`,
   newest first, `LIMIT 1` + tight date filter.
 - **`Logs.Issue` keeps only the FIRST trace per fingerprint** — its trace can name the wrong client and line.
@@ -2088,15 +2093,20 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
   `customerId=3`; Prudential adds the `Customers` join + `CLIENT_UUID_PRUDENTIAL_CUSTOMER`; no other client
   changed, no else/default, and the guard still throws for a genuinely-missing SO. PHP 7.2-safe
   (`$options['join'] ?? []`), signature unchanged.
+  - **S&P Global branch added 2026-10-29 (TRUE-83380, library `1bea35a4`, PR #917).** S&P's extra row is a
+    customerless copy (`customerId` NULL) from the 2025-09-10 data load, never touched by the sync. Scope:
+    `Customers` join + `Customers.c_netsuiteInternalCustomerId = (int)($nsInvoice->entity->internalId ?? 0)`
+    (the invoice's NetSuite customer, same idea as `syncSalesOrderFromNetsuite`). Fires only when NetSuite
+    changes an invoice on one of those old orders. Details: [S&P Global profile](../../../clients/spglobal/profile.md).
+  - `syncItemFulfillmentFromNetsuite`'s SO lookup is also unscoped, but on 2 rows it falls back to
+    `syncSalesOrderFromNetsuite` (scoped), so it does not throw.
   - Same trap hit the closed-order delete 2026-10-05 (SALES_ORDERS freeze, Issue 940) — 4th path; see [closed-order rule](../../library/features/netsuite-closed-order-rule.md). Every SO-by-NS-id lookup needs this scope.
-  - **⚠ OPEN follow-up (own ticket): six OTHER clients also keep two SalesOrders per NetSuite id, and the
-    fix does NOT scope them.** A read-only sweep found the same dual-SO pattern in **Growrk (2789), Canon
-    (2890), Endeavorhealth (4828), Spglobal (2893), Browardsheriff (160), Masonite (17)** (Nychh/Elite/Quad/Sbasite
-    = 0; Compass 28377). Dormant today — only Compass is actually throwing the invoice guard (0 "refusing to
-    import invoice" events for any other client in the last 5 days) — but if any of those six ever bills an
-    invoice against a duplicated SO it will false-throw the same way, and **their duplicates have a DIFFERENT
-    cause than Compass's Office-Depot dual-source.** Also a data-quality question: why do six non-Compass
-    clients have duplicate SalesOrders rows at all.
+  - **⚠ OPEN follow-up (own ticket): five OTHER clients also keep two SalesOrders per NetSuite id, and the
+    fix does NOT scope them.** A read-only sweep (2026-09-17) found the same dual-SO pattern in **Growrk (2789),
+    Canon (2890), Endeavorhealth (4828), Browardsheriff (160), Masonite (17)** (Nychh/Elite/Quad/Sbasite
+    = 0; Compass 28377). Dormant until NetSuite changes an invoice on a duplicated SO; then it false-throws
+    the same way (S&P hit exactly this on 2026-10-28, now scoped above). **Their duplicates have a DIFFERENT
+    cause than Compass's Office-Depot dual-source.**
 - **⚠ "refusing to import invoice … billed sales order not imported" can mean the SO is OLDER than the
   SO cursor (Adyen, 2026-10-01, issue 910).** Invoices from 2026-07-02 on billed 10 SOs last-modified
   2025-11-04 → 2026-06-15, but `NETSUITE_LAST_SYNC_CURSOR_SALES_ORDERS` was 2026-07-31, so the SO sync
@@ -2201,6 +2211,7 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
   [NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
 
 ## Change history
+- 2026-10-29 — S&P Global INVOICES froze at `1-RUNNING` (from 2026-10-28) on an invoice billing an SO with a customerless duplicate row; billed-SO lookup now scoped by the invoice's NetSuite customer for S&P (library `1bea35a4`, prod-verified, all 6 S&P sections IDLE). Section run order + abort cascade, poison-record tip. TRUE-83380. (bala)
 - 2026-10-26 — Prudential INVOICES froze on 9 old invoices whose Jan/Feb 2025 SOs were never imported; unblocked by a NetSuite `memo` bump on each SO (no code). Gotcha + blocker query added. TRUE-83152. (bala)
 - 2026-10-19 — Compass ITEM_FULFILLMENTS froze on a serial moved between IFs in NetSuite (1062, Issue 468); update branch now repoints within the item + prunes stale assignment links; SuiteQL paging throws on bad pages + unique ORDER BY. Fix rule 1 widened (no hand cleanup). Library only, not deployed. TRUE-82938. (jcardinal)
 - 2026-10-18 — NYCHH SALES_ORDERS froze on unknown warehouse NS location 1 (Issue 1041) after the 10-14 cursor rewind; fixed by adding the Location row, not the skip flag. Data only. TRUE-82859. (jcardinal)

package/knowledge/2.0/apps/api2/INDEX.md

@@ -28,6 +28,6 @@
 | [V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front-controller "Prong A" that detects a retriable concurrency failure (deadlock 1213/1205 or unique-race 1062) and replays the whole request in-process |
 | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n |
 | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | Why a reverse hasMany collection (a parent's child records that FK back to it) is absent unless named in the fetch `fields` list; open when a related collection |
-| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | How 2.0 apps deploy (CodePipeline → CodeConnections → EB), the per-env `_underscore` branch resolution, the production auto-deploy + Rolling/batch-1 policy, and |
+| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | How 2.0 apps deploy (CodePipeline → CodeConnections → EB), the standard env branch set in every 2.0 repo, the per-env `_underscore` branch resolution, the produ |
 | [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | The three things that must be right for a new 2.0 API environment to serve requests (config conventions, `Core.DatabaseHosts` data, DB seeding incl. |
 | [api2 instance frozen by php-fpm out of memory (triage and recovery)](workflows/php-fpm-out-of-memory-triage.md) | How to spot, confirm and recover an api2 EB instance that the kernel OOM-killer has frozen (every call 60 s then 504, EB "No Data", yet EC2 checks pass and CPU |

package/knowledge/2.0/apps/api2/features/v2-rest-query-contract.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: feature
 status: active
-updated: 2026-10-28
+updated: 2026-10-29
 owners: [tcox, bala, apeterson, jcardinal, rgirish]
 files:
   - api2/Component/Api/V2/V2.php
@@ -422,6 +422,10 @@ where=(field:op:value,LOGIC,field:op:value,(nested,OR,nested))
   through a join alias: `ojoin=Cases:Cases.id=Tickets.caseId&where=(Cases.uuid:eq:<uuid>)` (or a
   `TableViewJoins` row that gives the alias). **Self-join (FK to the same table, e.g.
   `parentLocation.uuid`) needs an alias:** `ojoin=Locations@p:p.id=Locations.parentLocationId&where=(p.uuid:eq:<uuid>)`.
+  **Any plain field of a joined table works, not only `uuid` (incl. `c_` fields):**
+  `join=Customers:Customers.id=SalesOrders.customerId&where=(Customers.c_netsuiteInternalCustomerId:eq:27446)`
+  on `/v2/sales-orders` (verified prod 2026-10-29). Qualified names are used as is (~L6998); only `._`
+  routes to HAVING (~L3571/8648); `join` emits an INNER JOIN (~L3613).
   A path through an **inherent child**
   (`ticketUsers.user.uuid`) does work — but only when the **first** segment is an inherent child
   of the route (`buildChildPathWhereSql`). A path that starts with a forward FK hop fails
@@ -698,6 +702,7 @@ If you need the behaviour, loop uuids client-side. Point 3 is the durable, non-o
   [V2 request logging](request-logging.md).
 
 ## Change history
+- 2026-10-29 — `where` on a joined table's plain/`c_` field works (`Customers.c_netsuiteInternalCustomerId`), verified prod (TRUE-83380). (bala)
 - 2026-10-28 — `sort` orders only the main record query; nested child sets come back unordered, sort them client-side (TRUE-81848). (bala)
 - 2026-10-27 — `where` unsupported on PUT/PATCH; uuid-less PUT/PATCH with no identifying field now 400 `EV-1` instead of a full-table load + `EV-14` (TRUE-83276). (jcardinal)
 - 2026-10-24 — blox JS `null` in `where` → `= ''` → 1525 500, send `"null"`; dotted FK path fails in `fields` too; cross-cluster FK → filter client-side (TRUE-82916). (jcardinal)

package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md

@@ -6,8 +6,8 @@ project: API
 client: shared
 type: workflow
 status: active
-updated: 2026-10-12
-owners: ["jcardinal", "mhammontree", "bala", "apeterson"]
+updated: 2026-10-29
+owners: ["jcardinal", "mhammontree", "bala", "apeterson", "rgirish"]
 files:
   - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
   - api2/ebs/register_instance_to_shared_application_load_balancer.php
@@ -19,13 +19,20 @@ related:
   - ../../../standards/frontend-deploy.md
 ---
 
-How 2.0 apps deploy (CodePipeline → CodeConnections → EB), the per-env `_underscore` branch resolution, the production auto-deploy + Rolling/batch-1 policy, and the Source-stage GitHub-error / stale-instance traps; open when deploying, promoting a change, or triaging a pipeline failure.
+How 2.0 apps deploy (CodePipeline → CodeConnections → EB), the standard env branch set in every 2.0 repo, the per-env `_underscore` branch resolution, the production auto-deploy + Rolling/batch-1 policy, and the Source-stage GitHub-error / stale-instance traps; open when deploying, promoting a change, or triaging a pipeline failure.
 
 ## Summary
 
 2.0 apps (`api2`, `_underscore`) deploy through **AWS CodePipeline**. The Source stage pulls from GitHub via a **CodeConnections** connection (the modern replacement for CodeStar Connections), backed by the **"AWS Connector for GitHub" GitHub App** installed on the `agilantsolutions` org. A single pipeline can have multiple Source actions sharing one connection (e.g. `agilantsolutions/api2` and `agilantsolutions/_underscore` on the same connection ARN). Deployment target is Elastic Beanstalk.
 
-**Branch model (which branch deploys where).** The application repos (`_underscore`, `api2`, `toga2-supply`) deploy from **long-lived per-env branches** — `_beta`, `_production` (and `_stage`, etc.) — not from `_main`. **api2 pulls `_underscore`'s matching `_<env>` branch at EB build time**, so promoting a shared `_underscore` change to an env means merging it into that `_underscore` `_<env>` branch (the api2 build then picks it up). **`dbchanges2` has only `_main`** — its migrations are applied **per-environment by the team's process**, decoupled from the app branches; a code deploy does not run migrations. So a full feature promotion to an env = apply the `dbchanges2` migrations to that env **and** deploy the app-repo `_<env>` branches.
+**Branch model (which branch deploys where).** The application repos (`_underscore`, `api2`, `toga2-supply`) deploy from **long-lived per-env branches** — `_beta`, `_production` (and `_stage`, etc.) — not from `_main`. **api2 pulls `_underscore`'s matching `_<env>` branch at EB build time**, so promoting a shared `_underscore` change to an env means merging it into that `_underscore` `_<env>` branch (the api2 build then picks it up). **`dbchanges2` migrations are applied per-environment by the team's process**, decoupled from the app branches (its working branch is still `_main`); a code deploy does not run migrations. So a full feature promotion to an env = apply the `dbchanges2` migrations to that env **and** deploy the app-repo `_<env>` branches.
+
+**Standard env branch set (all 2.0 repos, TRUE-83396, 2026-10-29).** Every 2.0 repo in `agilantsolutions` has these 8 branches: `_dev`, `_qa`, `_stage`, `_uat`, `_hotfix`, `_production`, `_demo`, `_sandbox`. Repos: `api2`, `_underscore`, `worker2`, `dbchanges2`, `apiproxy`, `saml`, `toga2-hub`, `toga2-commerce`, `toga2-supply`, `toga2-view`, `toga25-supply`, `toga25-desk`, `toga25-iq`, `toga2-template`, `toga25-blox`.
+- New branches were cut from each repo's `_production`. Branches that already existed were left as they were (not reset).
+- `dbchanges2`, `apiproxy`, `toga25-iq`, `toga2-template`, `toga25-blox` had no `_production`: it was cut from `_main`, then the other 7 from it. Their GitHub default branch is still `_main`.
+- `toga-blox` does not exist in the org (closest: `toga-blox-npm`) — skipped, has none of the 8.
+- Older env branches still exist next to them (`_sandbox-dev`, `_sandbox-client`, `_qa-alpha`/`-beta`/`-gamma`/`-hotfix`/`-task`, `_qc-*`, `_client-*`, `_alpha`/`_beta`/`_gamma`, `toga25`). The new `_sandbox` and `_qa` are separate branches, not renames.
+- **No pipeline watches any of the 8 new names except `_production` and `_stage`** (checked 2026-10-29, accounts 654654170868, 975050298201, 502614707982; us-east-1 + us-west-2). Existing watches: `_production` → `api-production-1/2`, `webhook-production`, `Worker-Production`, `saml-production`, non-prod `API-Demo`; `_stage` → `API-Stage`, `API-QC-Performance`, `API-QC-Security`. Once a pipeline points at a new branch, every push to it deploys.
 
 > **Not every EB env pulls `_underscore` from its `_<env>` branch — check `git.<env>.json`.** The beta env **`API-Sandbox-Dev`** does **not** deploy `_underscore` from `_beta`. Its `.platform/hooks/prebuild/git.sh` `rm -Rf`s and **re-clones `_underscore` from the branch named in `.ebextensions/git.<env>.json`** — for this env, `git.sandbox-dev.json` → branch **`_sandbox-dev`** — **overwriting the CodePipeline-supplied copy.** So merging a change into `_beta` does nothing for this env; you must push it to **`_sandbox-dev`**. Always confirm the exact env↔branch mapping in the env's `git.<env>.json` before assuming `_<env>` applies.
 
@@ -119,6 +126,7 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 - **On-instance composer install (when a dep is missing post-deploy).** If a Composer dep is missing on the running instance (e.g. `Class "FPDF" not found` because `composer.lock` wasn't committed), you can install it on the box over SSH/PuTTY — but `composer require setasign/fpdf:^1.8` must have **no space** after the colon (`fpdf: ^1.8` parses `^1.8` as a separate package and errors), and run it as the web app user / `chown` the result so the webserver can read `vendor/`. The real fix is to **commit `composer.lock`** so EB's `composer install` picks it up — the on-instance install is a stopgap that a redeploy wipes.
 
 ## Change history
+- 2026-10-29 — Added the standard 8 env branches in all 2.0 repos and which ones pipelines watch (TRUE-83396) (rgirish)
 - 2026-10-12 — Added prod Approve-stage gotcha and `_underscore`-first merge order (TRUE-82705) (jcardinal)
 
 ## Related

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-27
+updated: 2026-10-29
 owners: ["jcardinal", "mhammontree", "bala", "rgirish", "snaredla", "amadda"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -640,6 +640,13 @@ production deploy.**
 - `_Query::fetchRow()` returns an object, not an array — access `$row->field`.
 - Keep the metric POST non-fatal (`throwExceptionsOnFailure` off) so a monitoring outage never breaks the
   worker job it rides in.
+- **"Stopped reporting / is late" can be a false alarm: the push to OneUptime timed out.** Signal:
+  `Core.WorkerJobs.output` ends with `OneUptime push FAILED.` while the check itself says `alarm=OK`, and
+  `executionTime` is ~10s higher (`$PUSH_TIMEOUT_SECONDS` in `Monitor/Compass.php` / `Operations.php`).
+  To tell a OneUptime problem from a real outage, count `ok` vs `FAILED` per minute with a `LIKE` on
+  `output` for `action LIKE 'Monitor/%'`: many monitors failing together with `alarm=OK`, while non-monitor
+  jobs run normally, = OneUptime side. Seen 2026-10-29 from 09:42 CT: about half of all pushes failed for
+  30+ minutes. No code change.
 - **Two branches adding a monitor to the same `Monitor/<Client>.php` conflict if both insert at the
   same spot.** Blank-line padding does not help — git merges by surrounding context. Put the new
   method somewhere else (e.g. after the last helper) so the hunks sit far apart. Check with
@@ -658,6 +665,7 @@ production deploy.**
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
 
 ## Change history
+- 2026-10-29 — Gotcha: "stopped reporting" incidents from failed OneUptime pushes (`OneUptime push FAILED.` + `alarm=OK` in `WorkerJobs.output`) and how to tell them from a real outage (TRUE-83380). (bala)
 - 2026-10-27 — `EmailedAsnImport` moved off php-imap2 to `_Component_Email_Inbound_Imap` (webklex), TLS now validated, no ext-imap (TRUE-82933). (jcardinal)
 - 2026-10-27 — Added `OfficeDepotLostOrders` (TRUE-82736); legacy `Logs`/`Common` via `DB_VISION_LOGS`; per-run `resetQueryCache`; worker2 project status/severity ids + export fields. (amadda)
 - 2026-10-27 — SFTP monitors: reuse `Sftp/Import` config + login when it already reads the SFTP; linked NYCDOE `StuckAsnFiles` (TRUE-82941). (mhammontree)

package/knowledge/clients/spglobal/profile.md

@@ -11,11 +11,12 @@ project: Worker
 client: spglobal
 type: profile
 status: active
-updated: 2026-10-21
-owners: ["jcardinal", "mhammontree", "snaredla", "rgirish"]
+updated: 2026-10-29
+owners: ["jcardinal", "mhammontree", "snaredla", "rgirish", "bala"]
 files:
   - store/_/app/framework.php
   - store/common/spglobal/welcome/get.php
+  - library/app/api/toga2.php
 related:
   - ../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
   - ../../1.0/apps/library/features/app-cloud-aws-credentials.md
@@ -25,11 +26,16 @@ S&P Global TOGa-Supply NetSuite integration client on the shared 1.0 worker+libr
 
 ## Summary
 
-- TOGa-Supply NetSuite integration client on the shared 1.0 NetSuite → TOGa Supply per-client sync (`worker` + `library`).
-- NetSuite transactions sync into its 2.0 `Client_<Id>` tenant via the TOGa2 API through a thin `worker/crons/toga2/netsuite/sync_togasupply_*.php` wrapper.
+- TOGa-Supply NetSuite integration client on the shared 1.0 NetSuite → TOGa Supply per-client sync (`worker` + `library`). 2.0 DB `Client_Spglobal`; library client uuid const `App_Api_Toga2::CLIENT_UUID_SPGLOBAL`.
+- NetSuite transactions sync into its 2.0 tenant via the TOGa2 API through a thin `crons/toga2/netsuite/sync_togasupply_*.php` wrapper.
 - Profile seeded 2026-08-29 during root-cause work on stuck togasupply integrations: S&P Global was exposed to the item-fulfillment "Undefined index" on an empty `orderLine` (`syncItemFulfillmentFromNetsuite`), fixed in shared code. Mechanics + diagnosis playbook: [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
 
 ## Integration touchpoints
 
 - **`worker` / `library` (1.0)** — shared togasupply sync engine + `App_Api_Toga2` bridge.
 - **`store` (1.0) — PunchOut from SAP Ariba** (Classic Buyer + Guided Buying) lands on `spglobal.togacommerce.com`: spglobal host block in `store/_/app/framework.php`, client code in `store/common/spglobal/`. Not vision (that is Canon's flow). A store HTTP 500 shows in Guided Buying as "Unable to connect to supplier's website". Outage 2026-10-20 (TRUE-82959): store missing `aws/aws-sdk-php`, see [1.0 AWS credentials](../../1.0/apps/library/features/app-cloud-aws-credentials.md#gotchas).
+
+## Gotchas
+
+- **Customerless duplicate SalesOrders.** Many NetSuite SO ids have 2 rows in `Client_Spglobal.SalesOrders`: a copy with `customerId` NULL from the 2025-09-10 data load (the sync never touches it) and the synced row (`customerId` 1 = S&P GLOBAL, NetSuite customer 27446). Older invoices link to the customerless copies. Any SO lookup by `c_netsuiteInternalSalesOrderId` alone returns 2 rows for those orders.
+- **INVOICES froze on this 2026-10-28 (TRUE-83380, fixed + prod-verified 2026-10-29).** NetSuite changed an invoice on one of those old orders; `syncInvoiceFromNetsuite`'s billed-SO lookup got 2 rows and threw "refusing to import invoice ... is not imported in toga" every run, freezing INVOICES at `1-RUNNING` and the three sections after it. Fix: an S&P branch scopes the lookup by the invoice's NetSuite customer (`Customers.c_netsuiteInternalCustomerId`). See [per-client sync, billed-SO scope gotcha](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1190",
+  "version": "1.0.1192",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",