toga-ai 1.0.649 → 1.0.651

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -37,6 +37,8 @@
 | [Persona Name Translation (PersonaTranslations sidecar)](features/persona-name-translation.md) | Serves Persona **names** in multiple languages by adding a per-language **sidecar** table `PersonaTranslations`, reusing the platform's existing metadata-driven | _underscore/Model/Client/PersonaTranslation.php, dbchanges2/Client/2026-07-22b - PersonaTranslations.sql, dbchanges2/Core/2026-07-22a - PersonaTranslationsRecord.sql, dbchanges2/Client/2026-07-22c - PersonaTranslationsAcl.sql, dbchanges2/Client_CompassCanada/2026-07-22 - PersonaTranslationsFrench.sql, toga2-commerce/src/pages/Account/view/MySettingsView.tsx |
 | [Record Change Audit Log (Logs_<Client>.Record / RecordField) — reading a field's history](features/record-change-audit-log.md) | Every 2.0 client schema has a sibling **logs** schema `Logs_<Tenant>` (e.g. | _underscore/Model/Client/Logs/Record.php, _underscore/Model/Client/Logs/RecordField.php, _underscore/Model/Client/Logs/CustomRecordField.php |
 | [Recursive Item Fulfillments (upstream mirroring)](features/recursive-item-fulfillments.md) | In a multi-tier supply chain a sales order (SO) spawns a purchase order (PO) that becomes another SO downstream, and so on. | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillmentItem.php, _underscore/Model/Client/ItemFulfillmentItemUnit.php, _underscore/Model/Client/ItemFulfillmentPackage.php, _underscore/Model/Compass/AdvanceShippingNotice.php, dbchanges2/Core/2026-02-13 - 75601 - RecursiveItemFulfillmentCreation.sql, dbchanges2/Core/2026-06-04 - RecursiveItemFulfillmentPut.sql, dbchanges2/Client_Compass/2026-07-02a - FixSA133377TrackingSerialAndDuplicateIF.sql, dbchanges2/Client_Compass/2026-08-18 - FixSA135471HeroItemHalfQuantityFulfillment.sql |
+| [Sales-order \"PO Number\" sourcing — one surface element, `_purchaseOrders`, tenant SQL underneath](features/sales-order-po-number-sourcing.md) | The sales-order modal's **PO Number** field is **not** tenant-split at the display layer. | _underscore/Model/Client/SalesOrder.php, _underscore/Model/Compass/SalesOrder.php, toga25-supply/src/pages/SalesOrders/helpers/surfaceBundleToTenantFields.ts, toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts, dbchanges2/Core/2026-08-25a - PoNumberDetailFieldPurchaseOrdersValueKey.sql, dbchanges2/Core/2026-07-23a - PoNumberDetailFieldValueKey.sql, dbchanges2/Client_Quad/2026-08-19a - PoDetailsButtonMostRecentPoSelection.sql |
+| [SO↔PO bridge tables are TWO tables in OPPOSITE directions (upstream vs downstream)](features/sales-order-purchase-order-bridge-direction.md) | There are **two** bridge tables linking sales orders and purchase orders, and they mean **opposite things**. | _underscore/Model/Client/PurchaseOrders/SalesOrder.php, _underscore/Model/Client/SalesOrders/PurchaseOrder.php, _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/SalesOrder.php |
 | [Per-client sales-order status filter (Surface FILTER_SET → table meta `filterOptions`)](features/sales-order-status-filter-surface.md) | The status-filter dropdown on the sales-orders table is **per client**, driven by a Surface `FILTER_SET` rather than by the raw contents of the client's `SalesO | _underscore/Model/Client/TableView.php, _underscore/Model/Core/Surface.php, _underscore/Model/Client/SalesOrder.php, _underscore/Model/Quad/SalesOrder.php, _underscore/Model/Prudential/SalesOrder.php, toga-blox/src/components/Table/hooks/useFetchTablePageMeta.ts, toga-blox/src/api/types.ts, dbchanges2/Core/2026-08-24b - SalesOrderStatusFilterSurfaceSeed.sql, dbchanges2/Client_Compass/2026-08-24b - SalesOrderStatusFilterHides.sql, dbchanges2/Client_CompassCanada/2026-08-24b - SalesOrderStatusFilterHides.sql, dbchanges2/Client_Quad/2026-08-24b - SalesOrderStatusFilterHides.sql, dbchanges2/Client_Nychh/2026-08-24b - SalesOrderStatusFilterHides.sql |
 | [_String helpers — ASCII-safe HTML entity encoding (and the parseBetween trap)](features/string-html-entity-helpers.md) | `_String` is the 2.0 framework's static string utility class. | _underscore/String.php |
 | [Surface Resolver (_Model_Core_Surface::resolve — replaces Page::meta)](features/surface-resolver.md) | The runtime for the platform-wide **Surface** UI presentation layer: 9 `_underscore` models plus a cached resolver, `_Model_Core_Surface::resolve(&$api, string | _underscore/Model/Client/Language.php, dbchanges2/Core/2026-08-21 - SalesOrderDecisionSurfacesReseed.sql, dbchanges2/Core/2026-08-24 - RestoreApproveDenyRowActionsVisibility.sql, dbchanges2/Client_Quad/2026-08-24 - ProdPortApprovePoNumberEnabledRule.sql, dbchanges2/Client_Compass/2026-08-24 - ProdPortApprovalsGateAndApproveStepTwoRule.sql, dbchanges2/Client_CompassCanada/2026-08-24 - ProdPortApprovalsGateAndApproveStepTwoRule.sql, dbchanges2/Client_Nychh/2026-08-24 - FixInventoryGroupingsUnitsTopologyOverrideIds.sql, toga25-supply/src/App.tsx, toga25-supply/src/surface/useFetchSurfaceMeta.ts, toga25-supply/src/contexts/AuthContext.tsx, dbchanges2/Core/2026-07-21a - SalesOrderDecisionSummarySurfaceSeed.sql, dbchanges2/Core/2026-07-21b - SalesOrderDecisionActionSurfaceSeed.sql, dbchanges2/Core/2026-08-21a - SalesOrderDecisionSurfacesReseed.sql, dbchanges2/Client_Compass/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Client_Quad/2026-07-21c - SalesOrderDecisionSummaryOverride.sql, dbchanges2/Client_CompassCanada/2026-07-21c - SalesOrderDecisionSummaryTotalConcat.sql, dbchanges2/Client_Compass/2026-08-04a - ApprovalDetailsAssignedManagerPreferredStage.sql, _underscore/Model/Core/Surface.php, _underscore/Model/Client/AclRecordScript.php, _underscore/Model/Core/RecordScript.php, dbchanges2/Core/2026-06-30a - SurfaceMetaGroupAndSalesOrderSections.sql, dbchanges2/Client/2026-06-30a - SurfaceMetaGroupAcl.sql, dbchanges2/Client_Quad/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Client_CompassCanada/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Core/2026-06-29b - SurfaceMetaPublicReadAcl.sql, dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql, dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql, dbchanges2/Client_Compass/2026-07-15f - SalesOrderRecordActionsRemoveDeadConfigRuleOverrides.sql, dbchanges2/Core/2026-07-17h - Update - ClearApprovalsFilterButtonConfig.sql, dbchanges2/Client_Compass/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Compass/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_Quad/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Quad/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, _underscore/Model/Core/SurfaceElement.php, _underscore/Model/Core/Action.php, _underscore/Model/Core/Vocabulary.php, _underscore/Model/Core/VocabularyTerm.php, _underscore/Model/Core/Message.php, _underscore/Model/Client/SurfaceOverride.php, _underscore/Model/Client/MessageTranslation.php, _underscore/Model/Client/ThemeToken.php, _underscore/Model/Core/Page.php, api2/Component/Api/V2/V2.php |

package/knowledge/2.0/apps/_underscore/features/sales-order-po-number-sourcing.md

@@ -0,0 +1,88 @@
+---
+title: "Sales-order \"PO Number\" sourcing — one surface element, `_purchaseOrders`, tenant SQL underneath"
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: active
+updated: 2026-08-25
+owners: [apeterson]
+files:
+  - _underscore/Model/Client/SalesOrder.php
+  - _underscore/Model/Compass/SalesOrder.php
+  - toga25-supply/src/pages/SalesOrders/helpers/surfaceBundleToTenantFields.ts
+  - toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts
+  - dbchanges2/Core/2026-08-25a - PoNumberDetailFieldPurchaseOrdersValueKey.sql
+  - dbchanges2/Core/2026-07-23a - PoNumberDetailFieldValueKey.sql
+  - dbchanges2/Client_Quad/2026-08-19a - PoDetailsButtonMostRecentPoSelection.sql
+related:
+  - ./sales-order-purchase-order-bridge-direction.md
+  - ./calculated-sql-fields.md
+  - ./surface-resolver.md
+  - ../../toga25-supply/features/surface-frontend.md
+  - ../../dbchanges2/features/surface-layer-schema.md
+---
+
+## Summary
+
+The sales-order modal's **PO Number** field is **not** tenant-split at the display layer. As of
+2026-08-25 every tenant renders it the same way:
+
+```
+Core.SurfaceElements id 7  (Order Details "PO Number", messages.label = salesOrder.field.poNumber)
+  → config.valueKey = "_purchaseOrders"
+  → FE adapter toga25-supply/src/pages/SalesOrders/helpers/surfaceBundleToTenantFields.ts
+  → renders the order's `_purchaseOrders` calculated field (comma-joined string)
+```
+
+**All tenant variation lives underneath, in the SQL behind `_purchaseOrders`** — not in the
+surface, not in the front end. This supersedes the earlier (TRUE-81190) understanding that the
+*display path* itself was tenant-split.
+
+## How it works
+
+### The display binding
+`SurfaceElements` id 7 carries `config = {"valueKey": "_purchaseOrders"}`. `isPersonaValue` is
+deliberately **not** set: the value is a scalar comma-joined string, not a persona array.
+
+`surfaceBundleToTenantFields.ts` exposes `PO_NUMBER_VALUE_KEY` and resolves it against the order
+via `resolvePathInsensitive(valueKey)`, normalising an empty string to an em-dash.
+
+### The tenant layer (`_purchaseOrders`)
+| Tenant | Implementation | Direction read |
+|---|---|---|
+| **Compass USA** | overrides in `_underscore/Model/Compass/SalesOrder.php:238` — a `UNION ALL` of its own vendor POs plus Office Depot's POs traversed SO→PO→SO→PO across the multi-tier chain | downstream **+ upstream** (only tenant reading upstream) |
+| **Quad Graphics** | no override → base implementation | downstream only |
+| **Compass Canada** | no `Model/Compasscanada/` directory exists at all → base implementation | downstream only |
+| base (`_Model_Client_SalesOrder:348`) | the shared calculated field | **downstream only** (`SalesOrders_PurchaseOrders`) |
+
+### `poSelection` is a different thing
+The `poSelection` config on the `sales-order-record-actions` surface picks **which** linked PO the
+*editable* flow treats as primary (Enter-PO modal prefill, `approveButtonRule`, completion state).
+Core default is `"first"`; Quad is overridden to `"mostRecent"` by
+`Client_Quad/2026-08-19a`. It reads the **downstream join**, not `_purchaseOrders`. Do not conflate
+the two — changing one does not move the other.
+
+## Gotchas
+
+- **A surface-config change is only real if the FE actually honours the binding.** Before
+  2026-08-25 the FE constant *matched* the surface's `valueKey` and then **discarded** it,
+  substituting a hardcoded field path. The config was read and thrown away — so shipping the
+  migration alone would have silently done nothing. Whenever you repoint a `valueKey`, grep the FE
+  for the old literal and confirm it resolves *through* the config, not around it.
+- **Surface seed `id`s are deterministic; surface `uuid`s are not.** Seeds insert with `UUID()`, so
+  the element uuid differs per environment. A uuid quoted in an older migration will **not** match
+  the uuid in a live payload. Key surface updates on the literal `id` (here, `id = 7`).
+- **Downstream-only base implementation is a trap for upstream tenants.** On an upstream-only
+  tenant `_purchaseOrders` resolves cleanly (no `EV-8`) and returns `null` forever. See
+  [SO↔PO bridge direction](./sales-order-purchase-order-bridge-direction.md) and
+  [NYCHH PO Number](../../../clients/nychh/features/po-number-upstream-direction.md).
+- Dropping `isPersonaValue` is intentional; re-adding it will make the FE try to treat the string
+  as a persona array.
+
+## Change history
+- 2026-08-25 — Repointed `SurfaceElements` id 7 to `_purchaseOrders` for all clients
+  (`Core/2026-08-25a`, reverting `Core/2026-07-23a`) and made the FE adapter honour the surface
+  `valueKey` instead of a hardcoded path; recorded the corrected per-tenant map, which supersedes
+  the tenant-split display understanding (apeterson)

package/knowledge/2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md

@@ -0,0 +1,72 @@
+---
+title: "SO↔PO bridge tables are TWO tables in OPPOSITE directions (upstream vs downstream)"
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: active
+updated: 2026-08-25
+owners: [apeterson]
+files:
+  - _underscore/Model/Client/PurchaseOrders/SalesOrder.php
+  - _underscore/Model/Client/SalesOrders/PurchaseOrder.php
+  - _underscore/Model/Client/ItemFulfillment.php
+  - _underscore/Model/Client/SalesOrder.php
+related:
+  - ./sales-order-po-number-sourcing.md
+  - ./recursive-item-fulfillments.md
+  - ./fulfillable-item-propagation.md
+  - ../../api2/features/v2-rest-query-contract.md
+  - ../../../clients/nychh/features/po-number-upstream-direction.md
+---
+
+## Summary
+
+There are **two** bridge tables linking sales orders and purchase orders, and they mean
+**opposite things**. Both names are legal under the bridge-table naming convention (sibling
+tables may be named in either order), and the two API routes differ only in word order — so
+**nothing at the API surface tells you which direction you are querying**. Querying the wrong
+one returns `200` with an empty array, which reads like a permission or data bug and is not.
+
+| Table | Direction | Meaning | Model | API route | `Core.Records` id |
+|---|---|---|---|---|---|
+| `PurchaseOrders_SalesOrders` | **UPSTREAM** | a sales order created **FROM** a purchase order — the *customer's* PO | `_Model_Client_PurchaseOrders_SalesOrder` | `purchase-order-sales-orders` | 276 |
+| `SalesOrders_PurchaseOrders` | **DOWNSTREAM** | a purchase order created **FROM** a sales order — a *vendor* PO | `_Model_Client_SalesOrders_PurchaseOrder` | `sales-order-purchase-orders` | 275 |
+
+Read the name as **"`<source>`\_`<thing created from it>`"**. The *first* table name is the
+origin; the second is what was generated from it.
+
+## How it works
+
+- Both `Records` rows are `aclDatabase = CLIENT`, `childPolicy = MATCH_UPSERT`, so both are
+  per-tenant data and both accept nested writes that match-or-create the link row.
+- Which direction a tenant actually populates depends on how their orders enter the platform:
+  - Customer sends TOGa a PO which becomes a sales order → **upstream** rows.
+  - TOGa raises vendor POs to fulfil a sales order → **downstream** rows.
+  - A multi-tier tenant (Compass USA) has **both**, chained SO→PO→SO→PO.
+- The only in-code statement of the distinction today is a docblock at
+  `_underscore/Model/Client/ItemFulfillment.php:136-140`.
+- The base `_purchaseOrders` calculated field on `_Model_Client_SalesOrder` (line ~348) reads the
+  **downstream** table only. See
+  [sales-order PO Number sourcing](./sales-order-po-number-sourcing.md).
+
+## Gotchas
+
+- **An empty array is the symptom.** `200` / `totalRecordCount 0` with **no** `EZ-*` or `EV-*`
+  message is the signature of *"the join matched nothing"* — i.e. wrong direction — not ACL and
+  not a missing migration. A permission failure would be `EZ-1`; an unknown field `EV-8`/`EV-9`.
+- **Check both directions before debugging anything else.** Count rows in
+  `PurchaseOrders_SalesOrders` and `SalesOrders_PurchaseOrders` for the order. If upstream = 1 and
+  downstream = 0, the caller is on the wrong route. This diagnosis is one query and saves hours.
+- **The routes are near-homographs.** `sales-order-purchase-orders` vs
+  `purchase-order-sales-orders`. Read them aloud before wiring a front-end call.
+- **Downstream-only assumptions are baked into shared code.** Any tenant whose POs are purely
+  upstream (verified: NYCHH; suspected: Compass Canada, which has no `Model/Compasscanada/`
+  overrides at all) silently gets `null`/empty from the shared downstream-reading helpers rather
+  than an error. Upstream-only tenants need an explicit upstream counterpart field.
+
+## Change history
+- 2026-08-25 — Documented the two-direction bridge after an NYCHH "empty array" debug that was
+  purely a wrong-direction query; recorded route/model/record-id mapping and the empty-vs-EZ
+  diagnostic signature (apeterson)

package/knowledge/2.0/apps/dbchanges2/INDEX.md

@@ -8,3 +8,4 @@
 | [Auditing a client DB that drifted from its models (partially applied module migration)](workflows/client-schema-drift-audit.md) | A recurring 2.0 failure mode: **one client's database drifts from what the PHP models declare**, usually because a `_modules/<module>/` migration was applied to | dbchanges2/Client_Growrk/2026-05-28.sql, dbchanges2/Client_Growrk/2026-08-10c - GrowrkServiceRequestCustomFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10d - GrowrkServiceRequestTypeAndDispositionSeeds.sql, dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Growrk/2026-08-10 - GrowrkUnitInventoryFieldsCatchUp.sql, dbchanges2/Client_Growrk/2026-08-10b - GrowrkUnitItemDescriptionAcl.sql, dbchanges2/Client_Growrk/_modules.txt |
 | [Local vs prod MySQL config parity — why “it passed locally” is not evidence](workflows/local-vs-prod-mysql-config-parity.md) | Several migration failures that look like "prod-only bugs" are actually **per-machine MySQL server-configuration differences**. |  |
 | [Repairing non-prod metadata drift (works in prod, broken in beta/dev-sandbox)](workflows/nonprod-metadata-drift-repair.md) | Almost all 2.0 platform behavior is **metadata** — `Core.Records`/`RecordFields`, `Core.RecordScripts`, `Core.ApiPayloadInterceptors`, and per-client `Acl*` row | dbchanges2/Core/2026-06-30a - ItemFulfillmentStageDefaultInterceptor.sql, dbchanges2/Client_Compass/2026-08-06 - RemoveBrokenSalesOrderItemPostPostInterceptor.sql, dbchanges2/Core/2026-07-16a - TrackingNumberSignatureTypeRecordField.sql, dbchanges2/_modules/netsuite/2026-07-10a - UnitInventoryFields.sql, dbchanges2/Client_Aig/_modules.txt, dbchanges2/Client/2026-07-22c - TrackingNumberMeasureIdsFieldPermission.sql, api2/Config/beta.ini, api2/Config/sandbox-dev.ini, dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql, dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql |
+| [Verifying whether a dbchanges2 migration actually ran in an environment](workflows/verifying-a-migration-ran.md) | dbchanges2 has no execution ledger you can query — "did this file run here?" has to be answered from the **rows the file would have produced**. | Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql, Client/2026-08-13 - UnitsForItemsPO_SalesOrderItemsJoin.sql |

package/knowledge/2.0/apps/dbchanges2/workflows/verifying-a-migration-ran.md

@@ -0,0 +1,58 @@
+---
+title: "Verifying whether a dbchanges2 migration actually ran in an environment"
+framework: "2.0"
+repo: dbchanges2
+project: Database Changes
+client: shared
+type: workflow
+status: active
+updated: 2026-08-25
+owners: [apeterson]
+files:
+  - Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql
+  - Client/2026-08-13 - UnitsForItemsPO_SalesOrderItemsJoin.sql
+related:
+  - ../architecture.md
+  - ../features/surface-layer-schema.md
+---
+
+## Summary
+
+dbchanges2 has no execution ledger you can query — "did this file run here?" has to be answered
+from the **rows the file would have produced**. This procedure gets a reliable answer, and names
+the two ways the naive answer is wrong.
+
+## Steps
+
+1. **Preferred — match the hardcoded uuid.** If the migration inserts a literal v4 uuid, select on
+   it. Present ⇒ this file ran. This is the only check that identifies the *file* rather than the
+   *effect*.
+2. **Fallback — match the structural signature.** When the uuid column is unavailable or the row
+   was seeded with `UUID()`, select on the **exact column-value combination** the file writes.
+   Worked example — `Client/2026-08-13 - UnitsForItemsPO_SalesOrderItemsJoin.sql` was confirmed on
+   `Client_Nychh` by a `TableViewJoins` row with `sortOrder = 23`, `joinRecordId = 15`,
+   `joinOnRecordFieldId = 60`, `parentRecordFieldId = 368`, `type = OUTER`, on the tableview slug
+   `units-for-items-for-purchase-orders`.
+3. **For a guarded file, find its distinguishing side-effect.** Pick something in the file that a
+   pre-existing row would *not* already have — typically the ACL grant pair it adds. Worked example
+   — `Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql` was confirmed on `Client_Nychh` by the
+   grant pair **exactly Base + API, `isWritable = 0`**; Compass already had `_purchaseOrders` as
+   `CustomRecordFields` id 77 under a *different* uuid, so the field row alone proved nothing.
+4. **State the conclusion as "the effect is present in env X"**, not "the file ran", unless step 1
+   succeeded.
+
+## Gotchas
+
+- **A guarded `INSERT ... WHERE NOT EXISTS` is a NO-OP where the row already exists under a
+  different uuid.** Therefore: *row absent* ≠ "the file did not run", and *row present* ≠ "this
+  file created it". Both directions of the naive inference are unsound.
+- Seed rows created with `UUID()` differ per environment, so cross-environment uuid comparison is
+  meaningless. Compare **ids and structure**, not uuids. (Same reason surface migrations key on the
+  literal `id` — see [Surface Layer Schema](../features/surface-layer-schema.md).)
+- Fan-out folders (`Client/`) run per tenant, so a file can be present in one client schema and
+  genuinely absent in another. Always verify against the specific schema, not "the environment".
+
+## Change history
+- 2026-08-25 — Captured the uuid-first / structural-signature-fallback procedure and the
+  guarded-insert unsoundness, after confirming `Client/2026-08-11b` and `Client/2026-08-13` on
+  `Client_Nychh` in sandbox-client (apeterson)

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -18,7 +18,7 @@
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php, worker2/Worker/Monitor/Fleet.php, library/app/worker.php, library/app/cloud.php |
-| [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's **enhanced-health** status | worker2/Worker/Infrastructure/CloudWatch.php |
+| [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's health and reports it to a | worker2/Worker/Infrastructure/CloudWatch.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Notification/Email.php, worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql, dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql |
 | [Error-Issue Auto-Resolution & Reopen (frequency-decay lifecycle)](features/error-issue-auto-resolution.md) | The error system could escalate and de-escalate an Issue's *urgency* but had no concept of an Issue being **resolved**. | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, _underscore/Model/Core/Logs/Issue.php, tools/mvc/errors/get.php, tools/mvc/errors/issue/get.php, dbchanges2/Logs/2026-08-05a - Issue status baseline and auto-resolution.sql |

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-19
+updated: 2026-08-25
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -19,209 +19,253 @@ related:
 ## Summary
 
 `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads
-each Elastic Beanstalk (EB) environment's **enhanced-health** status and reports it to a
-OneUptime **Incoming Request** monitor, which pages the team. It is a **"dumb reporter,
-smart monitor"** heartbeat (the same Pattern-B contract as
+each Elastic Beanstalk (EB) environment's health and reports it to a OneUptime **Incoming
+Request** monitor, which pages the team. It is a **"dumb reporter, smart monitor"**
+heartbeat (the same Pattern-B contract as
 [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)): the worker decides
 pass/fail and POSTs decided **string tokens** (`alarm=HIGH/OK`, `probe=DEGRADED/OK`) that
 OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
 
-The load-bearing design decision this doc records is the **paging (alarm) criteria**: page
-on EB `Degraded`/`Severe`, but **suppress the common "an occasional HTTP 500 tripped
-Degraded/Severe" false alarm** unless 5xx errors actually dominate the request mix (by
-share) or reach a real absolute volume. The 5xx gate applies to **both** `Degraded` and
-`Severe`, and only when EB attributes the problem **solely** to 5xx — a mixed cause always
-pages.
+The load-bearing design decision this doc records is the **paging (alarm) criteria**, and
+the key **2026-08-25** change: the 5xx page is now driven by a **CloudWatch windowed
+request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. A non-5xx
+`Degraded`/`Severe` still pages immediately off EB health; but whether a *5xx* problem pages
+is decided by the true per-environment 5xx **rate over a trailing window** (default 5 min),
+not by EB's ~10-second `ApplicationMetrics` snapshot.
+
+**Why the change:** the old gate read EB `describeEnvironmentHealth`'s `ApplicationMetrics`,
+a ~10-**second** window. At low traffic a single 500 in a near-empty bucket reads as 100%
+5xx and false-pages. This actually fired: worker-production paged "100% requests failing"
+while the healthd log showed **16 500s / 516 req = 3.1%** over 17 minutes — the 100% was one
+500 alone in a 1-request 10s bucket. The windowed CloudWatch gate averages over minutes and
+across instances, so a lone 500 can no longer dominate.
 
 **Critical behavior:** the method is **non-fatal and never throws** — it always returns a
 string. worker2 has **no DLQ and a 3600s SQS visibility timeout**, so any uncaught 500
 becomes a poison-message storm; an outer `catch(\Throwable)` is the backstop and the push
-itself runs non-fatal.
+itself runs non-fatal. CloudWatch read failures are caught **internally** and return `null`.
 
 ## Key files / entry points
 
 - `worker2/Worker/Infrastructure/CloudWatch.php` — `_Worker_Infrastructure_CloudWatch`,
   action `ElasticBeanstalkHealth`, dispatched via
   `_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
-  environments, oneuptimeUrl})`.
+  environments, oneuptimeUrl, …thresholds})`.
+- New private `fetchWindowedRequestCounts()` reads the CloudWatch metrics (below).
 - AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
   a read-only role per account, one assume reused across regions) — see
   [Cross-account AWS access](./cross-account-aws-access.md), which also documents the
-  `environments` region→env-names map parameter shape.
+  `environments` region→env-names map parameter shape. The **same assumed credentials** build
+  both the `ElasticBeanstalkClient` and the new `CloudWatchClient`.
 - `oneuptimeUrl` arrives as a **cron parameter and is a push credential** — never log it and
   never record its value in a doc (see gotchas).
 
 ## Alarm / paging criteria (the decision logic)
 
-Per environment, `alarm=HIGH` (page) when the EB `HealthStatus` is `Degraded` **or**
-`Severe`. `Suspended` never pages.
+Two independent paths now feed `alarm=HIGH`:
 
-The **5xx gate applies to both `Degraded` and `Severe`.** A non-5xx or mixed-cause
-`Degraded`/`Severe` — and any env with no stated cause — **always pages**; the gate only
-ever suppresses an env EB attributes **solely** to 5xx.
+1. **EB-health path (non-5xx).** EB `HealthStatus` `Degraded` or `Severe` that is **not**
+   attributable solely to 5xx (latency, instances down, a failed deploy, a mixed cause, or
+   no stated cause) **pages immediately** — unchanged, still driven off EB health.
+   `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx event
+   apart from everything else.
+2. **CloudWatch windowed 5xx-rate path.** For a solely-5xx `Degraded`/`Severe`, the page is
+   decided **not** by EB but by the true windowed rate from CloudWatch. Given the window's
+   `requests` (ApplicationRequestsTotal) and `fivexx` (ApplicationRequests5xx), page when:
+   - `fivexx >= min5xxAbsolute`  **(absolute-volume floor)** — pages regardless of share, or
+   - `requests >= minRequests` **AND** `sharePercent >= min5xxPercent` **(share gate)** — the
+     `minRequests` floor keeps a near-idle window off the share gate so a lone 500 can't page.
 
-- **Non-5xx or mixed-cause `Degraded`/`Severe`** (latency, instances down, a failed deploy,
-  or a 5xx trickle *alongside* a real non-5xx failure) **always pages.**
-- **Solely-5xx `Degraded`/`Severe` pages only on a genuine flood** — either:
-  - the **absolute** 5xx count in the window is at/above `HTTP_5XX_ALARM_ABSOLUTE_COUNT`
-    (default 20), **or**
-  - the **5xx share** of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80).
-- **Zero requests in the window → no page** (nothing observed; also avoids divide-by-zero).
+Because the 5xx decision reads CloudWatch directly, a real 5xx flood pages **even if EB
+hasn't escalated yet**, and a benign low-traffic 5xx blip does not page even though EB flags
+`Degraded`.
 
-**Why the gate now covers `Severe` too:** EB escalates an environment to `Severe` precisely
-when the 5xx share is high, so real 5xx floods reach `Severe` *first* and never touch the
-gated `Degraded` branch. When the gate applied only to `Degraded`, the 80% threshold was
-effectively dead — a real 50%-5xx `Severe` event paged despite the setting. Gating both
-tiers makes the threshold actually govern.
+### Cron parameters (replaces the former class constants)
 
-**Why "solely" 5xx, not "any" 5xx cause:** a mixed-cause env (a 5xx trickle plus a real
-non-5xx failure such as a failed deploy) must not be silenced by a sub-threshold ratio. The
-ratio gate only applies when **every** stated cause references 5xx (and there is at least
-one); anything else pages.
+The four thresholds are **optional cron parameters** spread as **named arguments** from
+`CronJobs.parameters`, with method-signature defaults:
 
-**Why an absolute floor in addition to the ratio:** the ratio alone is magnitude-blind —
-60k failing out of 100k reads as 60% and would be suppressed, though it is plainly a flood.
-`HTTP_5XX_ALARM_ABSOLUTE_COUNT` pages such an env regardless of share; tune it to the busiest
-monitored environment's request volume.
+| Parameter | Default | Meaning |
+|---|---|---|
+| `windowMinutes` | `5` | Trailing window (minutes) over which the CloudWatch rate is summed |
+| `minRequests` | `100` | Minimum windowed request count before the **share** gate applies |
+| `min5xxPercent` | `20.0` | 5xx share (percent) at/above which a solely-5xx env pages, once `minRequests` is met |
+| `min5xxAbsolute` | `20` | Absolute windowed 5xx count at/above which a solely-5xx env pages regardless of share |
 
-### Fail-safe: a blind read pages, never reads healthy
+Removed this iteration: `shouldPageForHealth()` and the old
+`HTTP_5XX_ALARM_RATIO` / `HTTP_5XX_ALARM_ABSOLUTE_COUNT` constants (superseded by the
+windowed gate and the four parameters).
 
-If the health read (`describeEnvironmentHealth`) throws `AwsException` for an environment,
-the monitor sets **`alarm=HIGH`** for that environment (a blind read must never masquerade
-as healthy) **and** sets **`probe=DEGRADED`** (coverage gap, distinct from a measured-bad
-signal — same alarm-vs-probe split as the multi-client monitors in
-[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)).
+### Per-monitor threshold tuning (one cron watches many environments)
 
-### Configurable constants (top of the class)
+A single `ElasticBeanstalkHealth` cron watches environments of **very different scale**, so
+thresholds are tuned per monitor. Measured 3h baselines: api-production-1 ~922–1737 req/5min,
+apiproxy ~400–1650, but eu-west-1 envs only ~40–90 req/5min; 5xx was 0 across all.
 
-| Constant | Default | Meaning |
-|---|---|---|
-| `HEALTH_ALARM_STATUSES` | `['Degraded','Severe']` | HealthStatus values that page |
-| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a solely-5xx `Degraded`/`Severe` pages |
-| `HTTP_5XX_ALARM_ABSOLUTE_COUNT` | `20` | Absolute 5xx count in the window at/above which a solely-5xx `Degraded`/`Severe` pages regardless of share |
-| `HTTP_5XX_CAUSE_MARKERS` | `['5xx','http 5']` | Case-insensitive substrings identifying a 5xx-attributed EB `Cause` |
-
-### Implementation shape
-
-- `shouldPageForHealth(healthStatus, causes, applicationMetrics): [bool, ?float]` — returns
-  the page decision and the observed 5xx ratio.
-- `causesAttributeSolelyTo5xx(causes): bool` — true only when **every** EB `Cause` references
-  5xx **and there is at least one** cause. Replaces the former `causesAttributeTo5xx()`, which
-  returned true if *any* cause mentioned 5xx and so let a mixed-cause env be gated.
-- `causeIsFivexx(cause): bool` — helper that substring-matches `HTTP_5XX_CAUSE_MARKERS`
-  against a single EB `Cause` string (case-insensitive); `causesAttributeSolelyTo5xx` requires
-  it to hold for all causes.
-- The ratio is computed as `StatusCodes.Status5xx / RequestCount` from `ApplicationMetrics`
-  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text. The
-  absolute-count floor reads `StatusCodes.Status5xx` directly.
-- `describeEnvironmentHealth` is called with
-  `AttributeNames = ['HealthStatus','Status','Color','Causes','ApplicationMetrics']`.
-- Per-environment report + OneUptime payload now include the per-env `alarm` token and the
-  observed `fivexxRatio`.
-
-## How EB enhanced-health is read (durable AWS reference)
-
-Facts verified against current AWS docs; they inform why the logic above is shaped as it is.
+- **Worker + worker 1.0 monitors** keep the **method defaults** (`100 / 20% / 20`) — the low
+  absolute floor suits ~150 req/5min traffic.
+- **API and API-Proxy monitors** (which each also include the low-traffic eu-west-1 env) are
+  overridden to **`windowMinutes=5, minRequests=30, min5xxPercent=20, min5xxAbsolute=100`** —
+  the lower request floor keeps the small env on the share gate, while the higher absolute
+  floor stops the busy envs paging on minor blips (20 scattered 500s on ~1300 req = <2%). The
+  override is applied by the dbchanges2 migration below.
+
+### Fail-safe: an unmeasurable solely-5xx event still pages
+
+The windowed metrics are **per-instance only** (see below), and some envs never publish them
+(the 1.0 worker env `agilant-worker` publishes only `EnvironmentHealth`). For such an env the
+gate has **no data** and returns **zeros** (not `null`). To avoid silently missing a real
+flood, when EB reports a **solely-5xx `Degraded`/`Severe`** but the window is **empty
+(unmeasurable)**, the monitor **pages** with gate token **`health-5xx-unmeasured`** rather
+than trusting the empty read.
+
+A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`** (coverage
+gap), distinct from a measured-bad `alarm=HIGH` — same alarm-vs-probe split as the
+multi-client monitors in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md).
+
+## Reading the windowed 5xx rate from CloudWatch (durable AWS reference)
+
+- **EB application-request metrics are PER-INSTANCE ONLY.** `ApplicationRequestsTotal`,
+  `ApplicationRequests5xx` (and `…2xx/3xx/4xx`) are published under dimensions
+  `{EnvironmentName, InstanceId}` in namespace `AWS/ElasticBeanstalk`. **There is no
+  environment-only rollup.** You must sum across instances yourself with metric-math over a
+  search expression:
+
+  ```
+  SUM(SEARCH('{AWS/ElasticBeanstalk,EnvironmentName,InstanceId} MetricName="ApplicationRequestsTotal" EnvironmentName="<env>"','Sum',<periodSeconds>))
+  ```
+
+  The `SEARCH` auto-discovers every instance's series and `SUM` folds them into one, so the
+  window total is correct as instances come and go. Query total and 5xx as two expressions
+  over `windowMinutes`.
+- **Not every environment publishes these metrics.** worker-production, api-production-*, and
+  apiproxy-production-* all publish them; the **1.0 worker env `agilant-worker` publishes
+  ONLY `EnvironmentHealth`** (no app-request metrics) → the gate reads zeros there → the
+  `health-5xx-unmeasured` fail-safe above covers it.
+- **IAM already sufficient.** The `WorkloadsRuntime` role grants `cloudwatch:*` in **both**
+  account `654654170868` (worker/api) and `502614707982` (legacy worker), so no IAM change
+  was needed for the CloudWatch reads.
+
+### EB enhanced-health facts (still relevant — the EB-health path)
 
 - **HealthStatus severity ladder:** `Ok → Info → Warning → Degraded → Severe` (plus the grey
-  states `Pending`/`Unknown`/`Suspended`/`NoData`). `Degraded` is the "high failure" tier
-  and is **routinely tripped by benign transients** (Auto Scaling scale-up, a mid-deploy
-  dip), which is exactly why `Degraded` alone is noisy. `Severe` = "very high failure /
-  environment effectively not serving."
-- **`ApplicationMetrics` returns RAW COUNTS, not percentages** — despite the API-reference
-  prose saying "percentage"/"per second". Fields: `Duration` (Integer seconds, usually 10),
-  `RequestCount` (Integer, **total** requests over the window),
-  `StatusCodes.{Status2xx,Status3xx,Status4xx,Status5xx}` (Integer counts). AWS's own
-  example: 2xx 3391 + 5xx 843 = RequestCount 4234. With no traffic, `RequestCount=0` and
-  `StatusCodes` may be **absent** → treat as "no data," not "zero failures." The 5xx-ratio
-  denominator is `RequestCount` (the authoritative total).
+  states `Pending`/`Unknown`/`Suspended`/`NoData`). `Degraded` is routinely tripped by benign
+  transients (Auto Scaling scale-up, a mid-deploy dip), which is why a solely-5xx `Degraded`
+  is now gated on the CloudWatch rate rather than paged outright.
 - **Why an env is Degraded comes from `Causes`:** EB `Causes` strings literally contain e.g.
   `"19.9 % of the requests are failing with HTTP 5xx."` A substring check reliably
-  **classifies** 5xx-driven vs. not (fail-safe: a wording change → treated as a normal
-  `Degraded` → pages). The **ratio itself must come from `ApplicationMetrics`**, never by
-  parsing the number out of `Causes`.
-- **A failed application DEPLOYMENT is NOT reliably reflected in HealthStatus** — it can read
-  `Warning`/`Degraded`, or fail before any red request-failure signal. The authoritative
-  structured deploy signal is per-instance `describeInstancesHealth` →
-  `Deployment.Status ∈ {'In Progress','Deployed','Failed'}`. (This detector was evaluated and
-  removed this iteration — see Design history.)
-- **IAM:** `describeEnvironmentHealth` / `describeInstancesHealth` / `describeEvents` are all
-  covered by the managed `AWSElasticBeanstalkReadOnly` policy and require enhanced health
-  enabled. No per-call charge.
+  **classifies** solely-5xx vs. not; a wording change fails safe (treated as non-5xx →
+  pages). `describeEnvironmentHealth` is called with `AttributeNames` including
+  `HealthStatus`, `Status`, `Color`, `Causes`.
+- **A failed application DEPLOYMENT is NOT reliably reflected in HealthStatus** — the
+  authoritative structured deploy signal is per-instance `describeInstancesHealth` →
+  `Deployment.Status`. (A deploy that fails without ever degrading health remains an accepted
+  residual gap.)
+- **IAM:** `describeEnvironmentHealth` is covered by the managed `AWSElasticBeanstalkReadOnly`
+  policy and requires enhanced health enabled.
+
+## Deploy order (mandatory) — code BEFORE the migration
+
+worker2 code **must** deploy **before** the dbchanges2 threshold migration runs.
+
+- new-code + old-params → **fine** (the params are optional with defaults).
+- **old-code + new-params → FATAL.** The dispatcher spreads `CronJobs.parameters` as **named
+  arguments**; an old method signature that lacks `windowMinutes`/`minRequests`/… throws
+  **"Unknown named parameter"** and **every run fails** → the monitor goes dark → OneUptime
+  raises an offline incident.
+
+Correct order: **1) deploy worker2 → 2) apply the SQL → 3) add the OneUptime criterion.**
+
+## dbchanges2 migration (the API/API-Proxy override)
+
+`dbchanges2/Core/2026-08-25a - ApiApiproxy Beanstalk Health 5xx Thresholds.sql` applies the
+API/API-Proxy override with a JSON_SET that **adds only the four keys**, preserving
+`awsAccountId`/`environments`/`oneuptimeUrl`:
+
+```sql
+UPDATE CronJobs
+SET parameters = JSON_SET(parameters,
+    '$.windowMinutes', 5, '$.minRequests', 30,
+    '$.min5xxPercent', 20, '$.min5xxAbsolute', 100)
+WHERE action = 'Infrastructure/CloudWatch/ElasticBeanstalkHealth'
+  AND name IN (<the two API monitor names>);
+```
+
+`Core.CronJobs.parameters` is confirmed a native `JSON` column, so `JSON_SET` merges rather
+than overwrites.
 
 ## OneUptime monitor config (external, not code)
 
 Paging depends on the OneUptime Incoming-Request monitor's **string-match** criteria on the
-POSTed body — `Contains "alarm":"HIGH"` (and, if desired, `Contains "probe":"DEGRADED"`).
-This is external OneUptime configuration, not worker2 code.
-
-## Design history — rejected direction (this session)
+POSTed body — `Contains "alarm":"HIGH"` pages.
 
-An earlier iteration set `HEALTH_ALARM_STATUSES = ['Severe']` (dropping `Degraded` entirely)
-and **added a separate deployment-failure detector** (`describeInstancesHealth` →
-`Deployment.Status = 'Failed'`) to still catch deploy failures. That was **reverted** after a
-scope change: the team decided they **do** want to page on `Degraded` (to catch a flood of
-500s and non-500 degradations), with only the occasional-500 case suppressed via the 80%
-ratio. The deployment-failure detector was **removed** — deploy failures that surface as
-`Degraded` are now covered by the `Degraded` alarm. **Residual gap accepted for this
-iteration:** a deploy that fails fast *without ever degrading health* is not caught.
+**Known alerting gap (recommended fix):** the current monitors match `"status":"error"`,
+`"alarm":"HIGH"`, and heartbeat-gap rules, but **nothing matches `"probe":"DEGRADED"`**. So a
+CloudWatch-metrics **read failure while EB reads Ok** (body = `alarm:OK, probe:DEGRADED,
+status:reporting`) is **invisible to alerting**. Add a criterion matching
+`Contains "probe":"DEGRADED"` → a **Degraded-severity, auto-resolving** incident, placed
+**before** the "online" recovery rule (which also matches `alarm:OK`). The code already emits
+the `probe` token; this is a OneUptime config change the developer applies.
 
 ## Known gaps / follow-ups (not done)
 
-- **Per-region client construction is not individually guarded.** Each
-  `new ElasticBeanstalkClient` is not in its own try/catch, so a bad region/creds aborts the
-  **whole run** via the outer backstop instead of paging just that region's envs as blind.
-  Candidate follow-up now that blind reads page.
-- **No first-party test harness exists in worker2** (all tests are vendor/).
-  `shouldPageForHealth()` and `causesAttributeSolelyTo5xx()` are pure and ideal to unit-test
-  (ratio at exactly 0.80; absolute count 19 vs 20; `Severe` over a solely-5xx cause; a
-  mixed 5xx + non-5xx cause; empty causes; null metrics; zero requests) — deferred pending a
-  harness.
+- **No first-party test harness in worker2** (all tests are vendor/). The gate decision and
+  `causesAttributeSolelyTo5xx()` are pure and ideal to unit-test (share at exactly
+  `min5xxPercent`; absolute at `min5xxAbsolute-1` vs `min5xxAbsolute`; `requests` just below
+  `minRequests`; empty/zero window; mixed 5xx + non-5xx cause) — deferred pending a harness.
+- **Deploy fails without degrading health** — still not caught (accepted residual).
 
 ## Gotchas / known issues
 
+- **Deploy worker2 code BEFORE the threshold migration** — old-code + new-params is fatal
+  ("Unknown named parameter" on every run → monitor dark). See Deploy order.
 - **Keep the method non-fatal — never let it throw.** worker2 has no DLQ and a 3600s SQS
-  visibility timeout, so any uncaught 500 becomes a poison-message storm. The push runs
-  non-fatal and an outer `catch(\Throwable)` is the backstop.
-- **A blind health read must page, not read healthy** — an `AwsException` on
-  `describeEnvironmentHealth` sets `alarm=HIGH` + `probe=DEGRADED` for that env.
-- **`ApplicationMetrics` is raw counts, not percentages** — divide `Status5xx` by
-  `RequestCount`; `RequestCount=0`/absent `StatusCodes` means "no data," not "zero failures."
-- **Classify 5xx-attribution from `Causes` text, but take the ratio from `ApplicationMetrics`
-  — never parse the percentage out of `Causes`.**
+  visibility timeout; an outer `catch(\Throwable)` is the backstop and CloudWatch failures are
+  caught internally (return `null`).
+- **EB app-request metrics are per-instance only** — there is no env rollup; you must
+  `SUM(SEARCH(...))` across instances or you undercount.
+- **An empty window ≠ zero failures.** An env with no published app-request metrics (e.g. 1.0
+  `agilant-worker`) reads zeros; a solely-5xx EB event over an empty window pages via the
+  `health-5xx-unmeasured` fail-safe rather than reading healthy.
+- **`minRequests` floor guards the near-idle edge** the old code couldn't — the share gate is
+  suppressed until the window has real volume, so a lone 500 in a quiet window no longer pages.
+- **OneUptime has no `probe:DEGRADED` criterion** — a metrics-read failure while EB is Ok is
+  currently unalerted until that criterion is added.
 - **`oneuptimeUrl` is a push credential** — it arrives as a cron parameter; never log it or
   record its value in a doc.
-- **Near-idle ratio edge (residual, accepted).** With the tiny-sample guard removed, a
-  near-idle env whose window holds a tiny all-error sample (e.g. its only request being a
-  500 = 100%) still trips the ratio and pages. Optional future guard: require a minimum 5xx
-  count before the ratio applies (the absolute floor gates high volume, not this low-volume
-  edge).
 
 ## Change history
+- 2026-08-25 — Replaced the EB `ApplicationMetrics` 5xx gate (a ~10s window that false-paged
+  "100% failing" on a lone 500 in a near-empty bucket — real worker-production incident: 16
+  500s / 516 req = 3.1% over 17 min) with a **CloudWatch windowed request-rate gate**. Reads
+  true per-env `ApplicationRequestsTotal`/`ApplicationRequests5xx` over a trailing window
+  (default 5 min) via `SUM(SEARCH('{AWS/ElasticBeanstalk,EnvironmentName,InstanceId}…','Sum',
+  period))` (these metrics are per-instance only — no env rollup). The 5xx page is now
+  **decoupled from EB `HealthStatus`**: pages when `5xx >= min5xxAbsolute` OR `requests >=
+  minRequests AND share >= min5xxPercent`; a non-5xx `Degraded`/`Severe` still pages
+  immediately. Added four optional cron params (`windowMinutes=5, minRequests=100,
+  min5xxPercent=20.0, min5xxAbsolute=20`) spread as named arguments; removed
+  `shouldPageForHealth()` and the `HTTP_5XX_ALARM_RATIO`/`HTTP_5XX_ALARM_ABSOLUTE_COUNT`
+  constants. Added a `health-5xx-unmeasured` fail-safe (solely-5xx over an empty/unpublished
+  window pages rather than reading healthy). Per-monitor tuning: worker + worker-1.0 keep
+  defaults; API/API-Proxy overridden to `30 / 20% / 100` (dbchanges2
+  `2026-08-25a - ApiApiproxy Beanstalk Health 5xx Thresholds.sql`, JSON_SET on the native
+  `CronJobs.parameters` JSON column). Recorded the mandatory deploy order (code before the
+  migration — old-code+new-params throws "Unknown named parameter") and the OneUptime gap
+  (no `probe:DEGRADED` criterion, so a metrics-read failure while EB reads Ok is unalerted).
+  WorkloadsRuntime already grants `cloudwatch:*` in accounts 654654170868 and 502614707982 —
+  no IAM change. (jcardinal)
 - 2026-08-19 — Overhauled the 5xx alarm gating in `shouldPageForHealth()`. The 80% share
-  gate now applies to **both** `Degraded` and `Severe` (was `Degraded`-only, which left the
-  threshold dead because EB escalates real 5xx floods straight to `Severe` — a 50%-5xx
-  `Severe` had been paging despite the 0.80 setting). This Severe-gating trade-off was made
-  deliberately with an independent architecture second opinion on record (cto returned
-  DISAGREE-WITH-ALTERNATIVE; developer accepted). Narrowed 5xx classification: replaced
-  `causesAttributeTo5xx()` (any cause mentions 5xx) with `causesAttributeSolelyTo5xx()` (every
-  cause references 5xx, ≥1) + a `causeIsFivexx()` helper, so a mixed cause (5xx trickle beside
-  a real non-5xx failure) always pages — a security-review gap the Severe change would have
-  widened. Added `HTTP_5XX_ALARM_ABSOLUTE_COUNT` (20): a solely-5xx env pages regardless of
-  share once the absolute 5xx count reaches the floor (closes the ratio's magnitude-blindness,
-  e.g. 60k of 100k). Removed `HTTP_5XX_MIN_REQUEST_COUNT` (was 20) and its tiny-sample
-  page-anyway guard — a small number of 5xx is now judged purely on the ratio; below the
-  absolute floor, page only if share ≥ 0.80, and zero requests → no page. Accepted residual:
-  a near-idle env with a tiny all-error sample still trips the ratio. (jcardinal)
+  gate now applied to **both** `Degraded` and `Severe` (was `Degraded`-only, which left the
+  threshold dead because EB escalates real 5xx floods straight to `Severe`). Narrowed 5xx
+  classification: replaced `causesAttributeTo5xx()` (any cause mentions 5xx) with
+  `causesAttributeSolelyTo5xx()` (every cause references 5xx, ≥1). Added
+  `HTTP_5XX_ALARM_ABSOLUTE_COUNT` (20). Removed `HTTP_5XX_MIN_REQUEST_COUNT` tiny-sample
+  guard. (jcardinal)
 - 2026-08-17 — Created. Documented the `ElasticBeanstalkHealth` alarm/paging criteria (page
-  on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless the 5xx share ≥
-  `HTTP_5XX_ALARM_RATIO` 0.80 with an `HTTP_5XX_MIN_REQUEST_COUNT` 20 tiny-sample guard;
-  `Severe` and non-5xx `Degraded` always page), the blind-read fail-safe (`alarm=HIGH` +
-  `probe=DEGRADED`), the four configurable constants, and the `shouldPageForHealth` /
-  `causesAttributeTo5xx` helper shape. Captured durable EB enhanced-health facts (severity
-  ladder; `ApplicationMetrics` returns raw counts not percentages; `Causes` classifies but
-  `ApplicationMetrics` sets the ratio; failed deploys aren't reliably in HealthStatus —
-  `describeInstancesHealth.Deployment.Status` is authoritative; `AWSElasticBeanstalkReadOnly`
-  covers the reads). Recorded the reverted `['Severe']`-only + deployment-failure-detector
-  direction and the accepted residual gap (a deploy that fails without degrading health).
-  (jcardinal)
+  on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless share ≥ ratio), the
+  blind-read fail-safe (`alarm=HIGH` + `probe=DEGRADED`), and durable EB enhanced-health
+  facts (severity ladder; `ApplicationMetrics` raw counts; `Causes` classifies; failed
+  deploys aren't reliably in HealthStatus). (jcardinal)
+</content>
+</invoke>

package/knowledge/INDEX.md

@@ -18,10 +18,10 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 62 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 66 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
 - **worker2** (Worker) — 56 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 25 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
-- **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
+- **dbchanges2** (Database Changes) _(framework core)_ — 9 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 7 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
 - **saml** (SAML SSO Gateway) — 4 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
 - **toga2-view** (TOGa View Frontend) — 12 doc(s) → [2.0/apps/toga2-view/INDEX.md](2.0/apps/toga2-view/INDEX.md)

package/knowledge/clients/compass-usa/profile.md

@@ -22,6 +22,7 @@ updated: 2026-08-25
 owners: [jcardinal, bala, tcox, apeterson, dfranks]
 files: []
 related:
+  - ../../2.0/apps/_underscore/features/sales-order-po-number-sourcing.md
   - features/persona-model-and-levy-gating.md
   - features/people-file-user-lifecycle.md
   - workflows/persona-refactor-migration.md

package/knowledge/clients/nychh/INDEX.md

@@ -2,4 +2,5 @@
 
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
+| [NYCHH PO links are UPSTREAM — the downstream SO→PO route returns empty](features/po-number-upstream-direction.md) | 2.0 | NYCHH's sales orders are created **from the customer's purchase order**, so their SO↔PO links live in the **upstream** table `PurchaseOrders_SalesOrders` (route | _underscore/Model/Client/SalesOrder.php, toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts, dbchanges2/Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql |
 | [NYC Health & Hospitals](profile.md) | 2.0 | NYC Health & Hospitals (NYCHH) is a TOGA 2.0 client on the `_underscore` platform, prod schema `Client_Nychh`. | dbchanges2/Client_Nychh/2026-08-18 - InventoryUnitsItemColumns.sql, dbchanges2/Client_Nychh/2026-08-24 - FixInventoryGroupingsUnitsTopologyOverrideIds.sql |

package/knowledge/clients/nychh/features/po-number-upstream-direction.md

@@ -0,0 +1,59 @@
+---
+title: "NYCHH PO links are UPSTREAM — the downstream SO→PO route returns empty"
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: nychh
+type: client-feature
+status: active
+updated: 2026-08-25
+owners: [apeterson]
+files:
+  - _underscore/Model/Client/SalesOrder.php
+  - toga25-supply/src/pages/SalesOrders/view/SalesOrderRecordModalLayout/hooks/usePurchaseOrderDetails.ts
+  - dbchanges2/Client/2026-08-11b - SalesOrderPurchaseOrdersField.sql
+related:
+  - ../../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
+  - ../../../2.0/apps/_underscore/features/sales-order-po-number-sourcing.md
+  - ../profile.md
+---
+
+## Summary
+
+NYCHH's sales orders are created **from the customer's purchase order**, so their SO↔PO links live
+in the **upstream** table `PurchaseOrders_SalesOrders` (route `purchase-order-sales-orders`,
+record 276). They have **no** downstream `SalesOrders_PurchaseOrders` rows.
+
+This produced a symptom that looked like a bug and was not: the sales-order modal's PO Number came
+back as an empty array while the sales-orders **list** column showed a PO number for the same
+order. The front end was calling the **downstream** route; the list column's table-view join uses
+the **upstream** table.
+
+## How it works
+
+- Verified on `Client_Nychh` (sandbox-client) for the affected order: **upstream = 1 row,
+  downstream = 0 rows**.
+- The API returned `200` with `totalRecordCount 0` and **no** `EZ-*`/`EV-*` messages — the
+  signature of a join matching nothing, not of a permission or field-resolution failure.
+- **ACL was clean and was ruled out:** `AclRecordPermissions` on records **275 and 276** both grant
+  `roleId` 1 and 3 full CRUD in `Client_Nychh`; the JWT carried client roles `[1,2]`. This is *not*
+  the Compass Canada-style role mis-seed.
+- Both relevant migrations were confirmed to have run on `Client_Nychh`
+  (`Client/2026-08-11b`, `Client/2026-08-13`) — see
+  [verifying a migration ran](../../../2.0/apps/dbchanges2/workflows/verifying-a-migration-ran.md).
+
+## Gotchas
+
+- 🚨 **Open consequence.** The shared `_purchaseOrders` calculated field reads the **downstream**
+  table (`_underscore/Model/Client/SalesOrder.php:348`). Now that the modal's PO Number binds to
+  `_purchaseOrders`, NYCHH resolves it without an `EV-8` but gets **`null` forever**. NYCHH needs an
+  **upstream counterpart** — either an upstream-aware override of `_purchaseOrders` for this tenant
+  or an upstream variant field. **No tenant precedent exists for upstream-only:** Compass USA reads
+  both, Quad and Compass Canada are downstream-only.
+- Do not "fix" this by adding downstream rows. The direction is correct for NYCHH's business model;
+  the shared field is what is incomplete.
+
+## Change history
+- 2026-08-25 — Diagnosed the empty-array PO Number as a wrong-direction query (upstream data,
+  downstream route); ruled out ACL and migrations; recorded the still-open `_purchaseOrders`
+  downstream-only gap for this tenant (apeterson)

package/knowledge/clients/nychh/profile.md

@@ -23,6 +23,8 @@ files:
 related:
   - ../../2.0/apps/_underscore/features/tracking-number-bridges.md
   - ../../2.0/apps/_underscore/features/tableview-joins.md
+  - ./features/po-number-upstream-direction.md
+  - ../../2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
 ---
 
 ## Summary
@@ -52,6 +54,11 @@ table views. Client-specific DB change-sets live in `dbchanges2/Client_Nychh/`.
   tag added after the ItemShip never bumps the fulfillment sync. See
   [NYCHH Asset-Tag Backfill](../../2.0/apps/worker2/features/nychh-asset-tag-backfill.md).
 
+- **PO links are UPSTREAM (`PurchaseOrders_SalesOrders`), not downstream.** The downstream
+  `sales-order-purchase-orders` route returns an empty array for NYCHH by design, and the shared
+  `_purchaseOrders` field (downstream-only) resolves to `null` for this tenant — open gap. See
+  [NYCHH PO Number direction](./features/po-number-upstream-direction.md).
+
 ## Notes
 - Tracking data: record 318 (item-level) is currently empty for this client; their tracking
   populates the IF/shipment level (record 317). The rebuilt views use 318 (per the Compass

package/knowledge/clients/quad/INDEX.md

@@ -2,5 +2,6 @@
 
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
+| [Quad Enter PO Details — single-vendor-by-country selection and the EV-12 `vendorItem` failure](features/enter-po-details-vendor-selection-ev12.md) | 2.0 | Adding a PO number for Quad through **Enter PO Details** can `400` with **`EV-12` "unable to find a unique match"** on field **`vendorItem`**. | _underscore/Model/Quad/PurchaseOrder.php |
 | [Quad: PO + ASN email importer (import_po_and_asn.php)](features/po-asn-email-import.md) | 1.0 | Quad Graphics' supplier emails Purchase Order and Advance Shipping Notice CSVs into a monitored Microsoft 365 mailbox. | worker/crons/toga2/quad/import_po_and_asn.php |
 | [Quad Graphics](profile.md) | 2.0 | Quad Graphics is a TOGA 2.0 client on the `_underscore` platform, prod schema `Client_Quad`. |  |

package/knowledge/clients/quad/features/enter-po-details-vendor-selection-ev12.md

@@ -0,0 +1,62 @@
+---
+title: "Quad Enter PO Details — single-vendor-by-country selection and the EV-12 `vendorItem` failure"
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: quad
+type: client-feature
+status: active
+updated: 2026-08-25
+owners: [apeterson]
+files:
+  - _underscore/Model/Quad/PurchaseOrder.php
+related:
+  - ../../../2.0/apps/api2/features/v2-api-error-codes.md
+  - ../../../2.0/apps/_underscore/features/sales-order-po-number-sourcing.md
+  - ../../../2.0/standards/backend-php.md
+  - ../profile.md
+---
+
+## Summary
+
+Adding a PO number for Quad through **Enter PO Details** can `400` with
+**`EV-12` "unable to find a unique match"** on field **`vendorItem`**. The failure is a **code
+defect plus a country-scoped catalog coverage gap** — not a bad data reset and not ACL. Quad's
+`Client_Quad` data was verified healthy (592 `VendorItems`, 18 active vendors, 15 with a complete
+`Vendors → Addresses → States → Countries` chain).
+
+## How it works
+
+1. The FE payload contains **no items**. Quad's `prePost` interceptor
+   (`_underscore/Model/Quad/PurchaseOrder.php:9` → `populatePurchaseOrderFromSalesOrder`) builds
+   them from the sales order's lines.
+2. `getVendorByCountry` (line ~186) picks **one vendor for the whole PO**: first the vendor covering
+   the **most** of the order's items *in the ship-to country*, else **any** vendor in that country.
+3. Line ~79 then restricts the `VendorItems` join to that **single** vendor.
+4. When the chosen vendor has no `VendorItems` row for a line, lines ~127-139 fall back to sending a
+   **descriptor** (`vendor` + `item` + `vendorPartNumber`).
+
+## Gotchas
+
+- 🚨 **The descriptor fallback cannot work — its comment is wrong.** The comment claims the API can
+  "create/match" the descriptor. The field's child policy is **`MATCH`**, which only *looks up* and
+  **never creates**. Zero matches (or 2+) ⇒ `EV-12`. This is the primary defect.
+- **Real-world instance:** order ships to Poland; picked vendor **Bechtle Direct Polska** (PL,
+  correct); item **MGDN4HN/A** is mapped **only** to General Technologies (IN, India). No Polish
+  supplier for the part ⇒ `EV-12`. Reproduced on a second order with a different vendor/item pair —
+  it is **systemic**, not one-off.
+- 🚨 **Secondary defect — nondeterministic vendor choice.** `getVendorByCountry`'s ranking query is
+  `ORDER BY vendorItemCount DESC LIMIT 1` with **no tie-break column**. On tied counts the winner is
+  arbitrary and can **change after a database reload** (row order/ids move). This is exactly the
+  hazard in [backend-php standards](../../../2.0/standards/backend-php.md) ("a `LIMIT 1` on a
+  non-unique key MUST have a deterministic `ORDER BY`"). Fix is to append `, v.id`.
+- **A vendor with a broken address link is invisible to vendor selection.** The queries require the
+  full `Vendors → Addresses → States → Countries` chain; a vendor missing any hop is silently
+  skipped and the PO is silently handed to a different vendor.
+- **Recommended remediation (not yet applied):** raise a useful error naming the chosen vendor and
+  the unmapped item instead of an opaque `EV-12`, and add the deterministic tie-break.
+
+## Change history
+- 2026-08-25 — Diagnosed EV-12 on `vendorItem` as the `MATCH`-policy descriptor fallback that can
+  never create, driven by single-vendor-by-country selection against a country-scoped catalog gap;
+  also recorded the missing `LIMIT 1` tie-break in `getVendorByCountry` (apeterson)

package/knowledge/clients/quad/profile.md

@@ -20,6 +20,8 @@ owners: ["jcardinal", "bala", "apeterson", "ajean"]
 files: []
 related:
   - ./features/po-asn-email-import.md
+  - ./features/enter-po-details-vendor-selection-ev12.md
+  - ../../2.0/apps/_underscore/features/sales-order-po-number-sourcing.md
   - ../../2.0/apps/_underscore/features/tracking-number-bridges.md
   - ../../2.0/apps/_underscore/features/item-fulfillment-stage-lifecycle-and-order-status.md
   - ../../2.0/apps/_underscore/features/surface-resolver.md

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.649",
+  "version": "1.0.651",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",