toga-ai 1.0.270 → 1.0.272

package/knowledge/1.0/standards/backend-php.md

@@ -5,8 +5,8 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-06-08
-owners: [jcardinal]
+updated: 2026-07-06
+owners: [jcardinal, rgirish]
 files: []
 related:
   - ../apps/library/architecture.md
@@ -395,6 +395,34 @@ class Browser_Datagrid_Accounts extends Browser_Datagrid {
 * The `@` error-suppression operator is acceptable only for defensive file I/O (e.g. reading an optional cache file); do not use it to hide real errors.
 * Use structured logging to capture application behavior and errors.
 
+### PHP warnings throw — guard fragile I/O, do not rely on `!== false`
+
+The bootstrap sets `error_reporting(E_ALL)` and `App_Error::handleError` (`library/app/error.php`)
+converts **every** PHP warning into an `ErrorException`. Consequences you must design around:
+
+- Warning-emitting calls **throw at the call site** rather than returning their documented
+  failure value. `file_get_contents()` on an unreachable/403/timed-out URL raises an
+  `ErrorException` **inside** the call — so a following `if ($x !== false)` guard never runs.
+- In a loop (e.g. a cron), one such throw aborts the **entire** remaining batch, not just the
+  current item. This is especially dangerous in two-phase "stamp then act" crons where earlier
+  phases have already committed state.
+
+Wrap fragile I/O and return the failure value yourself:
+
+```php
+function safeFetchPdf(string $url) {
+    try {
+        return file_get_contents($url);
+    } catch (\ErrorException $e) {
+        error_log('PDF fetch failed: ' . $e->getMessage());
+        return false;
+    }
+}
+```
+
+Then guard each dependent step **independently** (not a joint `&&`) so one failure degrades
+gracefully instead of dropping unrelated work or crashing the loop.
+
 ## Security Best Practices
 
 ### Input Validation and Sanitization

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -4,6 +4,7 @@
 |-----|---------|-------|
 | [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php |
 | [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | A cron watchdog that emails when the ClickUp integration looks disconnected during business hours. | worker2/Worker/Clickup/Health.php, worker2/Database/ClickupHealthWatchdog.sql |
+| [ClickUp Design Sprint Automation (Final Design Outcome)](features/clickup-design-sprint-automation.md) | `_Worker_Clickup_Design` is meant to drive the design-sprint workflow in ClickUp via the API, replacing a set of native ClickUp automations. | worker2/Worker/Clickup/Design.php, worker2/Worker/Clickup.php |
 | [ClickUp GitHub-tab Auto-linking & Ticket-id Branch Naming](features/clickup-github-autolink.md) | How ClickUp surfaces branches/PRs/commits in a ticket's **GitHub tab**, and the branch / PR-title naming convention that triggers it. |  |
 | [ClickUp Project & Opportunity Multi-List Routing](features/clickup-project-routing.md) | Routes ClickUp tasks into the correct **secondary multi-list memberships** based on their custom-field values, via the `clickup` webhook. | worker2/Worker/Clickup/Project.php, worker2/Worker/Clickup.php |
 | [ClickUp Rich-Text Custom Fields via Quill Delta (API)](features/clickup-richtext-api.md) | ClickUp custom text fields (type `text` and long-text) support rich formatting only through a **Quill Delta** written to the undocumented `value_richtext` key o | test/@dave/clickup_md2delta.js, .claude/skills/plan-ticket/scripts/clickup.js |

package/knowledge/2.0/apps/worker2/features/clickup-design-sprint-automation.md

@@ -0,0 +1,111 @@
+---
+title: ClickUp Design Sprint Automation (Final Design Outcome)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-07-06
+owners: ["ajean"]
+files:
+  - worker2/Worker/Clickup/Design.php
+  - worker2/Worker/Clickup.php
+related:
+  - ./clickup-work-type-automation.md
+  - ./team-sprint-management.md
+---
+
+## Summary
+`_Worker_Clickup_Design` is meant to drive the design-sprint workflow in ClickUp via the API,
+replacing a set of native ClickUp automations. When a task's **Final Design Outcome**
+drop-down is set to an actionable value, the worker either spawns an iteration task or
+completes the task, and a daily cron reminds the Design Approver of pending reviews.
+
+**Critical current state (as of 2026-07-06):** the worker's
+`handleFinalDesignOutcome` has been **silently no-opping in production** — the native
+ClickUp automations it was supposed to replace were never disabled and had been doing the
+real work all along, masking the failure. Do not assume the API path is live. See Gotchas
+and the open action item below before touching this code.
+
+## Key files / entry points
+- `Worker/Clickup/Design.php` → `_Worker_Clickup_Design` — the design-sprint engine.
+  - `handleFinalDesignOutcome(...)` — the entry that decides spawn vs. complete (currently a
+    silent no-op in prod).
+  - `resolveFinalDesignOutcomeOptionId(...)` (Design.php:298-317) — maps the drop-down value
+    to an option ID.
+  - `spawnIterationTask(...)` / `carryDesignApprover(...)` — duplicate the task and copy the
+    Design Approver.
+  - `completeTask(...)` — closes an approved task.
+  - `SendReviewReminders(...)` — daily reminder cron action.
+  - `TRUE_DESIGN_SPRINTS_FOLDER_ID` constant (Design.php:27,50).
+- `Worker/Clickup.php` (dispatcher ~lines 822-875) — routes ClickUp webhooks; the Design
+  Approver / status guard lives at 822-833.
+
+## How it works
+1. A ClickUp webhook reaches `_Worker_Clickup`; the dispatcher (Clickup.php ~822-875) routes
+   design-folder events into `_Worker_Clickup_Design`.
+2. The worker acts on **only two** Final Design Outcome values:
+   - **"Iterated – New Task"** (option `3f835ac8…`) → `spawnIterationTask` — duplicates the
+     ticket into a new iteration task and carries over the Design Approver.
+   - **"Approved"** (option `2d17868b…`) → `completeTask`.
+3. The full Final Design Outcome option set is: In Progress, In Review, **Iterated – New
+   Task**, **Approved**, Cancelled. Only the two bolded values are actionable.
+4. A separate ClickUp automation (`a64c863a`) sets the outcome to **"In Review"** when a
+   task's *status* changes to Iterated. "In Review" is **not** actionable — so
+   `status → Iterated` alone never triggers the API. The worker acts only when the Design
+   Approver **manually** selects one of the two actionable outcome values.
+5. Daily reminder cron `Core.CronJobs` id 21 ("Design Review Reminders", `0 9 * * 1-5`,
+   action `Clickup/Design/SendReviewReminders`) nudges the approver about pending reviews.
+
+## Data model
+Configuration lives as ClickUp identifiers (folder IDs, custom-field IDs, drop-down option
+UUIDs), not in the DB. The custom-field / option-ID constants are defined in `_underscore`
+(`_production`) — confirmed present (worker2 PR #80, _underscore PR #624); the
+undefined-constant theory was ruled out.
+
+## Client variations
+None — internal TOGA Technology (`true`) design-sprint tooling.
+
+## Gotchas / known issues
+- **Silent no-op masked by native automations (root cause of TRUE-79685).** The worker's
+  docstring claims it replaced the native automations, but they were never disabled. Native
+  automation `c25fd345` ("When Final Design Outcome is Iterated, duplicate the ticket and
+  close the original") was doing the duplicate+close and **hiding** that the worker never
+  spawned. Only after disabling **all** native automations did setting Final Design Outcome →
+  "Iterated – New Task" on TRUE-80046 (prod job 295376, isSuccess=1) produce **no** spawn,
+  exposing the failure.
+- **OPEN — two prime suspects for the silent no-op** (need a combined code + live-API session
+  to confirm/fix):
+  1. **Folder-gate ID mismatch.** `TRUE_DESIGN_SPRINTS_FOLDER_ID = '90115939842'`
+     (Design.php:27,50) vs. the file-header comment naming `90020178491`. `90115939842` is a
+     valid folder (the daily reminder cron GETs `/folder/90115939842/list` successfully) but
+     may not be the folder the automation-test tasks live in — in which case the folder gate
+     returns early on every task. **Verify:** compare `getTaskDetails` `folder.id` for
+     TRUE-80046 against the constant.
+  2. **Strict `===` mismatch in `resolveFinalDesignOutcomeOptionId`** (Design.php:298-317). If
+     `getTaskDetails` returns the `drop_down` value in a shape (string `orderindex`, or an
+     option-id) that matches neither `option->id === value` nor `option->orderindex === value`,
+     it returns `null` → silent no-op. Fix should be an explicit **cast/validate at the
+     ClickUp-payload boundary**, not merely switching `===` to `==`.
+- **Latent bug — spurious "Design Review" email on freshly spawned tasks.** Spawned iteration
+  tasks transiently pass through "iterated" status during creation (a ClickUp list
+  automation), and `spawnIterationTask → carryDesignApprover` copies the Design Approver onto
+  the new task. If that copy lands while the new task is momentarily "iterated", the dispatcher
+  guard (Clickup.php:822-833: `custom_field.name == 'Design Approver' && status == 'iterated'`)
+  fires a **spurious** "Action Needed: Design Review" email. Candidate fix: gate
+  `sendDesignApproverEmail` on the change being user-driven (history `user.id != ClickBot -1`).
+- **Native automations must be disabled once the worker is fixed.** The worker was meant to
+  replace: `c25fd345`, `272f743e` (Automation #39, native "email Aaron when iterated"),
+  `a64c863a`, `7739cc41`, `08570f31`, `70b09962`. Keep sprint-readiness automations only:
+  `10df1786`, `f6214678`. **INTERIM:** if the design team needs the workflow before the worker
+  is fixed, re-enable **only** `c25fd345` (temporary — this reintroduces the Iteration-reset
+  and spurious-email issues).
+
+## Change history
+- 2026-07-06 — Diagnosed the "design-sprint API automation not running" report (TRUE-79685;
+  tests TRUE-80026/80046): `handleFinalDesignOutcome` is silently no-opping in prod, masked by
+  never-disabled native automation `c25fd345`. Narrowed to two suspects (folder-gate ID
+  mismatch, strict-`===` option resolution) pending a code+live-API fix session. Also
+  identified a latent spurious-email bug on spawned tasks and clarified that only
+  "Iterated – New Task" and "Approved" outcomes are actionable. (ajean)

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 11 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 12 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 13 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **togadesk** (TOGa Desk) — 8 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
 - **togaview** (TOGa View) — 6 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
 - **webhook** (Webhook) — 1 doc(s) → [1.0/apps/webhook/INDEX.md](1.0/apps/webhook/INDEX.md)
@@ -17,7 +17,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 22 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 25 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 26 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 7 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/compass-usa/INDEX.md

@@ -7,4 +7,5 @@
 | [Compass MITS PO → SO Item Linking](features/mits-po-to-so-item-linking.md) | 2.0 | MITS sends Compass inbound Purchase Orders (`POST /v2/purchase-orders`) against a Sales Order (`mitsSalesOrder`). | _underscore/Model/Compass/PurchaseOrder.php, worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php |
 | [Compass MITS PO Transmission to Vendors](features/mits-po-transmission-to-vendors.md) | 2.0 | The 1.0 worker cron `2_transmit_mits_purchase_orders_to_vendors.php` transmits Compass PurchaseOrders to their vendors (Office Depot, Strategic Systems, Compass | worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php, library/app/client/compass.php |
 | [Compass USA](profile.md) | 2.0 | Compass USA is a TOGA client running a multi-tier supply-chain commerce operation. |  |
+| [Compass ODP Order Pipeline to NetSuite (numbered worker crons)](workflows/odp-order-pipeline-to-netsuite.md) | 1.0 | The end-to-end **Compass Office Depot (ODP) order → NetSuite** pipeline as it actually runs through the 1.0 `worker` crons under `worker/crons/toga2/compass/`, | worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php, worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php, worker/crons/toga2/compass/edi/1_download_edi_s3_create_po_toga.php, worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, library/app/client/compass.php |
 | [Compass Order Lifecycle & Data-Integrity Invariants](workflows/order-lifecycle-and-data-integrity.md) | 2.0 | End-to-end map of how a Compass order flows through the `Client_Compass` (2.0) database and the **expected raw-data shape** at each link/ASN/IF level. |  |

package/knowledge/clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md

@@ -0,0 +1,113 @@
+---
+title: Compass ODP Order Pipeline to NetSuite (numbered worker crons)
+framework: "1.0"
+repo: worker
+project: Worker
+client: compass-usa
+type: workflow
+status: active
+updated: 2026-07-01
+owners: ["rgirish"]
+files:
+  - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
+  - worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php
+  - worker/crons/toga2/compass/edi/1_download_edi_s3_create_po_toga.php
+  - worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php
+  - library/app/client/compass.php
+related:
+  - clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md
+  - clients/compass-usa/features/mits-po-transmission-to-vendors.md
+  - clients/compass-usa/features/mits-po-to-so-item-linking.md
+  - clients/compass-usa/profile.md
+---
+
+## Summary
+The end-to-end **Compass Office Depot (ODP) order → NetSuite** pipeline as it actually runs
+through the 1.0 `worker` crons under `worker/crons/toga2/compass/`, and the **diagnostic
+method** for "this Compass order never reached NetSuite" support tickets. This is the *cron
+mechanics* companion to the 2.0 [Compass Order Lifecycle & Data-Integrity
+Invariants](order-lifecycle-and-data-integrity.md) (which covers the raw-data shape / repair
+side of the same chain). All facts verified against the live `prod` / `Client_Compass` schema
+and the cron source. The crons are `App_` (1.0) framework code; the data lives in the 2.0
+`Client_Compass` / `Logs_Compass` schema.
+
+## The pipeline (customerId / vendorId, in run order)
+
+```
+Commerce → Compass SalesOrder            (SalesOrders.customerId = 2)
+  [cron 1_transmit_compass_sales_orders_to_mits]  → MITS (API)
+MITS → PurchaseOrder                      (PurchaseOrders.vendorId = 1 = OFFICE DEPOT; POST /v2/purchase-orders; PO number e.g. 50305906-1)
+  [cron 2_transmit_mits_purchase_orders_to_vendors] → ODP (cXML; email fallback per integration)   [sets PurchaseOrders.dtSubmitted]
+ODP → 850 EDI dropped to S3 (AS2)         (bucket agilant-as2, prefix OfficeDepot/)
+  [cron edi/1_download_edi_s3_create_po_toga]     → creates ODP SalesOrder (customerId = 1); deletes the S3 file after processing
+ODP SalesOrder (customerId = 1)
+  [cron 5_create_netsuite_sales_orders_from_office_depot_purchase_orders, hourly] → NetSuite SO
+     writes back c_dtTransmittedToNetsuite + c_netsuiteInternalSalesOrderId ONTO the ODP SO (customerId=1)
+```
+
+## The crons
+
+1. **`workflow/1_transmit_compass_sales_orders_to_mits.php`** — transmits the Compass
+   `SalesOrder` (customerId=2) to MITS via API.
+2. **`workflow/2_transmit_mits_purchase_orders_to_vendors.php`** — transmits the MITS-created
+   PO to ODP. ODP integration is **cXML** (auth audience `"Office Depot (Cxml)"`); the script
+   also supports an **email fallback** path per the vendor's integration type. Sets
+   `PurchaseOrders.dtSubmitted` when transmitted (~2 min after PO creation). (See the dedicated
+   [MITS PO Transmission to Vendors](../features/mits-po-transmission-to-vendors.md) feature for
+   the item-less-PO gotcha and the CXML-vs-EMAIL routing detail.)
+3. **`edi/1_download_edi_s3_create_po_toga.php`** — pulls the ODP **850** from S3 bucket
+   `agilant-as2`, prefix `OfficeDepot/` (delivered via AS2), and creates a downstream **ODP
+   SalesOrder** (customerId=1 = Office Depot). **Deletes the file from S3 after processing** —
+   so an absent S3 object is expected once ingested, not evidence of a miss.
+4. **`workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php`** — runs
+   **hourly**; picks up ODP SalesOrders
+   (`OfficeDepotSalesOrders.customerId = 1 AND c_dtTransmittedToNetsuite IS NULL`), creates the
+   SO in NetSuite, and writes back `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId`
+   **onto the ODP SalesOrder (customerId=1), never the Compass SO (customerId=2)**.
+   **Exclusion filters** (an order legitimately waiting is not a bug):
+   - `CompassSalesOrders.number NOT LIKE 'MA%'` — MA orders are excluded.
+   - **computer-kit orders** (Bundles 187–196) wait for a **2nd PO** before syncing.
+   - a **30-minute delay** after the first ODP PO is created.
+
+## Constants & identities (`library/app/client/compass.php`)
+- `App_Client_Compass::VENDOR_ID__OFFICE_DEPOT = 1` — `Vendors.id = 1` = "OFFICE DEPOT".
+- `App_Client_Compass::CUSTOMER_ID__OFFICE_DEPOT = 1` — `Customers.id = 1` = "Office Depot".
+- Compass customer SOs are `customerId = 2`. (Agilant/NetSuite SOs are `customerId = 3`; see the
+  lifecycle doc.)
+
+## Diagnosing "this Compass order never reached NetSuite" (support-ticket method)
+
+The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**, so checking the
+wrong record makes every order look "stuck." Use the **join**, never a number match:
+
+- **Do NOT judge the Compass SO (customerId=2) by its NetSuite fields.**
+  `c_dtTransmittedToNetsuite` / `c_netsuiteInternalSalesOrderId` are **ALWAYS NULL on
+  customerId=2 SOs by design** — only the downstream ODP SO (customerId=1) carries them.
+  (Verified: of 1,827 customerId=2 SOs since 2026-06-01, **zero** have NetSuite fields set.)
+- **Do NOT search for the downstream ODP SalesOrder by the MITS PO number**
+  (e.g. `number LIKE '%50305906%'`). ODP SalesOrders are numbered with **ODP's own** order
+  numbers (e.g. `471134097001`), not the MITS PO number. Traverse the bridge instead.
+- **Link tables are directional and distinct:**
+  - `SalesOrders_PurchaseOrders` = **SO→PO** (a SalesOrder to the PO it generated).
+  - `PurchaseOrders_SalesOrders` = **PO→SO** (a PurchaseOrder to the downstream ODP SalesOrder
+    created from ODP's 850).
+  - To find the ODP SO for a Compass PO, **traverse `PurchaseOrders_SalesOrders`.**
+- **To confirm a PO actually reached ODP:** check `PurchaseOrders.dtSubmitted` is set **AND**
+  that downstream ODP SO(s) exist via the `PurchaseOrders_SalesOrders` join — their existence
+  proves ODP received the PO and 850'd back.
+- **`PurchaseOrderNotes` is NOT the record of truth.** That table is **empty across ALL POs** —
+  do not read its emptiness as evidence a transmission script never ran. (Transmission evidence
+  is `dtSubmitted` + the `Logs_Compass.Record` "transmitted to Vendor" note — see the MITS PO
+  transmission feature.)
+
+## Change history
+- 2026-07-01 — Documented the numbered ODP→NetSuite worker-cron pipeline (crons 1/2/edi-1/5),
+  the AS2 850 S3 hand-off (`agilant-as2` / `OfficeDepot/`), the customerId=1-vs-2 NetSuite
+  writeback rule + cron-5 exclusion filters, and the "order not in NetSuite" diagnostic method
+  (join not number-match; empty `PurchaseOrderNotes` is not evidence). Production-support
+  investigation; no code change. (rgirish)
+
+## Related docs
+- [Compass Order Lifecycle & Data-Integrity Invariants](order-lifecycle-and-data-integrity.md)
+- [Compass MITS PO Transmission to Vendors](../features/mits-po-transmission-to-vendors.md)
+- [Compass MITS PO → SO Item Linking](../features/mits-po-to-so-item-linking.md)

package/knowledge/clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md

@@ -15,6 +15,7 @@ related:
   - clients/compass-usa/features/item-fulfillment-tracking-tableview.md
   - clients/compass-usa/features/mits-po-transmission-to-vendors.md
   - 2.0/apps/_underscore/features/item-fulfillment-stage-lifecycle-and-order-status.md
+  - clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md
 ---
 
 ## Summary

package/knowledge/clients/prudential/INDEX.md

@@ -5,6 +5,6 @@
 | [Prudential: Dell ASN units PRE/POST interceptor (legacy key + flat tracking)](features/dell-asn-units-interceptor.md) | 2.0 | After the tracking-number bridge migration, the ASN unit route was renamed (`advance-shipping-notice-units` → `advance-shipping-notice-item-units`), so the inhe | _underscore/Model/Prudential/AdvanceShippingNotice.php, dbchanges2/Client_Prudential/2026-06-10 - AsnUnitsInterceptor.sql |
 | [Prudential: Dell LCH IOP transmissions (LCHRequestV2)](features/dell-lch-iop-transmissions.md) | 1.0 | Outbound order transmissions from the 1.0 worker tier to Dell's Lifecycle Hub (LCH) ITSM integration. | library/app/api/delllch.php, worker/crons/toga2/prudential/transmissions_to_dell.php, worker/crons/toga2/prudential_beta/transmissions_to_dell_india.php, worker/crons/toga2/prudential_beta/transmissions_to_dell_ireland.php, worker/crons/toga2/prudential_beta/transmissions_to_dell_usa.php |
 | [Prudential: Service Request Regional Address Validation](features/service-request-address-validation.md) | 2.0 | The `prePost` interceptor on `_Model_Prudential_ServiceRequest` validates `deliverToAddress` fields differently depending on which Prudential regional customer | _underscore/Model/Prudential/ServiceRequest.php |
-| [Prudential Order Shipped Email — transmit_ordershipped_updates_prudential.php](features/transmit-ordershipped-email.md) | 1.0 | Cron script that transmits "Order Shipped" updates to ServiceNow (RITM) and sends a shipped notification email to the end user. | worker/crons/toga2/prudential/transmit_ordershipped_updates_prudential.php |
+| [Prudential Order Shipped Email — transmit_ordershipped_updates_prudential.php](features/transmit-ordershipped-email.md) | 1.0 | Cron script that transmits "Order Shipped" updates to ServiceNow (RITM) and sends a shipped notification email to the end user. | worker/crons/toga2/prudential/transmit_ordershipped_updates_prudential.php, worker/crons/toga2/prudential/transmit_closecomplete_updates_prudential.php, worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php, worker/crons/toga2/prudential_beta/generate_sales_and_purchase_orders_from_service_requests.php |
 | [Prudential Financial](profile.md) | 2.0 | Prudential is a TOGA client whose device-fulfillment flow is driven by **Dell** via the Dell API (`Client_Prudential.Apis.id = 2`). |  |
 | [Prudential: Dell ASN failed POST backfill replay](workflows/dell-asn-backfill-replay.md) | 2.0 | When Dell ASN POSTs fail in bulk (e.g. |  |

package/knowledge/clients/prudential/features/transmit-ordershipped-email.md

@@ -6,10 +6,13 @@ project: Worker
 client: prudential
 type: client-feature
 status: active
-updated: 2026-06-18
+updated: 2026-07-06
 owners: ["rgirish"]
 files:
   - worker/crons/toga2/prudential/transmit_ordershipped_updates_prudential.php
+  - worker/crons/toga2/prudential/transmit_closecomplete_updates_prudential.php
+  - worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php
+  - worker/crons/toga2/prudential_beta/generate_sales_and_purchase_orders_from_service_requests.php
 related:
   - ../profile.md
 ---
@@ -49,22 +52,68 @@ Iterates the same POs collected in Phase 1. For each:
 `c_dtTransmittedOrderShippedUpdateToPrudential IS NOT NULL` AND `c_dtEmailSentOrderShipped IS NULL`
 as "Order Shipped Email Not Sent". This appears daily in the Prudential REQ Processing Report email.
 
+## Sibling scripts sharing this pattern
+The same two-phase (transmit-then-email) shape and the same PDF-fetch fragility exist in three
+sibling crons, all fixed the same way this session (2026-07-06):
+- `transmit_closecomplete_updates_prudential.php` — the "Order Closed/Complete" analogue. Phase 1
+  stamps `c_dtTransmittedOrderClosedUpdateToPrudential`; Phase 2 stamps `c_dtEmailSentOrderClosed`.
+  Same stranding flaw (Phase-1 SELECT filters only on the transmitted column).
+- `generate_sales_and_purchase_orders_from_service_requests.php` (prod and `prudential_beta/`) —
+  got the PDF crash-proofing only. Their re-processing is gated by SalesOrder/PurchaseOrder
+  **existence**, not a timestamp, so no self-heal was added there (it would risk duplicate SO/PO
+  or duplicate emails).
+
 ## Gotchas / known issues
 
-**PDF fetch crash kills entire batch (fixed 2026-06-18):**
-The New-type branch fetches 3 PDFs from S3 via `file_get_contents()`. If any S3 URL returns
-a non-200 (e.g. 403 Forbidden), PHP emits a warning which `App_Error::handleError` converts
-to an `ErrorException` — crashing the script mid-loop. Phase 1 has already stamped
-`c_dtTransmittedOrderShippedUpdateToPrudential` for all POs in the batch, but Phase 2 dies
-before writing `c_dtEmailSentOrderShipped` for any remaining POs.
+**ROOT CAUSE — `file_get_contents` in a 1.0 cron THROWS, it does not return false (discovered 2026-07-06):**
+The 1.0 framework bootstrap sets `error_reporting(E_ALL)` and `App_Error::handleError`
+(`library/app/error.php` ~line 348) converts **every** PHP warning into an `ErrorException`.
+So `file_get_contents()` on an unreachable / 403 / timed-out S3 URL **throws at the call site**,
+before any `if ($pdf !== false)` guard can ever run. This is why the pre-existing `!== false`
+guards (and the 2026-06-18 guard) never worked: the exception is raised **inside**
+`file_get_contents`, not returned by it. Failures are often transient (the same URL returns 200
+on a later check), which is why the "Order Shipped email not sent" issue kept recurring after
+multiple prior "fixes" that only swapped individual PDF filenames.
+
+**Fix — `safeFetchPdf()` + independent per-PDF guards (2026-07-06):**
+Added a file-scope helper `function safeFetchPdf(string $url)` that wraps `file_get_contents`
+in `try/catch (\ErrorException)` and returns `false` on failure — so a failed fetch now really
+does return false. Every PDF fetch was converted to `safeFetchPdf()`, and each
+`file_put_contents` + `$mail->addAttachment` is guarded by its **own independent**
+`if ($pdfGuideX !== false)` — NOT a joint `&&`, so one failed PDF drops only that attachment and
+the other attachments and the email itself still go out. **The email must always send; a
+missing/failed PDF must never crash the loop or block the send.** Applied across all four files.
+
+**Two-phase stranding is permanent — recovery MUST be date-bounded (discovered 2026-07-06):**
+Phase 1 stamps the transmitted column and the Phase-1 SELECT filters **only** on that column
+being NULL. So any Phase-2 crash strands that PO's email **forever** (transmitted flag already
+set → never re-selected). Prod inspection found ~8,070 ordershipped + ~11,343 closecomplete POs
+historically stranded (transmitted set, email NULL) — but **most are OLD rows never
+back-populated and are NOT owed an email**; only ~21 shipped / ~60 close were genuinely recent
+(last 7 days). **Any stranding-recovery query MUST be date-bounded**, or you will mass-email
+customers about months-old orders. `prudential_exception_report.php` already flags these
+(transmitted set + email NULL). This session deliberately did NOT auto-recover stranded POs —
+see Scope decision below.
+
+**Scope decision — fix-forward only, no automated self-heal (decided 2026-07-06):**
+An earlier iteration built a bounded 7-day self-heal SELECT; the developer decided to keep the
+fix to **forward crash-proofing only** ("just make sure we send emails; if the PDF fails, send
+without attachment"). Already-stranded emails are NOT auto-recovered. The generate scripts got
+crash-proofing only for the existence-gated reason above. Do not "helpfully" re-add an unbounded
+self-heal.
 
-Fix applied: added `!== false` guard around all three `file_get_contents` calls in the
-New branch (matching the guard the non-New branch already had). If any PDF fetch fails,
-the email sends without attachments rather than crashing.
+**Wrong Mac PDF URL (partially fixed 2026-06-18, fully fixed 2026-07-06):**
+`PrudentialMacSetupGuide_15.6_v4.pdf` (403 since ~May 5 2026) should be
+`PrudentialMacSetupGuide.pdf`. The 2026-06-18 fix corrected it only in `ordershipped`; the stale
+URL was still live in 3 of the sibling files and was corrected this session.
 
-**Wrong Mac PDF URL (fixed 2026-06-18):**
-The New branch was pointing to `PrudentialMacSetupGuide_15.6_v4.pdf` (403 since ~May 5 2026)
-instead of `PrudentialMacSetupGuide.pdf` (the working URL used by the non-New branch).
+**Pre-existing issues surfaced this session (NOT fixed — awareness only):**
+- SQL-injection risk: unescaped `$reqNumber` concatenated into a `LIKE '%...%'` query in the
+  Phase-2 body of both `transmit_ordershipped` and `transmit_closecomplete`.
+- `transmit_closecomplete` calls `App_Database::escapeString()` (~line 158) which does not exist
+  on `App_Database` (only `sqlEscape` does) — latent error in the `createAssetAndRetryPut` path.
+- Separate inbound issue: Dell SR re-POST hitting `ServiceRequests.number` UNIQUE constraint →
+  API 400 "Duplicate entry SR129013" (REQ4444748). Not part of the email fix.
 
 **Re-queueing stuck POs:**
 If `c_dtEmailSentOrderShipped` is NULL but `c_dtTransmittedOrderShippedUpdateToPrudential`
@@ -83,4 +132,5 @@ stamp (crash hits the next iteration). With multiple New-type POs, a crash on PO
 all subsequent POs in the same run.
 
 ## Change history
-- 2026-06-18 — Fixed 403 crash: corrected Mac PDF URL (`_15.6_v4` → `PrudentialMacSetupGuide.pdf`) and added `false` guard on all S3 fetches in New branch (rgirish)
+- 2026-07-06 — Found the real root cause: `file_get_contents` throws (E_ALL→ErrorException) so the `!== false` guards (incl. the 2026-06-18 one) were never effective. Added `safeFetchPdf()` (try/catch ErrorException → false) + independent per-PDF guards; email now always sends even if a PDF fails. Extended the same fix to `transmit_closecomplete` and both `generate_sales_and_purchase_orders_from_service_requests` scripts, and finished the stale Mac PDF URL fix in the 3 files the 2026-06-18 change missed. Documented the permanent two-phase stranding flaw and the date-bounded-recovery requirement; scope kept to fix-forward (no auto self-heal) by developer decision (rgirish)
+- 2026-06-18 — Fixed 403 crash: corrected Mac PDF URL (`_15.6_v4` → `PrudentialMacSetupGuide.pdf`) and added a `!== false` guard on all S3 fetches in New branch. NOTE (2026-07-06): this guard was ineffective — the exception is thrown inside `file_get_contents`, not returned (rgirish)

package/knowledge/clients/prudential/profile.md

@@ -12,7 +12,7 @@ project: _Underscore
 client: prudential
 type: profile
 status: active
-updated: 2026-06-30
+updated: 2026-07-06
 owners: ["jcardinal", "rgirish"]
 files: []
 related:
@@ -45,10 +45,15 @@ order-status transmissions.
   PRE/POST translation that keeps their feed working after the tracking-number bridge migration.
 - Service request region is resolved from `Customers.name` string (`'USA'`/`'India'`/`'Ireland'`).
   Renaming those customer records in the DB would silently break regional validation routing.
-- `transmit_ordershipped_updates_prudential.php` runs in two phases. Phase 1 stamps
-  `c_dtTransmittedOrderShippedUpdateToPrudential` immediately; Phase 2 sends the email and stamps
-  `c_dtEmailSentOrderShipped`. Any exception in Phase 2 leaves `c_dtEmailSentOrderShipped` NULL
-  and the PO is never re-processed automatically. See the exception report and the transmit-ordershipped-email feature doc.
+- `transmit_ordershipped_updates_prudential.php` **and** `transmit_closecomplete_updates_prudential.php`
+  run in two phases. Phase 1 stamps the transmitted column immediately
+  (`c_dtTransmittedOrderShipped…` / `c_dtTransmittedOrderClosed…`); Phase 2 sends the email and
+  stamps a separate column (`c_dtEmailSentOrderShipped` / `c_dtEmailSentOrderClosed`). The Phase-1
+  SELECT filters only on the transmitted column, so any Phase-2 crash strands that PO's email
+  **forever** — and any recovery query MUST be date-bounded (most historically-stranded rows are
+  old and NOT owed an email). Note: PDF fetches via `file_get_contents` in these 1.0 crons **throw**
+  (E_ALL→ErrorException), so `!== false` guards do not work — use the `safeFetchPdf()` pattern. See
+  the exception report and the transmit-ordershipped-email feature doc.
 
 ## Related docs
 - Dell ASN units interceptor.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.270",
+  "version": "1.0.272",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",