toga-ai 1.0.157 → 1.0.158

package/knowledge/clients/compass-canada/features/grand-and-toy-asn-import.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: compass-canada
 type: client-feature
 status: active
-updated: 2026-06-19
+updated: 2026-06-22
 owners: ["bala"]
 files:
   - worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php
@@ -28,8 +28,9 @@ flow, adapted for Canadian carriers and bilingual email.
 ## Key files / entry points
 - `worker/crons/toga2/compasscanada/workflow/4_import_grand_and_toy_advance_shipping_notices.php`
   — the scheduled cron (1.0 worker tier). Reads mailbox `compasscanada.status@togatech.com`
-  (OAuth2 creds in config `[compasscanada]`), parses the 24-column CSV, posts to api2
-  `/advance-shipping-notices`. Scheduled in `worker/schedules/cron.worker.sync.json` (every 4h).
+  (OAuth2 creds — currently hardcoded with a TODO; must move to `App_Registry::get('config')['compasscanada']['gt_asn_oauth2']`
+  before production), parses the 24-column CSV, posts to api2 `/advance-shipping-notices`.
+  Scheduled in `worker/schedules/cron.worker.sync.json` (every 4h).
 - `_underscore/Model/Compass/AdvanceShippingNotice.php` — `postPost` interceptor that builds the
   ItemFulfillment chain on each ASN POST. Compass Canada inherits it via the empty
   `_underscore/Model/Compass/Canada/AdvanceShippingNotice.php`.
@@ -38,21 +39,27 @@ flow, adapted for Canadian carriers and bilingual email.
   user emails/notifications, and stamps `c_dtInTransitEmailSent = NOW()`.
 
 ## How it works
-1. Cron gets an O365 OAuth2 token (creds from `App_Registry::get('config')['compasscanada']`),
+1. Cron gets an O365 OAuth2 token (creds currently hardcoded; see Gotchas for migration path),
    opens the INBOX.
 2. Per attachment: decode CSV, gate on exactly 24 columns (otherwise email the team the file +
-   skip).
-3. Per row: skip electronic-delivery carriers (`Digital` / `E-delivered` / `E-verified`);
-   validate PO / part / tracking; look up SO + contact user + CC addresses; resolve carrier →
-   Ground shipping method; upsert `TrackingNumbers`; resolve the item by
-   `VendorItems.vendorPartNumber`; find-or-create a `Units` row per serial.
-4. POST `/advance-shipping-notices` with the tracking number nested at header + item + unit
-   level (so the interceptor sees it during the POST).
-5. The Compass ASN `postPost` interceptor (when enabled) creates the `ItemFulfillment` (one per
+   skip). Uses `preg_split('/\r?\n/', ...)` on the first line so Windows CRLF attachments parse
+   correctly.
+3. **Before the row loop** — build three lookup maps once per file: `$purchaseOrderUuidByNumber`
+   (`SELECT number, uuid FROM PurchaseOrders`), `$shippingMethodByCarrierId`, `$carrierIdByName`.
+4. Per row: validate PO / part / tracking (early `continue` with error recording on missing
+   critical fields); look up SO + contact user + CC addresses; resolve carrier →
+   Ground shipping method (falls back to `DEFAULT_GROUND_SHIPPING_METHOD_ID = 3`); upsert
+   `TrackingNumbers`; resolve item by `Items.partNumber` joined through `VendorItems`; find-or-create
+   a `Units` row per serial.
+5. POST `/advance-shipping-notices` with ASNIU (unit) payload so the interceptor can act.
+6. After the POST: query each generated ASNIU UUID back from DB, insert
+   `AdvanceShippingNoticeItemUnits_TrackingNumbers`. This runs for EVERY newly created ASNIU,
+   regardless of whether the TrackingNumber record was new or pre-existing.
+7. The Compass ASN `postPost` interceptor (when enabled) creates the `ItemFulfillment` (one per
    SO number, reused if it exists), `ItemFulfillmentItems`, `ItemFulfillmentItemUnits`, and the
    IF/IFI/IFIU `_TrackingNumbers` bridges.
-6. Only when a tracking number is newly inserted: write Notifications (1.0 `db_store` + 2.0),
-   resolve the user's language, send the EN/FR in-transit email via POST
+8. Only when a tracking number is newly inserted: write Notifications (1.0 `db_store` + 2.0),
+   resolve the user's language, send the EN/FR in-transit email via
    `/email-templates/sendEmail`, then stamp `c_dtInTransitEmailSent = NOW()`.
 
 ## Data model
@@ -73,30 +80,47 @@ a bilingual EN/FR in-transit email (USA is English only) and uses Canadian carri
 
 ## Gotchas / known issues
 - **CSV "Part Number" is the VENDOR part number (`VendorItems.vendorPartNumber`), NOT
-  `Items.partNumber`.** Resolve the item through `VendorItems`. (e.g. vendor `IMFP3260` →
-  `Items.partNumber` `C2CY5UC#ABA`.)
+  `Items.partNumber` directly.** Resolve the item through the `VendorItems` join. (e.g. vendor
+  `IMFP3260` → `Items.partNumber` `C2CY5UC#ABA`.)
 - **G&T's ASN file inconsistently drops the `IM` prefix** (`FP3260` vs `IMFP3260`) on some lines
   → item-not-found → those lines are skipped and reported in the import-report email. This is a
   G&T-side export issue; toga transmits the correct `IMFP…` to G&T as the cXML `SupplierPartID`
   (see `2_transmit_mits_purchase_orders_to_vendors.php`), so it is NOT a MITS bug.
+- **Build lookup maps OUTSIDE the while loop.** `$purchaseOrderUuidByNumber`, `$shippingMethodByCarrierId`,
+  and `$carrierIdByName` must be built once before the row loop — they were originally inside,
+  causing a full-table scan of `PurchaseOrders`, `ShippingMethods`, and `ShippingCarriers` on
+  every CSV row.
+- **ASNIU→TrackingNumber link is NOT gated on `$isNewTrackingNumber`.** The ASNIU is always
+  freshly created on each import row; the `AdvanceShippingNoticeItemUnits_TrackingNumbers` insert
+  must run regardless of whether the TrackingNumber record itself was new or pre-existing.
+  Gating it on "new TN only" means re-sent / re-processed CSV rows create orphaned ASNIUs with
+  no tracking link.
+- **`$shippingMethodByCarrierId[$selectedCarrierId]` needs a null-coalesce fallback.**  On PHP 8,
+  accessing a missing key without `?? DEFAULT_GROUND_SHIPPING_METHOD_ID` produces a deprecation
+  warning that becomes an error. Always write `$shippingMethodByCarrierId[$selectedCarrierId] ?? DEFAULT_GROUND_SHIPPING_METHOD_ID`.
+- **CSV CRLF attachments:** use `preg_split('/\r?\n/', $decodedAttachment)[0]` (not
+  `explode("\n", ...)`) for the header-line column-count check. `explode("\n")` leaves a trailing
+  `\r` on the first line when the file uses Windows line endings, making `str_getcsv` miscount
+  columns and silently skip the file.
 - **The ASN→IF interceptor must be ENABLED per client.** `Core.ApiPayloadInterceptors` for
-  `recordId 55`, `prePostProcessing=POST`, `httpMethod=POST` must be `isActive=1` (and the client
-  `AclRecordScripts` must grant the caller's roles). It was OFF for Compass Canada; enabling it is
-  required or ASNs post but the IF/IFI/IFIU chain never builds.
-- **Tracking must be nested in the ASN POST payload** (header/item/unit), not linked via SQL
-  after the POST — the interceptor runs during the POST, so post-POST links are invisible to it.
-- **`/email-templates/sendEmail` must be POST** (not GET) so a large `orderItems` HTML payload
-  does not hit the URL-length limit (HTTP 414). POST requires the `sendEmail` record script
-  registered for the POST method (`Core.RecordScripts`), else the engine returns `EV-8`.
+  `recordId 55`, `prePostProcessing=POST`, `httpMethod=POST` must be `isActive=1`. It was OFF for
+  Compass Canada; enabling it is required or ASNs post but the IF/IFI/IFIU chain never builds.
+- **OAuth2 credentials are currently hardcoded** in the cron with a `// TODO` comment. Before
+  going to production, move them to `config.<env>.ini` under `[compasscanada]` and read via
+  `App_Registry::get('config')['compasscanada'][…]`. The `config.<env>.ini` must be deployed
+  with the code or the OAuth2 token request fails silently.
 - Each CSV line creates one ASN; the interceptor dedups to **one ItemFulfillment per SO**.
 - A new carrier needs its **own** Ground `ShippingMethod`, or the TrackingNumber ends up with the
   right carrier but another carrier's Ground method id.
 - `App_Database::fetchOne()` (1.0) throws on a 0-row result — guard item lookups with `numRows()`
   before `fetchOne()`.
-- The cron reads OAuth2 creds from config `[compasscanada]`; the updated `config.<env>.ini` must
-  be deployed with the code or the token request fails.
 
 ## Change history
+- 2026-06-22 — Full variable rename (no abbreviations); moved PO/carrier/shipping-method lookup maps
+  outside the CSV row loop (was causing N full-table queries per file); fixed ASNIU→TN link to run
+  unconditionally for new ASNIUs (not only when TN was new); added PHP 8 null-coalesce on carrier→
+  method lookup; fixed CSV CRLF detection with `preg_split`; added `DEFAULT_GROUND_SHIPPING_METHOD_ID`
+  named constant; function renamed `buildOrderItemsHtmlGrandAndToy` (camelCase). (bala)
 - 2026-06-19 — Built the G&T ASN import cron + one-time backfill script; matched items by
   `VendorItems.vendorPartNumber`; added Canadian carriers + per-carrier Ground methods; EN/FR
   in-transit email via POST; enabled the Compass ASN→IF interceptor for Compass Canada. (bala)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.157",
+  "version": "1.0.158",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",