toga-ai 1.0.865 → 1.0.866

package/knowledge/INDEX.md

@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 1.0 framework
 
 - **library** (Library) _(framework core)_ — 26 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
-- **worker** (Worker) — 37 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
+- **worker** (Worker) — 38 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
 - **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)

package/knowledge/clients/compass-usa/INDEX.md

@@ -18,6 +18,7 @@
 | [Compass ODP EDI 850 Line-Item Resolution (VA part number to IN SKU fallback)](features/odp-edi-850-item-resolution.md) | 1.0 | How each PO1 line on an inbound Office Depot EDI 850 resolves to a real Compass catalog item before cron 3a writes anything; open for ODP 850 import failures, u |
 | [Compass Office Depot EDI 855 Acknowledgement + Over-Quantity PO Guard](features/odp-edi-855-acknowledgement-and-overquantity-guard.md) | 1.0 | How Compass acknowledges ODP EDI 850 POs with an X12 855 and rejects a duplicate/over-quantity PO before import; open for 855 accept/reject behaviour, over-quan |
 | [Compass ODP EDI File Retention & Import Retry (cron 3a)](features/odp-edi-file-retention-and-retry.md) | 1.0 | When cron 3a keeps vs deletes an ODP EDI file in S3, and why an api2 outage used to silently lose orders; open for lost ODP imports, retry behaviour, or the sti |
+| [Compass: ODP / Strategic Systems ASN CSV email import (and the scientific-notation tracking-number loss)](features/odp-strategic-systems-asn-csv-import.md) | 1.0 | The 1.0 worker cron that reads Office Depot / Strategic Systems ASN CSVs from email, why it routes by column count, and why a tracking number can arrive as `8.7 |
 | [Compass: MA refresh-order monitor (Monitor/Compass/MaRefreshOrderExceptions)](features/oneuptime-ma-refresh-order-monitor.md) | 2.0 | worker2 OneUptime push monitor counting Compass MA employee-refresh orders whose ODP SO has no NetSuite order yet; open when working on Compass MA monitoring or |
 | [Compass USA — Fulfilled vs Partially Fulfilled (order total kept; per-line rule built then rejected)](features/order-fulfillment-status-per-line.md) | 2.0 | Why Compass USA shows Fulfilled on partly-shipped orders and why the per-line fix was built, measured, then reverted; open before touching Compass USA `_status` |
 | [Compass PEOPLE-File User Lifecycle (duplicate accounts, reactivation grace window, raw-SQL deactivation)](features/people-file-user-lifecycle.md) | 2.0 | The nightly Compass PEOPLE-file cron's Users-row lifecycle (insert/reactivate/deactivate), why it creates duplicate accounts, and its raw-SQL/index traps; open |

package/knowledge/clients/compass-usa/features/odp-strategic-systems-asn-csv-import.md

@@ -0,0 +1,84 @@
+---
+title: "Compass: ODP / Strategic Systems ASN CSV email import (and the scientific-notation tracking-number loss)"
+framework: "1.0"
+repo: worker
+project: Worker
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-09-22
+owners: ["rgirish"]
+files:
+  - worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php
+  - worker/crons/toga2/compass/DOA_Scripts/DOA_compass_statuses.php
+related:
+  - ./tracking-number-origin-and-asn-link.md
+  - ./asn-to-item-fulfillment.md
+  - ./multi-tier-tracking-and-serial-mirroring.md
+  - ../../compass-canada/features/grand-and-toy-asn-import.md
+  - ../../quad/features/po-asn-email-import.md
+  - ../../../1.0/apps/worker/architecture.md
+---
+The 1.0 worker cron that reads Office Depot / Strategic Systems ASN CSVs from email, why it routes by column count, and why a tracking number can arrive as `8.77536E+11` with its digits already gone. Open before touching tracking-number parsing in this file or its DOA copy.
+
+## Summary
+
+Office Depot (ODP) and Strategic Systems email Compass USA ASN files as **CSV attachments**. The 1.0 worker cron `crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php` reads them and writes `Client_Compass.TrackingNumbers` rows. `DOA_Scripts/DOA_compass_statuses.php` is a **near-identical copy** of the same parsing logic — every change here must be made in both files.
+
+Two facts drive most of the trouble in this file:
+
+1. **The file is routed by COLUMN COUNT, not by filename.** 25 columns → ODP parser, 15 → Strategic Systems, anything else throws.
+2. **It is a CSV, so a tracking number that arrives as `8.77536E+11` is unrecoverable.** The characters in the file *are* `8.77536E+11`. There is no underlying cell value to read back. No PHP change can restore the digits.
+
+## How it works
+
+### Key files / entry points
+- `worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php` — the scheduled importer. Reads attachments with `fgetcsv` / `str_getcsv`.
+- `worker/crons/toga2/compass/DOA_Scripts/DOA_compass_statuses.php` — copy of the same logic, same parsing, same guards. No shared helper exists; the two files drift.
+- `PhpSpreadsheet` is imported at the top of the importer but is used only to **write the validation-error report workbook** (`setCellValueExplicit` / `createWriter('Xlsx')`). It does **not** read the incoming feed. Do not read the import as a spreadsheet path.
+
+### File routing is by column count
+`3b_import_..._advance_shipping_notices.php` line ~266 counts the header's columns and dispatches:
+
+| Header column count | Constant | Parser |
+|---|---|---|
+| 25 | `ODP_HEADER_COLUMN_COUNT` | `processOfficeDepotFile()` |
+| 15 | `SS_HEADER_COLUMN_COUNT` | `processStrategicSystems()` |
+| anything else | — | throws `"Unsupported file format: $filename"` |
+
+Filename and extension are ignored. **If a vendor adds or drops one column, the whole file silently reroutes to the wrong parser** (25 ↔ 15 is a wide gap today, but the check is exact equality, so any drift toward the other constant is a real risk). This mirrors the Compass Canada G&T importer's 24-column gate — see [Grand & Toy ASN Import](../../compass-canada/features/grand-and-toy-asn-import.md).
+
+### Tracking-number parsing (4 guard sites)
+Both files parse tracking numbers in two branches each — ODP and Strategic Systems — so there are **four** copies of the same block. The correct order, now used in all four:
+
+1. Replace the multi-value separators `<BR>`, `<br>`, `;` with commas.
+2. `explode(',')`, trim, strip noise characters (`"`, `'`, `VYLI`, `#`), `array_filter` the empties.
+3. **Then** test each element for `e+` and, on a match, append to `$validationErrors`.
+
+`$validationErrors` already feeds an unprocessed-rows report emailed to the team, so a rejected row is visible, not silent.
+
+## Gotchas
+
+- **⚠ A scientific-notation tracking number is a DATA bug, not a display bug — and the digits are gone before our code runs.** `Client_Compass.TrackingNumbers.number` literally stores the 11-character string `8.77536E+11` (e.g. id 80112, `shippingCarrierId` NULL). 44 such rows exist, Sep 2025 → 2026-09-22. The cause is upstream: a 12-digit number like `877536271001` becomes `8.77536E+11` when a spreadsheet saves the column as a number rather than text. Only 6 significant digits survive. Reported on order **SA137640**.
+
+- **⚠ NEVER "recover" a scientific-notation number with `number_format()` — it pads zeros and fabricates a believable-but-wrong tracking number.** A guard added 2026-10-17 (commit `be8ff1d14`) ran `number_format($n, 0, '', '')` on any value containing `e+`. Verified by running it: `number_format("8.77536E+11")` returns **`877536000000`**, while the real value was **`877536271001`**. The result is a 12-digit number that looks perfectly valid, so nobody can spot it as fake — strictly worse than failing the row. Removed 2026-09-22; all four sites now reject instead. **A silent wrong tracking number is worse than a loud rejected row.**
+
+- **⚠ Guard-before-split threw a PHP fatal on multi-value rows.** The Strategic Systems branch tested the **whole raw string** for `e+` *before* splitting on `<BR>` / `;`. Input `"8.77536E+11<BR>8.77517E+11"` reached `number_format()` as a non-numeric string: `TypeError: number_format(): Argument #1 ($num) must be of type float, string given`. The ODP branch in the same file split first and was correct — the two branches disagreed. Always **split, clean, then guard each element**.
+
+- **CSV vs XLSX is the deciding fact for any fix here.** The feed is a CSV, which has no cell types, so the file itself contains only the rounded characters — nothing to recover. If the sender switched to `.xlsx`, the cell could be read as a string and the full number would survive. **Decided 2026-09-22:** ask Office Depot for `.xlsx` as the primary fix; fallback ask is CSV with the tracking column formatted as text (weaker — breaks again the moment anyone re-opens the file). **BLOCKED as of 2026-09-22 waiting on ODP to confirm the format**; no further integration work until they answer. The four reject-guards are worth keeping either way.
+
+- **The 44 corrupted rows cannot be repaired in SQL.** The digits do not exist anywhere in our data. They must be re-sourced from ODP or the carrier.
+
+- **One corrupted number on many orders is mirroring, not a rounding collision.** `8.77517E+11` appears on 69 distinct orders, `8.77536E+11` on 21, `8.73895E+11` on 18. Each was written once at the Agilant tier and then mirrored up the 3-tier drop-ship chain — see [multi-tier tracking & serial mirroring](./multi-tier-tracking-and-serial-mirroring.md). Do not read a high order count as many separate bad reads.
+
+- **⚠ OPEN — a third, unidentified writer creates TrackingNumbers rows.** Five corrupted rows landed **2026-09-22 06:22–06:32 with NULL `shippingCarrierId` AND NULL `trackingUrl`**. Both CSV importers always set a carrier (ODP from the `ODP_SHIPPING_CARRIER` column; Strategic Systems from the tracking number or the file), so neither wrote them. The fulfillments they attached to were created 2026-09-19, three days earlier. **Unresolved: find what job runs at 06:22.** By contrast the July 2025 corrupted rows carry Convey/Veyer URLs (`track.getconvey.com/veyer/us/odn/...`, Office Depot's tracking service), confirming the CSV path genuinely produced some of them. Note this is a *third* writer on top of the two already documented — see [where a TrackingNumbers row comes from](./tracking-number-origin-and-asn-link.md).
+
+- **⚠ OPEN — wider tracking-number data quality problem (168 bad rows).** Of **73,909** rows in `Client_Compass.TrackingNumbers`, **168** fail `^[A-Za-z0-9]{8,40}$`: 44 scientific notation, 79 with leading spaces, 45 shorter than 8 characters, plus keyboard typos such as `!Z8696XA0140259245` (`!` instead of `1` — a shift-key slip on a UPS number). Max observed length is 114. A CHECK constraint on `TrackingNumbers.number` is the only fix that covers **every** write path including the unknown 06:22 one, but the existing rows must be cleaned first and the upper bound validated against carriers that use longer formats.
+
+- **Only two files in the whole `crons/toga2/compass/` tree create TrackingNumbers rows**, and they are copy-paste siblings with no shared helper. When you fix one guard, grep the other file for the same block.
+
+- The Compass **Canada** G&T importer already rejects scientific-notation tracking as a row-level guard; its read-only inbox inspector reports it as a dry-run category. That is the pattern Compass USA now matches.
+
+## Change history
+- 2026-09-22 — Removed the lossy `number_format()` "recovery" from all 4 guard sites in both files; now split-then-guard, appending to `$validationErrors` instead. Fixes the fabricated-tracking-number bug and the multi-value PHP fatal. Written and linted (`php -l` passes), **not deployed**. (rgirish)
+- 2026-09-22 — Documented CSV-vs-XLSX as unrecoverable-at-source, column-count file routing, the unknown 06:22 writer, and the 168-row data-quality survey. (rgirish)

package/knowledge/clients/compass-usa/features/tracking-number-origin-and-asn-link.md

@@ -6,18 +6,20 @@ project: Worker
 client: compass-usa
 type: client-feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-22
+owners: ["bala", "rgirish"]
 files:
   - library/app/api/toga2.php
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
   - worker/crons/toga2/netsuite/sync_togasupply_compass_usa.php
   - worker/crons/toga2/compass/update_salesorder_status_from_odp.php
+  - worker/crons/toga2/compass/workflow/3b_import_strategic_systems_advance_shipping_notices.php
   - worker2/Worker/Monitor/Compass.php
 related:
   - ../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md
   - ../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
   - ./asn-to-item-fulfillment.md
+  - ./odp-strategic-systems-asn-csv-import.md
   - ./contact-email-resolution.md
   - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
 ---
@@ -26,6 +28,8 @@ Why a Compass TrackingNumbers row is born from the invoice sync (not the ASN) an
 ## Summary
 For Compass USA a `TrackingNumbers` row is **born from the NetSuite invoice sync**, not from the ASN. The ASN turns up days or weeks later and is the **only** thing that ties that tracking number to a sales order, and therefore to a customer's email address.
 
+The invoice sync is the **main** origin, but it is **not the only** one — see *Who else writes a TrackingNumbers row* below. At least three writers exist and one is still unidentified.
+
 Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is the invoice date.** It is the wrong clock to time any shipping notification, backlog age, or SLA from. Anything that asks *"when did we get this tracking number?"* and means *"when could we first have told the customer?"* is wrong for Compass by up to two weeks.
 
 ## How it works
@@ -52,6 +56,17 @@ Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is th
    ```
 4. Only once that chain exists can the in-transit email be built or sent.
 
+### Who else writes a `TrackingNumbers` row (added 2026-09-22)
+The invoice sync is the main writer, not the only one. Known and unknown writers:
+
+| Writer | Sets carrier? | Notes |
+|---|---|---|
+| NetSuite invoice sync (`syncInvoiceFromNetsuite`) | yes | the normal origin described above |
+| ODP / Strategic Systems ASN **CSV email** cron + its `DOA_Scripts` copy | **always** | [ODP / Strategic Systems ASN CSV import](./odp-strategic-systems-asn-csv-import.md) |
+| **UNKNOWN job running ~06:22** | **no** — NULL `shippingCarrierId` *and* NULL `trackingUrl` | ⚠ unidentified, see gotcha |
+
+So **"which writer made this row?" is answerable from the carrier columns**: both CSV importers always set a carrier, so a row with NULL `shippingCarrierId` and NULL `trackingUrl` did **not** come from them.
+
 ### How big the gap is (measured on prod)
 - **2026-09-03, 11:15:56–11:40:20** — `Logs_Compass.Api` shows **98** `POST /v2/invoice-tracking-numbers` from the worker instances. **100** `TrackingNumbers` rows created in that window, while only **4** ASNs were created in the whole of 09-03.
 - Since **2026-08-01**, **53** tracking rows had their ASN land **more than 7 days** after the tracking row, with a **maximum gap of 13 days**.
@@ -66,6 +81,8 @@ Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is th
 ## Gotchas
 - **⚠ Never window, age, or alarm a Compass shipping feature on `TrackingNumbers.dtCreated`.** Use `AdvanceShippingNotices.dtCreated` for a window, and `GREATEST(TrackingNumbers.dtCreated, AdvanceShippingNotices.dtCreated)` when you need "how long has this been sendable". Getting this wrong produced both a real missed-email gap and two false HIGH alarms — see [Compass in-transit emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md).
 - **⚠ Correction to an earlier belief.** It was previously recorded that invoice-created tracking numbers "never touch the ASN chain" and are therefore out of scope for in-transit email. That is wrong. They are the *normal* origin for Compass tracking numbers; the ASN chain simply attaches to them later. The narrower true statement: a tracking number that **never** gets an ASN can never be emailed, and a naive `TrackingNumbers WHERE c_dtInTransitEmailSent IS NULL` count (with no ASN join) reads those as a backlog that never drains.
+- **⚠ OPEN — an unidentified job writes TrackingNumbers rows around 06:22.** Five rows landed 2026-09-22 06:22–06:32 with NULL `shippingCarrierId` and NULL `trackingUrl`, attached to fulfillments created three days earlier (2026-09-19). Neither CSV importer wrote them (both always set a carrier) and the shape does not match the invoice sync either. Do not assume a Compass tracking row came from one of the two known paths until this is identified.
+- **⚠ Tracking-number values are not validated anywhere — 168 of 73,909 rows are malformed.** Scientific notation, leading spaces, sub-8-character values, and typos all persist. Because an unknown writer exists, a **CHECK constraint on `TrackingNumbers.number` is the only fix that covers every path**. Details and counts: [ODP / Strategic Systems ASN CSV import](./odp-strategic-systems-asn-csv-import.md).
 - **A tracking number with no ASN has no customer.** There is no fallback route from invoice to contact, because `Invoices.salesOrderId` is NULL. Waiting for the ASN is the design, not a bug.
 - Credentials for the sync ride on `App_Api_Toga2` class constants (e.g. `App_Api_Toga2::CLIENT_UUID_COMPASS`) — reference the constant, never the value.
 
@@ -73,5 +90,9 @@ Consequence, and the reason this doc exists: **`TrackingNumbers.dtCreated` is th
 - [Compass partial in-transit/delivered emails](../../../1.0/apps/worker/features/compass-partial-in-transit-delivered-emails.md)
 - [NetSuite TOGaSupply per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md)
 - [ASN → ItemFulfillment](./asn-to-item-fulfillment.md)
+- [ODP / Strategic Systems ASN CSV import](./odp-strategic-systems-asn-csv-import.md)
 - [Contact email resolution](./contact-email-resolution.md)
 - [OneUptime worker2 monitoring](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md)
+
+## Change history
+- 2026-09-22 — Recorded that the invoice sync is not the only writer: added the ASN CSV cron and an unidentified ~06:22 writer, plus the tracking-number data-quality gap. (rgirish)

package/knowledge/clients/compass-usa/profile.md

@@ -37,6 +37,7 @@ related:
   - features/oneuptime-ma-refresh-order-monitor.md
   - workflows/catalog-fold-reversal.md
   - features/asn-to-item-fulfillment.md
+  - features/odp-strategic-systems-asn-csv-import.md
   - features/multi-tier-tracking-and-serial-mirroring.md
   - features/order-fulfillment-status-per-line.md
   - features/cost-centers.md

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.865",
+  "version": "1.0.866",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",