toga-ai 1.0.857 → 1.0.858
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: TRUE-82180-staples-asn-late
|
|
4
|
+
title: Staples ASNs sent days-to-weeks late — root cause investigation
|
|
5
|
+
author: mhammontree
|
|
6
|
+
repos: [worker, library, dbchanges]
|
|
7
|
+
framework: "1.0"
|
|
8
|
+
client: staples
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-09-21
|
|
11
|
+
updated: 2026-09-21
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: TRUE-82180-staples-asn-late
|
|
15
|
+
**Date:** 2026-09-21
|
|
16
|
+
**Project/Repo:** worker (1.0)
|
|
17
|
+
**Task:** Investigate why 856 ASNs are sent to Staples days-to-weeks after we ship and after we invoice — example PO 478KVU invoiced 9/8, ASN sent 9/18.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
|
|
23
|
+
- **Root cause identified and proven.** `worker/crons/sync/staples/staples_asn_netsuite.php` L48 sets
|
|
24
|
+
`$modifiedAfterEpochTime = $currentEpochTime - 3600` and the cron runs `50 * * * *`. The selection window
|
|
25
|
+
equals the run interval, so each shipment gets exactly ONE chance. No state, no backlog, no retry. A missed
|
|
26
|
+
shipment is only sent later when something edits the item fulfillment in NetSuite.
|
|
27
|
+
- **Confirmed the driver is a NetSuite `lastModifiedDate onOrAfter` search** — `App_NetSuite::listAllItemFulfillmentsForCustomer()`
|
|
28
|
+
in `library/app/netsuite.php` L1633-1679. It already accepts an after AND before timestamp, so no library
|
|
29
|
+
change is needed to change the window.
|
|
30
|
+
- **Proved it on the example order.** `Logs`.`FileLog` (env `legacy`), job `STAPLES_ASN_OUT_ORDER`:
|
|
31
|
+
- PO 478KVU: 850 imported 2026-09-01 12:45; ASN cXML says `shipmentDate="2026-09-02T21:00:00.000-07:00"`
|
|
32
|
+
(9/3 Central) and `noticeDate="2026-09-17T11:53:21.000-07:00"`.
|
|
33
|
+
- `noticeDate` is `$order->lastModifiedDate`. The fulfillment was edited 9/17 13:53 Central; the 14:52 run
|
|
34
|
+
sent it. 15 days after ship.
|
|
35
|
+
- **Sized the problem.** First-ASN vs `shipmentDate`, "late" = >1 day. By ship date:
|
|
36
|
+
09-01 49 on-time/17 late; 09-02 5/16; 09-03 4/13; 09-07 11/30; 09-08 15/11; 09-09 63/11; 09-10 9/9;
|
|
37
|
+
09-13 0/24; 09-14 0/27; 09-15 8/5. ~half late every day. 40+ orders shipped 8/18–9/07 all first-ASN'd in
|
|
38
|
+
one burst 9/15–9/18.
|
|
39
|
+
- **Found a second, separate bug: duplicate sends.** Dedupe at L453 checks SFTP `/archive` only, never `/out`.
|
|
40
|
+
PO 478KVU written 10 times in 10 hours. PO 755KVB written as `856_755KVB_1_*` repeatedly on 9/11 AND again
|
|
41
|
+
as `856_755KVB_2_1789602977` on a later day — same single item, same tracking `488192721813`. The
|
|
42
|
+
`<Package identifier="...">` is `mt_rand(1000,10000)` so it differs every send.
|
|
43
|
+
- **Proved `partialPackIdentifier` is unusable as a dedupe key.** It restarts at 1 every run and depends on
|
|
44
|
+
iteration order, so the `856_<PO>_<n>` prefix check against `/archive` cannot recognise an already-sent
|
|
45
|
+
shipment.
|
|
46
|
+
- **Ruled out the overlap guard.** `App_Framework::cronInitialization()` (`library/app/framework.php` L689-711)
|
|
47
|
+
calls `exitIfProcessRunning()` BEFORE inserting the `Common`.`CronJobExecutions` check-in row. Check-in rows
|
|
48
|
+
exist at `:50` for all 24 hours on 9/2, 9/3, 9/4. Every run started.
|
|
49
|
+
- **Ruled out reuse of existing tables.** `Common`.`AdvanceShipNotices` (163,145 rows) is an old INBOUND vendor
|
|
50
|
+
ASN feed (ACER/LENOVO/APPLE/LEXMARK), dead since 2022-05-17, wrong grain and wrong direction.
|
|
51
|
+
`TOGA_Staples` DB is the Staples retail-services app (Stores, Incidents, ServiceRequests) — unrelated.
|
|
52
|
+
- **CTO agent review completed** — verdict DISAGREE-WITH-ALTERNATIVE. Agreed on the state table in `Common`,
|
|
53
|
+
rejected widening the lookback, insisted on a different order of work.
|
|
54
|
+
- **Leadership write-up delivered** as a Claude doc:
|
|
55
|
+
https://claude.ai/code/artifact/95957a53-cd23-4ba6-a850-83b143b037b0
|
|
56
|
+
(private — Mark must share it from the Share menu before anyone else can open it).
|
|
57
|
+
|
|
58
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
59
|
+
|
|
60
|
+
- **CTO claim that a stale `$itemCxmlConfirmation` is emitting package-less ASNs — FALSE, disproven.**
|
|
61
|
+
Query over all `STAPLES_ASN_OUT_ORDER` rows since 2026-08-20: `totalAsns 8831, noPackageTag 0, noItems 0`.
|
|
62
|
+
Every ASN sent contains both `<Package ` and `<ShipNoticeItem `. Do not re-open this theory. The leaked
|
|
63
|
+
variable is still a real code defect, it is just not firing in production.
|
|
64
|
+
- **CTO claim of cross-fulfillment item bleed — real in code, NOT demonstrated in data.** `$itemCXMLArray` is
|
|
65
|
+
initialised at L136, outside the `foreach ($partialItemFulFil as $order)` loop, so a second fulfillment would
|
|
66
|
+
carry the first's items. But only 2 POs since 9/1 have more than one pack (755KVB, 856KWB) and 755KVB's
|
|
67
|
+
`_1` and `_2` files are byte-identical apart from the random package identifier — one item each. Not proven.
|
|
68
|
+
- **Widening the lookback window to 7 days — rejected on numbers.** ~30 shipments/day x ~20 serials, one SOAP
|
|
69
|
+
`get` per serial (L326) plus `getSalesOrder` + `get itemFulfillment` + `listAllItemFulfillmentsForSalesOrder`
|
|
70
|
+
per fulfillment ≈ 4,600 SOAP calls per hourly run, 40–75 min runtime. It would collide with the next `:50`
|
|
71
|
+
run and `exitIfProcessRunning` would then start eating hours. Also re-writes the full cXML BLOB into
|
|
72
|
+
`FileLog` every pass (the FileLog write at L446 happens BEFORE the dedupe check at L453).
|
|
73
|
+
- **Using `CronJobExecutions.dtCheckOut` to detect runs that died mid-loop — does not work.** The script never
|
|
74
|
+
calls `App_Framework::cronFinished()` or `App_Database::closeConnections()`, so EVERY row has
|
|
75
|
+
`dtCheckOut = NULL`. A completed run and a crashed run are indistinguishable. Do not read NULL as "crashed".
|
|
76
|
+
- **Looking for "ASN Not Generated" emails to find failures — there are none, ever.** `Vision_Log`.`Email`
|
|
77
|
+
searched back to 2026-08-25 for `emailObject LIKE '%ASN Not Generated%'`: 0 rows. The else-branch at L469
|
|
78
|
+
has never fired. Late orders are dropped BEFORE reaching that check.
|
|
79
|
+
- **`toga_query` blocks the `REPLACE` keyword.** A `LENGTH(x)-LENGTH(REPLACE(x,...))` counting query is
|
|
80
|
+
rejected with "Forbidden keyword 'REPLACE' found in query". Use `LOCATE`/`SUBSTRING` or fetch and eyeball.
|
|
81
|
+
- **`Logs` schema is on the `legacy` environment, not `production`.** `toga_query` with
|
|
82
|
+
`environment='production'` gives "Table 'Logs.FileLog' doesn't exist". Also `FileLog` has `dtStamp`, NOT
|
|
83
|
+
`dtCreated`.
|
|
84
|
+
|
|
85
|
+
## Not tried yet (candidates for next session)
|
|
86
|
+
|
|
87
|
+
- Confirm the actual per-order silent skip. Strongest candidate is the `if ($matchedFile)` gate at L160 —
|
|
88
|
+
it requires OUR OWN 855 ack to already be in SFTP `/archive`, i.e. Staples must have collected it before we
|
|
89
|
+
will send the ASN. Customer-controlled, no log, no email. Needs SFTP pickup timing to confirm.
|
|
90
|
+
- Test the mid-run-death theory: per hour, compare the count of fulfillments the NetSuite search returned
|
|
91
|
+
against the `FileLog` row count for that run. A truncated tail proves a warning killed the cron.
|
|
92
|
+
- Size the item-bleed risk: find sales orders with multiple fulfillments AND multiple line items, and check
|
|
93
|
+
whether any delivered ASN carried another fulfillment's items.
|
|
94
|
+
- Nothing has been written. No code changed, no SQL written, no branch commits.
|
|
95
|
+
|
|
96
|
+
## Current file state
|
|
97
|
+
|
|
98
|
+
| File | Status | Notes |
|
|
99
|
+
|------|--------|-------|
|
|
100
|
+
| `worker/crons/sync/staples/staples_asn_netsuite.php` | **unchanged** | Read only. 496 lines. Branch `TRUE-82180` checked out, working tree clean. |
|
|
101
|
+
| `dbchanges/` | **unchanged** | Branch `TRUE-82180` checked out. No file created. Target would be `dbchanges/Common/MH/<date>.sql`. |
|
|
102
|
+
| `library/` | **unchanged** | On branch `TRUE-81955` (another ticket). Not needed for this work. |
|
|
103
|
+
| Claude doc (leadership findings) | created | https://claude.ai/code/artifact/95957a53-cd23-4ba6-a850-83b143b037b0 — private, not yet shared. |
|
|
104
|
+
|
|
105
|
+
## Decisions made
|
|
106
|
+
|
|
107
|
+
- **Ticket switched to pseudocode-only, for Jeff to review.** Mark changed the scope late in the session: the
|
|
108
|
+
ticket asks for an investigation, not a fix. Pseudocode + the findings breakdown go in the ClickUp task's
|
|
109
|
+
pseudocode field. **No code is to be written until Jeff reviews.**
|
|
110
|
+
- **This may be solved the 2.0 way instead of fixing 1.0.** Open possibility raised by Mark. Do not assume the
|
|
111
|
+
1.0 design below is the agreed path — it is the 1.0 option, not the decision.
|
|
112
|
+
- **If fixed in 1.0: a state table, not a wider window.** Rationale: the expensive work (per-serial SOAP calls)
|
|
113
|
+
must be driven by "not yet sent", not by "recently modified". Rejected: widening the lookback (see failures
|
|
114
|
+
above); rejected: keeping SFTP folder listings as the record of what was sent.
|
|
115
|
+
- **Home would be `Common` (`db_common`), table `StaplesAdvanceShipNotices`.** Rationale: `Common` already
|
|
116
|
+
holds cross-app NetSuite state (`App_Model_Common_NsSalesOrderItem`, `NsSerialNumber`, `NsTrackingNumber`)
|
|
117
|
+
and `CronJobExecutions`. Rejected `Logs` — it is append-only under retention pressure (API logs already cut
|
|
118
|
+
to 30 days), and mutable state must not sit where a purge job can delete it. Rejected `TOGA_Staples` —
|
|
119
|
+
wrong business domain. Name must NOT be `AdvanceShipNotices`, that is taken.
|
|
120
|
+
- **Key on `itemFulfillmentInternalId` with a UNIQUE constraint; rename the file to
|
|
121
|
+
`856_<PO>_<itemFulfillmentInternalId>`.** Rationale: stable NetSuite-assigned id, one ASN per fulfillment,
|
|
122
|
+
and the `/out` + `/archive` checks become exact matches instead of prefix guesses. This is the fix for the
|
|
123
|
+
duplicates.
|
|
124
|
+
- **Sequencing, if built (CTO's main correction).** 1) Fix the leaked per-iteration variables, the unchecked
|
|
125
|
+
`$conn->put()` return at L467, move the `FileLog` write after the dedupe, add `cronFinished()` +
|
|
126
|
+
`closeConnections()`, log every skip with a reason. 2) Add the table and switch the filename key. 3) Seed the
|
|
127
|
+
table as already-sent from `FileLog`, run one cycle log-only. 4) Only then enable retry, with an attempt cap
|
|
128
|
+
and an age alert. Rationale: enabling retry before step 1 would re-send bad ASNs at scale, and an empty table
|
|
129
|
+
on day one would dump months of history on Staples.
|
|
130
|
+
|
|
131
|
+
## Blockers
|
|
132
|
+
|
|
133
|
+
1. **Waiting on Jeff's review of the pseudocode.** Scope is now pseudocode only; no implementation.
|
|
134
|
+
2. **Unanswered, needed from Staples:** does a repeated `ShipNoticeRequest` with `operation="new"` and the same
|
|
135
|
+
`shipmentID` get ignored, or treated as a new shipment? We have been sending repeats for weeks, so this
|
|
136
|
+
also decides whether existing damage needs cleanup.
|
|
137
|
+
3. **Unanswered, business decision:** should the 855 gate at L160 exist at all? It hands our ASN timing to the
|
|
138
|
+
customer's file-pickup schedule.
|
|
139
|
+
|
|
140
|
+
## Exact next step
|
|
141
|
+
|
|
142
|
+
> Write the pseudocode into the ClickUp task TRUE-82180 pseudocode field (Mark has already put the findings
|
|
143
|
+
> breakdown there), covering the 4-step sequence in "Decisions made", and hold for Jeff's review. Do NOT write
|
|
144
|
+
> code in `worker` or `dbchanges` — both branches are clean and must stay that way until Jeff decides 1.0 vs 2.0.
|
|
145
|
+
|
|
146
|
+
---
|
|
147
|
+
_Saved by /session-save on 2026-09-21_
|
package/package.json
CHANGED