toga-ai 1.0.958 → 1.0.960
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +49 -11
- package/knowledge/clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md +3 -1
- package/knowledge/clients/nychh/features/netsuite-transfer-order-import.md +3 -1
- package/knowledge/clients/nychh/profile.md +3 -1
- package/knowledge/sessions/2026-10-06-TRUE_80519-mhammontree.md +91 -0
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-06
|
|
10
10
|
owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
@@ -582,17 +582,15 @@ purchase orders, invoices and item receipts were disabled again on 2026-09-03**
|
|
|
582
582
|
is exactly why every NYCHH cursor stops at **2026-09-03**. Before concluding "the sync is broken",
|
|
583
583
|
check the launcher's toggles at the relevant commit.
|
|
584
584
|
|
|
585
|
-
### ⚠ NYCHH
|
|
585
|
+
### ⚠ NYCHH runs every 5 min (NOT once a day) — and its live state is in the CLIENT db, not Core
|
|
586
586
|
|
|
587
|
-
Two traps
|
|
587
|
+
Two NYCHH traps:
|
|
588
588
|
|
|
589
|
-
-
|
|
590
|
-
`
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
rewinding a cursor, run the wrapper by hand rather than waiting (the 10-day ceiling above means a
|
|
595
|
-
deep rewind would otherwise take weeks).
|
|
589
|
+
- **`cronInitialization(86400)` is a TIME LIMIT, not a cadence (corrected 2026-10-06).** The arg is
|
|
590
|
+
`$timeLimitSeconds` → `set_time_limit()`; the only gate is `exitIfProcessRunning()` (overlap
|
|
591
|
+
guard). The "RUN EVERY DAY" comment is stale. Prod `Logs.Event` shows `sync_togasupply_hh.php`
|
|
592
|
+
running on the `*/5` schedule (a frozen section errors every ~5 min). The 2026-10-01 "once a day"
|
|
593
|
+
note was wrong.
|
|
596
594
|
- **Read `Client_Nychh.Parameters`, never the `NETSUITE_HH_*` keys in `Core`.** Core still carries
|
|
597
595
|
legacy `NETSUITE_HH_*` rows whose last-sync dates sit in **May 2026**; they are **stale leftovers
|
|
598
596
|
that nothing writes**. The live cursors and execution modes are
|
|
@@ -983,6 +981,45 @@ php-reviewer clean (additive; forecast callers unaffected), sql-reviewer SAFE TO
|
|
|
983
981
|
mapping, no injection). **Verified prod:** `NETSUITE_EXECUTION_MODE_INVOICES` `1-RUNNING → 1-IDLE`
|
|
984
982
|
(window still 1s; self-heals ×3 from 2024-02-22).
|
|
985
983
|
|
|
984
|
+
### Reverse retire: stale TransferOrder → SalesOrder (`retireStaleTransferOrderForSalesOrder`, DEPLOYED + prod-verified 2026-10-06)
|
|
985
|
+
|
|
986
|
+
Mirror of the SO→TO retire above. Trigger: an order imported as a TO before the stocking-flag rule
|
|
987
|
+
(2026-09-03) is now a SalesOrder in NetSuite → both exist on the same NetSuite id → fulfillment
|
|
988
|
+
reconcile re-matches lines to the SO and tries to delete the TO-side line → FK 1451.
|
|
989
|
+
|
|
990
|
+
- **Called** (i) end of `syncSalesOrderFromNetsuite`; (ii) in `syncItemFulfillmentFromNetsuite` right
|
|
991
|
+
after an existing SalesOrder is found, **BEFORE** the IF header/lines/units snapshot is read. (ii) is
|
|
992
|
+
required: that path skips `syncSalesOrderFromNetsuite`, and NYCHH runs only IF + IA feeds.
|
|
993
|
+
- **Gates:** `is_object($nsOrder)`, `IS_ENABLED_INTEGRATION_TRANSFER_ORDERS`, order is NOT a TO now.
|
|
994
|
+
- **Find:** TO by `TransferOrders.c_netsuiteInternalSalesOrderId` (index `idx_to_netsuite_so_id`).
|
|
995
|
+
- **Throws on:** >1 TO; item receipts on the TO; line lineNumber/product mismatch; duplicate SO
|
|
996
|
+
lineNumber; unmatched TO line that has fulfillments or PO links.
|
|
997
|
+
- **Moves:** IF lines (shared `moveFulfillmentItemToOrderLine` — PUT, or delete-as-duplicate via
|
|
998
|
+
`deleteFulfillmentItemWithChildren` after moving IA links); PO links upstream→upstream,
|
|
999
|
+
downstream→downstream, header + line (`moveOrderPurchaseOrderLinks`, consts
|
|
1000
|
+
`RETIRE_TRANSFER_ORDER_LINE_PO_LINKS` / `RETIRE_TRANSFER_ORDER_HEADER_PO_LINKS`); deletes SO↔TO
|
|
1001
|
+
pairing rows; repoints IF headers `transferOrderId`→`salesOrderId`; deletes TO lines, then TO.
|
|
1002
|
+
- **No transaction.** A throw can leave a partial move; every step checks before writing, so the next
|
|
1003
|
+
run resumes.
|
|
1004
|
+
- All reads via fail-loud paged `readAllForClosedOrderDelete` → any new route must be in
|
|
1005
|
+
`CLOSED_ORDER_DELETE_ROUTES`.
|
|
1006
|
+
- **ACL:** needs DELETE on records 312/313 (TransferOrders/Items) + CRUD on 28. NYCHH already has it
|
|
1007
|
+
via the Base role (see [NYCHH profile](../../../clients/nychh/profile.md)). Other clients: fails loud
|
|
1008
|
+
if missing → add a grant.
|
|
1009
|
+
|
|
1010
|
+
### ⚠ Any `ItemFulfillmentItems` delete must move its inventory-adjustment links first (2026-10-06)
|
|
1011
|
+
|
|
1012
|
+
`ItemFulfillmentItems_InventoryAdjustmentItems` is FK RESTRICT. Deleting a fulfillment line that an
|
|
1013
|
+
adjustment line points at → `EV-11` / 1451 → section frozen at `1-RUNNING`. Full child list to
|
|
1014
|
+
clear/move before any IF line delete: units, unit tracking, item tracking, **IA line links**.
|
|
1015
|
+
|
|
1016
|
+
`moveFulfillmentItemInventoryAdjustmentLinks()` (private) runs before all 4 IF-line delete sites:
|
|
1017
|
+
stale-line delete in `syncItemFulfillmentFromNetsuite` (candidates = current lines on the same
|
|
1018
|
+
fulfillment with same product, minus stale ones), its dup/orphan self-heal (dup → kept row; orphan
|
|
1019
|
+
+ linked → throw), `deleteItemFulfillmentDuplicateLines` (→ kept row), and the duplicate path of
|
|
1020
|
+
`retireStaleSalesOrderForTransferOrder`. >1 candidate → link to all (same never-drop rule as
|
|
1021
|
+
`pairInventoryAdjustmentLinesToFulfillmentLines`); 0 candidates with links → throw.
|
|
1022
|
+
|
|
986
1023
|
### ⚠ TWO code paths create `PurchaseOrderItems` — only one stamped `fulfillmentType` (fixed 2026-09-17)
|
|
987
1024
|
|
|
988
1025
|
`fulfillmentType` was stamped by `syncPurchaseOrderFromNetsuite()` (from the order-level
|
|
@@ -2079,9 +2116,10 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
|
|
|
2079
2116
|
[NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
|
|
2080
2117
|
|
|
2081
2118
|
## Change history
|
|
2119
|
+
- 2026-10-06 — NYCHH ITEM_FULFILLMENTS frozen 09-30→10-06 (Event issue 483, FK 1451 on IA→IF line link) fixed: reverse retire TO→SO + IA-link move before every IF-line delete (library only). Corrected "NYCHH once a day" — it runs every 5 min. (jcardinal)
|
|
2082
2120
|
- 2026-10-05 — Closed-order delete now per-client SO scoped (Compass dual-SO freeze, Issue 940); details in [closed-order rule](../../library/features/netsuite-closed-order-rule.md). Library only. (jcardinal)
|
|
2083
2121
|
- 2026-10-01 — `Items.description` now from the NetSuite item record, refreshed every touch ([doc](../../library/features/netsuite-item-description-sync.md); library + worker deploy together). Gotchas: invoice billing an SO older than the SO cursor (Adyen); receipt lines never rebuilt on an existing receipt (Elite). (rgirish)
|
|
2084
|
-
- 2026-10-01 — NYCHH diagnosis traps recorded: the wrapper is gated to **once a day** by
|
|
2122
|
+
- 2026-10-01 — NYCHH diagnosis traps recorded: (WRONG, corrected 2026-10-06 — it runs every 5 min) the wrapper is gated to **once a day** by
|
|
2085
2123
|
`cronInitialization(86400)` despite its `*/5` schedule entry, and the live cursors/modes live in
|
|
2086
2124
|
`Client_Nychh.Parameters` — the `NETSUITE_HH_*` keys in `Core` are stale (May 2026) leftovers.
|
|
2087
2125
|
Also noted that the “interval already at minimum” throw is **commented out**
|
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: nychh
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-06
|
|
10
10
|
owners: [jcardinal, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
@@ -140,6 +140,7 @@ php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient
|
|
|
140
140
|
|
|
141
141
|
## Gotchas / known issues
|
|
142
142
|
|
|
143
|
+
- **These link rows block deletes of the IF line (FK RESTRICT).** Since 2026-10-06 every IF-line delete in the sync moves them first (`moveFulfillmentItemInventoryAdjustmentLinks`, link-to-all on >1 candidate, throw on 0) — see [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md). A new IF-line delete path must call it too.
|
|
143
144
|
- **Old code cross-producted lines (fixed 2026-09-30, deployed — `df03dd8f` on `_production` since 2026-09-29).** Every IA line of a product linked to every IF line of it: IA #1312 (NS 5211365, TOGa id 387) had 108 line-bridge rows vs 23 correct (210-BKVQ 9×9 = 81 vs 9). Prod: 615 rows over 8 IAs; only 1312 + 1318 wrong. Self-heals via reconcile on re-import (1312 108→23, 1318 4→2) — no SQL. Re-run: `php crons/toga2/netsuite/process_netsuite_inventory_adjustment.php hh --id=5211365,5211772`. 2026-10-02 re-runs of IAs 4754924, 5190495, 5746859, 5746881 linked their missing lines.
|
|
144
145
|
|
|
145
146
|
- **`custbody_stock_adjustment` is a MULTI-SELECT — one adjustment links MANY fulfillments.** The old code used `custbody_stock_adjustment[0]` and captured only the **first**: on adj **1320** (NS id 5218734) that linked **1 of 17** header + **1 of 16** lines. Always read every id from the scalar and loop.
|
|
@@ -156,6 +157,7 @@ php crons/toga2/netsuite/diagnose_hh_import_inventory_adjustment.php <togaClient
|
|
|
156
157
|
- **⚠ OPEN — the IA dedup / self-heal deletes are NOT idempotent against already-gone children.** A half-finished prior run leaves a child already deleted; the retry's `send()` **throws on the 404** and wedges the run. Fix (not yet done): treat not-found as done in the delete path.
|
|
157
158
|
|
|
158
159
|
## Change history
|
|
160
|
+
- 2026-10-06 — IA links now moved before any IF-line delete (FK 1451 froze ITEM_FULFILLMENTS 09-30→10-06). (jcardinal)
|
|
159
161
|
- 2026-10-02 — Marked line-pairing fix deployed (`df03dd8f`); blank link = NetSuite/human issue; chain imports missing only; SuiteQL LIKE gotcha. (jcardinal)
|
|
160
162
|
- 2026-09-30 — Line pairing: per product, exact-qty 1:1 when both sides have many (was full cross-product); paged reads with totalRecordCount guard; synced-IA-line count guard. Verify harness deleted → `process_netsuite_inventory_adjustment.php`. CTO AGREE, php-reviewer CLEAN. (jcardinal)
|
|
161
163
|
- 2026-09-21 — Removed the wrong "adjustment qty must EQUAL fulfillment qty" match rule (an adjustment corrects stock independently of what shipped — proven adj 6868145: 400 adjusted vs 350 shipped); the IA→fulfillment line link is now by **PRODUCT only**, and a product with no fulfillment line **logs + skips** (service items, proven adj 6966700). Paged all 3 attach reads + the fulfillment dedup lookup (api2's 25-row unpaged cap was re-creating duplicate `ItemFulfillmentItems`). New `deleteItemFulfillmentDuplicateLines()` cleans a linked fulfillment's truncation duplicates before matching (serial-subset guard spares item-group members). Prod-verified: INVENTORY_ADJUSTMENTS reached IDLE, cursor draining, `Logs.Issue` 827 cleared. (jcardinal)
|
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: nychh
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-06
|
|
10
10
|
owners: [jcardinal, bala, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/toga2.php
|
|
@@ -208,6 +208,7 @@ WARNING: NYCHH's `NETSUITE_LAST_SYNC_CURSOR_ITEM_RECEIPTS` is stale at **2024-08
|
|
|
208
208
|
|
|
209
209
|
## Gotchas / known issues
|
|
210
210
|
|
|
211
|
+
- **Orders imported as TOs before the 2026-09-03 stocking-flag rule may now ALSO exist as a SalesOrder** (same `c_netsuiteInternalSalesOrderId`). The reverse retire (TO→SO) cleans them up on the next SO or IF sync of that order — see [per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md). Open 2026-10-06: #233366 (SO 172 / TO 1811), #290752 (SO 167 / TO 1958; no IFs, waits for SALES_ORDERS feed).
|
|
211
212
|
- **Location 2 count (1,832 of 1,939) is not a real ship pattern** — 2026-09-24 overwrite bug + NetSuite-entered orders under 28908. Location 2 address = 55 Water St 10041. See destination resolution above.
|
|
212
213
|
- **`Client_Nychh` has TWO `Countries` rows with `c_netsuiteCountry = '_unitedStates'`** — id 1 (US, 52 states, ~20.8k addresses) and id 2 (stray `_t`, only NY, ~38.1k addresses). A by-NetSuite-country map keeps the last one (`_t`), so any non-NY state misses. Froze ITEM_FULFILLMENTS at 1-RUNNING 2026-10-02 (`Logs.Event` issue 935, NS order 3600216, "ship-to state CT has no TOGa State in country _unitedStates") — fixed by `findStateUuidForNetsuiteAddress()` fallback. Stray `_t` country cleanup is open.
|
|
213
214
|
- **Real-TrnfrOrd item receipts are disabled — see the DEFERRED section above.** A receipt whose `createdFrom` is a real Transfer Order is skipped by design; do not read that as the importer being broken.
|
|
@@ -217,6 +218,7 @@ WARNING: NYCHH's `NETSUITE_LAST_SYNC_CURSOR_ITEM_RECEIPTS` is stale at **2024-08
|
|
|
217
218
|
- **`custbody_stocking_order` must reach the shim.** `fetchSalesOrderById()` now carries the field via `customFieldList`; a missing custom field read returns null → the order routes as a transfer order (unchecked = transfer). Verify the field is populated in NetSuite before trusting the split.
|
|
218
219
|
|
|
219
220
|
## Change history
|
|
221
|
+
- 2026-10-06 — Reverse retire stale TO→SO (stocking orders double-imported as TO + SO); froze ITEM_FULFILLMENTS 09-30→10-06. Fixed + prod-verified (TO 508 retired). (jcardinal)
|
|
220
222
|
- 2026-10-02 — Destination: mapped address-book entry > text match (hyphen fix) > Location 2; TO-owned `shipToAddressId`; state lookup falls back across countries (issue 935 hotfix). (jcardinal)
|
|
221
223
|
- 2026-09-24 — Keep user destination on client-customer TOs; ship-to text match for new NS-entered TOs. (jcardinal)
|
|
222
224
|
- 2026-09-24 — PO links now via the shared two-direction reconcile; vendor POs moved off upstream 325/329 onto downstream 326/330; 326/330 ACL fixed (403 EZ-1 resolved); missing customer POs are created; new `process_netsuite_order.php`. Prod-verified. (jcardinal)
|
|
@@ -18,7 +18,7 @@ project: _Underscore
|
|
|
18
18
|
client: nychh
|
|
19
19
|
type: profile
|
|
20
20
|
status: active
|
|
21
|
-
updated: 2026-10-
|
|
21
|
+
updated: 2026-10-06
|
|
22
22
|
owners: ["jcardinal", "apeterson", "bala", "akhokhani"]
|
|
23
23
|
files:
|
|
24
24
|
- dbchanges2/Client_Nychh/2026-09-02a - TransferOrderNetsuitePushInterceptor.sql
|
|
@@ -66,6 +66,7 @@ TOGA 2.0 client on `_underscore`, prod schema `Client_Nychh`. Supply-chain / ful
|
|
|
66
66
|
- **Worker tiers:** `worker2` (2.0) runs the NYCHH asset-tag backfill; `worker`/`library` (1.0) run the NetSuite→TOGa Supply sync and the asset-tag verification/backfill diagnostics.
|
|
67
67
|
|
|
68
68
|
## Key features (this client)
|
|
69
|
+
- **ITEM_FULFILLMENTS froze `1-RUNNING` 2026-09-30 → 2026-10-06 (Event issue 483) — fixed + prod-verified 2026-10-06 (library).** NS order 4512137 (#165198, stocking) existed twice: stale TransferOrder 508 (pre-2026-09-03 stocking rule) + SalesOrder 168. IF reconcile tried to delete TO line 7409 that an IA line linked to → FK 1451. New reverse retire (TO→SO) + IA-link move before IF-line deletes; TO 508 retired, cursor past `4605943`. Remaining doubles: #233366 (SO 172 / TO 1811 — retires when its IF 2392 syncs), #290752 (SO 167 / TO 1958, no IFs — waits for SALES_ORDERS feed). See [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
|
|
69
70
|
- **Transfer-order "Order Shipped" email fires on the first tracking number, not the NetSuite shipped stage (2026-10-05, prod).** Warehouse marks shipped after delivery. NYCHH is the only client emailing off the NS shipped stage. See [TO notification emails](./features/transfer-order-notification-emails.md).
|
|
70
71
|
- **Prod NetSuite feeds (2026-10-02):** only ITEM_FULFILLMENTS + INVENTORY_ADJUSTMENTS run (worker `511a7fd9`, 2026-09-30), cursors rewound to 2022/2023.
|
|
71
72
|
- **TO destination + own ship-to (2026-10-02, library `8f974d93`, _underscore `6839dd3e`).** NetSuite-entered TOs landed on catch-all Location 2 (text match failed on Queens hyphens + multi-address sites). Destination now = mapped NetSuite address-book entry → text match → Location 2; TOs own `shipToAddressId`; Location auto-create uses default shipping. Mapping: 61 entries → 18 sites. See [TO import](./features/netsuite-transfer-order-import.md), [address-book mapping](../../1.0/apps/worker/workflows/netsuite-addressbook-location-mapping.md).
|
|
@@ -119,6 +120,7 @@ TOGA 2.0 client on `_underscore`, prod schema `Client_Nychh`. Supply-chain / ful
|
|
|
119
120
|
## Notes
|
|
120
121
|
- **`Items.title` is NULL on most NYCHH rows — `description` is the populated label field.** Confirmed 2026-09-03 on client-sandbox (a `transfer-order-items` line returned `"title": null` with a real `partNumber`). Every screen that labels an item for this tenant needs the `title || description` chain — the Transfer Order record modal and the Create modal's item picker both carry it. The real fix is populating `Items.title`.
|
|
121
122
|
- **Role ids in `Client_Nychh` (verified 2026-09-01):** `1 = Base`, `2 = Developer`, `3 = API`. Role ids differ per client DB — resolve by **name** in migrations, never hard-code these.
|
|
123
|
+
- **The Agilant API (`Client_Nychh.Apis` id 1) holds BOTH Base (1) and API (3) via `Apis_Roles` (2026-10-06).** Record ACL = union, so checking only the API role misleads. Base grants full CRUD on 28/312/313 (ItemFulfillments, TransferOrders, TransferOrderItems); API role has no DELETE on 312/313 and no row for 28. Interceptors: record 312 has POST/POST + POST/PUT only (none on DELETE).
|
|
122
124
|
- **PO Number is upstream-sourced** (`PurchaseOrders_SalesOrders`), which is why the downstream SO→PO route and the shared `_purchaseOrders` field came back empty. Details, and the now-closed-on-`_beta` fix: [PO Number](./features/po-number-upstream-direction.md).
|
|
123
125
|
- Tracking data: record 318 (item-level) is currently empty for this client; their tracking populates the IF/shipment level (record 317). The rebuilt views use 318 (per the Compass pattern) and will show tracking once item-level tracking is backfilled.
|
|
124
126
|
- Bridge ACL gap fixed 2026-06-19 (`dbchanges2/Client_Nychh/2026-06-19b`): the IF/IR tracking bridge records 317-322 had AclRecordPermissions but no AclLogicGroups, so reads of `item-fulfillment-tracking-numbers` 403'd (EZ-1). Added the logic-group chain for all 6 bridges. See [Tracking-Number Bridge Migration](../../2.0/apps/_underscore/features/tracking-number-bridges.md).
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: TRUE_80519
|
|
4
|
+
title: NYCDOE gated SFTP delete built (1.0 hands ASN files to 2.0, 2.0 deletes after parse)
|
|
5
|
+
author: mhammontree
|
|
6
|
+
repos: [worker2, dbchanges2, library]
|
|
7
|
+
framework: "both"
|
|
8
|
+
client: nycdoe
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-10-06
|
|
11
|
+
updated: 2026-10-06
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: TRUE_80519
|
|
15
|
+
**Date:** 2026-10-06
|
|
16
|
+
**Project/Repo:** worker2 + dbchanges2 (2.0), library (1.0) — branch TRUE-80519 in all
|
|
17
|
+
**Task:** Build the Jeff-approved gated delete for NYCDOE vendor ASN files: 1.0 hands each file to the 2.0 import instead of deleting it, and 2.0 deletes it from the SFTP only after a successful parse — so 1.0 → 2.0 cutover is just flipping cron jobs (NetSuite/Phase 3 later).
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
- **Jeff approved plan §6 flow (2026-10-06)**: move 1.0 → 2.0 by turning cron jobs off/on.
|
|
23
|
+
- **worker2/Worker/Sftp/Import.php** — `RunForFile(clientId, username, filePath)` and `DeleteSource(clientId, filesUuid, filePath=null)` built; `Run()` gained `deleteSourceOnSuccess` + `retiredFileNamePatterns`, 5-min mtime guard. Commit `2a7392e`.
|
|
24
|
+
- **worker2/Worker/Sftp/Parse.php** — claims its row (pending→processing, committed), commits before queuing DeleteSource, saves status outside the job rollback ('pending' for errno 1213/1205/2006/2013, else 'failed').
|
|
25
|
+
- **Real Phase 2 bug found + fixed**: worker2 rolls back the whole job on throw (Route.php transactionStart; Controller/Index.php transactionRollback), so the old `c_status='failed'` save was undone — rejected files stayed 'pending' forever. Regression test `rejectsAFileWithAMismatchedHeader` FAILS on old Parse.php ("Failed asserting that two strings are identical") and passes on new.
|
|
26
|
+
- **Tests**: `tests/Integration/ImportDecisionCest.php` (new) + ParseCest updates + helper `runAsWorkerJob/getFileStatus/setFileStatus` — `OK (18 tests, 57 assertions)` with `vendor/bin/codecept run Integration` (PHP 8, xampp_8x).
|
|
27
|
+
- **dbchanges2** `0d7f065`: `Client_Nycdoe/2026-09-14a` ENUM + `'released'`, non-unique `INDEX checksum`; `Core/2026-09-14a ... NYCDOE Lenovo.sql` gets `"retiredFileNamePatterns": ["DOE-OPEN-LENOVO-*"]`. Prod check: Client_Nycdoe.Files has no c_ columns and Core.CronJobs has no Sftp/Import/Run rows → safe to edit in place.
|
|
28
|
+
- **library (1.0)** `6accf93d` + Mark's `fe2fbbaa`: `App_ASNProcessor::isGatedDeleteMode()`, `isFileAlreadyLogged()` (gated only), `releaseSourceFile()`; lenovo/apple/lexmark save FileLog after parse/email and BEFORE release.
|
|
29
|
+
- **Legacy safety proven**: offline PHP 7.2 stub run of the real base class — 12/12 PASS (default config: every file processed, no FileLog query, old DeleteSftpFile with same 4 params; gated: RunForFile without password, logged file skipped). Exact skip SQL ran on legacy prod and matched FileLog id 899789. Only `sync/nycdoe/legacy_import_asn.php` is scheduled (crons/infrastructure/import_asn.php is not).
|
|
30
|
+
- **Reviews**: CTO AGREE (+5 changes applied); sql-reviewer clean; cso SAFE TO SHIP (both lows fixed); php-reviewer final pass "clean".
|
|
31
|
+
- **All 3 repos pushed** — `git status -sb` shows `TRUE-80519...origin/TRUE-80519` in sync (checked 2026-10-06).
|
|
32
|
+
- **/capture** published KB commit `15439596` (5 docs: asn-sftp-ingestion-2-0, servicenow-integration, per-client-database-connections, codeception-integration-tests, committed-secrets-and-config-debt).
|
|
33
|
+
|
|
34
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
35
|
+
- **Saving `c_status='failed'` with a model save and then throwing** — the worker2 job wrapper rolls back every write on throw; the status is lost. Save it after `_Database::transactionRollback()` and commit `DB_CLIENT`.
|
|
36
|
+
- **Treating every `_Exception_Database` as retriable ('pending')** — `_Query` throws it for ALL SQL errors incl. data-too-long/FK/enum; that made an endless 30-min retry loop. Only errnos 1213, 1205, 2006, 2013 are transient.
|
|
37
|
+
- **Matching the ledger on Files.name or name+size** — same content under a new name never deletes; same name with new content deletes before import. Use checksum.
|
|
38
|
+
- **Trusting a manufacturer from 1.0** — 1.0 Lexmark hands over Acer files (shared CDW folder); resolve patterns by SFTP username across all CronJobs rows.
|
|
39
|
+
- **Deleting every file that matches no 2.0 pattern** — rejected by Mark: only `retiredFileNamePatterns` are deleted; everything else is kept on the SFTP.
|
|
40
|
+
- **Skipping already-logged files in the default (immediate) mode** — old code wrote FileLog BEFORE parsing, so a file it logged but never finished could be released unprocessed at deploy time. Skip is gated-mode only.
|
|
41
|
+
- **Saving FileLog AFTER `releaseSourceFile()`** — if the FileLog save fails after the release, the file can be deleted with no log copy. Log first, then release (both inside `if ($filePutContentsResults !== false)`).
|
|
42
|
+
- **`DeleteSource` deleting the path in Files.sourceUrl for a same-content copy** — that is the FIRST copy's path; the triggered copy stayed forever. Pass the triggered `filePath`.
|
|
43
|
+
- **Ledger / lock reads via a default `_Query`** — goes to a read replica + query cache. Use `setisReadHostEnabled(false)` + `_Database::useQueryCache(false)`.
|
|
44
|
+
- **`GET_LOCK` copied from Github.php as-is** — that works only because DB_TEAM has no replica; the client DB must be pinned to the writer.
|
|
45
|
+
- **A rollback test without resetting `_Database::$_transactionStarts`** — `transactionCommit()`/`transactionRollback()` with no argument leave per-alias flags `true`, so a later `transactionStart()` never begins a transaction; the test passed on the OLD code too. Use helper `runAsWorkerJob()`.
|
|
46
|
+
- **`sed -i` on a CRLF file in Git Bash** — produced mixed endings (LF lines + inserted CRLF). Re-normalize with `tr -d '\r' | sed 's/$/\r/'`, or use the Edit tool.
|
|
47
|
+
- **`git -C` + a `sed` with `\U` in a path** — sed uppercased the rest of the line (`\U` = uppercase). Use the Edit tool for paths with backslashes.
|
|
48
|
+
- **Changing `zend.exception_ignore_args` in worker2 `.platform` ini** — Mark: devops decision, not ours. Fixed in code instead (connect() rethrows without the phpseclib exception).
|
|
49
|
+
|
|
50
|
+
## Not tried yet (candidates for next session)
|
|
51
|
+
- Deploy (Mark): dbchanges2 → `_underscore` (Rohan's window) → worker2 → library + worker with NO ini change (immediate mode = old behavior).
|
|
52
|
+
- Flip `[nycdoe] asn_delete_mode = gated` in `worker/config.worker.ini` + worker deploy; flip the 4 Core CronJobs only at cutover.
|
|
53
|
+
- First gated run checks: one `Logs.FileLog` row per file, no re-processing/re-email, `Files.c_status` reaching 'succeeded', `RunForFile`/`DeleteSource` results in Core.WorkerJobs, vendor folder emptying.
|
|
54
|
+
- **Decide how 2.0 handles manual ASN files** — `WR260197522-RPL Manual ASN.csv` (2026-09-14) came through the Lenovo folder; 2.0 keeps (does not import) unmatched names.
|
|
55
|
+
- Cutover follow-ups: `Run()` has no retry for a lost Parse or lost DeleteSource job; a row stuck 'processing' is only failed by the next 1.0 trigger after 30 min.
|
|
56
|
+
- SFTP file-age monitor (OneUptime, phpseclib3) + alert on `c_status='failed'`.
|
|
57
|
+
- Separate tickets: SSH host-key pinning in `connect()`; rotate the FTP passwords likely stored by old `DeleteSftpFile` jobs in Core.WorkerJobs.parameters.
|
|
58
|
+
- Confirm whether Client_Nycdoe actually has a read replica (code is safe either way — pinned to writer).
|
|
59
|
+
|
|
60
|
+
## Current file state
|
|
61
|
+
| File | Status | Notes |
|
|
62
|
+
|------|--------|-------|
|
|
63
|
+
| `worker2/Worker/Sftp/Import.php` | Committed `2a7392e`, pushed | RunForFile, DeleteSource, Run params, writer reads, lock, path check |
|
|
64
|
+
| `worker2/Worker/Sftp/Parse.php` | Committed `2a7392e`, pushed | claim, status outside rollback, transient errnos |
|
|
65
|
+
| `worker2/tests/Integration/ImportDecisionCest.php` | New, committed `2a7392e`, pushed | 10 decision/path/retired tests |
|
|
66
|
+
| `worker2/tests/Integration/ParseCest.php`, `tests/Support/Helper/Integration.php` | Committed `2a7392e`, pushed | claim + kept-failed tests, runAsWorkerJob helper |
|
|
67
|
+
| `worker2/tests/local.env.php` | Recreated, gitignored | `dev-markhammontree-laptop` (was missing) |
|
|
68
|
+
| `dbchanges2/Client_Nycdoe/2026-09-14a - Add SFTP import status fields to Files.sql` | Committed `0d7f065`, pushed | 'released' + checksum index |
|
|
69
|
+
| `dbchanges2/Core/2026-09-14a - Add SFTP Import cron - NYCDOE Lenovo.sql` | Committed `0d7f065`, pushed | retiredFileNamePatterns; isActive = 0 |
|
|
70
|
+
| `library/app/asnprocessor.php` | Committed `6accf93d` + `fe2fbbaa`, pushed | gated-mode helpers |
|
|
71
|
+
| `library/app/asnprocessor/{lenovo,apple,lexmark}.php` | Committed `6accf93d`, pushed | skip + log-then-release |
|
|
72
|
+
| `C:\Users\mhammontree\.claude\plans\phase-2-asn-parsing-plan.md` | Updated (local) | change history 2026-10-06 |
|
|
73
|
+
| KB docs (5) | Published `15439596` | see What WORKED |
|
|
74
|
+
|
|
75
|
+
## Decisions made
|
|
76
|
+
- **Flow (Jeff):** 1.0 triggers 2.0 per file; 2.0 deletes after a good parse; cutover = flip cron jobs. NetSuite later.
|
|
77
|
+
- **Unmatched files (Mark):** delete only `retiredFileNamePatterns` (data in CronJobs params, not hardcoded); keep all other unmatched files in case a vendor sent a real file under a wrong name. Rejected: delete all unmatched (old 1.0 behavior).
|
|
78
|
+
- **Legacy safety (Mark):** the deployment must never change 1.0 execution — new behavior only behind `asn_delete_mode = gated`; missing = old path, no new query, same job payload.
|
|
79
|
+
- **FileLog = "done" marker:** saved after parse/email, before release; skip only in gated mode.
|
|
80
|
+
- **Transient vs permanent errors:** only deadlock / lock wait / lost connection retry; everything else 'failed' + manual release ('released' status + queue DeleteSource).
|
|
81
|
+
- **Devops owns PHP ini** (`zend.exception_ignore_args`) — recorded in KB, not changed.
|
|
82
|
+
- **CronJobs rows stay `isActive = 0` in shadow mode** — used only as config by RunForFile.
|
|
83
|
+
|
|
84
|
+
## Blockers
|
|
85
|
+
none — Mark is creating PRs and deploying.
|
|
86
|
+
|
|
87
|
+
## Exact next step
|
|
88
|
+
> After deploy (still immediate mode), set `[nycdoe] asn_delete_mode = gated` in `worker/config.worker.ini`, deploy worker, then on the first 1.0 run check legacy `Logs.FileLog` (one row per file) and `Client_Nycdoe.Files.c_status` / Core.WorkerJobs results for `Sftp/Import/RunForFile` and `Sftp/Import/DeleteSource`.
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
_Saved by /session-save on 2026-10-06_
|
package/package.json
CHANGED