toga-ai 1.0.915 → 1.0.916

package/knowledge/clients/nycdoe/features/asn-sftp-ingestion-2-0.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-09-25
+updated: 2026-09-29
 owners: [mhammontree]
 files:
   - worker2/Worker/Sftp/Import.php
@@ -17,6 +17,7 @@ files:
   - dbchanges2/Client_Nycdoe/2026-09-22b - Add ASN parsing fields to ServiceRequests.sql
   - dbchanges2/Client_Nycdoe/2026-09-22c - Add customerPurchaseOrder field to Cases.sql
   - worker2/tests/Integration/ParseCest.php
+  - dbchanges2/Core/2026-09-14a - Add SFTP Import cron - NYCDOE Lenovo.sql
 related:
   - servicenow-integration.md
   - ../profile.md
@@ -46,7 +47,21 @@ The 2.0 replacement for 1.0's `legacy_import_asn.php` + `asnprocessor/*` (see [S
 | Acer | CDW 47-col, 3 names differ from Lexmark | `ACER_HEADER` | `YYYY/MM/DD` |
 | Apple | own | `APPLE_HEADER` | `order_date` `M/D/YY` or `M/D/YYYY` |
 
-Schema (`dbchanges2/Client_Nycdoe/`): `2026-09-14a` Files `c_status`/`c_manufacturer`/`c_dtProcessed`; `2026-09-22a` ServiceRequestStages seed (New/Scheduled/On Hold/Delivered/Cancelled); `2026-09-22b` ServiceRequests fields; `2026-09-22c` Cases `c_customerPurchaseOrder`; `2026-09-25a` Cases `longDescription`.
+### Activation / cutover (NOT built — design under review)
+- **`Sftp/Import/Run` is read-only on the SFTP** (`nlist`/`stat`/`get` only). 1.0 still deletes each file within minutes and runs `*/5` vs 2.0 `*/15`, so a naive 2.0 shadow run sees a random subset. See 1.0 Stage 1b in [ServiceNow / ASN Integration](servicenow-integration.md).
+- **DECIDED (Mark, 2026-09-25/29): 1.0 must not delete a file until 2.0 has imported and parsed it.** Replaces the 2026-08-28 "copy to temp subfolder" bridge idea.
+- **The 4 `dbchanges2/Core/2026-09-14a - Add SFTP Import cron - NYCDOE <Mfr>` rows ship `isActive=0`** (ordered rollout); they stay off through shadow mode.
+- **PROPOSED design (2026-09-29, for review with Jeff; needs a fresh `cto` pass — the 2026-09-25 CTO agreement was for an earlier polling variant):**
+  1. 1.0 replaces its `DeleteSftpFile` runTask with `Sftp/Import/RunForFile(clientId, username, filePath)` — no password in SQS.
+  2. `RunForFile` loads **all** `Sftp/Import/Run` CronJobs rows for clientId+username (the CDW username returns both Lexmark and Acer rows — never trust a manufacturer from 1.0), matches `fnmatch` patterns (no match → delete directly, old behavior), fetches, saves `Files`, queues Parse with `deleteSourceOnSuccess=true`.
+  3. Parse on success queues new `Sftp/Import/DeleteSource(clientId, filesUuid)`, which re-checks `c_status` (`succeeded`/`released`) **and** the remote file's sha256 vs `Files.checksum` before deleting.
+  4. Checksum already in `Files`: `succeeded` → delete; stale `pending` → re-queue Parse; `failed` → keep the file.
+  5. 1.0 skips files already in `Logs.FileLog` (job+fileName+fileTimestamp) but still re-queues `RunForFile`.
+  6. Kill switch `[nycdoe] asn_delete_mode` = `gated` | `immediate`; missing = `immediate`.
+  7. Cutover: turn the CronJobs rows on with `deleteSourceOnSuccess`, remove the 1.0 processors.
+  8. Add `released` to `Files.c_status` for manual release of failed files.
+
+Schema (`dbchanges2/Client_Nycdoe/`): `2026-09-14a` Files `c_status` (ENUM `pending`/`processing`/`succeeded`/`failed`, not shipped)/`c_manufacturer`/`c_dtProcessed`; `2026-09-22a` ServiceRequestStages seed (New/Scheduled/On Hold/Delivered/Cancelled); `2026-09-22b` ServiceRequests fields; `2026-09-22c` Cases `c_customerPurchaseOrder`; `2026-09-25a` Cases `longDescription`.
 
 Tests: `worker2/tests/Integration/ParseCest.php` (all 4 mfrs + header rejection + comma repair) — see [worker2 Codeception integration tests](../../../2.0/apps/worker2/features/codeception-integration-tests.md).
 
@@ -60,4 +75,5 @@ Tests: `worker2/tests/Integration/ParseCest.php` (all 4 mfrs + header rejection
 - **Schema drift: `_Model_Client_Case` declares `longDescription` but `Client_Nycdoe.Cases` lacked the column** → any direct-model Case load/save 1054s. Fixed for DOE (`2026-09-25a`). Other tenants' `Cases` may have the same gap — unchecked (ask Rohan).
 
 ## Change history
+- 2026-09-29 — Activation plan: gated-delete decision, proposed RunForFile/DeleteSource design, CronJobs rows set isActive=0 (mhammontree)
 - 2026-09-25 — Phase 2 parsing built for all 4 mfrs; header validation, comma repair, real-sample header/date corrections; integration tests added (mhammontree)

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -38,6 +38,7 @@ files:
   - dbchanges/Core/SK/
   - dbchanges/TOGaDeskSupport/SK/
   - worker2/Worker/Monitor/Nycdoe.php
+  - worker2/Worker/Utilities/Sftp/DeleteFile.php
 related:
   - ../profile.md
   - edi-850-receive-and-997-ack.md
@@ -110,6 +111,10 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘      [UNIQUE ded
 
 - **Stage 1a (API):** vendor detected by string match on `u_asset_mfg` (apple=1, acer=2, lenovo=3, lexmark=4; unknown → Lexmark). Serial 'S'-prefix stripped for Lexmark/Lenovo. Ship-to resolved via `App_Api_NYCDOEV2::resolveShippingAddress()`; on failure a tracking `NYCDOETicket` is recorded but **ASN creation is skipped**. `dedupeKey = vendorId|PO|DOE-part|serialNumber`. Dedup is SELECT-before-INSERT with per-row try/catch (unique index is the race backstop).
 - **Stage 1b (SFTP):** the only source of **non-serialized** items (in practice the Lexmark feed). Serial-less lines of the same PO+part are summed per file into one queue row with `dedupeKey = vendorId|PO|DOE-part|<file token>`. Dedup is INSERT + catch "Duplicate entry" (race-safe).
+  - **Per file, in order** (`App_ASNProcessor_{Lenovo,Apple,Lexmark}::processFiles()`): read over plain FTP (`App_FTP`: `ftp_connect`/`ftp_rawlist`/`ftp_get`) → save legacy `Logs.FileLog` (jobs `ASN_LENOVO` / `ASN_APPLE` / `ASN_LEXMARK`) → parse → cache to disk → email → queue `App_Worker::runTask('Utilities/Sftp/DeleteFile/DeleteSftpFile')` (`lenovo.php:176-184`, `apple.php:105-113`, `lexmark.php:102-110`). worker2 `Worker/Utilities/Sftp/DeleteFile.php` then deletes the file over SFTP (phpseclib3). ⚠ The SFTP password travels **inside the SQS message** (source: hardcoded constants in `asnprocessor/*.php`) — see the credentials gotcha.
+  - **Delete is gated only on the disk-cache write** (`$filePutContentsResults !== false`), **not on a good parse** — a bad file is still deleted within minutes, and `Logs.FileLog` is the only copy left.
+  - **No Acer processor runs.** `asnprocessor/acer.php` is dead code (not called from `legacy_import_asn.php`, never appends to `$output`, never deletes). Acer files do arrive — via CDW on the shared CDW/Lexmark SFTP account — and Lexmark `processFiles()` has **no file-name filter**, so 1.0 processes every Acer file **as Lexmark** (Lexmark vendorId). Proof: 458 `ASN_LEXMARK` FileLog rows with `acer` in `fileName` (2025-08-18 → 2026-09-25), 0 `ASN_ACER`. Any 1.0-vs-2.0 count must compare 1.0 Lexmark against 2.0 Lexmark + Acer.
+  - **Vendors upload and forget** — they never re-send or notice if a file sits on the SFTP (Mark, 2026-09-25). A deleted-but-unparsed file is gone unless recovered from `Logs.FileLog`.
 - **Stage 2:** queue → ASN/Items/Units/TrackingNumbers in a transaction. Serialized: one Unit per serial. Non-serialized: the file's quantity is a **cumulative** total — `unsent item qtyOrder = cumulative − SUM(committed qtyOrder)`; post-freeze growth becomes a **fresh item row**, a decrease fires a discrepancy email and changes nothing. Exactly one Unit per serial-less item row regardless of quantity. Failed rows retry 5× then alert `devteam@goagilant.com`.
 - **Stage 3 (THE FREEZE):** creates the NetSuite SO and stamps `AdvanceShippingNoticeItems.netSuiteInternalSalesOrderId`. **Once stamped, that item's `qtyOrder` is immutable** — there is no update path to a sent SO line; growth must be a new item row → new SO line.
 - **Stage 4:** reconciles the NetSuite **PO** (`netSuiteInternalPurchaseOrderId` — not the SO id) and sends serial inventory assignments. Already multi-PO aware (`listPurchaseOrdersCreatedFromSalesOrder`, groups items by PO, stamps each item's `netSuiteInternalPurchaseOrderId`) **but only stamps items where the PO is NULL and never re-stamps** — so an item stamped to PO-A but later *received* on PO-B is never corrected. **TOGA does NOT create NetSuite POs anywhere in the DOE flow.** Stage 3 creates only the SO (and, when needed, a special-order inventory Item); POs are generated inside NetSuite by procurement/special-order. A single SO can therefore be fulfilled across **multiple POs** (back-orders, multi-vendor, re-cut lines) — a legitimate NetSuite outcome, so the durable fix lives downstream (resolve receipts SO-wide), not in preventing PO splits.
@@ -138,6 +143,7 @@ The 1.0 worker cannot run on a dev/Windows box; use the toga DB MCP + `Logs.API`
 - **NetSuite SOAP request/response payloads** are logged in legacy `Logs.API` — filter `sourceJob LIKE '%3_create_installation_ticket%'`; `requestPayload` contains the PO internalId. (The multi-PO case was verified this way: the receipt search for PO 6971859 returned `totalRecords=1`, only `38S0500`.)
 - **`Bridge_NetSuite`** (legacy) has `SalesOrders` / `SalesOrderItems` / `SerialNumbers` but **no `PurchaseOrders` table** — POs are pulled live from the NetSuite API.
 - **Stuck-vs-received check, per ASN item:** compare `listItemReceiptsCreatedFromSalesOrder($stampedSO)` against `listItemReceiptsCreateFromPurchaseOrder($stampedPO)`, dumping each receipt line's `itemName` plus its `inventoryAssignment` serials. If the SO-wide call returns 0 but the stamped PO returns the receipt with the matching serials, it is the cross-wired-stamp regression below.
+- **Legacy `Logs.FileLog` (~47k rows) has no `fileName` index** — only `PRIMARY(id)`, `FileLog_job(job)`, `dtStamp`. Filter by `job` first (uses `FileLog_job`; `ASN_LENOVO` 1,899 / `ASN_LEXMARK` 954 / `ASN_APPLE` 113 rows as of 2026-09-25), then `fileName`. Select `id` only — never `fileData` (longblob) in a scan.
 - **`App_Model_Core_AdvanceShippingNotice` does NOT map `dtCreatedInstallationTicket`** — its `__get` throws "No field or quick query defined for key …". Read that column via **raw SQL**, not the model.
 
 ### Monitoring the ServiceNow endpoint (worker2 liveness probe — TRUE-80587)
@@ -157,7 +163,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 
 ## Gotchas
 
-- **⚠ SECURITY (cso review 2026-09-18 — NOT fixed, needs its own ticket): live production ServiceNow credentials are hardcoded in committed PHP.** `library/app/api/nycdoev2.php` and `library/app/api/nycdoe.php` hold the ServiceNow **`clientSecret` and `refreshToken` as literals** (stage values too). A test-repo diagnostic script (committed 2026-06-18) embeds the same prod credential — one rotation covers both. Treat as compromised: move them to config/env and **rotate with DOE**. Separately, `Vision_Log.ErrorLog` rows dump `$_SERVER` including live AWS access key id / secret in plain text — its own ticket, same rotation treatment. **No credential values are recorded here or anywhere in the KB.**
+- **⚠ SECURITY (cso review 2026-09-18 — NOT fixed, needs its own ticket): live production ServiceNow credentials are hardcoded in committed PHP.** `library/app/api/nycdoev2.php` and `library/app/api/nycdoe.php` hold the ServiceNow **`clientSecret` and `refreshToken` as literals** (stage values too). A test-repo diagnostic script (committed 2026-06-18) embeds the same prod credential — one rotation covers both. Treat as compromised: move them to config/env and **rotate with DOE**. Separately, `Vision_Log.ErrorLog` rows dump `$_SERVER` including live AWS access key id / secret in plain text — its own ticket, same rotation treatment. Vendor FTP/SFTP credentials are also hardcoded constants in `library/app/asnprocessor/*.php`, and Stage 1b sends the SFTP password in plain SQS message bodies to worker2's delete task. **No credential values are recorded here or anywhere in the KB.**
 
 - **⚠ OPEN BUG / NAMED PATTERN — "Lenovo Off-Layout ASN File" (silent column shift + unit collapse).** Investigated 2026-08-11 (read-only; **no code fix exists**). Lenovo sends NYCDOE ASNs in **two different column layouts**, and `library/app/asnprocessor/lenovo.php` picks its column mapping by **column COUNT with no header-name validation** (`:110`, `if ($colCount >= 30`), so the off-layout file is parsed with the wrong mapping and silently mis-read.
   - **The two layouts.** The normal daily feed `DOE-Lenovo-ASN-MM-DD-YYYY.CSV` (~2 MB) has **30** columns and matches the code's mapping. A second file, `DOE-Open-Lenovo-MM-DD-YYYY.csv` (**lowercase** `.csv`, ~45 KB), has **31** columns in a **different order**: `Contact Email` moves from column 29 to column **6**, and a trailing `Full Shipment` column is appended. Everything from `Street Address` onward is therefore read **one column too early**. The "extra comma" repair heuristic at `:119-127` does not catch this (unchanged since March 2026 — commits `3442ddcf`, `6956dfbf`, `864cf073`, so not a regression).
@@ -262,6 +268,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
+- 2026-09-29 — Stage 1b file-delete path (delete not gated on parse; SFTP password rides in SQS), Acer-as-Lexmark, FileLog index facts (mhammontree)
 - 2026-09-29 — Assignment-group scoping corrected (inverted `isValidDepartmentName`, three competing group lists) with detail in hold-status-sync; `Logs.API` 30-day retention shipped. (mhammontree)
 - 2026-09-25 — TRUE-82332: cross-link to the close-clobber fix; test-repo copy of the ServiceNow credential noted (mhammontree)
 - 2026-09-25 — Corrected "no 2.0 tenant" (TRUE-80519 2.0 ASN rewrite now exists); DOE-Open-Lenovo marked retired (mhammontree)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.915",
+  "version": "1.0.916",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",