toga-ai 1.0.1202 → 1.0.1203

package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md

@@ -6,8 +6,8 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-10-19
-owners: ["dfranks", "jcardinal", "kyalamarthi", "ajean", "rgirish", "bala"]
+updated: 2026-10-29
+owners: ["dfranks", "jcardinal", "kyalamarthi", "ajean", "rgirish", "bala", "mhammontree"]
 files:
   - library/app/api/netsuite/rest.php
   - library/ssl/netsuite_ec_key.pem
@@ -19,6 +19,7 @@ related:
   - ../../worker/features/forecast2-netsuite-reconciliation.md
   - ../../../clients/nychh/features/netsuite-inventory-adjustment-fulfillment-link.md
   - ../../worker/features/compass-ma-sales-order-exception-report.md
+  - ../../../clients/nycdoe/features/servicenow-integration.md
 ---
 
 Working reference for the Agilant NetSuite REST/SuiteQL integration (`App_Api_Netsuite_Rest`): auth, SuiteQL mechanics, table schemas, type/status codes, performance; open when writing or debugging a NetSuite query.
@@ -292,6 +293,16 @@ SOAP one). First caller: [Compass MA exception report](../../worker/features/com
 - Returns `[UPPERCASED_PO => [row, ...]]`, row = `id, tranid, otherrefnum, status` (status letter,
   see the codes above). One PO can map to several SOs; the caller picks.
 
+### Receipts by sales order: `listItemReceiptPartsBySalesOrderIds()`
+
+`listItemReceiptPartsBySalesOrderIds(array $salesOrderIds): array` — one SuiteQL per
+`RECEIPT_SALES_ORDER_ID_CHUNK_SIZE` (900) SO ids. Chain: PO lines with `createdfrom` = SO →
+`ItemRcpt` lines with `createdfrom` = PO (`mainline = 'F'`, `item > 0`, `quantity <> 0`). Returns
+rows `so, itemname (BUILTIN.DF(item)), firstreceipt (MIN trandate, YYYY-MM-DD)` per SO + item.
+REST replacement for the SOAP SO → PO → receipt fan-out; first caller is the DOE received flag
+([NYCDOE ServiceNow / ASN Integration](../../../clients/nycdoe/features/servicenow-integration.md) →
+Stage 5 *Received gate*). Checked 2026-10-29 on DOE data: finds every receipt the SOAP chain finds.
+
 ## Gotchas
 
 - **PHP 7.2 prod constraint.** `library`/`worker` run PHP 7.2 (Amazon Linux 1 EB). Banned syntax: arrow functions (`fn() =>`), typed properties (`public int $x`), `??=`, `match()`, numeric separators (`1_000`). Lint with `C:\xampp7\php\php.exe -l` before deploying — `C:\xampp8\php` (PHP 8.0) is for running probes on the laptop only, **not** for compat checking.
@@ -307,6 +318,8 @@ SOAP one). First caller: [Compass MA exception report](../../worker/features/com
 - `../../worker/features/forecast2-netsuite-reconciliation.md` — the audit/trueup tooling that consumes this API.
 
 ## Change history
+- 2026-10-29 — Added `listItemReceiptPartsBySalesOrderIds()` (first receipt date per SO + item,
+  900 SOs per query) for the DOE received flag (mhammontree, TRUE-83394)
 - 2026-10-16 — Added `listSalesOrdersByOtherRefNums()` (batched SO lookup by PO number, handles
   `A + B` / `A / B` combined `otherrefnum`). REST-only, survives the SOAP retirement. (bala, TRUE-82888)
 - 2026-10-01 — Added `nexttransactionlink` as the order → child-document join (`createdfrom` is a

package/knowledge/1.0/apps/worker/workflows/running-a-worker-cron-locally.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: workflow
 status: active
-updated: 2026-10-23
-owners: ["bala"]
+updated: 2026-10-29
+owners: ["bala", "mhammontree"]
 files:
   - worker/_/app/framework.php
   - library/app/framework.php
@@ -68,6 +68,15 @@ Use Laragon's PHP 7.2.33 build (old worker tier). worker1.5 prod runs PHP 8.5, s
 `phpexcel/` does not parse on PHP 8 ([PHP 8.5 compatibility](../../library/features/php-8-5-runtime-compatibility.md)).
 
 ## Gotchas
+- **worker1.5 scripts — four extra traps** (TRUE-83394):
+  1. xampp_8x `php.ini` `include_path` lists `_underscore` before `library`, so `_.php` loads the
+     2.0 bootstrap. Pass `-d include_path=".;C:/WWW/library"`.
+  2. library `_.php` finds the app root from `SCRIPT_FILENAME` — run the file by its real path
+     inside `worker1.5`.
+  3. worker1.5 has no local `vendor/` — run `composer install` first.
+  4. **No `ENVIRONMENT` = prod `config.worker.ini`.** An error then lands in prod V2
+     `Logs.Issue` and auto-creates a ClickUp task (happened 2026-10-29: issue 1A6). Set
+     `ENVIRONMENT` before every run.
 - **`App_Framework_Worker::initialize()` cannot run twice in one process** — the second
   `define('SENTRY_PROJECT', ...)` fatals. One cron run per PHP process; a wrapper script cannot
   require two crons.
@@ -85,6 +94,8 @@ Use Laragon's PHP 7.2.33 build (old worker tier). worker1.5 prod runs PHP 8.5, s
 - [App_Email side effects & test mode](../../library/features/email-test-mode-and-write-free-sends.md)
 
 ## Change history
+- 2026-10-29 — worker1.5 local-run traps: include_path order, SCRIPT_FILENAME, no vendor/, unset
+  ENVIRONMENT → prod Logs.Issue + ClickUp task (mhammontree, TRUE-83394)
 - 2026-10-23 — PhpSpreadsheet clean on PHP 8 (TRUE-83159); worker1.5 prod is 8.5, phpexcel does not parse there (bala)
 - 2026-10-23 — Deprecations no longer exit since library PR #905 (TRUE-83107) (bala)
 - 2026-10-16 — Created from the cron 7 local test harness (prod-pointing configs, Sentry, SMTP,

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -10,7 +10,10 @@ updated: 2026-10-29
 owners: [mhammontree, sking, bala, jcardinal, rgirish]
 files:
   - worker1.5/crons/sync/nycdoe/3_create_installation_ticket.php
+  - worker1.5/crons/sync/nycdoe/3a_flag_received_asn_items.php
   - worker1.5/schedules/cron.worker.json
+  - library/app/api/netsuite/rest.php
+  - dbchanges/Core/MH/2026-10-29-asn-items-netsuite-received.sql
   - worker1.5/ebs/cron.worker.php
   - library/app/framework.php
   - worker/crons/sync/nycdoe/import_asn.php
@@ -56,6 +59,7 @@ related:
   - ../../../1.0/apps/worker/architecture.md
   - ../../../1.0/apps/worker1.5/architecture.md
   - ../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md
+  - ../../../1.0/apps/library/features/netsuite-suiteql-api-reference.md
 ---
 
 Full DOE ServiceNow/ASN integration map — ticket mirror, ASN → NetSuite SO/TOGa Desk pipeline, outbound status/POD, EDI; open for any DOE cron/pipeline work or a stranded-ASN/duplicate-fulfillment investigation.
@@ -88,6 +92,7 @@ The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents +
 | Queue → ASN records (Stage 2) | `legacy_process_asn_queue.php` | every 15 min |
 | NetSuite SO create (Stage 3, **freeze**) | `1_send_asn_to_netsuite.php` | every 2 h at :07 |
 | NetSuite PO/serials (Stage 4) | `2_send_serials_to_netsuite.php` | every 2 h at :27 |
+| Stage 5 received flag ("DOE/NS Integration - Step 3a Flag Received Lines") | `3a_flag_received_asn_items.php` — stamps `AdvanceShippingNoticeItems.dtNetSuiteReceived` (see Stage 5 → *Received gate*) | every 10 min |
 | TOGa Desk repair orders (Stage 5; schedule name "DOE/NS Integration - Step 3") | `3_create_installation_ticket.php` | `*/5`, but each run lasts ~590 s (own `$stopTime`) + overlap guard → effectively every ~10 min |
 | NetSuite Item Fulfillment (Stage 6) | `4_create_ns_item_fulfillment.php` (**paused**, `active: 0`); `5_DOA_…` unscheduled (moving to `crons/sync/DOA/`, see below) | not running |
 | ETA → ServiceNow | `send_ticket_updates.php` | every 6 min |
@@ -131,8 +136,15 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘      [UNIQUE ded
 - **Stage 4:** reconciles the NetSuite **PO** (`netSuiteInternalPurchaseOrderId` — not the SO id) and sends serial inventory assignments. Already multi-PO aware (`listPurchaseOrdersCreatedFromSalesOrder`, groups items by PO, stamps each item's `netSuiteInternalPurchaseOrderId`) **but only stamps items where the PO is NULL and never re-stamps** — so an item stamped to PO-A but later *received* on PO-B is never corrected. **TOGA does NOT create NetSuite POs anywhere in the DOE flow.** Stage 3 creates only the SO (and, when needed, a special-order inventory Item); POs are generated inside NetSuite by procurement/special-order. A single SO can therefore be fulfilled across **multiple POs** (back-orders, multi-vendor, re-cut lines) — a legitimate NetSuite outcome, so the durable fix lives downstream (resolve receipts SO-wide), not in preventing PO splits.
 - **Stage 5:** for units `WHERE togadeskRepairOrderId IS NULL`, reads serials from NetSuite item receipts (so TOGa Desk RO serials are **driven by NetSuite**, not the ASN unit), creates/reuses TOGa Desk manufacturers/models/assets/serial-numbers/MSO and creates **a new repair_orders row unconditionally per loop iteration** — one RO per Unit (a 50-cable line = 1 unit = 1 RO). Existing-serial lookup is scoped **per item row** (`$asnItemId`). Item receipts are resolved **Sales-Order-wide** via `App_NetSuite::listItemReceiptsCreatedFromSalesOrder($soId)` (fans across ALL POs from `listPurchaseOrdersCreatedFromSalesOrder`, dedupes receipts by `internalId`), cached **per sales-order internal id** (`$itemReceiptsBySo`, keyed by `netSuiteInternalSalesOrderId`) — a shared SO is still fetched once — with the `getItemReceipt` detail cache kept per (globally-unique) receipt `internalId`. Both the inner item query **and** the outer ASN-candidate query require `netSuiteInternalSalesOrderId <> 0` (a sentinel guard mirroring the PO guard) so a null/zero-SO row cannot poison the cache or match the wrong SO's receipts. Self-healing: units previously stuck under an unstamped PO clear on subsequent 5-min cron runs.
   - **Receipt resolution is SO-ONLY — the stamped PO id never finds a receipt** (traced 2026-07-29, `3_create_installation_ticket.php`: candidate query ~L248-278, per-item ~L311-341, receipt resolution ~L354-360, part-number match ~L418-422, per-unit gate ~L511-535). The chain is strictly `AdvanceShippingNoticeItems.netSuiteInternalSalesOrderId` → `listPurchaseOrdersCreatedFromSalesOrder` (all POs created from that SO) → `listItemReceiptsCreateFromPurchaseOrder` for each → dedupe (this is what `listItemReceiptsCreatedFromSalesOrder` wraps, `library/app/netsuite.php` ~L739 / ~L762 / ~L797, details via `getItemReceipt` ~L1697). **`netSuiteInternalPurchaseOrderId` is NEVER used to look up receipts** — it is only a non-null / non-zero **eligibility gate**. The schema is misleading here: the PO column *looks* authoritative and is not.
-  - **Item receipts are never persisted.** They are fetched **live from NetSuite on every run** (no table, no bridge row), and the receipt-line → ASN-line linkage is derived **at runtime by part-number string match** after stripping the `DOE-` prefix and `_` characters. So NetSuite is the only source of truth for what was received, and a part-number formatting drift breaks the match silently.
-  - **Candidate pick is `ORDER BY RAND() LIMIT 1` ON PURPOSE — do not change it to a fixed order** (rgirish, 2026-10-23). Newest-first was tried and the script stuck for days on one ASN with bad data. Cost: ~1,700–1,800 pending candidate ASNs (1,698 on 2026-10-23, 1,795 on 2026-10-29; oldest 2023-10-02, most will never complete) and only a handful finish per run, so a new, fully received ASN can wait hours or days by chance (2026-10-22: 4 ASNs completed in 7.5 h). Symptom "received in NetSuite, no repair order in Desk" with correct SO/PO stamps = this wait, not a stamp mismatch. A DOE report that lists POs by repair order will miss the PO until then.
+  - **Item receipts are never persisted.** They are fetched **live from NetSuite on every run** (no table, no bridge row), and the receipt-line → ASN-line linkage is derived **at runtime by part-number string match**. So NetSuite is the only source of truth for what was received, and a part-number formatting drift breaks the match silently.
+  - **One shared part-match rule (TRUE-83394):** `App_NYCDOE::receiptPartMatchesAsnPart()` (`library/app/nycdoe.php`) — `receiptPartKey()` (drop every `_`, then strip a **leading** `DOE-`) `===` `asnPartKey()` (strip a leading `DOE-`). Used by Stage 5 and the 3a flag cron; if the two ever disagree, a received line is never picked or never ticketed. Before TRUE-83394 Stage 5 removed `DOE-` anywhere and compared with loose `==`.
+  - **Received gate (TRUE-83394, built 2026-10-29; SQL applied to prod 1.0 Core 2026-10-29, library + worker1.5 deploy together).** `Core.AdvanceShippingNoticeItems.dtNetSuiteReceived` (datetime NULL, + `KEY idx_installTicketReady (installTicketStatus, dtNetSuiteReceived, advanceShippingNoticeId)`) = first NetSuite receipt date for the line's SO + part. NULL = no receipt seen yet.
+    - **Writer:** `3a_flag_received_asn_items.php` (every 10 min) loads pending unstamped lines (same gate as Stage 5), runs one SuiteQL per 900 SOs via `App_Api_Netsuite_Rest::listItemReceiptPartsBySalesOrderIds()` (SO → PO lines `createdfrom` SO → receipt lines `createdfrom` PO; `MIN(trandate)` per SO + item), matches with the shared part rule, stamps. **Never clears a stamp. No watermark** — it rechecks every unstamped line each run, so the first run backfilled and a missed run heals itself. Errors are not caught → `Logs.Issue` (alert on failure).
+    - **Reader:** Stage 5's driver and per-ASN item queries add `dtNetSuiteReceived IS NOT NULL`, **except** 1 pick in 10 (`UNFILTERED_PICK_ONE_IN`, a safety net if the flag is ever wrong) and manual single-ASN runs (no filter).
+    - **SuiteQL finds what SOAP finds** (checked 2026-10-29): all 106,083 DOE receipt lines since 2023-09 have `BUILTIN.DF(item) = item.itemid`, no sub-items; 24,099 POs from DOE SOs, none linked to the SO only at header level.
+    - **Stuck line = received but still pending.** A stamped line that never gets an RO (partial receipt, serial already on another line of the same PO) now gets picked over and over. Planned (Mark, 2026-10-29): after ~1 week live, count pending lines with `dtNetSuiteReceived` older than 30 days; only then open a backlog task. Pre-2025 purge skipped for now.
+  - **Candidate pick is `ORDER BY RAND() LIMIT 1` ON PURPOSE — do not change it to a fixed order** (rgirish, 2026-10-23; kept in TRUE-83394). Newest-first was tried and the script stuck for days on one ASN with bad data; oldest-first would keep hitting stuck lines. Before the received gate the pick ran over ~1,700–1,800 pending ASNs (oldest 2023-10-02), most never received, so a new fully received ASN waited hours or days by chance (2026-10-22: 4 ASNs completed in 7.5 h). Symptom "received in NetSuite, no repair order in Desk" with correct SO/PO stamps = this wait, not a stamp mismatch. After the gate deploys, check `dtNetSuiteReceived` first: NULL → 3a has not seen the receipt (part-key or SO-chain issue); stamped → normal wait.
+    - **Queue makeup 2026-10-29 (why the gate):** 1,764 pending ASNs / 4,598 pending lines / 1,959 SOs. Only 363 lines (923 units) have a NetSuite receipt for their SO + part, nearly all 2026, 200 received < 30 days ago. 4,235 lines have no receipt at all (e.g. ASN 14654 / SO 195834: 4 of 6 lines never got a vendor PO). "Old duplicates" is mostly false — `dbchanges/Core/SK/2026-07-30-dedupe-asn-install-units.sql` (run 2026-08-05) already stamped 31,716 dup units. Pre-2025: 536 ASNs / 1,871 lines; they got 904 tickets only in a burst Jul 9–16 2026, none since.
     - **Default response for ONE such ASN: no action** — no manual run, no code change; let the cron pick it (Brian Yard, 2026-10-29). Run by hand (below) only when asked, or for a batch/outage gap.
     - **Fast triage (SuiteQL, 2 queries):** `transaction` + `transactionline` `WHERE id IN (<SO>,<PO>) OR createdfrom IN (<SO>,<PO>)` shows the SO → PO → ItemRcpt chain at once; `inventoryassignment WHERE transaction = <receipt id>` shows the received serial. PO created from SO + receipt holds the ASN serial = clean chain → it is this wait, not [Pre-ASN Manual Receipt Mismatch](#gotchas).
     - Worked case (TRUE-83394): `WR270047873` = ASN 27376 (Apple, 1 unit, serial `CK7W74WQCG`), late from the TRUE-82940 Stage 1b outage; SO 7686363 → PO 7686364 → receipt 7693049 all clean on 10-28; still no RO on 10-29 → left to the cron.
@@ -161,7 +173,7 @@ Both crons were dead in prod for months before this ticket (see gotcha). Ship **
 - These crons stay on until 2.0 takes over sending — see [profile](../profile.md) → *Platforms & data*.
 
 ### Debugging & DB topology (DOE 1.0 sync)
-The 1.0 worker cannot run on a dev/Windows box; use the toga DB MCP + `Logs.API` instead of running prod code locally.
+Prefer the toga DB MCP + `Logs.API` over running a DOE cron locally — a local run talks to prod by default ([Running a 1.0 worker cron locally](../../../1.0/apps/worker/workflows/running-a-worker-cron-locally.md)).
 - **⚠ `Logs.API` retention is 30 days — SHIPPED 2026-09-28 (TRUE-82049)**; mechanics, storage facts and the not-yet-run backlog purge in [API Request Logging & Retention](../../../1.0/apps/library/features/api-request-logging-and-retention.md) — so it is no longer a *historical* debugging avenue, only a recent-window one. Bugs that burn slowly (a hold not re-pushed for 22 days; 9,914 PATCHes traced back to 2026-08-20) can no longer be reconstructed after the fact and must be caught by **alerting** instead. NYCDOE stays 1.0, so its logs remain in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0.
 - **`repair_orders.dtUpdated` and `.lastupdateddate` are both time-skewed in `TOGaDeskSupport`** — never select or diff on them, and never compensate with a fixed offset. Detail: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
 - **`dbchanges` does not describe the real 1.0 schema** — some live columns were created outside the repo. Verify indexes/columns against the database, not the repo: [Authoring & Shipping a 1.0 dbchanges SQL File](../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
@@ -268,7 +280,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
         OR (AdvanceShippingNoticeItems.togadeskRepairOrderId IS NULL AND NOT EXISTS (SELECT 1 FROM AdvanceShippingNoticeUnits WHERE AdvanceShippingNoticeUnits.advanceShippingNoticeItemId = AdvanceShippingNoticeItems.id))
       )
     ```
-    Note this only confirms the row is an eligible *candidate* for the `*/5 min` Stage 5 cron (which then picks ONE ASN at random via `ORDER BY RAND() LIMIT 1` across all pending ASNs) — it does not by itself confirm a ticket was created; wait for the cron and check `togadeskRepairOrderId`/`dtInstallTicketStatusChanged` to confirm ticketing completed.
+    Note this is the pre-TRUE-83394 shape (the live query also needs `dtNetSuiteReceived IS NOT NULL` on 9 picks in 10) and only confirms the row is an eligible *candidate* for the Stage 5 cron (which then picks ONE ASN at random via `ORDER BY RAND() LIMIT 1`) — it does not by itself confirm a ticket was created; wait for the cron and check `togadeskRepairOrderId`/`dtInstallTicketStatusChanged` to confirm ticketing completed.
   - Unlike instances 1–2 below, no duplicate/replacement SO was involved here — the mismatch is a same-ASN sibling PO the warehouse had already been using for other lines on this ASN.
 
 - **⚠ OPEN BUG (TRUE-83064, 2026-10-22; data repair COMPLETE 2026-10-23, root-cause code fix NOT done) — one customer PO split into two ASNs (two vendors) → wrong manufacturer + stranded cross-received units.** Customer PO `WR270089579` came in as two ASNs, one under the Acer vendor and one under the Lexmark vendor, for the same Acer part `DOE-NX.JHWAA.004-NYCDOE` (likely the Acer-as-Lexmark SFTP path, Stage 1b above).
@@ -318,6 +330,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
+- 2026-10-29 — TRUE-83394 (reopened): Stage 5 received gate — `dtNetSuiteReceived` column (SQL applied to prod), new `3a_flag_received_asn_items.php` cron, Step 3 filters on it except 1 in 10, shared part-match rule; queue makeup; SuiteQL = SOAP check; stuck-set count planned after ~1 week. CTO agreed. library + worker1.5 deploy together. (mhammontree)
 - 2026-10-29 — TRUE-83394: `WR270047873` received in NetSuite but no Desk RO = Stage 5 random-pick wait (1,795 candidates); default = no action for one ASN (Brian Yard); SuiteQL chain triage tip. No code change. (mhammontree)
 - 2026-10-27 — TRUE-82940: `apple.php:74` PHP 8.5 outage (10-22 → 27) fixed, not deployed; errors land in V2 `Logs.Issue`; Stage 1b cadence is 15 min (worker1.5); gated mode sends one `RunForFiles` per account per run. (mhammontree)
 - 2026-10-23 — Daily open/closed report row; manual re-run gotcha (duplicate email, 24 h window shift) (TRUE-83159) (bala)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1202",
+  "version": "1.0.1203",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",