toga-ai 1.0.899 → 1.0.900

package/knowledge/clients/nycdoe/INDEX.md

@@ -4,6 +4,6 @@
 |-----|-----------|---------|
 | [NYCDOE 2.0 ASN SFTP Ingestion + Parsing (Sftp/Import → Sftp/Parse → Case/ServiceRequest)](features/asn-sftp-ingestion-2-0.md) | 2.0 | 2.0 rewrite of DOE's vendor-SFTP ASN intake (TRUE-80519 Phases 1–2): per-manufacturer fetch to `Files`, header-validated parse into one Case per customer PO + o |
 | [NYCDOE EDI 850 Receive & 997 Acknowledgement](features/edi-850-receive-and-997-ack.md) | 1.0 | DOE's EDI 850-in → NetSuite SO + inline 997-out ack; open for the 997 build, its "always accept" stub, or a DOE ack complaint. |
-| [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, and the outbound cron's sel |
+| [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, the held→complete close clo |
 | [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | Full DOE ServiceNow/ASN integration map — ticket mirror, ASN → NetSuite SO/TOGa Desk pipeline, outbound status/POD, EDI; open for any DOE cron/pipeline work or |
 | [New York City Department of Education](profile.md) | 1.0 | NYC DOE client profile (1.0 worker tier live; 2.0 ASN rewrite in progress), ServiceNow + EDI; open for DOE's platform scope, endpoints, API clients, and where i |

package/knowledge/clients/nycdoe/features/hold-status-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: nycdoe
 type: client-feature
 status: active
-updated: 2026-09-18
+updated: 2026-09-25
 owners: [mhammontree, sking]
 files:
   - worker/crons/sync/nycdoe/send_ticket_updates.php
@@ -23,7 +23,7 @@ related:
   - ../../../1.0/apps/worker/architecture.md
 ---
 
-Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, and the outbound cron's selection/throttle model; open before changing DOE hold logic, the selection query, or "which SNOW field is authoritative."
+Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, the held→complete close clobber (TRUE-82332), and the outbound cron's selection/throttle model; open before changing DOE hold/close logic, the selection query, or "which SNOW field is authoritative."
 
 ## Summary
 
@@ -73,6 +73,17 @@ When a DOE repair order is on hold, `send_ticket_updates.php` pushes **the full
 - The "Assigned → In Progress" auto-start block is still guarded so it never pushes `state:2` over a held order (TRUE-79922). Named constants: `SN_STATE_IN_PROGRESS` / `SN_STATE_ON_HOLD` / `SN_HOLD_REASON_DEFAULT` in `send_ticket_updates.php`.
 - The existing **empty-body patch-response check** (~L493) still **throws**, so any remaining rejection surfaces loudly. Keep it.
 
+### Close vs. hold-release — held→complete never resolved (TRUE-82332)
+**Bug (silent, 2026-09-18 → fix):** for a COMPLETE order the close block sets `state` 6 (8 for `STATUS_COMPLETE_EXCEPTION`) + `close_code` + `close_notes`. The later hold-release block (`!$orderIsOnHold && $snState == 'On Hold'` → `state` 2) **overwrote** it with 2 (In Progress). The PATCH succeeded, `dtSynced` advanced to the newest note, `close_code` now matched ServiceNow (so the `close_code != remote` check never rebuilt the close), and the order left the queue (COMPLETE is eligible only while notes > `dtSynced`). No error, no alert.
+- **Why it surfaced 2026-09-18:** before TRUE-82129 hold pushes did not reach ServiceNow, so almost nothing was On Hold remotely at completion. After it, every held→complete order hit this.
+- **Evidence:** 45 incidents 2026-09-18..09-25, each one close PATCH with `"state":2`, none with 6/8; non-held closes in the same window sent 6. Example INC2131474: GET `Logs.API` id 173203251 "On Hold" → PATCH id 173203252 (2026-09-22 15:08:32) `close_code` "Repaired" + `state` 2 → next GET "In Progress", no more traffic.
+- **Fix (worker `a04c5452a`, branch TRUE-82332, not deployed as of 2026-09-25):** (a) the release block sets `state` 2 only when `!isset($payload['state'])`; (b) the close block also fires when the ServiceNow state is not in `$serviceNowClosedStatesArray`, even if `close_code` matches — so a re-queued stuck ticket gets its close re-sent.
+- **Rule: a later block in the payload builder must not overwrite a `state` an earlier block already set.** Close outranks release.
+- **No PATCH storm:** the earlier "complete + SN Resolved/Closed/Canceled" check retires the ticket before the close block; `dtSynced` moves only after a successful PATCH.
+- **OPEN:** not confirmed ServiceNow accepts On Hold → Resolved (6) in one step. If rejected: empty-body 200 → `$failedPatches`, `dtSynced` not moved, retried every 6-min run, alert email max 1/hr, until someone steps in.
+
+**Recovering tickets already stuck (one-off, run AFTER the worker deploy):** they left the queue, so re-queue them — set `Common.NYCDOETickets.dtSynced` = newest `repair_order_notes.notesdate` − 1 second, pinned in the `WHERE` on `id` + `ticketNumber` + current `dtSynced`. Only the newest note becomes "unsent"; verify it is already in ServiceNow (`referenceId` set / not internal) so no duplicate `work_notes`. **Run it before the deploy and it re-sends `state` 2 and drops them again.** Tickets since closed by hand in ServiceNow are harmless: the "both closed" check retires them with one GET, no PATCH. One-off nudge — not a `dbchanges` file.
+
 > **DECISION (TRUE-81044) — do NOT map TogaDesk hold reasons onto ServiceNow hold-reason codes.** `hold_reason` stays pinned to `10` ("Other") and the human-readable reason is carried entirely in `u_other_reason_on_hold` / `work_notes`, sourced from TogaDesk. Rationale: ServiceNow's own states are not reliable, so mapping our six `HOLD_*` constants onto their code set buys nothing and adds a second source of truth. **Do not re-propose a reason-code mapping.**
 
 ### Outbound selection, watermark & throttle (TRUE-82027 current shape)
@@ -122,7 +133,7 @@ All three `error_log()` calls were removed (the per-ticket technician sys_id mes
 - **The invalid-assignment-group email had never sent, ever** — the block built the email and never called `$email->send()` (every sibling DOE cron does, e.g. `2_send_serials_to_netsuite.php:346`). Added, behind the same throttle on its own key `system|ticket_sync_invalid_group_alert`.
 
 ### Verifying a TRUE-82027-class change (no sandbox — verified in production)
-There is **no sandbox for the SNOW integration**; verification happens in prod. Deploy order is **dbchanges → library → worker**, and dbchanges `_production` does **not** auto-apply — the migration lands on the next worker EB deploy. After deploy, check legacy `Logs.API` filtered `endpoint LIKE '%nycd3%'` for:
+There is **no sandbox for the SNOW integration**; verification happens in prod. Deploy order is **dbchanges → library → worker**, and dbchanges `_production` does **not** auto-apply — the migration lands on the next worker EB deploy. After deploy, check legacy `Logs.API` (query shapes in *Reference facts* — `LIKE` scans time out) for:
 1. A held order PATCHing `state:3` with **no new note**.
 2. INC2087375's `u_eta` moving from 2026-09-07 to 2026-09-17.
 3. `SELECT COUNT(*) FROM Common.NYCDOETickets WHERE dtLastChecked IS NOT NULL` climbing.
@@ -160,7 +171,8 @@ Making DOE holds persist took a change in **three repos**, with a **deploy-order
 The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets`) is **intentionally kept** as a failover for SNOW API unreliability — not removed by this work.
 
 ### Reference facts (DOE sync debugging)
-- **SNOW request/response logs:** logged to the legacy `Logs` schema, **`API` table** (`App_Model_Logs_API`, `db_logs`, `_databaseNameOverride 'Logs'`). `requestPayload` / `responsePayload` are queryable CHAR columns — the primary debugging avenue. Filter `endpoint LIKE '%nycd3%'`; the custom DOE endpoint base is **`/api/nycd3/`**. The state:3↔state:2 oscillation in TRUE-79922 was proven from this table (INC2074899, INC2185183 showed `state:3` then `state:2` pushed seconds apart in one run).
+- **SNOW request/response logs:** logged to the legacy `Logs` schema, **`API` table** (`App_Model_Logs_API`, `db_logs`, `_databaseNameOverride 'Logs'`). `requestPayload` / `responsePayload` are queryable CHAR columns — the primary debugging avenue. Time column is **`dtStamp`** (not `dtCreated`). The custom DOE endpoint base is **`/api/nycd3/`** on prod host **`supportadmin.schools.nyc`** (not `nycdoeprod.service-now.com`).
+  - **Query shape that works (TRUE-82332):** `endpoint LIKE '%nycd3%...'` and broad `requestPayload LIKE` scans time out (>300s) on the MCP. Use exact matches: PATCH `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/data/tasks/INCxxxx'`; GET `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/v2/data/tasks?domain=diit%2Calm&filter=sys_class_name%3Dincident%5Enumber%3DINCxxxx'`. For sweeps: `FORCE INDEX (dtStamp)`, windows ≤2 days, plus `sourceJob` / `method` filters. The state:3↔state:2 oscillation in TRUE-79922 was proven from this table (INC2074899, INC2185183 showed `state:3` then `state:2` pushed seconds apart in one run).
 - **`NYCDOETickets` (`App_Model_Common_NYCDOETicket`, `db_common` / legacy `Common`)** stores the inbound SNOW record in `rawData`, a **`FIELDTYPE_BLOBSTORAGE`** field (S3 bucket `asifiles`, key `prod/Common/NYCDOETickets/rawData/{id}`) — **NOT a queryable DB column.**
 - **⚠ `repair_orders.dtUpdated` is skewed — never build a selection query on it.** It is written exactly **240 minutes behind** the clock used by `repair_order_notes.notesdate` and `NYCDOETickets.dtSynced`. **Do NOT compensate with `+240 MINUTE`** — 240 is almost certainly UTC-vs-Eastern and becomes 300 at the DST change (2026-11-01), so a hardcoded offset breaks twice a year. TRUE-82027 sidestepped it by removing `dtUpdated` from the query. Mechanism + the second (720-minute) skew: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
 - **⚠ `Logs.API` history is capped at 30 days from 2026-09-17** (Jeff, sprint review) and existing logs are being purged. This investigation depended on history far older than that (9,914 PATCHes back to 2026-08-20; a hold last pushed 22 days earlier; the 2,392-call storm). **Slow-burn bugs like this can no longer be traced backwards — they must be caught by alerting.** NYCDOE stays 1.0 so its logs stay in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0. Agreed: hold the purge until TRUE-82027 is verified in production.
@@ -181,14 +193,16 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
 - **Diagnosing which writer dropped a hold:** `repair_order_history` (`repairid, userid, dtStamp, note`) records *user-driven* status changes. A hold reverting with **no** history row between the hold and the revert = a raw-`UPDATE` writer (`updateStatus`/`qqStatus`), **not** the worker inbound cron (`process_tickets.php` contains no reference to `ORDER_ASSIGNED_AWAITING_SCHEDULING`). Note that **since TRUE-80060** a *status-changing* `class.repair.php::addNotes` now **calls** `updateStatus()` (so it too can move the cache via that raw `UPDATE`); a **pure comment** still writes nothing to `repair_orders.status`. This history-gap technique is how TRUE-80060 was pinned to `qqStatus()` rather than the SNOW round-trip.
 - **⚠ OUTAGE (TRUE-82129, 2026-09-18) — `send_ticket_updates.php` died on its FIRST ticket of EVERY run, for a whole day.** `App_Database::fetchOne()` (like `fetchRow()` / `numRows()`) takes its result **by reference** (`&$res`, `library/app/database.php` L336/L352/L362). TRUE-82027 (commit `b63fb0136`) passed `App_Database::query(...)` into it **inline**, which raises the notice *"Only variables should be passed by reference"* — and 1.0's error handler **terminates the run on a notice**. Live from the 2026-09-18 10:18 deploy until fixed. **Fix:** assign the `query()` result to a variable first. Symptom set worth recognizing: zero ServiceNow calls after the deploy, `Common.CronJobExecutions.dtCheckOut` NULL on every run, and exactly **one** `NYCDOETickets.dtLastChecked` stamp per 6-minute run. General 1.0 trap — see the 1.0 back-end standard.
 - **⚠ "The outbound cron re-asserts On Hold every run" (business rule 2) was only true in theory between 2026-09-18 10:18 and the TRUE-82129 fix** — the cron was dying on its first ticket. Check the cron actually completes before trusting that guarantee.
-- **OPEN QUESTION — `dtSynced` is not a reliable outbound-only signal.** The 12 reported incidents (INC2298009, INC2298110, INC2297667, INC2299269-77) had `Common.NYCDOETickets.dtSynced` advance from ~2026-09-17/18 05:00–09:46 to ~08:57–09:00 on 2026-09-18 **while the outbound cron was not running**. Suspected `process_tickets.php` (inbound also writes `dtSynced`) but **not confirmed**. Confirm before reading `dtSynced` as proof of an outbound push.
-- **⚠ SECURITY — hardcoded live ServiceNow credentials in `library/app/api/nycdoev2.php` / `nycdoe.php`.** Found 2026-09-18, not fixed; detail in [ServiceNow / ASN Integration](servicenow-integration.md).
+- **OPEN QUESTION — `dtSynced` is not a reliable outbound-only signal.** The 12 reported incidents (INC2298009, INC2298110, INC2297667, INC2299269-77) had `Common.NYCDOETickets.dtSynced` advance from ~2026-09-17/18 05:00–09:46 to ~08:57–09:00 on 2026-09-18 **while the outbound cron was not running**. Suspected `process_tickets.php` (inbound also writes `dtSynced`) but **not confirmed**. More evidence (2026-09-25): some rows have `dtSynced` LATER than their newest note (INC2244995: `dtSynced` 2026-09-25 06:20 vs newest note 09-21 16:04), which the outbound note watermark alone cannot produce. Confirm before reading `dtSynced` as proof of an outbound push.
+- **⚠ SECURITY — hardcoded live ServiceNow credentials in `library/app/api/nycdoev2.php` / `nycdoe.php`.** Found 2026-09-18, not fixed; a test-repo diagnostic script also embeds the same credential; detail in [ServiceNow / ASN Integration](servicenow-integration.md).
+- **KNOWN REMAINING GAP (parked, TRUE-82332) — `u_technician` re-pushed with no change.** The technician email comparison appears never to match, so passes with nothing new still PATCH `u_technician` alone (INC2131474: two such PATCHes at 16:06:30 / 16:06:31). Wasted calls; not fixed.
 - **KNOWN REMAINING GAP (needs a follow-up ticket):** both `qqStatus()` hold gates live inside the `type IN (ONSITE_REPAIR, ONSITE_SERVICE)` branch. The `DEPOT_REPAIR` branch (`repairorder.php` from ~L799) has **no hold gate at all**, so a `DEPOT_REPAIR` order set to any hold status is still silently recomputed out of hold by `updateStatus()` — same class of bug, untouched by TRUE-80060.
 - **KNOWN REMAINING GAP (needs a follow-up ticket) — inbound note reconciliation is paged too small.** `process_tickets.php`'s inbound comment reconciliation fetches only `page_size=50, page_number=1` of the SNOW journal, then **deletes any local note not in that set** (the remainder-delete ~L1326–1330). An order with **>50** comments/work_notes can have local notes silently deleted. It is also inconsistent with `addNotes`, which fetches `page_size=100`.
 - **KNOWN REMAINING GAP — `buildDoubleFieldArray` collapses unsynced notes.** In `process_tickets`, `buildDoubleFieldArray` keys on `IFNULL(referenceId,0)`, so **all** NULL-`referenceId` notes collapse to key `'0'`, making the remainder-delete handle multiple unsynced notes inconsistently.
 - `qqStatus()` collapses all recognized hold sub-statuses to base `HOLD` in `repair_orders.status`; the specific hold **sub-reason persists only in `repair_order_notes.newstatus`**. Any status badge / report / filter reading `repair_orders.status` directly cannot distinguish the six hold variants after a recompute.
 
 ## Change history
+- 2026-09-25 — TRUE-82332: hold-release no longer overwrites the close `state`; close re-sent when SN not closed; stuck-ticket re-queue recipe; `Logs.API` query shapes. (mhammontree)
 - 2026-09-18 — TRUE-82129: fixed the by-reference `fetchOne()` notice that killed every run; batch loop + run caps replaced LIMIT 1 + per-ticket sleep; alert emails de-PII'd and escaped; unmapped-group email now grouped per run. (mhammontree)
 - 2026-09-17 — TRUE-82027: outbound push runs once per order per pass; `dtLastChecked` split from `dtSynced`; failure alerting instead of a throw. (mhammontree)
 - 2026-08-20 — TRUE-81044: `state` + `hold_reason` + `u_eta` + `u_other_reason_on_hold` pushed together; stopped writing `u_status_task`. (mhammontree)

package/knowledge/clients/nycdoe/features/servicenow-integration.md

@@ -157,7 +157,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 
 ## Gotchas
 
-- **⚠ SECURITY (cso review 2026-09-18 — NOT fixed, needs its own ticket): live production ServiceNow credentials are hardcoded in committed PHP.** `library/app/api/nycdoev2.php` and `library/app/api/nycdoe.php` hold the ServiceNow **`clientSecret` and `refreshToken` as literals** (stage values too). Treat as compromised: move them to config/env and **rotate with DOE**. Separately, `Vision_Log.ErrorLog` rows dump `$_SERVER` including live AWS access key id / secret in plain text — its own ticket, same rotation treatment. **No credential values are recorded here or anywhere in the KB.**
+- **⚠ SECURITY (cso review 2026-09-18 — NOT fixed, needs its own ticket): live production ServiceNow credentials are hardcoded in committed PHP.** `library/app/api/nycdoev2.php` and `library/app/api/nycdoe.php` hold the ServiceNow **`clientSecret` and `refreshToken` as literals** (stage values too). A test-repo diagnostic script (committed 2026-06-18) embeds the same prod credential — one rotation covers both. Treat as compromised: move them to config/env and **rotate with DOE**. Separately, `Vision_Log.ErrorLog` rows dump `$_SERVER` including live AWS access key id / secret in plain text — its own ticket, same rotation treatment. **No credential values are recorded here or anywhere in the KB.**
 
 - **⚠ OPEN BUG / NAMED PATTERN — "Lenovo Off-Layout ASN File" (silent column shift + unit collapse).** Investigated 2026-08-11 (read-only; **no code fix exists**). Lenovo sends NYCDOE ASNs in **two different column layouts**, and `library/app/asnprocessor/lenovo.php` picks its column mapping by **column COUNT with no header-name validation** (`:110`, `if ($colCount >= 30`), so the off-layout file is parsed with the wrong mapping and silently mis-read.
   - **The two layouts.** The normal daily feed `DOE-Lenovo-ASN-MM-DD-YYYY.CSV` (~2 MB) has **30** columns and matches the code's mapping. A second file, `DOE-Open-Lenovo-MM-DD-YYYY.csv` (**lowercase** `.csv`, ~45 KB), has **31** columns in a **different order**: `Contact Email` moves from column 29 to column **6**, and a trailing `Full Shipment` column is appended. Everything from `Street Address` onward is therefore read **one column too early**. The "extra comma" repair heuristic at `:119-127` does not catch this (unchanged since March 2026 — commits `3442ddcf`, `6956dfbf`, `864cf073`, so not a regression).
@@ -240,6 +240,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 
 - **The queue `dedupeKey` format is load-bearing.** A 2026 incident (WR260236464 duplicate repair order) was caused by dedupeKey format drift (4-part keys with serial vs 3-part without) letting old rows re-import; Stage 2 then created a sibling item row (frozen original not reusable), and Stage 5's per-item-row serial scoping duplicated the unit → duplicate RO. Any change to the key format needs a backfill or transitional matcher. Forensic fingerprint: a Unit with `trackingNumberId IS NULL` was added by Stage 5 from NetSuite, not by the importer.
 - **Two dedupe strategies coexist:** SFTP path = INSERT+catch (race-safe); API path = check-then-insert (unique index backstop).
+- **Held→complete incidents never resolving in ServiceNow** = the TRUE-82332 close clobber; mechanism, fix and re-queue recipe in [Hold-Status Sync](hold-status-sync.md).
 - **Timezone traps:** worker runs `America/Chicago`; `repair_orders.dtEta` is stored **Eastern**; ServiceNow `u_eta` is sent as **UTC** `YYYY-MM-DD HH:MM:SS` (converted via `App_Date::convertTimezone()`, only sent when first 19 chars differ from remote).
 - **EDI PO104 leading-zero suppression** (go-live 2026-06-07): DOE's IBM-based translator sends `.94` instead of `0.94` — exposure is in `App_Edi::translatePurchaseOrder850()`, not the cron.
 - **Installation schedule reports skip weekends but not holidays** — empty output near a holiday is expected, not a bug. Empty also just means no ROs with `dtEta` on the exact target date.
@@ -261,4 +262,5 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
 - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
 
 ## Change history
+- 2026-09-25 — TRUE-82332: cross-link to the close-clobber fix; test-repo copy of the ServiceNow credential noted (mhammontree)
 - 2026-09-25 — Corrected "no 2.0 tenant" (TRUE-80519 2.0 ASN rewrite now exists); DOE-Open-Lenovo marked retired (mhammontree)

package/knowledge/clients/nycdoe/profile.md

@@ -37,9 +37,8 @@ NYC DOE — TOGA client whose **live** integration runs in the **1.0 worker tier
 ## Endpoints
 | System | Endpoint | Notes |
 |---|---|---|
-| ServiceNow prod | `https://nycdoeprod.service-now.com` | Incidents + RITMs |
-| ServiceNow stage | `https://nycdoedev2.service-now.com` | used when `App_Registry::inTestMode()` |
-| ServiceNow disposals | `https://supportadmin.schools.nyc` (stage `nycdoedev3`) | disposal service requests |
+| ServiceNow prod (live) | `https://supportadmin.schools.nyc` (stage `nycdoedev3.service-now.com`) | `App_Api_NYCDOEV2` — all live INC/RITM GET/PATCH under `/api/nycd3/`, incl. disposals; V1 `App_Api_NYCDOE` uses the same prod host (stage `supportadminstg.schools.nyc`) |
+| ServiceNow legacy | `https://nycdoeprod.service-now.com` (stage `nycdoedev2`) | only the dormant `download_tickets*.php` — not live |
 | DOE EDI SFTP | `transfer.schools.nyc` (SSH key `doedimedu.pem`) | EDI 850 in, invoices out |
 | Vendor SFTP feeds | Apple / Lexmark / Lenovo (Lenovo: `ftp.asisystem.com`) | legacy ASN file ingestion |
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.899",
+  "version": "1.0.900",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",