toga-ai 1.0.749 → 1.0.751

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,8 +6,8 @@ project: Library
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-10
10
- owners: ["bala"]
9
+ updated: 2026-09-02
10
+ owners: ["bala", "jcardinal"]
11
11
  files:
12
12
  - library/app/systemmonitor/netsuiteintegration.php
13
13
  - worker/crons/infrastructure/system_monitors.php
@@ -104,8 +104,22 @@ that worker job does the syncing, this library class notices when it stops.
104
104
  checkpoints mean anything; and a client that is held / never transacts supply (AIG) must be
105
105
  left **off** this list or it produces permanent false-positive tickets.
106
106
 
107
+ - **⚠ The pending cursor-rename in the sync will SILENTLY BLIND this monitor (BUILT 2026-09-02, not yet
108
+ deployed).** The per-record resume cursor work on the sync renames each section's position key from
109
+ `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` to `NETSUITE_LAST_SYNC_CURSOR_<SECTION>` (value becomes
110
+ `"<lastmodifieddate>|<netsuite id>"`). This monitor's `NETSUITE_SYNC_PARAMETER_KEYS` still lists the old
111
+ `DATETIME` keys, so after the rename its `where..in` `/parameters` read matches **zero rows** and it
112
+ reports OK forever — no stale-checkpoint alerts. The sync developer **intentionally left the monitor
113
+ un-updated** this session (accepted gap). When the rename deploys, update `NETSUITE_SYNC_PARAMETER_KEYS`
114
+ to the `CURSOR` names **and** parse the `<date>|<id>` value (`strtotime` the part before `|`) before the
115
+ 48h compare. See the [per-client sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md).
116
+
107
117
  ## Change history
108
118
 
119
+ - 2026-09-02 — Flagged that the sync's pending `NETSUITE_LAST_SYNC_DATETIME_*` →
120
+ `NETSUITE_LAST_SYNC_CURSOR_*` key rename (BUILT, not yet deployed) will make this monitor match zero rows
121
+ and stop alerting; it was intentionally left un-updated. Update the key list + parse the `<date>|<id>`
122
+ value when the rename ships. (jcardinal)
109
123
  - 2026-07-10 — **Fixed the monitor timing out and silently not running since 2026-06-29.**
110
124
  Rewrote `getClientStatus()` to read all six checkpoints per client in one TOGa2
111
125
  `where..in` `/parameters` call (was 84 serialized calls ≈ 230s, over the 300s
@@ -10,7 +10,7 @@
10
10
  | [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
11
11
  | [Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers](features/forecast2-supporting-records-import.md) | The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`, `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employ | worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, worker/crons/sync/netsuite/netsuite_customers.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php, library/app/model/forecast2/consolidatedcustomer.php |
12
12
  | [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
13
- | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
13
+ | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/query.php, library/app/database.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
14
14
  | [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
15
15
  | [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php, worker/ebs/cron.worker.php, worker/.ebextensions/045_oneuptime_agent.config |
16
16
  | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
@@ -16,8 +16,11 @@ files:
16
16
  - dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql
17
17
  - library/app/api/toga2.php
18
18
  - library/app/api/netsuite/rest.php
19
+ - library/app/query.php
20
+ - library/app/database.php
19
21
  - library/app/framework.php
20
22
  - library/app/systemmonitor/netsuiteintegration.php
23
+ - dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql
21
24
  - test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php
22
25
  - test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php
23
26
  related:
@@ -89,6 +92,28 @@ its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in
89
92
  never *tune the window size down*. (Corollary: a window that has grown back to the cap is positive
90
93
  evidence the section is completing.)
91
94
 
95
+ ### Per-record resume cursor — checkpoint after EACH record, not window-end (BUILT 2026-09-02, NOT yet deployed)
96
+
97
+ The historical weakness of window-end-only checkpointing: a mid-window crash (poison record, or the
98
+ log-DB doom loop above) re-processes the **whole** window every run — the giant-fulfillment doom loop is a
99
+ direct consequence. This change converts the checkpoint to **per-record keyset (seek) resume** so a
100
+ transient crash resumes at the next record instead of redoing the window.
101
+
102
+ - **Key rename per section:** `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` → `NETSUITE_LAST_SYNC_CURSOR_<SECTION>`,
103
+ value format **`"<lastmodifieddate>|<netsuite internal id>"`**. The id breaks same-second ties.
104
+ - **Fetch change (`library/app/api/netsuite/rest.php`, all 6 header queries):** add
105
+ `ORDER BY lastmodifieddate, id` and change the lower bound to
106
+ `lastmodifieddate > :t OR (lastmodifieddate = :t AND id > :fromId)` (upper bound `<= :to` kept). A new
107
+ `$fromId` param was threaded into the 6 list funcs. Added `lastModifiedDate` to the fetched record object
108
+ for **PURCHASE_ORDERS, ITEM_RECEIPTS, ITEM_FULFILLMENTS, INVENTORY_ADJUSTMENTS** (SALES_ORDERS + INVOICES
109
+ already carried it).
110
+ - **Save after EACH main record** (one PUT of `lastmod|id`); on a clean window with 0 records, PUT
111
+ `"<windowTop>|0"` so empty date ranges still advance. No new `try/catch` — stays fail-loud.
112
+ - **Migration is MANDATORY with the code (see the reset trap in Gotchas):**
113
+ `dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql` (Client fan-out) renames the keys **and**
114
+ appends `|0` to carry the value across; safe to re-run. The new-client seed module
115
+ (`dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL`) was updated to CURSOR.
116
+
92
117
  ### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
93
118
 
94
119
  Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
@@ -305,6 +330,10 @@ Per-client `Parameters` table (2.0 client DB, e.g. `Client_Canon.Parameters`; co
305
330
 
306
331
  Parameters are stored **per client DB** but accessed **through the TOGa2 API**, not direct SQL.
307
332
 
333
+ **Pending rename (BUILT 2026-09-02, not yet deployed):** the position keys become
334
+ `NETSUITE_LAST_SYNC_CURSOR_<SECTION>` with value `"<lastmodifieddate>|<netsuite internal id>"` — see the
335
+ per-record resume cursor in *How it works*. The `NETSUITE_EXECUTION_MODE_*` keys are unchanged.
336
+
308
337
  ## Client variations
309
338
 
310
339
  - **Transfer-order routing is opt-in per wrapper (default OFF).** A client whose NetSuite
@@ -856,6 +885,15 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
856
885
  line). **Fix (`library`):** skip lines whose `adjustQtyBy` is null in **BOTH** passes over
857
886
  `inventoryList->inventory` (the item-build pass and the serial/unit pass). Shared fix; unblocked NYCHH
858
887
  inventory adjustments.
888
+ - **Prod-confirmed 2026-09-02** (commit `8bae8a2a`, 2026-08-29; the null-skip lands at
889
+ `toga2.php:5728` and `:5879`). Before it, NYCHH `INVENTORY_ADJUSTMENTS` was frozen `3-IDLE` (window
890
+ collapsed to ~3s), watermark stuck at **2022-12-28**, advancing only seconds/run — `Logs.Issue`
891
+ **542** (`clientId 3`), 35 occurrences 2026-08-28..09-01. The fix reached prod ~2026-09-01 afternoon
892
+ and **Issue 542 stopped 09-01 14:08**. This `3-IDLE`-crawl-with-frozen-watermark signature is the same
893
+ poison-record shrink as a `1-RUNNING` freeze — a section that throws on a record too old to fall
894
+ outside the collapsed window keeps re-throwing; once the window shrinks past that record's timestamp it
895
+ reports IDLE but crawls seconds/run and can never advance past it. Read it as "still throwing," not
896
+ "small/slow section" (see the item-fulfillment doom-loop gotcha below for the same signature).
859
897
  - **⚠ Item-receipt sync froze on a PARTIAL receipt's NULL-quantity PO lines — the item-receipt TWIN of the
860
898
  inventory-adjustment NULL-quantity fix above (fixed 2026-08-31).** For a partial item receipt NetSuite
861
899
  returns **all** the PO's linked lines but sets `quantity=null` on the lines **not received on this
@@ -877,14 +915,33 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
877
915
  `transactionId` (`Logs.Event` carries `clientId`, `Logs.Issue` does not) → `Logs_Nychh.Api` on that
878
916
  `transactionId` for the untruncated `EV-10`/1048 rejection and the payload showing the `quantity:null`
879
917
  lines.
880
- - **⚠ OPEN, not yet root-caused (found 2026-08-31): NYCHH ITEM_FULFILLMENTS watermark frozen at 2021-12-29.**
881
- `NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS='3-IDLE'` with the window collapsed to ~3s (it has been
882
- throwing recently) but currently **IDLE** — so it crawls a few seconds per run rather than sitting in a
883
- hard `1-RUNNING` freeze. Distinct from the fixed ITEM_RECEIPTS freeze above; needs its own investigation.
884
- Context: the 2026-08-31 full 19-client togasupply prod sweep found all other clients healthy
885
- (`864000-IDLE`) except Elite SALES_ORDERS `864000-RUNNING` (healthy mid-run, large window, not stuck) and
886
- AIG SALES_ORDERS `8-RUNNING` (the known held account, already documented). Only NYCHH ITEM_RECEIPTS was in
887
- a true `1-RUNNING` freeze.
918
+ - **NYCHH ITEM_FULFILLMENTS watermark frozen at 2021-12-29 (found 2026-08-31; ROOT-CAUSED 2026-09-02 —
919
+ the log-DB doom loop below).** `NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS='3-IDLE'` with the window
920
+ collapsed to ~3s (throwing recently) but currently **IDLE** — so it crawls a few seconds per run rather
921
+ than a hard `1-RUNNING` freeze. This is the exact signature of the doom loop in the next gotcha: a giant
922
+ fulfillment throws every run, the window shrinks ÷3 until it no longer spans that fulfillment, then goes
923
+ IDLE but can never advance past it. Context: the 2026-08-31 full 19-client togasupply prod sweep found
924
+ all other clients healthy (`864000-IDLE`) except Elite SALES_ORDERS `864000-RUNNING` (healthy mid-run,
925
+ large window, not stuck) and AIG SALES_ORDERS `8-RUNNING` (the known held account).
926
+ - **⚠ Item-fulfillment DOOM LOOP from a dropped LOG-DB connection during a giant fulfillment (found
927
+ 2026-09-02, NOT yet fixed — the per-record cursor below is the mitigation).** A single very large
928
+ fulfillment POSTs thousands of `/item-fulfillment-item-units` over api2 HTTP; that long unit-POST phase
929
+ lets the **worker's log-DB connection idle past MySQL `wait_timeout`**. The **next** fulfillment's refetch
930
+ — `fetchItemFulfillmentById` at `library/app/api/toga2.php:4440`, the **first line** of
931
+ `syncItemFulfillmentFromNetsuite` — tries to log that NetSuite REST call and dies at
932
+ `library/app/query.php:168` `mysqli_select_db` *"MySQL server has gone away / Connection timed out"*
933
+ (`Logs.Issue` **589**). The 1.0 framework escalates that warning to `exit`, so the run aborts mid-window;
934
+ the watermark only advances at window end (`finishModeIteration`), so the whole window — including the
935
+ giant fulfillment — **re-processes next run, times out at the same spot, and the window shrinks ÷3 each
936
+ run** (→ the `3-IDLE` crawl signature above once the window no longer spans the giant fulfillment).
937
+ - **Root of the crash:** the "gone away" recovery in `App_Query::execute` (`library/app/query.php`,
938
+ `database.php:431`) only guards the `mysqli_query` call (errno **2006** reconnect), **NOT** the
939
+ `mysqli_select_db` that runs before it — which is the first touch of the stale connection, so the
940
+ reconnect never gets a chance. A durable framework gap, not sync-specific.
941
+ - **Durable engine rule reconfirmed:** a shrinking window in this sync **ALWAYS** means a thrown
942
+ exception — there is **no** volume/time/memory backoff anywhere in the script. "Window not 864000" ⇒
943
+ something threw. The log-connection idle-out is a concrete cause to add to the poison-record and
944
+ shim-shape causes above.
888
945
  - **⚠ Item-fulfillment stale-item prune must delete tracking-number children before the unit AND the item
889
946
  (fixed 2026-08-30).** In `syncItemFulfillmentFromNetsuite`'s cleanup loop, deleting a stale
890
947
  item-fulfillment-item-**unit** failed api2 `EV-11` / MySQL **1451** (FK
@@ -967,8 +1024,51 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
967
1024
  A "silent freeze" here is a serial/data collision, not necessarily a catalog/item-matching problem —
968
1025
  read the untruncated Api-log rejection before assuming item matching.
969
1026
 
1027
+ - **⚠ The position key defaults to the EARLIEST date when MISSING — a code-only cursor rename WITHOUT
1028
+ the data migration resets every client to re-sync from scratch (found 2026-09-02).** When the section's
1029
+ position key is absent, the engine falls back to `MIN_DATETIME_TO_CHECK_FOR_NETSUITE_DATA`
1030
+ (`common_sync_togasupply.php` ~L573). So shipping the `NETSUITE_LAST_SYNC_DATETIME_*` →
1031
+ `NETSUITE_LAST_SYNC_CURSOR_*` rename in **code only** makes every client's old key invisible → each
1032
+ section restarts from the earliest date and re-syncs its entire history. The migration
1033
+ (`dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql`, Client fan-out) **must carry the value
1034
+ across** — it renames the key and appends `|0` in one `UPDATE ... REPLACE(...)`, re-runnable. **Ship the
1035
+ migration on every client DB before (or with) the code.**
1036
+ - **⚠ The NetSuite Sync Alert monitor will SILENTLY STOP after the cursor rename (known, accepted
1037
+ 2026-09-02).** `App_SystemMonitor_NetSuiteIntegration` (`library/app/systemmonitor/netsuiteintegration.php`)
1038
+ hardcodes the old `NETSUITE_LAST_SYNC_DATETIME_*` keys, so once the rename ships its `where..in`
1039
+ `/parameters` read matches **zero rows** and it stops alerting on stale checkpoints. The developer
1040
+ **intentionally left the monitor un-updated** this session — a documented, accepted gap. Update the
1041
+ monitor's key list to `NETSUITE_LAST_SYNC_CURSOR_*` (and parse the `<date>|<id>` value) when the rename
1042
+ deploys, or NetSuite sync stalls go unmonitored. Cross-ref:
1043
+ [NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
1044
+
970
1045
  ## Change history
971
1046
 
1047
+ - 2026-09-02 — **Per-record resume cursor BUILT (shared: `worker` + `library` + `dbchanges2`; pending
1048
+ review + deploy, NOT shipped).** Converts the section checkpoint from window-end-only to per-record
1049
+ keyset resume so a mid-window crash (poison record or the log-DB doom loop) resumes at the next record
1050
+ instead of re-processing the whole window. Position key `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` →
1051
+ `NETSUITE_LAST_SYNC_CURSOR_<SECTION>`, value `"<lastmodifieddate>|<netsuite id>"` (id breaks same-second
1052
+ ties); `rest.php`'s 6 header queries gain `ORDER BY lastmodifieddate, id` + a keyset lower bound + a
1053
+ `$fromId` param; cursor PUT after each record (and `"<windowTop>|0"` on an empty window); `lastModifiedDate`
1054
+ added to the PO/IR/IF/InvAdj fetched objects. Migration `Client/2026-09-02b - NetsuiteSyncCursorRename.sql`
1055
+ (fan-out) renames the keys and appends `|0` — **mandatory**, because a missing key defaults to the earliest
1056
+ date and would re-sync every client's full history. Monitor `netsuiteintegration.php` intentionally left on
1057
+ the old DATETIME keys (will stop alerting after the rename — accepted). (jcardinal)
1058
+ - 2026-09-02 — **Item-fulfillment doom loop root-caused (found, NOT fixed — the cursor above mitigates).** A
1059
+ giant fulfillment's long unit-POST phase idles the worker's log-DB connection past MySQL `wait_timeout`; the
1060
+ next fulfillment's `fetchItemFulfillmentById` (`toga2.php:4440`) dies logging its NetSuite call at
1061
+ `query.php:168` `mysqli_select_db` "server has gone away" (`Logs.Issue 589`) → 1.0 warning→`exit` → window
1062
+ re-processes and shrinks ÷3 each run (the `3-IDLE` crawl). Root cause: `App_Query::execute`'s gone-away
1063
+ recovery (`query.php`/`database.php:431`) guards only the `mysqli_query` (2006 reconnect), not the
1064
+ `mysqli_select_db` before it. Explains the NYCHH ITEM_FULFILLMENTS freeze at 2021-12-29. Reconfirmed the
1065
+ engine rule: a shrinking window ALWAYS means a throw (no volume/time/memory backoff exists). (jcardinal)
1066
+ - 2026-09-02 — **Prod-confirmed the NYCHH inventory-adjustment NULL-`adjustQtyBy` freeze + its fix (commit
1067
+ `8bae8a2a`, `toga2.php:5728`/`:5879`).** `INVENTORY_ADJUSTMENTS` had been frozen `3-IDLE`, watermark stuck
1068
+ 2022-12-28, crawling seconds/run — `Logs.Issue 542` (`clientId 3`), 35 occ 2026-08-28..09-01, EV-10/MySQL
1069
+ 1048 "quantityAdjustBy cannot be null" on NetSuite serialized-detail sub-lines. The null-skip reached prod
1070
+ ~2026-09-01 afternoon; Issue 542 stopped 09-01 14:08. Recorded that `3-IDLE`-crawl = the same poison-record
1071
+ shrink as `1-RUNNING`, not a slow section. (jcardinal)
972
1072
  - 2026-09-02 — **PURCHASE_ORDERS cleanup DELETE freeze fixed (shared, `library` only — WRITTEN +
973
1073
  REVIEWED, NOT yet committed/deployed).** `syncPurchaseOrderFromNetsuite`'s "remove dropped items"
974
1074
  cleanup deleted a NetSuite-dropped PO line with a bare throwing DELETE; a leftover item receipt
@@ -0,0 +1,110 @@
1
+ ---
2
+ type: session
3
+ slug: talos-surface-layer
4
+ title: Talos moved onto the surface layer; chat history, titles, launcher dim
5
+ author: apeterson
6
+ repos: [toga25-supply, toga-blox, dbchanges2]
7
+ framework: "2.0"
8
+ client: nychh
9
+ status: active
10
+ created: 2026-09-02
11
+ updated: 2026-09-02
12
+ ---
13
+
14
+ # Session: talos-surface-layer
15
+ **Date:** 2026-09-02
16
+ **Project/Repo:** toga25-supply + toga-blox + dbchanges2 (2.0)
17
+ **Task:** Move the Talos AI assistant (feature gate + suggestion chips) off hardcoded frontend config onto the backend surface layer — off for all clients in Core, granted only to NYCHH — and finish the chat history / auto-title / launcher work started earlier in the same thread.
18
+
19
+ ---
20
+
21
+ ## What WORKED
22
+
23
+ - **Loading bubble restored in host-render mode.** `toga-blox src/components/Talos/TalosPanel.tsx` now renders the synthetic thinking bubble in BOTH modes, because it is chrome derived from `isStreaming`/`progress`, not a message. Evidence: 2 new regression tests; blox Talos suite 31 passing.
24
+ - **Chat history drawer + new chat, data-driven.** New controlled props `threads`, `activeThreadId`, `onSelectThread`, `onNewChat`, `threadsStatus`. `history` re-enables only when `threads` AND `onSelectThread` are both given; `newChat` only when `onNewChat` is. The panel sorts rows newest-first. Evidence: tests incl. ordering and "host content replaces the list".
25
+ - **Opening an existing thread needs no transcript parser.** Verified in `node_modules/@langchain/langgraph-sdk/dist/ui/orchestrator.js` — `setThreadId` calls `#fetchHistoryForThread` and clears the stream when `fetchStateHistory: true`. Selecting a row is just "set threadId"; the existing LangGraph→blox mapper handles the rest.
26
+ - **Auto-title polling.** `toga25-supply src/hooks/useTalosThreads.ts` `titlePollInterval` — 4s `refetchInterval` while any recent thread is unnamed, `false` otherwise, with a 2-minute age window. Evidence: 7 tests incl. server-clock-ahead and the never-named case.
27
+ - **"Couldn't load your chats." over a full list fixed.** `threadsStatusFrom` returns "error" only when there is no data; blox gates all three drawer notes on `histRows.length === 0`. Evidence: regression test in each repo.
28
+ - **Talos on the surface layer, RUN AND VERIFIED ON LOCAL.** `dbchanges2 Core/2026-09-02a - TalosAssistantSurfaceSeed.sql` → 2 elements, both `isVisible = 0`, suggestions row `itemCount = 4`. `dbchanges2 Client_Nychh/2026-09-02a - TalosAssistantEnable.sql` Section A → 3 rows: two `IS_VISIBLE='1'` + one `CONFIG` with `itemCount = 3` and `roleRestated = 'suggestion'`. Three NYCHH chips render in the browser. Developer confirmed: "that workds".
29
+ - **Fail-closed FE gate.** `src/hooks/useTalosSurface.ts` — missing bundle, failed fetch, absent marker, or still-loading all resolve to disabled. `TalosStreamProvider` gained an `enabled` prop so a tenant without Talos never calls `/assistants`. 14 tests.
30
+ - **Launcher dims under a modal.** `toga-blox TableRecordModal.tsx` sets `data-blox-modal-open` on `<html>` (a counter); `Talos.module.css` dims `.launcher` and stops the orbit ring under `:root[data-blox-modal-open]`.
31
+ - **Surface bundle no longer cached to disk.** `src/App.tsx` `shouldDehydrateQuery` excludes the `talos`, `surface-meta` and `surface-meta-group` namespaces.
32
+ - **End state:** toga25-supply 87 tests / 12 files passing; `npx tsc --noEmit -p tsconfig.app.json` clean apart from one pre-existing error; toga-blox 31 Talos + TableRecordModal tests passing; blox `dist/` rebuilt.
33
+
34
+ ## What did NOT work — DO NOT RETRY THESE
35
+
36
+ - **A host-render branch that bypasses the panel's own message list.** The first `renderMessages` implementation replaced `msgs` entirely, which took the synthetic thinking bubble with it. Symptom: submit a question and get **zero** feedback until the answer appears — no loading UI, no streaming. The bubble is chrome, not conversation; never let a host slot remove it.
37
+ - **Trusting supply's typecheck to prove a blox change reached the browser.** `npx tsc --noEmit -p tsconfig.app.json` resolves blox **types from source** through the `node_modules/@agilant/toga-blox` symlink, but Vite loads the package at **runtime from `dist/`**. Typecheck was clean while the browser ran stale blox code. Always `npm run build` in blox + `rm -rf node_modules/.vite` in supply.
38
+ - **The earlier note that blox `npm run build` fails on a missing `tsconfig.app.json` is WRONG.** It builds fine (`tsc && copyfiles … && fix-esm-imports && fix-asset-imports`). Do not route around it.
39
+ - **`status: isError ? "error" : …` for a React Query list.** React Query sets `isError` when a **background** refetch fails while KEEPING the data it already had. With the 4s title poll running, one failed poll printed "Couldn't load your chats." above a full, usable list. Report error only when there is no data.
40
+ - **A drawer error message with no empty-list guard.** It rendered *alongside* the rows, producing a screen that says loading failed while showing the loaded data.
41
+ - **Building the title polling before checking that ANY thread had a title.** Every NYCHH thread reads "New conversation", including ones days old and far outside any poll window — so this was never a client timing bug. Naming is a per-assistant setting (`thread_name_generation_enabled`, default `true`, plus `thread_name_model`) on `GET`/`PATCH /api/assistants/{assistant_id}/config`. The polling is correct and will work the moment the server writes a name, but it could not fix this.
42
+ - **A cross-schema `JOIN Core.SurfaceElements` in a `Client_*` file.** Works on local/beta, **fails in production** — Core and Client_* are on separate clusters. Error there: `Table 'Core.Surfaces' doesn't exist`. Prod needs the read-ids-on-Core → insert-literals-on-client two-step.
43
+ - **Schema-qualifying a Core-only migration (`Core.Messages`, `Core.Surfaces`).** api2 can rename the Core DB via `[database] core_dbname`, so qualifying breaks exactly the environment that renamed it. Core-only files must use UNQUALIFIED names and be run while connected to Core.
44
+ - **Running an unqualified `SurfaceElements` query while connected to `Client_Nychh`.** Error: `Table 'Client_Nychh.SurfaceElements' doesn't exist`. `SurfaceElements`/`Surfaces`/`Messages` are **Core-only**; a cross-schema verify must qualify every Core table.
45
+ - **Assuming the launcher-not-dimmed bug was z-index.** It was not. `.launcher` has no z-index, blox `.header` has no z-index or position, supply's `.headerActions` has neither, and `.backdrop` is `position: fixed; z-index: 40` — so the launcher was already painted under the backdrop and its clicks already swallowed. The real cause: `.backdrop-right`/`.backdrop-left` is only `rgba(0, 0, 0, 0.2)`, and a 20% black wash over a **white** button still reads as active. Do not go looking for a z-index.
46
+ - **One surface element per suggestion chip.** Replaced before it shipped past local. A tenant could only switch on questions **Core had already seeded**, so changing a tenant's wording needed a new Core migration for every phrasing.
47
+ - **A bare JSON array as a `CONFIG` override value.** A `CONFIG` override **replaces** `config` wholesale — `_underscore Model/Core/Surface.php:839` assigns, it does not `array_merge`. A bare array passes the `is_array()` check and wipes `config.role`, so the frontend stops recognising the element and the chips **silently disappear with no error anywhere**. The override must restate `"role":"suggestion"`.
48
+ - **Putting override JSON in `SurfaceOverrides.value`.** It is `varchar(255)`; any real chip set overflows it. Use `c_longValue` — the resolver falls back to it when `value` is NULL/'' (`Surface.php:907`).
49
+
50
+ ## Not tried yet (candidates for next session)
51
+
52
+ - **Run the SQL on beta and prod.** Local only so far. Prod = Section B (read ids on the Core cluster) then Section C (literal-id insert on the client cluster). **Never Section A in prod.**
53
+ - **Wire Stop to `cancelTalosRun`** instead of `stream.stop()`. This is a **real defect**: `stream.stop()` only drops the client transport while the run keeps executing and billing. Needs `run_id` from the `metadata` SSE event (handoff §8.5).
54
+ - **`fromTalosTranscript()` + terminal-completion reconciliation** (handoff §8.11 / §8.8). Message ids in `src/providers/TalosRuntimeProvider.tsx` are `talos-${index}` placeholders; feedback needs a durable server `message_id` + `run_id`.
55
+ - **Message actions** (copy, thumbs up/down) via `ActionBarPrimitive`, and the thumbs-down dialog (handoff §9). Blocked on the durable ids above; copy alone is doable now.
56
+ - **Rename / delete thread UI.** `renameTalosThread` / `deleteTalosThread` already exist in `src/api/talos.ts` and are unused. Both endpoints are real (`PATCH`/`DELETE /threads/{thread_id}`). Decide the `If-Match` question first: the handoff doc shows it quoted, the live spec types it as a plain optional string, and our client sends it bare — recommendation was to **drop the header** rather than guess.
57
+ - **Make supply's header `inert` while a modal is open.** The launcher is still keyboard-reachable behind a modal; `pointer-events: none` stops the mouse only. Affects all header controls, so it is a deliberate supply-wide change.
58
+ - **Rule on reserved-literal ids vs `AUTO_INCREMENT`** for Core surface seeds. `knowledge/2.0/apps/dbchanges2/features/surface-layer-schema.md` currently documents BOTH branches. Note the shipped `Core/2026-08-12a` navigation seed already uses `AUTO_INCREMENT`, so the "never AUTO_INCREMENT" rule was already contradicted by merged code.
59
+ - **Per-role Talos gating / per-role chip sets.** Current grants are client-wide (`roleId` NULL) — every NYCHH user incl. devteam gets Talos. Two role-scoped `CONFIG` rows would resolve by `ORDER BY (roleId IS NULL) DESC, (personaId IS NULL) DESC, id ASC`, so the **higher id wins arbitrarily** for a multi-role user.
60
+
61
+ ## Current file state
62
+
63
+ | File | Status | Notes |
64
+ |------|--------|-------|
65
+ | `dbchanges2 Core/2026-09-02a - TalosAssistantSurfaceSeed.sql` | new, uncommitted, RUN ON LOCAL | Surface `talos-assistant` (DRAWER, not record-bound), 2 elements both `isVisible=0`; suggestions element holds Core's default 4 items. Unqualified table names. Teardown in footer. |
66
+ | `dbchanges2 Client_Nychh/2026-09-02a - TalosAssistantEnable.sql` | new, uncommitted, RUN ON LOCAL (Section A only) | 3 sections: A single-cluster, B read ids on Core, C literal insert on client. Writes 2 `IS_VISIBLE` grants + 1 `CONFIG` with NYCHH's 3 questions. |
67
+ | `toga25-supply db-migrations/PLAYBOOK.md` | modified | 2026-09-02 change-log entry incl. prod cross-cluster procedure and the `SurfaceOverrides` NULL-unique-key gotcha. |
68
+ | `toga25-supply src/hooks/useTalosSurface.ts` (+ `.test.ts`) | new | Fail-closed gate + tolerant `readItems` parser. 14 tests. |
69
+ | `toga25-supply src/hooks/useTalosThreads.ts` (+ `.test.ts`) | new | Thread list, `titlePollInterval`, `threadsStatusFrom`, `toThreadSummary`. 12 tests. |
70
+ | `toga25-supply src/providers/TalosStreamProvider.tsx` | modified | `enabled` prop; threads/selectThread/activeThreadId in context; refresh on threadId assignment + run-end. |
71
+ | `toga25-supply src/providers/TalosRuntimeProvider.tsx` | new (earlier in thread) | assistant-ui `ExternalStoreRuntime` bridge. Message ids are index placeholders. |
72
+ | `toga25-supply src/components/TalosThread/` | new (earlier in thread) | assistant-ui thread + safe markdown components. |
73
+ | `toga25-supply src/hooks/useTalosSession.ts` | modified | Takes `enabled`; returns `disabled` when gated off. |
74
+ | `toga25-supply src/layout/AppLayout/AppLayout.tsx` | modified | Gates launcher + panel on `useTalosSurface().isEnabled`; passes surface suggestions, threads, history handlers. |
75
+ | `toga25-supply src/App.tsx` | modified | `shouldDehydrateQuery` excludes `talos`, `surface-meta`, `surface-meta-group`. |
76
+ | `toga-blox src/components/Talos/TalosPanel.tsx` | modified | `renderMessages`, thread-list props, loading bubble in both modes, `NO_SUGGESTIONS` default (DEFAULT_SUGGESTIONS deleted). |
77
+ | `toga-blox src/components/Talos/types.ts` | modified | `TalosThreadSummary`, thread props, `renderMessages`. |
78
+ | `toga-blox src/components/Talos/Talos.module.css` | modified | `.historyNote`; launcher dim under `:root[data-blox-modal-open]`. |
79
+ | `toga-blox src/components/Talos/README.md` | modified | Suggestions are host-supplied; chat-history section. |
80
+ | `toga-blox src/components/TableRecordModal/TableRecordModal.tsx` | modified | `data-blox-modal-open` counter on `<html>`. |
81
+ | `toga-blox src/components/TableRecordModal/tableRecordModal.module.css` | modified (earlier in thread) | `--talos-dock-width` awareness. |
82
+ | `toga-blox dist/` | rebuilt | Required for the browser; gitignored. |
83
+
84
+ **Nothing is committed in any of the three repos.**
85
+
86
+ ## Decisions made
87
+
88
+ - **Talos is gated by the surface layer, OFF for every client in Core.** Rationale: Talos answers questions from a tenant's live Supply data through an external AI service; a tenant must not inherit that from a migration. Rejected: a build-time env flag (not per-tenant) and defaulting ON (a data-exposure decision made by a file instead of a person).
89
+ - **The feature switch is a marker element, not `Surfaces.isVisible`.** The override cascade is keyed by `surfaceElementId` and is never applied to a Surfaces row's own `isVisible`, so overriding the surface row silently does nothing. Same trick as the `config.role='sectionVisibility'` markers.
90
+ - **Chips are ONE element with a `config.items` array of `{label, prompt, icon}`.** Rationale: per-chip elements let a tenant only enable questions Core had already seeded. Rejected: one element per chip (needs a Core migration per phrasing) and an array of bare strings (collapses label/prompt/icon and loses the icons). Trades accepted: labels are not `Messages` keys so `MessageTranslations` cannot translate them; and chip SETS are per-scope rather than per-chip.
91
+ - **The chip payload lives in `config`, not an `Actions` row with `type='RUN_TALOS'`.** `RUN_TALOS` is in the enum but **no `Actions` row has ever been seeded in dbchanges2** and there is no FE action-registry entry for it. `config` is emitted verbatim into the bundle. Same trade `Core/2026-08-12a` made choosing `config.path` over `Actions`/`NAVIGATE`.
92
+ - **`AUTO_INCREMENT` + `NOT EXISTS` guards on natural keys, not a reserved literal id block.** Reserving requires verifying `MAX(id)` against every environment first; these files are purely additive and id-agnostic. Matches the `navigation` seed. Rejected: reserved literals (unverifiable without DB access, and the ids are read off Core at run time in Section B anyway).
93
+ - **The per-client file keeps `Client_Nychh.` qualification while the Core file drops `Core.`.** Run against the wrong cluster the qualified one fails loudly on a missing schema instead of silently granting the feature to another tenant; the Core schema name is not guaranteed.
94
+ - **The `NOT EXISTS` guard is load-bearing, not decoration.** `SurfaceOverrides`' unique key is `(surfaceElementId, surfaceId, personaId, roleId, languageId, attribute)` and MySQL treats NULLs in a unique key as **distinct** — client-wide grants have persona/role/language all NULL, so the index accepts duplicates.
95
+ - **Permission/feature surfaces are not persisted to localStorage.** `surface-meta` is fetched with `staleTime: Infinity`, so a persisted copy is never refetched and a grant — or a **revoke** — would not reach a browser for up to the persister's 24h default `maxAge`.
96
+ - **The modal-open marker is a root-element attribute and a counter.** An attribute because the launcher renders outside the modal's portal so no React state reaches it (same boundary-crossing trick as `--talos-dock-width`); a counter because modals stack and the marker must survive the inner one closing. Rejected: threading a prop from every modal layout.
97
+ - **`config.items` parsing is tolerant, not strict.** `config` is a hand-edited JSON column that differs per tenant, so one bad item costs one chip rather than every chip or a render crash.
98
+
99
+ ## Blockers
100
+
101
+ - **Thread auto-titles are broken server-side.** Every NYCHH thread reads "New conversation". Naming is per-assistant: `thread_name_generation_enabled` (default `true`) + `thread_name_model` on `GET`/`PATCH /api/assistants/{assistant_id}/config`. Either it is `false` on the `TOGa Supply` assistant or the Bedrock naming model is unreachable in that deployment. **A consumer app cannot fix this.** Open with `akhokhani`; NYCHH `assistant_id` `c7981164-1fe9-5878-b115-85c38dadad43`.
102
+ - **Nothing is committed in toga25-supply, toga-blox or dbchanges2.** blox `dist/` is gitignored, so no other consumer sees any of this until blox publishes.
103
+ - **Beta and prod SQL not run** — the feature is invisible in those environments (the fail-closed gate working, not a regression).
104
+
105
+ ## Exact next step
106
+
107
+ > Wire Stop to `cancelTalosRun` in `toga25-supply src/providers/TalosStreamProvider.tsx` — replace `stream.stop()` in the `stop` callback. It is a real defect: `stream.stop()` drops only the client transport while the run keeps executing and billing on Aegra. You need the `run_id` from the `metadata` SSE event (handoff §8.5), so capture it in the `useStream` `onCustomEvent`/metadata handler first, then call the existing `cancelTalosRun(token, { threadId, runId })` from `src/api/talos.ts`.
108
+
109
+ ---
110
+ _Saved by /session-save on 2026-09-02_
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.749",
3
+ "version": "1.0.751",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",