toga-ai 1.0.749 → 1.0.751
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/features/netsuite-sync-alert-monitor.md +16 -2
- package/knowledge/1.0/apps/worker/INDEX.md +1 -1
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +108 -8
- package/knowledge/sessions/2026-09-02-talos-surface-layer-apeterson.md +110 -0
- package/package.json +1 -1
|
@@ -6,8 +6,8 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: ["bala"]
|
|
9
|
+
updated: 2026-09-02
|
|
10
|
+
owners: ["bala", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/systemmonitor/netsuiteintegration.php
|
|
13
13
|
- worker/crons/infrastructure/system_monitors.php
|
|
@@ -104,8 +104,22 @@ that worker job does the syncing, this library class notices when it stops.
|
|
|
104
104
|
checkpoints mean anything; and a client that is held / never transacts supply (AIG) must be
|
|
105
105
|
left **off** this list or it produces permanent false-positive tickets.
|
|
106
106
|
|
|
107
|
+
- **⚠ The pending cursor-rename in the sync will SILENTLY BLIND this monitor (BUILT 2026-09-02, not yet
|
|
108
|
+
deployed).** The per-record resume cursor work on the sync renames each section's position key from
|
|
109
|
+
`NETSUITE_LAST_SYNC_DATETIME_<SECTION>` to `NETSUITE_LAST_SYNC_CURSOR_<SECTION>` (value becomes
|
|
110
|
+
`"<lastmodifieddate>|<netsuite id>"`). This monitor's `NETSUITE_SYNC_PARAMETER_KEYS` still lists the old
|
|
111
|
+
`DATETIME` keys, so after the rename its `where..in` `/parameters` read matches **zero rows** and it
|
|
112
|
+
reports OK forever — no stale-checkpoint alerts. The sync developer **intentionally left the monitor
|
|
113
|
+
un-updated** this session (accepted gap). When the rename deploys, update `NETSUITE_SYNC_PARAMETER_KEYS`
|
|
114
|
+
to the `CURSOR` names **and** parse the `<date>|<id>` value (`strtotime` the part before `|`) before the
|
|
115
|
+
48h compare. See the [per-client sync doc](../../worker/features/netsuite-togasupply-per-client-sync.md).
|
|
116
|
+
|
|
107
117
|
## Change history
|
|
108
118
|
|
|
119
|
+
- 2026-09-02 — Flagged that the sync's pending `NETSUITE_LAST_SYNC_DATETIME_*` →
|
|
120
|
+
`NETSUITE_LAST_SYNC_CURSOR_*` key rename (BUILT, not yet deployed) will make this monitor match zero rows
|
|
121
|
+
and stop alerting; it was intentionally left un-updated. Update the key list + parse the `<date>|<id>`
|
|
122
|
+
value when the rename ships. (jcardinal)
|
|
109
123
|
- 2026-07-10 — **Fixed the monitor timing out and silently not running since 2026-06-29.**
|
|
110
124
|
Rewrote `getClientStatus()` to read all six checkpoints per client in one TOGa2
|
|
111
125
|
`where..in` `/parameters` call (was 84 serialized calls ≈ 230s, over the 300s
|
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/checker.php, worker2/Component/Forecast/SaleImport/SaleImport.php, test/@dave/looper.php, test/@dave/reconcile_netsuite_totals.php, test/@dave/fixer.php, tools/bin/forecast/fixer.php, tools/bin/forecast/checker.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/reconcile_drift_2023plus.php, test/@dave/probe_invoice_gap_2026.php, test/@dave/probe_creditmemo_gap_detail.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
|
|
11
11
|
| [Forecast2 Supporting-Records Nightly Import (1.0 cron) — Customers & ConsolidatedCustomers](features/forecast2-supporting-records-import.md) | The nightly **1.0** pull that keeps the Forecast2 lookup/dimension tables (`Forecast.Accounts`, `Classifications`, `Customers`, `ConsolidatedCustomers`, `Employ | worker/crons/toga2/forecast2/import_supporting_records.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/schedules/cron.worker.infrastructure.json, worker/.ebextensions/009_setup_phpini.config, worker/crons/sync/netsuite/netsuite_customers.php, library/app/api/netsuite/rest.php, library/app/model/forecast2/customer.php, library/app/model/forecast2/consolidatedcustomer.php |
|
|
12
12
|
| [NetSuite Sales Order Sales Rep Sourcing (Staples & ODP EDI orders)](features/netsuite-sales-order-sales-rep-sourcing.md) | How the **sales rep** on a NetSuite Sales Order is determined for the two 1.0 `worker` EDI order-creation integrations (Staples cXML and Compass/ODP EDI). | worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php, worker/crons/sync/staples/sync_staples_cxml.php, test/@Mark/NetSuite/TRUE_80451_customer_salesrep_diag.php |
|
|
13
|
-
| [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
|
|
13
|
+
| [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/crons/toga2/netsuite/sync_togasupply_elite.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql, library/app/api/toga2.php, library/app/api/netsuite/rest.php, library/app/query.php, library/app/database.php, library/app/framework.php, library/app/systemmonitor/netsuiteintegration.php, dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql, test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php, test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php |
|
|
14
14
|
| [OneUptime Server monitor + disk/memory hygiene on the 1.0 worker EB host](features/oneuptime-server-monitor-host-hygiene.md) | The 1.0 `agilant-worker` EB environment runs on the **legacy Amazon Linux 1 PHP 7.2 platform** (Apache httpd/prefork, s3fs mounts, cron) and repeatedly went dow | worker/.ebextensions/040_disk_memory_hygiene.config, worker/.ebextensions/045_oneuptime_agent.config, worker/ebs/cron.worker.php, worker/ebs/mount-s3fs-folders.php, worker/ebs/apache_settings.php, worker/ebs/setup_phpini.php |
|
|
15
15
|
| [OneUptime external uptime monitoring for 1.0 workers](features/oneuptime-worker-uptime-monitoring.md) | Every 1.0 worker box self-reports its liveness to an external OneUptime monitor once per minute by curl-POSTing to a per-worker "Incoming Request" heartbeat URL | library/app/worker.php, worker/crons/worker/worker_heartbeat.php, worker/ebs/cron.worker.php, worker/.ebextensions/045_oneuptime_agent.config |
|
|
16
16
|
| [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
|
|
@@ -16,8 +16,11 @@ files:
|
|
|
16
16
|
- dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql
|
|
17
17
|
- library/app/api/toga2.php
|
|
18
18
|
- library/app/api/netsuite/rest.php
|
|
19
|
+
- library/app/query.php
|
|
20
|
+
- library/app/database.php
|
|
19
21
|
- library/app/framework.php
|
|
20
22
|
- library/app/systemmonitor/netsuiteintegration.php
|
|
23
|
+
- dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql
|
|
21
24
|
- test/@srija/Elite Testing/Service Requests/test_sync_togasupply_elite_section.php
|
|
22
25
|
- test/@srija/Elite Testing/Service Requests/test_diagnose_togasupply_elite.php
|
|
23
26
|
related:
|
|
@@ -89,6 +92,28 @@ its IDLE write. **There is no timeout, memory cap, or resource limit anywhere in
|
|
|
89
92
|
never *tune the window size down*. (Corollary: a window that has grown back to the cap is positive
|
|
90
93
|
evidence the section is completing.)
|
|
91
94
|
|
|
95
|
+
### Per-record resume cursor — checkpoint after EACH record, not window-end (BUILT 2026-09-02, NOT yet deployed)
|
|
96
|
+
|
|
97
|
+
The historical weakness of window-end-only checkpointing: a mid-window crash (poison record, or the
|
|
98
|
+
log-DB doom loop above) re-processes the **whole** window every run — the giant-fulfillment doom loop is a
|
|
99
|
+
direct consequence. This change converts the checkpoint to **per-record keyset (seek) resume** so a
|
|
100
|
+
transient crash resumes at the next record instead of redoing the window.
|
|
101
|
+
|
|
102
|
+
- **Key rename per section:** `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` → `NETSUITE_LAST_SYNC_CURSOR_<SECTION>`,
|
|
103
|
+
value format **`"<lastmodifieddate>|<netsuite internal id>"`**. The id breaks same-second ties.
|
|
104
|
+
- **Fetch change (`library/app/api/netsuite/rest.php`, all 6 header queries):** add
|
|
105
|
+
`ORDER BY lastmodifieddate, id` and change the lower bound to
|
|
106
|
+
`lastmodifieddate > :t OR (lastmodifieddate = :t AND id > :fromId)` (upper bound `<= :to` kept). A new
|
|
107
|
+
`$fromId` param was threaded into the 6 list funcs. Added `lastModifiedDate` to the fetched record object
|
|
108
|
+
for **PURCHASE_ORDERS, ITEM_RECEIPTS, ITEM_FULFILLMENTS, INVENTORY_ADJUSTMENTS** (SALES_ORDERS + INVOICES
|
|
109
|
+
already carried it).
|
|
110
|
+
- **Save after EACH main record** (one PUT of `lastmod|id`); on a clean window with 0 records, PUT
|
|
111
|
+
`"<windowTop>|0"` so empty date ranges still advance. No new `try/catch` — stays fail-loud.
|
|
112
|
+
- **Migration is MANDATORY with the code (see the reset trap in Gotchas):**
|
|
113
|
+
`dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql` (Client fan-out) renames the keys **and**
|
|
114
|
+
appends `|0` to carry the value across; safe to re-run. The new-client seed module
|
|
115
|
+
(`dbchanges2/_modules/netsuite/2026-08-05 - CLEAN NETSUITE CLINET.SQL`) was updated to CURSOR.
|
|
116
|
+
|
|
92
117
|
### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
|
|
93
118
|
|
|
94
119
|
Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
|
|
@@ -305,6 +330,10 @@ Per-client `Parameters` table (2.0 client DB, e.g. `Client_Canon.Parameters`; co
|
|
|
305
330
|
|
|
306
331
|
Parameters are stored **per client DB** but accessed **through the TOGa2 API**, not direct SQL.
|
|
307
332
|
|
|
333
|
+
**Pending rename (BUILT 2026-09-02, not yet deployed):** the position keys become
|
|
334
|
+
`NETSUITE_LAST_SYNC_CURSOR_<SECTION>` with value `"<lastmodifieddate>|<netsuite internal id>"` — see the
|
|
335
|
+
per-record resume cursor in *How it works*. The `NETSUITE_EXECUTION_MODE_*` keys are unchanged.
|
|
336
|
+
|
|
308
337
|
## Client variations
|
|
309
338
|
|
|
310
339
|
- **Transfer-order routing is opt-in per wrapper (default OFF).** A client whose NetSuite
|
|
@@ -856,6 +885,15 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
|
|
|
856
885
|
line). **Fix (`library`):** skip lines whose `adjustQtyBy` is null in **BOTH** passes over
|
|
857
886
|
`inventoryList->inventory` (the item-build pass and the serial/unit pass). Shared fix; unblocked NYCHH
|
|
858
887
|
inventory adjustments.
|
|
888
|
+
- **Prod-confirmed 2026-09-02** (commit `8bae8a2a`, 2026-08-29; the null-skip lands at
|
|
889
|
+
`toga2.php:5728` and `:5879`). Before it, NYCHH `INVENTORY_ADJUSTMENTS` was frozen `3-IDLE` (window
|
|
890
|
+
collapsed to ~3s), watermark stuck at **2022-12-28**, advancing only seconds/run — `Logs.Issue`
|
|
891
|
+
**542** (`clientId 3`), 35 occurrences 2026-08-28..09-01. The fix reached prod ~2026-09-01 afternoon
|
|
892
|
+
and **Issue 542 stopped 09-01 14:08**. This `3-IDLE`-crawl-with-frozen-watermark signature is the same
|
|
893
|
+
poison-record shrink as a `1-RUNNING` freeze — a section that throws on a record too old to fall
|
|
894
|
+
outside the collapsed window keeps re-throwing; once the window shrinks past that record's timestamp it
|
|
895
|
+
reports IDLE but crawls seconds/run and can never advance past it. Read it as "still throwing," not
|
|
896
|
+
"small/slow section" (see the item-fulfillment doom-loop gotcha below for the same signature).
|
|
859
897
|
- **⚠ Item-receipt sync froze on a PARTIAL receipt's NULL-quantity PO lines — the item-receipt TWIN of the
|
|
860
898
|
inventory-adjustment NULL-quantity fix above (fixed 2026-08-31).** For a partial item receipt NetSuite
|
|
861
899
|
returns **all** the PO's linked lines but sets `quantity=null` on the lines **not received on this
|
|
@@ -877,14 +915,33 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
|
|
|
877
915
|
`transactionId` (`Logs.Event` carries `clientId`, `Logs.Issue` does not) → `Logs_Nychh.Api` on that
|
|
878
916
|
`transactionId` for the untruncated `EV-10`/1048 rejection and the payload showing the `quantity:null`
|
|
879
917
|
lines.
|
|
880
|
-
-
|
|
881
|
-
`NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS='3-IDLE'` with the window
|
|
882
|
-
throwing recently) but currently **IDLE** — so it crawls a few seconds per run rather
|
|
883
|
-
hard `1-RUNNING` freeze.
|
|
884
|
-
|
|
885
|
-
|
|
886
|
-
|
|
887
|
-
|
|
918
|
+
- **NYCHH ITEM_FULFILLMENTS watermark frozen at 2021-12-29 (found 2026-08-31; ROOT-CAUSED 2026-09-02 —
|
|
919
|
+
the log-DB doom loop below).** `NETSUITE_EXECUTION_MODE_ITEM_FULFILLMENTS='3-IDLE'` with the window
|
|
920
|
+
collapsed to ~3s (throwing recently) but currently **IDLE** — so it crawls a few seconds per run rather
|
|
921
|
+
than a hard `1-RUNNING` freeze. This is the exact signature of the doom loop in the next gotcha: a giant
|
|
922
|
+
fulfillment throws every run, the window shrinks ÷3 until it no longer spans that fulfillment, then goes
|
|
923
|
+
IDLE but can never advance past it. Context: the 2026-08-31 full 19-client togasupply prod sweep found
|
|
924
|
+
all other clients healthy (`864000-IDLE`) except Elite SALES_ORDERS `864000-RUNNING` (healthy mid-run,
|
|
925
|
+
large window, not stuck) and AIG SALES_ORDERS `8-RUNNING` (the known held account).
|
|
926
|
+
- **⚠ Item-fulfillment DOOM LOOP from a dropped LOG-DB connection during a giant fulfillment (found
|
|
927
|
+
2026-09-02, NOT yet fixed — the per-record cursor below is the mitigation).** A single very large
|
|
928
|
+
fulfillment POSTs thousands of `/item-fulfillment-item-units` over api2 HTTP; that long unit-POST phase
|
|
929
|
+
lets the **worker's log-DB connection idle past MySQL `wait_timeout`**. The **next** fulfillment's refetch
|
|
930
|
+
— `fetchItemFulfillmentById` at `library/app/api/toga2.php:4440`, the **first line** of
|
|
931
|
+
`syncItemFulfillmentFromNetsuite` — tries to log that NetSuite REST call and dies at
|
|
932
|
+
`library/app/query.php:168` `mysqli_select_db` *"MySQL server has gone away / Connection timed out"*
|
|
933
|
+
(`Logs.Issue` **589**). The 1.0 framework escalates that warning to `exit`, so the run aborts mid-window;
|
|
934
|
+
the watermark only advances at window end (`finishModeIteration`), so the whole window — including the
|
|
935
|
+
giant fulfillment — **re-processes next run, times out at the same spot, and the window shrinks ÷3 each
|
|
936
|
+
run** (→ the `3-IDLE` crawl signature above once the window no longer spans the giant fulfillment).
|
|
937
|
+
- **Root of the crash:** the "gone away" recovery in `App_Query::execute` (`library/app/query.php`,
|
|
938
|
+
`database.php:431`) only guards the `mysqli_query` call (errno **2006** reconnect), **NOT** the
|
|
939
|
+
`mysqli_select_db` that runs before it — which is the first touch of the stale connection, so the
|
|
940
|
+
reconnect never gets a chance. A durable framework gap, not sync-specific.
|
|
941
|
+
- **Durable engine rule reconfirmed:** a shrinking window in this sync **ALWAYS** means a thrown
|
|
942
|
+
exception — there is **no** volume/time/memory backoff anywhere in the script. "Window not 864000" ⇒
|
|
943
|
+
something threw. The log-connection idle-out is a concrete cause to add to the poison-record and
|
|
944
|
+
shim-shape causes above.
|
|
888
945
|
- **⚠ Item-fulfillment stale-item prune must delete tracking-number children before the unit AND the item
|
|
889
946
|
(fixed 2026-08-30).** In `syncItemFulfillmentFromNetsuite`'s cleanup loop, deleting a stale
|
|
890
947
|
item-fulfillment-item-**unit** failed api2 `EV-11` / MySQL **1451** (FK
|
|
@@ -967,8 +1024,51 @@ Parameters are stored **per client DB** but accessed **through the TOGa2 API**,
|
|
|
967
1024
|
A "silent freeze" here is a serial/data collision, not necessarily a catalog/item-matching problem —
|
|
968
1025
|
read the untruncated Api-log rejection before assuming item matching.
|
|
969
1026
|
|
|
1027
|
+
- **⚠ The position key defaults to the EARLIEST date when MISSING — a code-only cursor rename WITHOUT
|
|
1028
|
+
the data migration resets every client to re-sync from scratch (found 2026-09-02).** When the section's
|
|
1029
|
+
position key is absent, the engine falls back to `MIN_DATETIME_TO_CHECK_FOR_NETSUITE_DATA`
|
|
1030
|
+
(`common_sync_togasupply.php` ~L573). So shipping the `NETSUITE_LAST_SYNC_DATETIME_*` →
|
|
1031
|
+
`NETSUITE_LAST_SYNC_CURSOR_*` rename in **code only** makes every client's old key invisible → each
|
|
1032
|
+
section restarts from the earliest date and re-syncs its entire history. The migration
|
|
1033
|
+
(`dbchanges2/Client/2026-09-02b - NetsuiteSyncCursorRename.sql`, Client fan-out) **must carry the value
|
|
1034
|
+
across** — it renames the key and appends `|0` in one `UPDATE ... REPLACE(...)`, re-runnable. **Ship the
|
|
1035
|
+
migration on every client DB before (or with) the code.**
|
|
1036
|
+
- **⚠ The NetSuite Sync Alert monitor will SILENTLY STOP after the cursor rename (known, accepted
|
|
1037
|
+
2026-09-02).** `App_SystemMonitor_NetSuiteIntegration` (`library/app/systemmonitor/netsuiteintegration.php`)
|
|
1038
|
+
hardcodes the old `NETSUITE_LAST_SYNC_DATETIME_*` keys, so once the rename ships its `where..in`
|
|
1039
|
+
`/parameters` read matches **zero rows** and it stops alerting on stale checkpoints. The developer
|
|
1040
|
+
**intentionally left the monitor un-updated** this session — a documented, accepted gap. Update the
|
|
1041
|
+
monitor's key list to `NETSUITE_LAST_SYNC_CURSOR_*` (and parse the `<date>|<id>` value) when the rename
|
|
1042
|
+
deploys, or NetSuite sync stalls go unmonitored. Cross-ref:
|
|
1043
|
+
[NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
|
|
1044
|
+
|
|
970
1045
|
## Change history
|
|
971
1046
|
|
|
1047
|
+
- 2026-09-02 — **Per-record resume cursor BUILT (shared: `worker` + `library` + `dbchanges2`; pending
|
|
1048
|
+
review + deploy, NOT shipped).** Converts the section checkpoint from window-end-only to per-record
|
|
1049
|
+
keyset resume so a mid-window crash (poison record or the log-DB doom loop) resumes at the next record
|
|
1050
|
+
instead of re-processing the whole window. Position key `NETSUITE_LAST_SYNC_DATETIME_<SECTION>` →
|
|
1051
|
+
`NETSUITE_LAST_SYNC_CURSOR_<SECTION>`, value `"<lastmodifieddate>|<netsuite id>"` (id breaks same-second
|
|
1052
|
+
ties); `rest.php`'s 6 header queries gain `ORDER BY lastmodifieddate, id` + a keyset lower bound + a
|
|
1053
|
+
`$fromId` param; cursor PUT after each record (and `"<windowTop>|0"` on an empty window); `lastModifiedDate`
|
|
1054
|
+
added to the PO/IR/IF/InvAdj fetched objects. Migration `Client/2026-09-02b - NetsuiteSyncCursorRename.sql`
|
|
1055
|
+
(fan-out) renames the keys and appends `|0` — **mandatory**, because a missing key defaults to the earliest
|
|
1056
|
+
date and would re-sync every client's full history. Monitor `netsuiteintegration.php` intentionally left on
|
|
1057
|
+
the old DATETIME keys (will stop alerting after the rename — accepted). (jcardinal)
|
|
1058
|
+
- 2026-09-02 — **Item-fulfillment doom loop root-caused (found, NOT fixed — the cursor above mitigates).** A
|
|
1059
|
+
giant fulfillment's long unit-POST phase idles the worker's log-DB connection past MySQL `wait_timeout`; the
|
|
1060
|
+
next fulfillment's `fetchItemFulfillmentById` (`toga2.php:4440`) dies logging its NetSuite call at
|
|
1061
|
+
`query.php:168` `mysqli_select_db` "server has gone away" (`Logs.Issue 589`) → 1.0 warning→`exit` → window
|
|
1062
|
+
re-processes and shrinks ÷3 each run (the `3-IDLE` crawl). Root cause: `App_Query::execute`'s gone-away
|
|
1063
|
+
recovery (`query.php`/`database.php:431`) guards only the `mysqli_query` (2006 reconnect), not the
|
|
1064
|
+
`mysqli_select_db` before it. Explains the NYCHH ITEM_FULFILLMENTS freeze at 2021-12-29. Reconfirmed the
|
|
1065
|
+
engine rule: a shrinking window ALWAYS means a throw (no volume/time/memory backoff exists). (jcardinal)
|
|
1066
|
+
- 2026-09-02 — **Prod-confirmed the NYCHH inventory-adjustment NULL-`adjustQtyBy` freeze + its fix (commit
|
|
1067
|
+
`8bae8a2a`, `toga2.php:5728`/`:5879`).** `INVENTORY_ADJUSTMENTS` had been frozen `3-IDLE`, watermark stuck
|
|
1068
|
+
2022-12-28, crawling seconds/run — `Logs.Issue 542` (`clientId 3`), 35 occ 2026-08-28..09-01, EV-10/MySQL
|
|
1069
|
+
1048 "quantityAdjustBy cannot be null" on NetSuite serialized-detail sub-lines. The null-skip reached prod
|
|
1070
|
+
~2026-09-01 afternoon; Issue 542 stopped 09-01 14:08. Recorded that `3-IDLE`-crawl = the same poison-record
|
|
1071
|
+
shrink as `1-RUNNING`, not a slow section. (jcardinal)
|
|
972
1072
|
- 2026-09-02 — **PURCHASE_ORDERS cleanup DELETE freeze fixed (shared, `library` only — WRITTEN +
|
|
973
1073
|
REVIEWED, NOT yet committed/deployed).** `syncPurchaseOrderFromNetsuite`'s "remove dropped items"
|
|
974
1074
|
cleanup deleted a NetSuite-dropped PO line with a bare throwing DELETE; a leftover item receipt
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: talos-surface-layer
|
|
4
|
+
title: Talos moved onto the surface layer; chat history, titles, launcher dim
|
|
5
|
+
author: apeterson
|
|
6
|
+
repos: [toga25-supply, toga-blox, dbchanges2]
|
|
7
|
+
framework: "2.0"
|
|
8
|
+
client: nychh
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-09-02
|
|
11
|
+
updated: 2026-09-02
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: talos-surface-layer
|
|
15
|
+
**Date:** 2026-09-02
|
|
16
|
+
**Project/Repo:** toga25-supply + toga-blox + dbchanges2 (2.0)
|
|
17
|
+
**Task:** Move the Talos AI assistant (feature gate + suggestion chips) off hardcoded frontend config onto the backend surface layer — off for all clients in Core, granted only to NYCHH — and finish the chat history / auto-title / launcher work started earlier in the same thread.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
|
|
23
|
+
- **Loading bubble restored in host-render mode.** `toga-blox src/components/Talos/TalosPanel.tsx` now renders the synthetic thinking bubble in BOTH modes, because it is chrome derived from `isStreaming`/`progress`, not a message. Evidence: 2 new regression tests; blox Talos suite 31 passing.
|
|
24
|
+
- **Chat history drawer + new chat, data-driven.** New controlled props `threads`, `activeThreadId`, `onSelectThread`, `onNewChat`, `threadsStatus`. `history` re-enables only when `threads` AND `onSelectThread` are both given; `newChat` only when `onNewChat` is. The panel sorts rows newest-first. Evidence: tests incl. ordering and "host content replaces the list".
|
|
25
|
+
- **Opening an existing thread needs no transcript parser.** Verified in `node_modules/@langchain/langgraph-sdk/dist/ui/orchestrator.js` — `setThreadId` calls `#fetchHistoryForThread` and clears the stream when `fetchStateHistory: true`. Selecting a row is just "set threadId"; the existing LangGraph→blox mapper handles the rest.
|
|
26
|
+
- **Auto-title polling.** `toga25-supply src/hooks/useTalosThreads.ts` `titlePollInterval` — 4s `refetchInterval` while any recent thread is unnamed, `false` otherwise, with a 2-minute age window. Evidence: 7 tests incl. server-clock-ahead and the never-named case.
|
|
27
|
+
- **"Couldn't load your chats." over a full list fixed.** `threadsStatusFrom` returns "error" only when there is no data; blox gates all three drawer notes on `histRows.length === 0`. Evidence: regression test in each repo.
|
|
28
|
+
- **Talos on the surface layer, RUN AND VERIFIED ON LOCAL.** `dbchanges2 Core/2026-09-02a - TalosAssistantSurfaceSeed.sql` → 2 elements, both `isVisible = 0`, suggestions row `itemCount = 4`. `dbchanges2 Client_Nychh/2026-09-02a - TalosAssistantEnable.sql` Section A → 3 rows: two `IS_VISIBLE='1'` + one `CONFIG` with `itemCount = 3` and `roleRestated = 'suggestion'`. Three NYCHH chips render in the browser. Developer confirmed: "that workds".
|
|
29
|
+
- **Fail-closed FE gate.** `src/hooks/useTalosSurface.ts` — missing bundle, failed fetch, absent marker, or still-loading all resolve to disabled. `TalosStreamProvider` gained an `enabled` prop so a tenant without Talos never calls `/assistants`. 14 tests.
|
|
30
|
+
- **Launcher dims under a modal.** `toga-blox TableRecordModal.tsx` sets `data-blox-modal-open` on `<html>` (a counter); `Talos.module.css` dims `.launcher` and stops the orbit ring under `:root[data-blox-modal-open]`.
|
|
31
|
+
- **Surface bundle no longer cached to disk.** `src/App.tsx` `shouldDehydrateQuery` excludes the `talos`, `surface-meta` and `surface-meta-group` namespaces.
|
|
32
|
+
- **End state:** toga25-supply 87 tests / 12 files passing; `npx tsc --noEmit -p tsconfig.app.json` clean apart from one pre-existing error; toga-blox 31 Talos + TableRecordModal tests passing; blox `dist/` rebuilt.
|
|
33
|
+
|
|
34
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
35
|
+
|
|
36
|
+
- **A host-render branch that bypasses the panel's own message list.** The first `renderMessages` implementation replaced `msgs` entirely, which took the synthetic thinking bubble with it. Symptom: submit a question and get **zero** feedback until the answer appears — no loading UI, no streaming. The bubble is chrome, not conversation; never let a host slot remove it.
|
|
37
|
+
- **Trusting supply's typecheck to prove a blox change reached the browser.** `npx tsc --noEmit -p tsconfig.app.json` resolves blox **types from source** through the `node_modules/@agilant/toga-blox` symlink, but Vite loads the package at **runtime from `dist/`**. Typecheck was clean while the browser ran stale blox code. Always `npm run build` in blox + `rm -rf node_modules/.vite` in supply.
|
|
38
|
+
- **The earlier note that blox `npm run build` fails on a missing `tsconfig.app.json` is WRONG.** It builds fine (`tsc && copyfiles … && fix-esm-imports && fix-asset-imports`). Do not route around it.
|
|
39
|
+
- **`status: isError ? "error" : …` for a React Query list.** React Query sets `isError` when a **background** refetch fails while KEEPING the data it already had. With the 4s title poll running, one failed poll printed "Couldn't load your chats." above a full, usable list. Report error only when there is no data.
|
|
40
|
+
- **A drawer error message with no empty-list guard.** It rendered *alongside* the rows, producing a screen that says loading failed while showing the loaded data.
|
|
41
|
+
- **Building the title polling before checking that ANY thread had a title.** Every NYCHH thread reads "New conversation", including ones days old and far outside any poll window — so this was never a client timing bug. Naming is a per-assistant setting (`thread_name_generation_enabled`, default `true`, plus `thread_name_model`) on `GET`/`PATCH /api/assistants/{assistant_id}/config`. The polling is correct and will work the moment the server writes a name, but it could not fix this.
|
|
42
|
+
- **A cross-schema `JOIN Core.SurfaceElements` in a `Client_*` file.** Works on local/beta, **fails in production** — Core and Client_* are on separate clusters. Error there: `Table 'Core.Surfaces' doesn't exist`. Prod needs the read-ids-on-Core → insert-literals-on-client two-step.
|
|
43
|
+
- **Schema-qualifying a Core-only migration (`Core.Messages`, `Core.Surfaces`).** api2 can rename the Core DB via `[database] core_dbname`, so qualifying breaks exactly the environment that renamed it. Core-only files must use UNQUALIFIED names and be run while connected to Core.
|
|
44
|
+
- **Running an unqualified `SurfaceElements` query while connected to `Client_Nychh`.** Error: `Table 'Client_Nychh.SurfaceElements' doesn't exist`. `SurfaceElements`/`Surfaces`/`Messages` are **Core-only**; a cross-schema verify must qualify every Core table.
|
|
45
|
+
- **Assuming the launcher-not-dimmed bug was z-index.** It was not. `.launcher` has no z-index, blox `.header` has no z-index or position, supply's `.headerActions` has neither, and `.backdrop` is `position: fixed; z-index: 40` — so the launcher was already painted under the backdrop and its clicks already swallowed. The real cause: `.backdrop-right`/`.backdrop-left` is only `rgba(0, 0, 0, 0.2)`, and a 20% black wash over a **white** button still reads as active. Do not go looking for a z-index.
|
|
46
|
+
- **One surface element per suggestion chip.** Replaced before it shipped past local. A tenant could only switch on questions **Core had already seeded**, so changing a tenant's wording needed a new Core migration for every phrasing.
|
|
47
|
+
- **A bare JSON array as a `CONFIG` override value.** A `CONFIG` override **replaces** `config` wholesale — `_underscore Model/Core/Surface.php:839` assigns, it does not `array_merge`. A bare array passes the `is_array()` check and wipes `config.role`, so the frontend stops recognising the element and the chips **silently disappear with no error anywhere**. The override must restate `"role":"suggestion"`.
|
|
48
|
+
- **Putting override JSON in `SurfaceOverrides.value`.** It is `varchar(255)`; any real chip set overflows it. Use `c_longValue` — the resolver falls back to it when `value` is NULL/'' (`Surface.php:907`).
|
|
49
|
+
|
|
50
|
+
## Not tried yet (candidates for next session)
|
|
51
|
+
|
|
52
|
+
- **Run the SQL on beta and prod.** Local only so far. Prod = Section B (read ids on the Core cluster) then Section C (literal-id insert on the client cluster). **Never Section A in prod.**
|
|
53
|
+
- **Wire Stop to `cancelTalosRun`** instead of `stream.stop()`. This is a **real defect**: `stream.stop()` only drops the client transport while the run keeps executing and billing. Needs `run_id` from the `metadata` SSE event (handoff §8.5).
|
|
54
|
+
- **`fromTalosTranscript()` + terminal-completion reconciliation** (handoff §8.11 / §8.8). Message ids in `src/providers/TalosRuntimeProvider.tsx` are `talos-${index}` placeholders; feedback needs a durable server `message_id` + `run_id`.
|
|
55
|
+
- **Message actions** (copy, thumbs up/down) via `ActionBarPrimitive`, and the thumbs-down dialog (handoff §9). Blocked on the durable ids above; copy alone is doable now.
|
|
56
|
+
- **Rename / delete thread UI.** `renameTalosThread` / `deleteTalosThread` already exist in `src/api/talos.ts` and are unused. Both endpoints are real (`PATCH`/`DELETE /threads/{thread_id}`). Decide the `If-Match` question first: the handoff doc shows it quoted, the live spec types it as a plain optional string, and our client sends it bare — recommendation was to **drop the header** rather than guess.
|
|
57
|
+
- **Make supply's header `inert` while a modal is open.** The launcher is still keyboard-reachable behind a modal; `pointer-events: none` stops the mouse only. Affects all header controls, so it is a deliberate supply-wide change.
|
|
58
|
+
- **Rule on reserved-literal ids vs `AUTO_INCREMENT`** for Core surface seeds. `knowledge/2.0/apps/dbchanges2/features/surface-layer-schema.md` currently documents BOTH branches. Note the shipped `Core/2026-08-12a` navigation seed already uses `AUTO_INCREMENT`, so the "never AUTO_INCREMENT" rule was already contradicted by merged code.
|
|
59
|
+
- **Per-role Talos gating / per-role chip sets.** Current grants are client-wide (`roleId` NULL) — every NYCHH user incl. devteam gets Talos. Two role-scoped `CONFIG` rows would resolve by `ORDER BY (roleId IS NULL) DESC, (personaId IS NULL) DESC, id ASC`, so the **higher id wins arbitrarily** for a multi-role user.
|
|
60
|
+
|
|
61
|
+
## Current file state
|
|
62
|
+
|
|
63
|
+
| File | Status | Notes |
|
|
64
|
+
|------|--------|-------|
|
|
65
|
+
| `dbchanges2 Core/2026-09-02a - TalosAssistantSurfaceSeed.sql` | new, uncommitted, RUN ON LOCAL | Surface `talos-assistant` (DRAWER, not record-bound), 2 elements both `isVisible=0`; suggestions element holds Core's default 4 items. Unqualified table names. Teardown in footer. |
|
|
66
|
+
| `dbchanges2 Client_Nychh/2026-09-02a - TalosAssistantEnable.sql` | new, uncommitted, RUN ON LOCAL (Section A only) | 3 sections: A single-cluster, B read ids on Core, C literal insert on client. Writes 2 `IS_VISIBLE` grants + 1 `CONFIG` with NYCHH's 3 questions. |
|
|
67
|
+
| `toga25-supply db-migrations/PLAYBOOK.md` | modified | 2026-09-02 change-log entry incl. prod cross-cluster procedure and the `SurfaceOverrides` NULL-unique-key gotcha. |
|
|
68
|
+
| `toga25-supply src/hooks/useTalosSurface.ts` (+ `.test.ts`) | new | Fail-closed gate + tolerant `readItems` parser. 14 tests. |
|
|
69
|
+
| `toga25-supply src/hooks/useTalosThreads.ts` (+ `.test.ts`) | new | Thread list, `titlePollInterval`, `threadsStatusFrom`, `toThreadSummary`. 12 tests. |
|
|
70
|
+
| `toga25-supply src/providers/TalosStreamProvider.tsx` | modified | `enabled` prop; threads/selectThread/activeThreadId in context; refresh on threadId assignment + run-end. |
|
|
71
|
+
| `toga25-supply src/providers/TalosRuntimeProvider.tsx` | new (earlier in thread) | assistant-ui `ExternalStoreRuntime` bridge. Message ids are index placeholders. |
|
|
72
|
+
| `toga25-supply src/components/TalosThread/` | new (earlier in thread) | assistant-ui thread + safe markdown components. |
|
|
73
|
+
| `toga25-supply src/hooks/useTalosSession.ts` | modified | Takes `enabled`; returns `disabled` when gated off. |
|
|
74
|
+
| `toga25-supply src/layout/AppLayout/AppLayout.tsx` | modified | Gates launcher + panel on `useTalosSurface().isEnabled`; passes surface suggestions, threads, history handlers. |
|
|
75
|
+
| `toga25-supply src/App.tsx` | modified | `shouldDehydrateQuery` excludes `talos`, `surface-meta`, `surface-meta-group`. |
|
|
76
|
+
| `toga-blox src/components/Talos/TalosPanel.tsx` | modified | `renderMessages`, thread-list props, loading bubble in both modes, `NO_SUGGESTIONS` default (DEFAULT_SUGGESTIONS deleted). |
|
|
77
|
+
| `toga-blox src/components/Talos/types.ts` | modified | `TalosThreadSummary`, thread props, `renderMessages`. |
|
|
78
|
+
| `toga-blox src/components/Talos/Talos.module.css` | modified | `.historyNote`; launcher dim under `:root[data-blox-modal-open]`. |
|
|
79
|
+
| `toga-blox src/components/Talos/README.md` | modified | Suggestions are host-supplied; chat-history section. |
|
|
80
|
+
| `toga-blox src/components/TableRecordModal/TableRecordModal.tsx` | modified | `data-blox-modal-open` counter on `<html>`. |
|
|
81
|
+
| `toga-blox src/components/TableRecordModal/tableRecordModal.module.css` | modified (earlier in thread) | `--talos-dock-width` awareness. |
|
|
82
|
+
| `toga-blox dist/` | rebuilt | Required for the browser; gitignored. |
|
|
83
|
+
|
|
84
|
+
**Nothing is committed in any of the three repos.**
|
|
85
|
+
|
|
86
|
+
## Decisions made
|
|
87
|
+
|
|
88
|
+
- **Talos is gated by the surface layer, OFF for every client in Core.** Rationale: Talos answers questions from a tenant's live Supply data through an external AI service; a tenant must not inherit that from a migration. Rejected: a build-time env flag (not per-tenant) and defaulting ON (a data-exposure decision made by a file instead of a person).
|
|
89
|
+
- **The feature switch is a marker element, not `Surfaces.isVisible`.** The override cascade is keyed by `surfaceElementId` and is never applied to a Surfaces row's own `isVisible`, so overriding the surface row silently does nothing. Same trick as the `config.role='sectionVisibility'` markers.
|
|
90
|
+
- **Chips are ONE element with a `config.items` array of `{label, prompt, icon}`.** Rationale: per-chip elements let a tenant only enable questions Core had already seeded. Rejected: one element per chip (needs a Core migration per phrasing) and an array of bare strings (collapses label/prompt/icon and loses the icons). Trades accepted: labels are not `Messages` keys so `MessageTranslations` cannot translate them; and chip SETS are per-scope rather than per-chip.
|
|
91
|
+
- **The chip payload lives in `config`, not an `Actions` row with `type='RUN_TALOS'`.** `RUN_TALOS` is in the enum but **no `Actions` row has ever been seeded in dbchanges2** and there is no FE action-registry entry for it. `config` is emitted verbatim into the bundle. Same trade `Core/2026-08-12a` made choosing `config.path` over `Actions`/`NAVIGATE`.
|
|
92
|
+
- **`AUTO_INCREMENT` + `NOT EXISTS` guards on natural keys, not a reserved literal id block.** Reserving requires verifying `MAX(id)` against every environment first; these files are purely additive and id-agnostic. Matches the `navigation` seed. Rejected: reserved literals (unverifiable without DB access, and the ids are read off Core at run time in Section B anyway).
|
|
93
|
+
- **The per-client file keeps `Client_Nychh.` qualification while the Core file drops `Core.`.** Run against the wrong cluster the qualified one fails loudly on a missing schema instead of silently granting the feature to another tenant; the Core schema name is not guaranteed.
|
|
94
|
+
- **The `NOT EXISTS` guard is load-bearing, not decoration.** `SurfaceOverrides`' unique key is `(surfaceElementId, surfaceId, personaId, roleId, languageId, attribute)` and MySQL treats NULLs in a unique key as **distinct** — client-wide grants have persona/role/language all NULL, so the index accepts duplicates.
|
|
95
|
+
- **Permission/feature surfaces are not persisted to localStorage.** `surface-meta` is fetched with `staleTime: Infinity`, so a persisted copy is never refetched and a grant — or a **revoke** — would not reach a browser for up to the persister's 24h default `maxAge`.
|
|
96
|
+
- **The modal-open marker is a root-element attribute and a counter.** An attribute because the launcher renders outside the modal's portal so no React state reaches it (same boundary-crossing trick as `--talos-dock-width`); a counter because modals stack and the marker must survive the inner one closing. Rejected: threading a prop from every modal layout.
|
|
97
|
+
- **`config.items` parsing is tolerant, not strict.** `config` is a hand-edited JSON column that differs per tenant, so one bad item costs one chip rather than every chip or a render crash.
|
|
98
|
+
|
|
99
|
+
## Blockers
|
|
100
|
+
|
|
101
|
+
- **Thread auto-titles are broken server-side.** Every NYCHH thread reads "New conversation". Naming is per-assistant: `thread_name_generation_enabled` (default `true`) + `thread_name_model` on `GET`/`PATCH /api/assistants/{assistant_id}/config`. Either it is `false` on the `TOGa Supply` assistant or the Bedrock naming model is unreachable in that deployment. **A consumer app cannot fix this.** Open with `akhokhani`; NYCHH `assistant_id` `c7981164-1fe9-5878-b115-85c38dadad43`.
|
|
102
|
+
- **Nothing is committed in toga25-supply, toga-blox or dbchanges2.** blox `dist/` is gitignored, so no other consumer sees any of this until blox publishes.
|
|
103
|
+
- **Beta and prod SQL not run** — the feature is invisible in those environments (the fail-closed gate working, not a regression).
|
|
104
|
+
|
|
105
|
+
## Exact next step
|
|
106
|
+
|
|
107
|
+
> Wire Stop to `cancelTalosRun` in `toga25-supply src/providers/TalosStreamProvider.tsx` — replace `stream.stop()` in the `stop` callback. It is a real defect: `stream.stop()` drops only the client transport while the run keeps executing and billing on Aegra. You need the `run_id` from the `metadata` SSE event (handoff §8.5), so capture it in the `useStream` `onCustomEvent`/metadata handler first, then call the existing `cancelTalosRun(token, { threadId, runId })` from `src/api/talos.ts`.
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
_Saved by /session-save on 2026-09-02_
|
package/package.json
CHANGED