toga-ai 1.0.141 → 1.0.143

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -181,6 +181,35 @@ None — platform-wide Forecast sync.
   at all. Give `ss_amq_drain` a dedicated `Not Scheduled` deployment as the on-demand slot (add more
   for concurrency) and keep a `Scheduled` one for the cron. Submit by **scriptId only** — the
   deployment id is prefix-doubled (`customdeploycustomdeploy_ss_amq_drain`).
+- **Under a create/edit burst the on-demand kick still drops with `NO_DEPLOYMENTS_AVAILABLE`, and the
+  scheduled drainer is the ONLY backstop — but its floor is 15 min, which is NOT acceptable as the normal
+  path.** When the automated process mass-creates records, enqueues outrun the `Not Scheduled` slot pool;
+  the enqueuer logs `AMQ drainer NOT kicked (cron will catch it)` and the row sits `Created` until the cron
+  sweeps it. Widening the slot pool (we run **1 Scheduled + ≥4 Not Scheduled** `ss_amq_drain` deployments)
+  raises the ceiling; `Created` is sweepable so a dropped kick is a *delay*, not a loss (only terminal
+  `Failed`, after `maxRetries`, is a true dead-letter). **But the push must stay near-real-time via the
+  kick — do not "fix" the read-races by deferring delivery to the cron.** And you **cannot add a settle
+  delay inside the SuiteScript**: server-side SuiteScript has **no `sleep()` and no `setTimeout`** (no event
+  loop); a `while(Date.now()<t){}` busy-wait would block the UE `afterSubmit` (the user's save) and trip the
+  execution-time limit. Any "let the record settle before reading" mitigation belongs on the **worker2 read
+  side** (PHP can sleep), never the enqueuer. (See the open-orders doc's REST-sublist read-after-write-lag
+  gotcha for why a settle/SuiteQL-source fix is wanted in the first place.)
+- **A `Released` UE deployment with an EMPTY audience fires for NO ONE — `Released` ≠ active.** Live
+  incident 2026-06-18: the Opportunity `ue_amq_enqueue` deployment was flipped `Testing → Released` at
+  ~4:41 pm but **Audience: All Roles / All Employees were still F**, so it enqueued nothing for ~1 h until
+  the audience was set to T at ~5:39 pm. Result: a batch of opportunities created in that window never
+  enqueued (no queue row, no worker2 job) and silently missed the Forecast import — diagnosed by the
+  enqueuer's `scriptnote` AUDIT log going quiet and the missing opps having zero log entries. When opps go
+  missing, **check the enqueuer deployment's Status AND Audience (All Roles/All Employees = T)**, not just
+  "is it Released." Backfill the gap window with `trueup_opportunities.php`.
+- **NetSuite SuiteQL renders timestamps in TWO different zones — do not compare them naïvely.** Stored
+  datetime fields (a custom record's `created`, transaction dates, `systemnote.date`) and the NetSuite UI
+  render in the **account/user TZ (Eastern here)**, but **`SYSDATE` and `scriptnote.date` render in
+  server/Pacific (UTC−7)**. Meanwhile webhook payload timestamps are UTC (`...Z`) and `WorkerJobs` is CDT
+  (UTC−5). Mixing these silently produces hours-off conclusions (it sent us down a wrong "out-of-order /
+  empty-record" path for a while). Anchor every comparison to UTC, and verify the zone of each source with a
+  known same-event cross-reference (e.g. a `scriptnote` AUDIT entry's internalId ↔ that event's UTC webhook
+  payload ↔ its `WorkerJobs.dtCreated`) before trusting a delta.
 - **worker2 routes EVERY request to `_Controller_Index::worker()`** — `worker2/_.php` sets
   `DEFAULT_CONTROLLER_METHOD = ['_Controller_Index','worker']`, which short-circuits `_Route`'s
   URI matching. There is no per-URL routing, and `.htaccess` denies direct `.php` files, so you
@@ -277,6 +306,14 @@ same **skip-if-unchanged** compare on the extracted values, and **actor-identity
 trigger a CU→NS write. The NS→CU change-detection above is the complementary backstop, not a substitute.
 
 ## Change history
+- 2026-06-18 — **Diagnosed the live opportunity-import gap as a deployment misconfig + characterized the
+  AMQ burst behavior.** The Opportunity `ue_amq_enqueue` deployment was `Released` but with an **empty
+  audience** (~4:41–5:39 pm) so it enqueued nothing; missing opps had no queue row and no worker2 job
+  (enqueuer `scriptnote` log went silent). Also recorded: under bursts the on-demand kick drops with
+  `NO_DEPLOYMENTS_AVAILABLE` (widened to 1 Scheduled + ≥4 Not Scheduled `ss_amq_drain` slots; cron is the
+  redundancy backstop but its 15-min floor is not an acceptable normal path); **SuiteScript has no
+  `sleep`/`setTimeout`** so settle delays must live on the worker2 read side; and the **SuiteQL mixed-TZ
+  rendering** trap (stored fields = account/Eastern; `SYSDATE`/`scriptnote` = server/Pacific). (dfranks)
 - 2026-06-18 — **Verified live in production** (no code change): real NetSuite SuiteScript webhooks
   (`user-agent: NetSuite/2026.1`) arrive at `webhook.togahub.com/netsuite` and process successfully
   (e.g. opp 6926224 PUT). Clarified that **`UPDATE_CLICKUP = false` is an intentional one-way guard**

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -100,6 +100,26 @@ None — uniform (platform-wide Forecast2 sync).
   handler sees them). NB: `App_ApiTransaction::execute(false)` returns a raw JSON **string** for a record
   GET — `json_decode` it directly; double-encoding it (`json_decode(json_encode($resp))`) silently yields
   an empty object and a false "empty record" reading.
+- **REST `item.items` sublist has read-after-write LAG → partial or empty syncs that freeze (this is the deeper root of the gotcha above).** The handler builds `OpenOrderItems` from the REST record GET's
+  `item.items` sublist, and that sublist **trails the committed record for seconds after edits** — it can
+  return *fewer lines than the order actually has*. Because `syncOpenLines` reconciles to exactly what it
+  read (upsert + delete-not-returned), a lagged read commits a **partial** line set (N of M) or an **empty**
+  one (0 of 1), and it **freezes there** — no later webhook fires to correct it, so the only repair is a
+  re-sync/trueup. **Confirmed live 2026-06-18 on SO 7170669** (tranId 279851): 14 PUT jobs ran **strictly
+  sequentially on one instance, in order** (`dtStarted`/`instanceId` proven — *not* concurrent, *not*
+  out-of-order); the **last** (authoritative) read at 18:03:49 committed **4 lines**, while the order has
+  **15** ($4,634 of $8,714 — short $4,080). A later REST GET returned all 15 (the sublist had caught up),
+  and SuiteQL `transactionline` showed 15 throughout. SO 7152319 is the 0-of-1 form. **Ruled OUT:**
+  concurrency/ordering (jobs were serial on one instance) and enqueue/drain (jobs all `isSuccess=1`) — so
+  "trust the current REST state" fails specifically because the REST *sublist isn't reliably current*.
+  **Fix direction (not yet implemented):** source the open lines from **SuiteQL `transactionline`** (which
+  was consistent exactly when the REST sublist lagged — same query `probe_open_order_lines.php` uses)
+  rather than the REST `expandSubResources` sublist; *or* re-read until the line count is stable before
+  committing. A `lastModifiedDate` freshness/ordering guard does **NOT** help — it isn't an ordering bug.
+  The fix **must stay real-time** — do **not** defer to the 15-min cron (see the AMQ real-time constraint /
+  `project_amq_realtime_requirement` in dev memory). Diagnose with `probe_open_order_lines.php` (SuiteQL vs
+  Forecast) + `probe_so_rest_lines.php` (live REST sublist count) + the `WorkerJobs` `dtStarted`/`instanceId`
+  timing (to confirm serial-vs-concurrent before blaming ordering).
 - **REST shape ≠ SOAP shape.** The cron reads the SOAP-shim shape; this handler reads the REST record
   (`status->refName`, line `quantityBilled`, `class->refName`, `entity->id`, `salesRep->id`,
   `shippingCost`). Verified against live orders via `test/@dave/probe_salesorder_rest_shape.php`.
@@ -154,6 +174,15 @@ and are a candidate to extract into a shared `_Component_Forecast_Db` before the
 
 ## Change history
 
+- 2026-06-18 — **Root-caused the open-order delta to REST `item.items` read-after-write lag** (not the
+  drainer/enqueue fixes, not ordering). SO 7170669 committed 4 of 15 open lines despite 14 *strictly
+  sequential, single-instance, in-order* PUTs — the last authoritative read returned a lagged partial
+  sublist; REST later caught up to 15 and SuiteQL showed 15 throughout. Disproved the earlier
+  out-of-order/concurrency hypothesis (verified via `WorkerJobs.dtStarted`/`instanceId`) and a
+  `lastModified` ordering guard (irrelevant — reads were in order). Fix direction recorded: source open
+  lines from **SuiteQL `transactionline`** instead of the REST sublist (or re-read until line count is
+  stable), keeping the path real-time (no cron deferral). New diagnostic: cross-check
+  `probe_open_order_lines.php` (SuiteQL) vs `probe_so_rest_lines.php` (REST sublist). (dfranks)
 - 2026-06-18 — **Verified live in production** after the SalesOrder enqueuer left Testing. 14 orders
   reconciled against NetSuite SuiteQL (`probe_open_order_lines.php`); 13 at $0.00 delta. Found one
   miss — SO 7151688 (open, ~$20.31, 0 rows) — traced to the **async-recalc race** above (see new

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.141",
+  "version": "1.0.143",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",