toga-ai 1.0.787 → 1.0.789

package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-08
+updated: 2026-09-09
 owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
 files:
   - worker/crons/toga2/netsuite/common_sync_togasupply.php
@@ -257,6 +257,45 @@ again, the sections still could **not advance** — the value was in the **wrong
   This is the same SuiteQL-vs-REST clock trap as the date-only-format bug above — see the
   [SuiteQL/REST shim doc](../../library/features/netsuite-suiteql-rest-shim.md).
 
+### ⚠ …and the cursor was READ on the wrong clock too — `strtotime()` (Chicago) shifted the fetch window +1h → SILENT record skips across ALL 6 sections + ALL clients (found + fixed 2026-09-09, NYCHH prod)
+
+The deepest of the cursor-clock bugs, and the one that actually **dropped records that never came
+back**. The two clock bugs above are on the **write** side of the cursor; this one is on the
+**read** side of the same value, and it was silent — no crash, no alert, no frozen section. It
+hit **every section and every client**, not just the by-reference ones.
+
+- **Root cause — read clock ≠ fetch clock.** The cursor value `"<lastmoddate>|<netsuiteId>"` stores
+  its datetime in the NetSuite account timezone = **America/New_York (Eastern)** — the same clock the
+  list-fetch `WHERE` bounds render in (`App_Api_Netsuite_Rest::listItemFulfillments` et al. call
+  `->setTimezone('America/New_York')`). But `startModeIteration()`
+  (`common_sync_togasupply.php` ~L622) read it back with **`strtotime()`**, which parses in the
+  **script timezone America/Chicago** (set in library `_.php`). Chicago is 1h behind Eastern, so the
+  same digits produced an epoch **1 hour later**; the list fetch then re-rendered that bound in
+  Eastern, shifting the window's **lower bound +1 hour**.
+- **Effect — a silent fetch-level skip.** Any record whose Eastern lastmod fell in that **1-hour
+  shadow** right after the previous checkpoint was **excluded from the fetch**; the window read empty,
+  and the `|0` empty-window jump advanced the cursor **past** it. Nothing threw — a throw still
+  freezes the section loudly; this is the quiet path that loses data. Combined with the giant-record
+  900s timeouts (below) that historically froze sections and were then manually stepped over, the
+  gaps compounded over time.
+- **Proven on NYCHH prod** (item fulfillment `6138135`, lastmod **2025-09-04 17:26:58 Eastern**): the
+  +1h bound (`> 17:29:03`) returned **0** records; the correct bound (`> 16:29:03`) returned **1**
+  (that record). The cursor trace showed the `|0` empty-window jump right over its time slot.
+- **Fix (worker only, deployed + verified):** parse the cursor as Eastern in `startModeIteration` —
+  `$epochLastIntegration = (new \DateTime($dtLastIntegration, new \DateTimeZone('America/New_York')))->getTimestamp();`
+  — and change all **6** empty-window `|0` jump writes (~L831/938/1024/1193/1281/1369) from
+  `App_Date::convertToSQLDate(...)` (renders Chicago) to
+  `App_Date::convertToTimezone($epoch, 'Y-m-d H:i:s', 'America/New_York')`, so the **whole** cursor
+  pipeline — read + per-record write + empty-window write — runs on **one** clock (Eastern). DST-safe
+  (named IANA zone). php-reviewer + cto both **AGREE**; cto flagged (and we applied) the
+  empty-window-write hardening to stop a permanent 1h backward bias building up on idle runs. Verified
+  after setting NYCHH's fulfillment cursor to `2025-09-04 00:00:00|0`: `6138135` imported.
+- **The rule (now proven THREE times — this bug + the two write-clock bugs above):** every touch of
+  the cursor value — the **read**, the per-record **write**, and the empty-window **write** — must use
+  the NetSuite account timezone (**Eastern**), never the script's Chicago clock and never the REST
+  GET's UTC. `strtotime()` and `App_Date::convertToSQLDate()` both use the script tz (Chicago) and are
+  the trap. Any manual cursor rewind value you set by hand must also be **Eastern**.
+
 ### ⚠ Per-record isolation is SECTION-SPECIFIC — and SALES_ORDERS has NONE (corrected 2026-08-17)
 
 Whether a bad record is "stepped over" or freezes the section depends entirely on whether **that
@@ -454,6 +493,19 @@ See the [tracking-number bridge doc](../../../2.0/apps/_underscore/features/trac
   0 of their fulfillments resolve an upstream sales order). `/units` writes and all three
   tracking-bridge models have **no** interceptor, and **DELETE fires none**.
 
+- **The two `GET /sales-orders` calls use a narrow `fields` projection, NOT `depth` (2026-09-09).**
+  Both the sales-order lookup and the missing-SO refetch in `syncItemFulfillmentFromNetsuite`
+  (`toga2.php` ~L4644-4706) request only `uuid`,
+  `salesOrderItems.{uuid,lineNumber,quantity,c_netsuiteLineUniqueKey}`, and
+  `salesOrderItems.item.{uuid,partNumber}` — the only fields the line-matcher reads. They previously
+  sent `depth=5`, whose huge nested read (units, tracking, PO links) made api2 exceed the worker's
+  **900s cURL** limit and threw **before** the `ItemFulfillments` row was written, freezing the
+  section (`Logs.Issue` 666, `apitransaction.php:418`). SO GETs dropped 100s+ → ~1s. Same class of fix
+  as the 2026-09-08 inventory-adjustments `depth=5→4` change; a `fields` projection is the proven
+  pattern here (already used by the `/item-fulfillments` GET in this same method) and dodges the api2
+  max-depth FK omission. Keep new fields explicit in the projection — a needed field left out of the
+  list silently reads `null`, not an error.
+
 ### Single-tracking-number fan-out to items (318) and units (319) — corrected 2026-08-26
 
 The earlier "maintains all three bridges" claim was **only true for 317 (header)**: in prod
@@ -1387,6 +1439,30 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
 
 ## Change history
 
+- 2026-09-09 — **The resume cursor was READ on the wrong clock — a silent +1h window skip dropped
+  records across ALL 6 sections and ALL clients (worker, deployed + verified).** `startModeIteration`
+  (`common_sync_togasupply.php` ~L622) parsed the Eastern-stored cursor with `strtotime()` (script tz
+  America/Chicago), so the fetch's lower bound rendered **1 hour high** and any record in that 1-hour
+  shadow was excluded; the empty window then `|0`-jumped past it — no crash, no alert. Fixed by parsing
+  the cursor as `America/New_York` and rewriting the 6 empty-window `|0` jump writes with
+  `App_Date::convertToTimezone(..., 'America/New_York')`, so read + per-record write + empty-window
+  write all run on one clock (Eastern). Proven on NYCHH IF `6138135` (Eastern 2025-09-04 17:26:58): the
+  +1h bound returned 0 records, the correct bound returned 1. php-reviewer + cto AGREE; cto's
+  empty-window-write hardening applied. Now the THIRD cursor-clock bug — the standing rule is every
+  touch of the cursor value uses Eastern, never Chicago (`strtotime`/`convertToSQLDate`) and never the
+  REST GET's UTC. See the read-clock gotcha in *How it works*. (jcardinal)
+- 2026-09-09 — **Fulfillment sales-order GET at `depth=5` hit the worker's 900s cURL timeout and froze
+  the section (library, deployed + verified).** Both `GET /sales-orders` calls in
+  `App_Api_Toga2::syncItemFulfillmentFromNetsuite` (`toga2.php` ~L4644-4706 — the lookup + the
+  missing-SO refetch) changed from `depth=5` to a narrow `fields` projection (uuid;
+  `salesOrderItems.{uuid,lineNumber,quantity,c_netsuiteLineUniqueKey}`;
+  `salesOrderItems.item.{uuid,partNumber}` — the only fields the matcher reads), dropping the huge
+  nested read (units, tracking, PO links) that pushed api2 past 900s and threw before the
+  `ItemFulfillments` row was written (`Logs.Issue` 666, `apitransaction.php:418`). Same class as the
+  2026-09-08 inventory-adjustments depth fix; the `fields` projection is the proven pattern (already
+  used by the `/item-fulfillments` GET in this method). Fulfillment SO GETs dropped 100s+ → ~1s. Both
+  fixes address the NYCHH missing-fulfillment investigation; library + worker still deploy together for
+  this sync path. (jcardinal)
 - 2026-09-08 - **A `Closed` NetSuite order never reaches the transfer-order sync** - the
   `status === 'Closed'` `continue` at `common_sync_togasupply.php` ~L746 sits **before** the
   `isTransferOrder()` branch, so `syncTransferOrderFromNetsuite`'s `case 'Closed'` was dead code and

package/knowledge/2.0/apps/dbchanges2/architecture.md

@@ -6,7 +6,7 @@ project: Database Changes
 client: shared
 type: architecture
 status: active
-updated: 2026-09-04
+updated: 2026-09-09
 owners: [jcardinal, mhammontree, bala, ajean]
 files:
   - Core/
@@ -72,19 +72,21 @@ team-maintained `id`s** — ask the developer for the next value before insertin
 > trailing-space quirk like `2026-06-03- BLANK_CLIENT_DATABASE.sql`). The **only** hard rule
 > is the leading `YYYY-MM-DD<letter>` so ordering holds.
 
-## One file per type per build (keep change files consolidated)
+## One file per realm per turn (keep change files consolidated)
 
-**Prefer a single `.sql` file per database type (folder) per build.** When a session produces
-several changes for the same target — e.g. three separate files under `Core/`, or four under
-`Client/` — **consolidate them into one dated file** for that folder instead. Multiple same-type
-files in one build are not forbidden, but they have become too common; keep it to one per type
-unless they genuinely must be separate (for example, an ordering gap where another folder's file
-must run between them).
+**Prefer a single `.sql` file per realm (folder) per turn.** A *realm* is the target database a
+folder maps to — `Core`, the shared `Client`, a single tenant like `Client_True`, `Logs`, a module,
+and so on. When one response produces several changes for the **same** realm — e.g. three separate
+files under `Core/`, or four under `Client/` — **consolidate them into one dated file** for that
+folder instead of writing several. This is a **strong preference**, not an absolute rule: multiple
+same-realm files in one turn are allowed when they genuinely must be separate (for example, an
+ordering gap where another folder's file must run **between** them).
 
-- Combine same-folder statements into one `YYYY-MM-DD<letter> - <Description>.sql`.
-- Different folders (types) stay in their own files — that split is required (see isolation).
-- The `dbchanges2-file-sprawl` hook warns when a second file of the same type is created in a
-  session, as a reminder to consolidate.
+- Combine same-realm statements into one `YYYY-MM-DD<letter> - <Description>.sql`.
+- Different realms (folders) stay in their own files — that split is **required** (see isolation);
+  never mix two realms in one file.
+- The `dbchanges2-file-sprawl` hook warns when a second file for the same realm is created in a
+  turn, as a reminder to consolidate.
 
 ## Folder → database mapping
 
@@ -362,12 +364,23 @@ its own header.)
    the relevant clients list `<module>` in their `_modules.txt`.
 5. Never edit or re-date an already-applied file — add a new dated file instead. Never put new
    work in a `HISTORIC` folder.
-6. **Never seed a `uuid` column with MySQL `UUID()`.** The 2.0 standard requires a fully-random
-   **v4** UUID; `UUID()` is time/MAC-based (v1) and violates it. Since a migration can't call
-   `_String::generateUuid()`, **pre-generate a v4 UUID literal** (e.g. a v4 generator) and
-   hardcode the literal string in the `VALUES` clause. One earlier migration
-   (`Core/2026-05-08 … CronJobs_WorkerCleanup_insert`) used `UUID()` — that is a known violation,
-   not a precedent to copy. First applied correctly in
+6. **Never seed a `uuid` column with MySQL `UUID()` — not even in a bulk insert.** The 2.0 standard
+   requires a fully-random **v4** UUID; `UUID()` is time/MAC-based (v1) and violates it. Since a
+   migration can't call `_String::generateUuid()`:
+   - **Single-row insert:** **pre-generate a v4 UUID literal** (a real v4 generator) and hardcode
+     the literal string in the `VALUES` clause.
+   - **Multi-row / bulk `INSERT … SELECT`:** use the inline `RANDOM_BYTES()` v4 expression (see
+     `features/rerunnable-additive-inserts.md` → *Generating a real UUID4 inline*), never `UUID()`.
+
+   **Every UUID must be freshly, fully random — never derive one from another.** Do **not** take a
+   known uuid and bump a digit (`…-0001`, `…-0002`, …), copy-and-tweak, or otherwise pattern them.
+   When a file needs several literals, generate each one independently at random. (Patterned uuids
+   are guessable and defeat the uniqueness guarantee.) The one intended exception is a fan-out
+   `Client/` grant, which **reuses the same** literal per row across tenants on purpose for safe
+   rollback — that is deliberate reuse, not incrementing (see the feature doc).
+
+   One earlier migration (`Core/2026-05-08 … CronJobs_WorkerCleanup_insert`) used `UUID()` — that is
+   a known violation, not a precedent to copy. First applied correctly in
    `Core/2026-07-21b - Insert - SprintLockScheduled CronJob.sql`.
 7. **Guard every registration/seed insert with `NOT EXISTS` (or `INSERT IGNORE`).** Rows that
    register platform behavior — `ApiPayloadInterceptors`, `Core.RecordFields`/`CustomRecordFields`,
@@ -562,8 +575,9 @@ Wrap the existing-set subquery in a **derived table** (with `DISTINCT` or `LIMIT
 
 ```sql
 -- CORRECT — the derived table snapshots the "already present" set before any insert
+-- (uuid: inline RANDOM_BYTES() v4 expression, never UUID() — see rule #6 / the feature doc)
 INSERT INTO TranscriptPromptTerms (uuid, category, term, isActive, c_trueAiModelId)
-SELECT UUID(), t.category, t.term, t.isActive, m.mid
+SELECT LOWER(CONCAT(HEX(RANDOM_BYTES(4)), '-', …)), t.category, t.term, t.isActive, m.mid
 FROM <base rows> t
 CROSS JOIN <active model ids> m
 WHERE m.mid NOT IN (
@@ -591,6 +605,15 @@ tables that `_Model_*` classes map to.
 
 ## Change history
 
+- 2026-09-09 — **Two team-wide authoring changes (jcardinal).** (1) **UUIDs must be freshly, fully
+  random — never derived from another.** Sharpened rule #6: do not bump a digit / copy-and-tweak /
+  pattern uuid literals; generate each independently at random. Reaffirmed **never `UUID()` — not
+  even in bulk** (bulk uses the `RANDOM_BYTES()` v4 expression); fixed the doc's own
+  `TranscriptPromptTerms` example, which still used `UUID()`. The fan-out `Client/` same-literal
+  reuse is the one intended exception (deliberate reuse for safe rollback, not incrementing).
+  (2) **One file per realm per turn** — renamed/strengthened the consolidation section: prefer a
+  single `.sql` file per realm (`Core`, `Client`, `Client_<Name>`, …) per turn; a strong preference,
+  not absolute; different realms still split (never mix two realms in one file). (jcardinal)
 - 2026-08-18 - Added **rule 10**: a fan-out `Client/` file must never reference a
   `customRecordFieldId` or a client-specific label, because `CustomRecordFields` ids are per-client
   `AUTO_INCREMENT` - the same numeric id names a different field in every tenant and mis-wires

package/knowledge/2.0/apps/dbchanges2/features/rerunnable-additive-inserts.md

@@ -6,7 +6,7 @@ project: Database Changes
 client: shared
 type: feature
 status: active
-updated: 2026-09-08
+updated: 2026-09-09
 owners: [bala, kyalamarthi, apeterson, ajean, jcardinal]
 files:
   - dbchanges2/Client_Compass/2026-08-26a - CompassCreativeStudioPersona.sql
@@ -52,6 +52,14 @@ and `SUBSTRING('89ab', …)` picks the variant nibble. Verified working on prod
 that is *not* v4-shaped (older migrations concatenated `RANDOM_BYTES` without the version/variant
 nibbles). They are fine and must not be "fixed" — but do not use them as the template for new SQL.
 
+**⚠ When a file needs several hardcoded literals, generate each one independently at random — never
+derive one from another.** Do not paste one uuid and bump a digit (`…-0001`, `…-0002`, …) or
+copy-and-tweak to make the next. Patterned uuids are guessable and undercut the uniqueness the column
+exists to give. Generate each literal fresh from a real v4 generator. (The **one** intended exception
+is the fan-out rule just below, which reuses the **same** literal per row across tenants on purpose —
+deliberate reuse for a safe rollback, not incrementing.) `UUID()` is never allowed, in single-row or
+bulk inserts — bulk uses the `RANDOM_BYTES()` expression above.
+
 ### ⚠ In a FAN-OUT grant migration, use HARDCODED uuid literals — generated uuids make rollback unsafe
 
 The generator above is right for a normal additive insert. It is **wrong for a `Client/` fan-out
@@ -310,6 +318,11 @@ concluding a migration is production-safe.
   [FIELD_SQL calculated fields](../../_underscore/features/calculated-sql-fields.md).
 
 ## Change history
+- 2026-09-09 — Added the rule that **every hardcoded uuid literal must be generated independently at
+  random** — never bump a digit / copy-and-tweak / pattern them (guessable, defeats uniqueness); the
+  fan-out same-literal reuse is the one intended exception. Restated **never `UUID()`, single-row or
+  bulk** (bulk uses the `RANDOM_BYTES()` v4 expression). Team-wide authoring change alongside
+  architecture rule #6. (jcardinal)
 - 2026-09-08 — Extended the **`Core.CronJobs` registration** shape with three mechanics confirmed on
   `Core/2026-09-08e - Insert QcBatch CronJob.sql`: reference `CronJobs` **unqualified** in a `Core/`
   file (a `Core.` prefix is a cross-database reference that breaks in prod); **always set

package/knowledge/clients/nychh/profile.md

@@ -18,7 +18,7 @@ project: _Underscore
 client: nychh
 type: profile
 status: active
-updated: 2026-09-08
+updated: 2026-09-09
 owners: ["jcardinal", "apeterson", "bala", "akhokhani"]
 files:
   - dbchanges2/Client_Nychh/2026-09-02a - TransferOrderNetsuitePushInterceptor.sql
@@ -85,6 +85,27 @@ table views. Client-specific DB change-sets live in `dbchanges2/Client_Nychh/`.
   `NETSUITE_LAST_SYNC_CURSOR_ITEM_RECEIPTS` is stale at 2024-08-06**, so re-enabling that feed
   replays ~2 years. See
   [NYCHH NetSuite → TransferOrders import](./features/netsuite-transfer-order-import.md).
+- **Missing NetSuite records (item fulfillments + others) root-caused and fixed (2026-09-09).** Two
+  separate bugs in the 1.0 NetSuite→TOGa Supply sync starved NYCHH imports, both deployed and verified
+  in prod: (1) the resume cursor was **read on the wrong clock** (`strtotime()` in Chicago vs the
+  Eastern-stored value), shifting the fetch window +1h and **silently skipping** any record in that
+  1-hour shadow — this hit all 6 sections and all clients; and (2) the fulfillment `GET /sales-orders`
+  ran at `depth=5` and **timed out at 900s**, freezing the section before the fulfillment row was
+  written. Proven on IF `6138135` (Eastern 2025-09-04 17:26:58). Fixes and the standing "cursor is
+  always Eastern" rule live in the
+  [per-client sync feature doc](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
+- **NYCHH sync scope + data-gap scale + backfill approach (2026-09-09).**
+  - **Scope = 22 NetSuite customers:** parent **28908** "NYC Health + Hospitals" + 21 hospital child
+    customers (`24145, 29273, 29276, 31584, 31910-31916, 31918-31925, 32229, 33674`).
+  - **Gap report (NetSuite scope vs `Client_Nychh` imported):** Item Fulfillments **2986 vs 2472**
+    (~514 missing, 17%); Sales+Transfer Orders 2121 NS vs 2098 TOGa (159 SalesOrders + 1939
+    TransferOrders — roughly complete); Invoices **2128 vs 213** (~1915 missing, 90% — but that is
+    mostly because the **invoices feed was toggled OFF on 2026-09-03**, not the timezone bug).
+    POs/Receipts/Adjustments are vendor/location-scoped (not measured) and several are toggled off.
+  - **Backfill approach:** prefer a **targeted re-import of the specific missing NetSuite ids** (diff
+    NetSuite scope vs TOGa) over a **blind cursor rewind** — a full rescan re-pulls thousands of heavy
+    serial records and stresses the shared NetSuite auth quota. Re-import is safe (matches on the
+    NetSuite internal id). Any manual cursor rewind value must be written in **Eastern**.
 - 🚩 **OPEN, not root-caused (2026-09-08): a duplicate item INSERT aborts the whole NYCHH sales-order
   sync run.** `Items` id 5429 (658-BFZH) was created and then re-inserted in the same run —
   `Duplicate entry '26-658-BFZH-1' for key 'Items.manufacturerId_partNumber_catalogId'` (EV-10). The

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.787",
+  "version": "1.0.789",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",