toga-ai 1.0.424 → 1.0.426

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-07-21
+updated: 2026-07-23
 owners: [dfranks, jcardinal]
 files:
   - test/@dave/checker.php
@@ -303,6 +303,36 @@ by reconciling a chosen tranDate range directly against NetSuite.
   the sale-import doc). The salesRep dimension itself is the JE line custom column `custcol_sales_rep_line`
   (wired under TRUE-79862).
 
+- **Delete-safety is a targeted BY-ID RE-FETCH from NetSuite, NOT a `deletedrecord` system-log
+  lookup — mirror this in any new reconciler.** The fixer never blind-deletes a Forecast row just
+  because a windowed pull didn't return it. Its delete path (`fixer.php` Sales path ~773-891, and
+  `checker.php`) works by **re-fetching each discrepant transaction by its internal id** from
+  NetSuite via the local `fetchSalesBulk()` shim (`WHERE transaction IN (...)` SuiteQL, same shape
+  as the legacy `listSales()`): it upserts the lines NetSuite returns for that id and **deletes only
+  the FC lines NetSuite no longer returns for that specific id**. If NS returns **nothing** for the
+  id, the txn is gone → its FC rows are deleted. This by-id re-fetch is the concrete implementation
+  of the team's **"no blind auto-delete"** rule — it guards against a transient window/read miss
+  masquerading as a deletion (a `lastmodifieddate` window or a paged full-window scan can transiently
+  omit a row; a per-id GET/IN-list re-confirms it). A NetSuite **`deletedrecord` system-log gate is
+  NOT existing behavior** and should not be invented as the delete confirmation — the authoritative
+  signal is "the by-id re-fetch returns no lines."
+- **Why an independent tranDate-range reconciliation is MANDATORY, not just a nicety — NetSuite event
+  capture cannot be guaranteed by any configuration.** Beyond the sublist line-field inline-edit blind
+  spot (below), a broader class of NetSuite changes fires **no** User Event SuiteScript at all, so the
+  AMQ enqueuer never runs, no webhook is posted, and `lastmodifieddate` is frequently **not** bumped:
+  - **Bulk / mass updates and CSV imports** — a CSV import only runs server SuiteScript (and workflows)
+    when **"Run Server SuiteScript and Trigger Workflows"** is enabled, which is **OFF by default**.
+  - **Changes made BY another script or workflow** — a UserEvent script **cannot be triggered by
+    another UserEvent script or a workflow**, so a server-side edit never re-fires the enqueuer UE.
+
+  Consequence: such edits drift `Forecast.Sales` (and any webhook-synced NetSuite data) **silently and
+  invisibly to BOTH** the real-time webhook path **and** a `lastmodifieddate`-windowed pull cron —
+  strictly broader than the already-known inline sublist line-field case. Durable implication: because
+  event capture is not guaranteeable, **correctness requires an independent reconciliation that reads
+  NetSuite by `tranDate` range and matches on internal id** (exactly what `checker`/`fixer` do) — this
+  is the load-bearing reason the reconciliation backstop exists, not an optimization. (Oracle docs:
+  server scripting on CSV import `section_4676525683`; how UE scripts are executed `section_1512409310`.)
+
 ## Data model
 
 `Forecast.Sales`, `Forecast.OpenOrderItems` on the **core2** cluster
@@ -572,6 +602,19 @@ None — Forecast2 is a single shared dataset.
 
 ## Change history
 
+- 2026-07-23 — **Recorded WHY the reconciliation backstop is mandatory + the by-id re-fetch
+  delete-safety mechanism** (TRUE-80262 planning; no code shipped, dfranks). Documented the broader
+  NetSuite event-capture blind spot beyond the inline sublist edit: **bulk/mass updates and CSV
+  imports fire no UE** (CSV runs server SuiteScript only when "Run Server SuiteScript and Trigger
+  Workflows" is on — OFF by default), and a **UE cannot be triggered by another UE/workflow**, so
+  script/workflow-driven edits drift `Forecast.Sales` invisibly to BOTH the webhook path and a
+  `lastmodifieddate`-windowed cron (and often don't bump `lastmodifieddate`). Durable implication:
+  event capture can't be guaranteed by any NetSuite config, so an independent `tranDate`-range
+  reconciliation matched on internal id is required for correctness (Oracle docs `section_4676525683`
+  / `section_1512409310`). Also documented the fixer/checker **delete-safety = targeted by-id re-fetch**
+  (`fetchSalesBulk` shim; delete FC lines NS no longer returns for that id) as the concrete "no blind
+  auto-delete" implementation — a `deletedrecord` system-log gate is NOT existing behavior and must not
+  be invented by a future reconciler. (dfranks)
 - 2026-07-21 — **Documented `reconcile_drift_2023plus.php` + the SOAP-era double-line bug (2023)**
   (folded in from a retired project-local CLAUDE.md). The SOAP-era importer ran twice for some
   invoice batches and inserted each invoice's lines twice under consecutive-but-different line

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-07-14
+updated: 2026-07-23
 owners: [dfranks]
 files:
   - worker2/Component/Forecast/SaleImport/SaleImport.php
@@ -360,6 +360,21 @@ evidence: two invoices whose ONLY change that day was a cost line-edit had **zer
 entries, while 8 other cost-edited invoices that also had normal create/edit activity **did** fire
 and synced. This is recurring/multi-user (~12/day; 35 `MCOSTESTIMATE` edits in 3 days by 9 users).
 
+**Broader than the inline line-edit — a whole CLASS of NetSuite changes fires no UserEvent, so no
+webhook is posted (and `lastmodifieddate` is often not bumped):**
+- **Bulk imports / mass updates** and **CSV imports** — a CSV import runs server SuiteScript (and
+  workflows) **only** when **"Run Server SuiteScript and Trigger Workflows"** is enabled, which is
+  **OFF by default**; so the AMQ enqueuer UE never runs for a bulk/CSV load.
+- **Changes made BY another script or workflow** — a **UserEvent script cannot be triggered by
+  another UserEvent script or by a workflow**, so a server-side edit never re-fires the enqueuer.
+
+So event capture **cannot be guaranteed by any NetSuite configuration** — these edits drift
+`Forecast.Sales` (and any webhook-synced NetSuite data) **silently and invisibly to BOTH** the
+real-time webhook **and** a `lastmodifieddate`-windowed pull cron. This is the durable reason the
+independent tranDate-range reconciliation (`checker`/`fixer`, see the reconciliation doc) is
+**mandatory, not optional**. (Oracle docs: CSV server scripting `section_4676525683`; how UE scripts
+are executed `section_1512409310`.)
+
 **Corollary gotcha: `lastmodifieddate` is NOT a reliable change signal for line-cost edits** — a
 line-field inline edit leaves it untouched, so **both** the webhook pipeline **and** any
 `lastmodifieddate`-windowed `list*()`/cron sync miss it entirely. **Only reconciliation**
@@ -574,6 +589,14 @@ success from a `Forecast.Sales` row alone.
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-07-23 — **Broadened the AMQ event-capture blind spot beyond inline line-edits** (TRUE-80262
+  planning; no code shipped, dfranks). Recorded that **bulk/mass updates and CSV imports** fire no UE
+  (CSV runs server SuiteScript only when "Run Server SuiteScript and Trigger Workflows" is on — OFF by
+  default) and that a **UE cannot be triggered by another UE or a workflow** — so script/workflow/bulk
+  edits post no webhook and often don't bump `lastmodifieddate`, drifting `Forecast.Sales` invisibly to
+  both the webhook path and a `lastmodifieddate`-windowed cron. Durable implication: NetSuite event
+  capture is not guaranteeable by any config, making the independent tranDate-range reconciliation
+  mandatory (Oracle docs `section_4676525683` / `section_1512409310`). (dfranks)
 - 2026-07-14 — **JE lines now resolve `customerId` (was hard-coded null) + corrected the
   engine's repo location to worker2** (TRUE-80129, dfranks). `buildJournalEntryRows` reads
   `JE_LINE_CUSTOMER_FIELD = 'entity'` — a **native** JE-line reference field (not a `custcol_*`,

package/knowledge/2.0/apps/worker2/features/monitoring-framework.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-29
-owners: [mhammontree]
+updated: 2026-07-23
+owners: [mhammontree, dfranks]
 files:
   - worker2/Worker/Monitor.php
   - worker2/Worker/Monitors/
@@ -54,6 +54,35 @@ orchestrator) · runtime config in the `Core.Monitors` table (no redeploy).
 Reused with **no changes**: `Core.CronJobs`, `Core.WorkerJobs`, the `WorkerCronScheduler`
 Lambda, the EB worker tier + SQS delivery (see [worker2 architecture](../architecture.md)).
 
+## Where monitor / data-quality results belong (design placement)
+
+worker2 has **two** monitoring patterns and neither writes to the central **`Logs` Issues/Events**
+tables. That is deliberate — **`Logs.Issues`/`Logs.Events` are strictly for application
+errors/exceptions** that escalate to ClickUp or email (business-routed by `IssueEmailAddresses`
+presence; see the escalation-cron work). **Periodic health-check, integration-health, and
+data-quality RESULTS do NOT go there** — they belong in the Monitor framework:
+
+- **Pattern A — internal DB-driven state machine** (this doc): a `Core.Monitors` row +
+  `_Worker_Monitor::Run(monitorId)` (`worker2/Worker/Monitor.php`) invokes a child `phpClass::Run()`
+  that returns `{isOk, message}`; the orchestrator manages ok/alert state, `consecutiveOkCount` /
+  `requiredConsecutiveOks` anti-flap, and sends alert/reminder/recovery mail via
+  `_Worker_Notification_Email::Send`, persisting state back to `Monitors`.
+- **Pattern B — "dumb reporter, smart monitor"** (see
+  [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)): a child measures one metric
+  and POSTs a JSON body to a OneUptime incoming-request (heartbeat) monitor; the **threshold/alerting
+  lives in OneUptime** (tunable without a deploy). This is the right home for integration-health /
+  data-quality checks.
+
+So when deciding where a recurring check/reconciliation **result** goes, the answer is the Monitor
+framework (Pattern B / OneUptime for integration-health and data-quality), **not** Issues/Events.
+Key Pattern-B idioms (durable): the worker decides the threshold and emits a **string token**
+OneUptime matches (`"alarm":"HIGH"` → offline/incident, because OneUptime can only string-match a
+pushed body, not compare numbers); on a data-source failure POST an explicit **`status:error`** so
+OneUptime distinguishes "metric high" from "checker is blind"; the `_ApiRequest` push uses
+`setLogging(false)` + `setThrowExceptionsOnFailure(false)` so a failed ping never fails the job or
+depends on the Logs DB; the heartbeat URL is a **push credential — never log it**; and the
+human-readable result is the return string persisted in `WorkerJobs.output`.
+
 ## How it works
 
 One `Core.CronJobs` row per monitor (`action = 'Monitor/Run'`, `parameters =
@@ -237,6 +266,14 @@ clients' data flows (Compass, Prudential, AIG, Rate, …) but live as separate c
   HTML email + dashboard deep-links · anti-flap on the alarm side.
 
 ## Change history
+- 2026-07-23 — **Recorded the design-placement decision: monitor / data-quality RESULTS belong in the
+  Monitor framework (Pattern B / OneUptime for integration-health & data-quality), NOT in the central
+  `Logs` Issues/Events tables** (which are strictly for escalating application errors/exceptions).
+  Summarized both patterns side by side and the durable Pattern-B idioms (worker decides the threshold
+  and emits a string token OneUptime matches; explicit `status:error` on data-source failure; the push
+  is `setLogging(false)`+`setThrowExceptionsOnFailure(false)` so a failed ping never fails the job; the
+  heartbeat URL is a push credential — never logged; result string persisted in `WorkerJobs.output`).
+  TRUE-80262 planning; no code shipped. (dfranks)
 - 2026-06-29 — Built the first child monitor, `_Worker_Monitors_RateEntitlement` (TRUE-79129) + change-set `2026-06-29a`; added the log-scan worked example, the `_underscore` outbound-API-log detection pattern, and confirmed child details (orchestrator never calls `initialize()` so children self-register non-Core connections; no commit on read-only `Run()`; heredoc cannot interpolate `self::CONST`). Re-flagged that `Core.Monitors` is still local-only — the new change-set hard-fails where the table is absent. (mhammontree)
 - 2026-06-10 — Documented the v1.0 monitoring framework (orchestrator, `Core.Monitors` table, recovery-side anti-flap state machine, child contract). First child monitor + staging/prod migration still pending. (mhammontree)
 

package/knowledge/clients/rate/INDEX.md

@@ -8,5 +8,5 @@
 | [Rate SAML SSO](features/saml-sso.md) | 2.0 | Rate uses Azure AD as its IdP (`login.rate.com`). | _underscore/Model/Rate/ClientAuthentication.php, saml/Controller/Index.php, toga2-view/src/hooks/useAuthenticationFlow.ts |
 | [Service Card Entitlement Display](features/service-card-entitlements.md) | 2.0 | Rate's home and services pages display one service card per purchased entitlement. | src/components/ServiceCard/ServiceCard.tsx, src/components/ServiceCard/index.ts, src/hooks/useBundleServices.ts, src/hooks/useActiveServices.ts, src/pages/Home/api/homeApi.ts, src/pages/Home/view/HomePage.tsx, src/pages/Home/viewModels/useHomePageViewModel.ts, src/pages/Services/view/ServicesPage.tsx, src/pages/Services/viewModels/useServicePageViewModel.ts, src/api/serviceAddressApi.ts, src/api/apiErrors.ts |
 | [Rate Service-Purchase Confirmation Emails (Tech / Warranty)](features/service-purchase-emails.md) | 2.0 | When a Rate customer purchases a service, a confirmation email is sent. | _underscore/Model/Rate/Entitlement.php, worker2/Worker/Notification/EmailTemplate.php, dbchanges2/Client_Rate/2026-06-30a - Rate purchase email templates.sql |
-| [Rate Whole Home Warranty Per-Address Purchase Guard](features/whole-home-warranty-purchase-guard.md) | 2.0 | > **⚠ DEPLOYING beta→production, NOT YET PROD-VERIFIED (as of 2026-07-23).** TRUE-79533 is > beta-verified (PM Paulina tested the WH purchase flow on beta) and | _underscore/Model/Rate/Entitlement.php, _underscore/Model/Client/Entitlement.php, _underscore/Model/Client/Address.php, dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql, dbchanges2/Core/2026-07-22a - EntitlementServiceAddressIdField.sql, dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql, test/@Mark/Rate/verify_wholehome_per_address_guard.php |
+| [Rate Whole Home Warranty Per-Address Purchase Guard](features/whole-home-warranty-purchase-guard.md) | 2.0 | > **⚠ DEPLOYING beta→production, NOT YET PROD-VERIFIED (as of 2026-07-23).** TRUE-79533 is > beta-verified (PM Paulina tested the WH purchase flow on beta) and | _underscore/Model/Rate/Entitlement.php, _underscore/Model/Client/Entitlement.php, _underscore/Model/Client/Address.php, dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql, dbchanges2/Core/2026-07-22a - EntitlementServiceAddressIdField.sql, dbchanges2/Client_Rate/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql, test/@Mark/Rate/verify_wholehome_per_address_guard.php |
 | [Rate](profile.md) | 2.0 | Rate is a mortgage/lending client. |  |

package/knowledge/clients/rate/features/whole-home-warranty-purchase-guard.md

@@ -14,7 +14,7 @@ files:
   - _underscore/Model/Client/Address.php
   - dbchanges2/Client/2026-07-22a - EntitlementServiceAddressId.sql
   - dbchanges2/Core/2026-07-22a - EntitlementServiceAddressIdField.sql
-  - dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql
+  - dbchanges2/Client_Rate/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql
   - test/@Mark/Rate/verify_wholehome_per_address_guard.php
 related:
   - clients/rate/profile.md
@@ -32,11 +32,12 @@ related:
 > **shared-interceptor merge hazard** gotcha and the **dangling-pin** / **unguarded prod
 > interceptor** gotchas before merging to `_production`.
 >
-> **🚨 LAUNCH BLOCKER (added 2026-07-23, tcox) — the standard `serviceAddressId` field has NO
-> READ ACL grant.** The custom→standard redesign shipped the column + `Core.RecordFields` row
-> but **not** the `AclFieldPermissions` read grant, so `GET /v2/entitlements?fields=serviceAddressId`
-> returns **403 `EZ-2`** for roles 1/2/4. Prod does **not** launch clean without the new
-> `Client/2026-07-23a` field-permission migration below — see
+> **🚨 LAUNCH BLOCKER — RESOLVED (migration built 2026-07-23, mhammontree; found 2026-07-23,
+> tcox) — the standard `serviceAddressId` field had NO READ ACL grant.** The custom→standard
+> redesign shipped the column + `Core.RecordFields` row but **not** the `AclFieldPermissions`
+> read grant, so `GET /v2/entitlements?fields=serviceAddressId` returned **403 `EZ-2`**. The fix
+> now ships as **`dbchanges2/Client_Rate/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql`**
+> (committed `b74f42a` on TRUE-79533) — see
 > **[Launch blocker: standard field needs a READ ACL grant](#launch-blocker-true-79533--the-standard-field-needs-a-read-acl-grant)**.
 
 ## Summary
@@ -144,7 +145,7 @@ during backend review (Jeff). This is the single biggest change since the beta b
 
 **The custom→standard redesign shipped the column and the `Core.RecordFields` row but NOT the
 `AclFieldPermissions` READ grant.** Probe-verified on **beta 2026-07-23**:
-`GET /v2/entitlements?fields=serviceAddressId` returns **403 `EZ-2`** for roles 1, 2, and 4
+`GET /v2/entitlements?fields=serviceAddressId` returns **403 `EZ-2`** (no read ACL row)
 (Entitlements has `Core.Records.aclDatabase = 'CLIENT'`, so the grant lives in each client DB).
 Without the grant, **production launches with the exact `EZ-2` failure beta hit** — the field is
 registered but unreadable, so the service cards cannot fetch the pin.
@@ -157,16 +158,20 @@ field is authorized through **`AclFieldPermissions`**, keyed by the **`Core.Reco
 because the `CustomRecordFields` row for `c_serviceAddressId` was deleted in the standard-field
 redesign. The read grant now has to be an **`AclFieldPermissions`** row instead.
 
-**Needed migration (proposed, on the TRUE-79533 branch):**
-`dbchanges2/Client/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql`, following the
-existing `Client/2026-07-02c - TrackingNumberNeedsReturnLabelFieldPermission.sql` precedent:
-- Resolve field ids via `Core.RecordFields` / `Core.Records` subselects (route `'entitlements'`,
-  fields `'serviceAddressId'` **and** its sibling `'number'`) — never hardcode ids.
-- **Mirror the sibling `number` field's per-role grants**, but force **`isWritable = 0`**:
-  `serviceAddressId` is written **server-side** by the guard's `postPost`, never by the API caller,
-  so it must be read-only.
-- Guard each INSERT with `NOT EXISTS` (verified against the Client blank DDL:
-  `AclFieldPermissions` is `UNIQUE (recordFieldId, roleId)`).
+**The migration that ships (built + committed `b74f42a` on TRUE-79533):**
+`dbchanges2/Client_Rate/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql`:
+- Resolves field ids via cross-schema `Core.RecordFields` / `Core.Records` subselects (route
+  `'entitlements'`, field `'serviceAddressId'`) — never hardcodes ids.
+- **Grants READ only (`isWritable = 0`)** — `serviceAddressId` is written **server-side** by
+  `_Model_Rate_Entitlement::postPost` via raw SQL, never by the API caller.
+- **Mirrors the reader ROLE(s) of the sibling standard FK `saleItemId` — which is `role 1` in
+  `Client_Rate`** (the same role the earlier `c_serviceAddressId` custom grant used). Although
+  the cross-role probe showed roles 1/2/4 all lacked a grant, **role 1 is the only actual reader**
+  of this field (Rate's toga2-view service cards), so the grant is role-1 only.
+- Guards each INSERT with `NOT EXISTS` — safe no-op until the field is registered
+  (`AclFieldPermissions` is `UNIQUE (recordFieldId, roleId)`).
+- **Scoped to `Client_Rate`, not a `Client/` fan-out** — Rate is the only client whose UI reads
+  the field today. A `Client/` fan-out grant is the follow-up if another client's UI needs it.
 
 ## Beta data migration — old-column pins are orphaned by the redesign (beta only, NO prod impact)
 
@@ -289,6 +294,17 @@ live carrier waterfall is opt-in via `RUN_LIVE=1` (defaults off). Verified **18/
   the deleted `2026-07-07a` interceptor registration was redundant — and why keeping it would have
   been a latent hazard: a fresh environment running both would **double-register**, firing
   `prePost` twice. **Recommend adding a `NOT EXISTS` guard to the `2026-07-15` file.**
+- **`AclFieldPermissions` has NO `isReadable` column — a ROW grants READ; `isWritable`
+  *additionally* grants WRITE** (reusable platform fact, verified against the api2 enforcement
+  code, not just schema). In `api2/Component/Api/V2/V2.php`, `getAclFieldPermissions()` (~line
+  5731) sets the field key regardless of `isWritable`, and the read-authorization checks test key
+  **presence** (`isset`/`array_key_exists`). Consequences: to make a field API-**readable** you
+  insert an `AclFieldPermissions` row (`isWritable = 0` for read-only); to make it **writable** set
+  `isWritable = 1`. And promoting a custom (`c_`) field to a **standard** field means its read grant
+  must be **re-created** as a standard `AclFieldPermissions` row (keyed by the `Core.RecordFields`
+  id) — the old `AclCustomFieldPermissions` grant (keyed by `CustomRecordFields` id) does **not**
+  carry over. This is exactly why the `c_serviceAddressId` grant did not cover the standard field
+  and the launch-blocker migration above was needed.
 - **`Core.RecordFields` has no `foreignRecordId` column** — FK resolution for `serviceAddressId`
   is entirely **model-side** (the `_Model_Client_Entitlement` FK declaration). The RecordField row
   only carries type/identifier/childPolicy/precision, mirroring sibling FKs `saleItemId`/`vendorId`.
@@ -297,6 +313,17 @@ live carrier waterfall is opt-in via `RUN_LIVE=1` (defaults off). Verified **18/
 
 ## Change history
 
+- 2026-07-23 — **READ-ACL grant BUILT — launch blocker resolved** (TRUE-79533, committed `b74f42a`).
+  Added `dbchanges2/Client_Rate/2026-07-23a - EntitlementServiceAddressIdFieldPermission.sql`: a
+  READ-only (`isWritable = 0`) `AclFieldPermissions` grant for the standard `serviceAddressId`,
+  resolving field ids cross-schema (route `entitlements`), `NOT EXISTS`-guarded. Final scope
+  corrected vs. the earlier proposal: **`Client_Rate` (not a `Client/` fan-out), role 1 only** —
+  mirroring the sibling standard FK `saleItemId`'s reader role (role 1 is the sole actual reader,
+  the toga2-view service cards). Recorded the reusable platform fact that **`AclFieldPermissions`
+  has no `isReadable` column — row presence grants READ, `isWritable` additionally grants WRITE**
+  (verified in `api2/Component/Api/V2/V2.php` `getAclFieldPermissions()` ~line 5731), which is why
+  a custom→standard promotion must re-create the read grant as an `AclFieldPermissions` row.
+  (mhammontree)
 - 2026-07-23 — **LAUNCH BLOCKER found: the standard `serviceAddressId` field has NO read ACL grant**
   (tcox). Probe-verified on beta: `GET /v2/entitlements?fields=serviceAddressId` → 403 `EZ-2` for
   roles 1/2/4 (`aclDatabase=CLIENT`). Standard fields are authorized via **`AclFieldPermissions`**

package/knowledge/sessions/2026-07-23-compass-retrofix2-sql-generator-jcardinal.md

@@ -0,0 +1,69 @@
+---
+type: session
+slug: compass-retrofix2-sql-generator
+title: Compass retroactive data-fix v2 — SQL generator rewrite
+author: jcardinal
+repos: [test, worker, library, _underscore]
+framework: "2.0"
+client: compass-usa
+status: active
+created: 2026-07-23
+updated: 2026-07-23
+---
+
+# Session: compass-retrofix2-sql-generator
+**Date:** 2026-07-23
+**Project/Repo:** test/@jeff/compass (script), operating on Client_Compass (2.0)
+**Task:** Rewrite the Compass USA retroactive data-fix as a NEW standalone script that GENERATES SQL to a flat file (never writes the DB), fixing SO/PO item existence + the two item-link bridges bottom-up, then recursing all ItemFulfillments up to the Compass SO — replacing the old direct-write `retrofix.php`.
+
+---
+
+## What WORKED
+- **New script built & lint-clean:** `D:\WWW\test\@jeff\compass\retrofix2.php` (`php -l` passes). Standalone CLI: `require_once('_.php'); App_Framework_Sandbox::initialize();`, read-only DB alias `db_prod2_compass`, writes SQL to `retrofix2.phase1.sql` / `retrofix2.phase2.sql`.
+- **Two-pass PHASE model** (`const PHASE = 1|2`): Phase 1 generates item-link SQL → developer APPLIES it → re-run with PHASE=2 (now reads corrected bridges) → generates IF-recursion SQL. Chosen because Phase 2's up-mapping walks the item bridges Phase 1 fixes, and generate-only SQL can't see un-applied fixes.
+- **Chain resolution** (`getChainForSo`): developer's LEFT-JOIN header query, deduped into distinct id sets per level (compassPoIds/odpSoIds/odpPoIds/agilantSoIds + odpPo→odpSo, compassPo→odpSo pairings). Validated against prod: a Compass SO can have MULTIPLE Compass POs; recent orders legitimately have partial chains.
+- **Phase 1 L1 group logic** mirrors `library/app/api/toga2.php:778-897,1216-1352` in id-space: re-pulls Agilant SO from NetSuite (`App_NetSuite::getSalesOrder` by `c_netsuiteInternalSalesOrderId`), builds `memberItemId=>[groupItemId]` via `getItemDetails(...,'itemGroup',false)` on group-header lines (line has neither `rate` nor `amount`), matches Agilant SOI→ODP POI by (item OR group-item, quantity) non-consuming (1 POI → many SOIs). Memoizes getItemDetails per NS internal id.
+- **Phase 2** ports `_underscore/Model/Client/ItemFulfillment.php::reconcileUpstreamLevel` as SQL: header walk, item walk grouped-by-upstream-SOI with bundle scaling `min(rawSum, rawSum*upQtyOrdered/downOrderedSum)`, broken-bridge guard, create/link upstream IF+IFI, propagate units + 3 tracking bridges (shared unitId/trackingNumberId), carries planned rows forward in-memory across the Agilant→ODP→Compass climb. New rows use generated uuid + `(SELECT id … WHERE uuid=…)` refs.
+- **emit() guard** hard-blocks any write to `TrackingNumbers` or `Units` (only exact-table match; bridge tables like `ItemFulfillmentItemUnits_TrackingNumbers` correctly allowed).
+- **Bug fixes this session (all lint-verified):**
+  - L4 false inserts — root cause: existing-links query used `WHERE b.purchaseOrderItemId IN (<Compass PO HEADER ids>)`; fixed to join `PurchaseOrderItems … purchaseOrderId IN (compassPoIds)`. Confirmed against SA133672 (SO 108392) which no longer emits its two spurious inserts.
+  - Phase 2 `ifIdExpr` "Undefined array key realId" — IFI handles use `ifiRealId`/`ifiUuid`, IF handles use `realId`/`uuid`; helper now accepts both.
+  - MySQL 1093 (self-referential UPDATE) — `ifIdExpr` now wraps uuid lookups in a derived table `(SELECT id FROM (SELECT id FROM t WHERE uuid=…) AS _rf2_<hash>)`; inline upstream-IF-link expr routed through it too.
+  - MySQL 1451 (FK on stale deletes) — deletes now strictly bottom-up: IFIU_TrackingNumbers → IFIU → ItemFulfillmentItems_TrackingNumbers → ItemFulfillmentItems (and unit-tracking → unit for leftover units).
+  - Order header now prints date: `# SA123456 (2026-06-15)` (worklist selects dateOrder) for resume tracking.
+
+## What did NOT work — DO NOT RETRY THESE
+- **Relying on the plain `try/catch(Throwable)` around NetSuite calls to catch the SSL drop** ("SoapClient::__doRequest(): SSL: An existing connection was forcibly closed by the remote host"). The warning is intercepted by TOGA's GLOBAL `App_Error::handleError → handleException`, which (Sentry active) reports and can terminate the instance / not re-throw cleanly — so the retry loop never got a reliable shot. FIX APPLIED: `nsCall` now installs a LOCAL `set_error_handler` (throws ErrorException) around each attempt and `restore_error_handler()` after — so the warning is caught locally and retried. Do not go back to relying on the global handler.
+- **Old `retrofix.php` itemId-only "purity" test** for item-group links — would delete legitimate group links (1 ODP PO item ↔ many Agilant SO items whose itemIds differ). Superseded by the NetSuite group-map approach. Do not reintroduce pure itemId matching at L1.
+
+## Not tried yet (candidates for next session)
+- **End-to-end verification not yet done:** the idempotency re-run test (apply Phase 1 → re-run → expect empty) and the engine-parity check for a Flow-B order have NOT been run.
+- **L4 Compass-SO↔Compass-PO bundle links** are intentionally NOT auto-created — bundle/config lines (`c_isConfiguration`/`parentSalesOrderItemId`/`bundleId`) are only logged as SKIP. Needs the bundle-split logic from `worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php` before enabling.
+- Confirm the fresh NetSuite retry fix (local error handler) actually survives a real SSL drop in a full batch run.
+- Phase 2 possible over-emission: already-linked IFIs may get redundant (harmless) upstream-link UPDATEs across multi-level climbs — validate on the parity test.
+- IF `number` generation (`F#####` via MAX+1) collision risk vs concurrent prod IF creation — apply Phase 2 SQL promptly.
+
+## Current file state
+| File | Status | Notes |
+|------|--------|-------|
+| D:\WWW\test\@jeff\compass\retrofix2.php | Created, lint-clean, working | Full 2-phase generator; all reported bugs fixed; NetSuite local-error-handler retry (5× backoff 3/6/9/12s) |
+| D:\WWW\test\@jeff\compass\retrofix2.phase1.sql | Generated (test window) | Output artifact; L4 spurious-insert bug fixed since last gen — regenerate |
+| D:\WWW\test\@jeff\compass\retrofix2.phase2.sql | Generated (test window) | Output artifact; 1093/1451 fixed since last gen — regenerate |
+| C:\Users\JCardinal\.claude\plans\i-need-to-continue-declarative-kahan.md | Created | Approved implementation plan (context, phases, open items, verification) |
+| D:\WWW\test\@jeff\compass\retrofix.php | Unchanged | Old direct-write version, kept for reference |
+
+## Decisions made
+- **Generate SQL, never write the DB** (developer applies manually) — user requirement; SELECTs + NetSuite GETs only.
+- **Two-pass PHASE constant** over single-pass in-memory bridge model — simpler/safer, matches rollout order (verify links first, then fulfillments). Rejected: single-pass modelling expected bridges in memory (too complex/error-prone to verify).
+- **NetSuite group resolution via live `getItemDetails`** (per user direction) rather than any persisted TOGA bundle table — re-pulls each Agilant SO from NetSuite.
+- **Flat SQL file, no transactions, `# SA###### (date)` comments** per user preference.
+- **L4 conservative** (insert pure links, log suspected bundles, delete only exact dups) — avoids destroying legit bundle links pending the script-1 bundle logic.
+
+## Blockers
+None blocking. Open dependency: Phase 2 requires the Phase 1 SQL to be APPLIED to the target DB first (two-pass), and full E2E verification is still pending.
+
+## Exact next step
+> Regenerate Phase 1 over a small test window (set `START_DATE`/`END_DATE` to ~1 day, `PHASE=1`) in `D:\WWW\test\@jeff\compass\retrofix2.php`, run it (live NetSuite + prod-read), then run the idempotency check: apply `retrofix2.phase1.sql` to a LOCAL Client_Compass copy, re-run PHASE=1, and confirm the second output is empty. Then repeat for PHASE=2.
+
+---
+_Saved by /session-save on 2026-07-23_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.424",
+  "version": "1.0.426",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",