toga-ai 1.0.231 → 1.0.233

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -405,11 +405,27 @@ None — Forecast2 is a single shared dataset.
405
405
  longer open — so it **removes** billed/closed orders rather than resurrecting them. It is therefore
406
406
  **not** a resurrection risk against the webhook `removeAll` path; it is in fact the current backstop
407
407
  deleter for orders the webhook never sees billed (the invoice-transform gap).
408
+ - **`fixer.php`'s OpenOrderItems path does NOT carry `locationId` / `quantityBackordered` /
409
+ `amountDue` — mirror any new open-order column here or the reconciler reverts it.** `fixer.php`
410
+ already handles anchor-line **`amountDue` on the Sales path**, guarded by a column-existence check
411
+ `$hasAmountDueCol = forecastColumnExists('Sales','amountDue')` (so it's a no-op until the prod
412
+ column lands). Its **OOI** path — the lookup SELECT, change-detection, the
413
+ `buildOoiInsertSql`/`buildOoiUpdateSql` helpers, and the SuiteQL `transactionline` query — does
414
+ **not** include `locationId`/`quantityBackordered`/`amountDue`. Per the compute-identically
415
+ invariant, any field added to the open-order **importer** (TRUE-79162) must also be added to this
416
+ OOI path (and ideally behind the same `forecastColumnExists` guard pattern) or `fixer.php`/`looper`
417
+ will overwrite the importer's value to NULL on its next run.
408
418
  - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
409
419
  against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
410
420
 
411
421
  ## Change history
412
422
 
423
+ - 2026-06-29 — **Recorded that `fixer.php`'s OOI path lacks `locationId`/`quantityBackordered`/
424
+ `amountDue`** (planning for TRUE-79162). The Sales path already handles anchor-line `amountDue`
425
+ behind a `forecastColumnExists('Sales','amountDue')` guard; the OpenOrderItems path (lookup SELECT,
426
+ change-detection, `buildOoiInsertSql`/`buildOoiUpdateSql`, SuiteQL `transactionline`) carries none
427
+ of the three new open-order fields. Per the compute-identically invariant, any field added to the
428
+ open-order importer must be mirrored here or the reconciler reverts it. (dfranks)
413
429
  - 2026-06-29 — **Found/fixed the NULL-rate phantom open-order profit drift; extended PROFIT comparison to all
414
430
  three categories.** Cost-only NS open-order lines (zero revenue) can have a **NULL `rate`**; the FIND
415
431
  profit SQL did `tl.rate - unitCost` and `NULL - x = NULL` silently dropped the line from `SUM`, inflating
@@ -12,7 +12,7 @@
12
12
  | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
13
13
  | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
14
14
  | [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
15
- | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
15
+ | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
16
16
  | [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
17
17
  | [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
18
18
  | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
@@ -6,17 +6,21 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-06-10
9
+ updated: 2026-06-29
10
10
  owners: [mhammontree]
11
11
  files:
12
12
  - worker2/Worker/Monitor.php
13
13
  - worker2/Worker/Monitors/
14
+ - worker2/Worker/Monitors/RateEntitlement.php
14
15
  - worker2/Worker/Notification/Email.php
16
+ - worker2/Worker/Rate.php
15
17
  - dbchanges2/Core/2026-05-21 - Monitors.sql
18
+ - dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql
16
19
  related:
17
20
  - ../architecture.md
18
21
  - ./creating-worker-actions.md
19
22
  - ../../dbchanges2/architecture.md
23
+ - ../../../clients/rate/features/aig-contract-creation.md
20
24
  ---
21
25
 
22
26
  ## Summary
@@ -26,10 +30,11 @@ Prudential asset imports, AIG closed claims, …). One orchestrator applies cons
26
30
  anti-flap logic and notification rules to any registered "is X healthy?" check. Replaces
27
31
  scattered/ad-hoc monitoring where failures surfaced only when a client complained.
28
32
 
29
- **Status (2026-06-10):** v1.0 framework + orchestrator landed; **first child monitor
30
- pending**. Migration applied to local Core_2 only **not yet on staging/production Core**
31
- (coordinate before merge). Dashboard/acknowledgment layer under discussion (see Pending
32
- scope).
33
+ **Status (2026-06-29):** v1.0 framework + orchestrator landed; **first child monitor built**
34
+ `_Worker_Monitors_RateEntitlement` (TRUE-79129, see [worked example](#worked-example--rateentitlement-the-first-child-monitor)).
35
+ `Core.Monitors` migration applied to local Core_2 only **still not on staging/production
36
+ Core** (coordinate before merge; the new Rate change-set hard-fails where the table is
37
+ absent). Dashboard/acknowledgment layer under discussion (see Pending scope).
33
38
 
34
39
  Design principles: async isolation (one cron per monitor — a slow monitor can't block
35
40
  others) · anti-flap on the recovery side only (N consecutive OKs before declaring
@@ -131,6 +136,19 @@ Index `Monitors_isActive_IDX (isActive)`. `Core.CronJobs` reused as-is — one r
131
136
  (recipients see it verbatim); register any non-Core DB connection inline at the top of
132
137
  `Run()` — the framework does **not** call `initialize()` on child monitors.
133
138
 
139
+ **Confirmed against the orchestrator (`Worker/Monitor.php`):**
140
+ - The orchestrator calls `$phpClass::Run()` with **no args** and does **not** call
141
+ `initialize()` on children. A child needing a non-Core connection (e.g. a `Logs_<Client>`
142
+ DB) must self-register it inside `Run()`, guarding against a double-register:
143
+ `if (!isset(_Database::$_registers[$alias])) { _Database::register(db, host, user, pass,
144
+ null, null, $alias); }`, with credentials from `_Config::databaseLogs(...)`. The
145
+ registration pattern mirrors `worker2/Worker/Rate.php::initialize()`.
146
+ - The orchestrator wraps the call in its own try/catch and turns any `Throwable` into an
147
+ alert — children must **not** catch (re-confirms the "fail loudly" rule above).
148
+ - A read-only `Run()` (e.g. a log scan) needs **no** `transactionCommit()`.
149
+ - **PHP heredoc gotcha:** `self::CONST` does **not** interpolate inside a heredoc. Bind
150
+ class constants to local variables before building a heredoc SQL string.
151
+
134
152
  ### Adding a new monitor (runbook)
135
153
 
136
154
  1. Write the child class (one check, returns `{isOk, message}`).
@@ -150,17 +168,59 @@ No Lambda, SQS, or orchestrator changes needed per monitor.
150
168
  - Pause a monitor: `isActive = 0` (cron row can stay active). Pause email only: `peopleToNotify = JSON_ARRAY()`.
151
169
  - Manual state reset: clear `state`/`consecutiveOkCount`/`lastNotificationDt`/`lastNotificationType` — sparingly; the state machine self-corrects.
152
170
 
171
+ ### Worked example — RateEntitlement (the first child monitor)
172
+
173
+ `_Worker_Monitors_RateEntitlement` (`worker2/Worker/Monitors/RateEntitlement.php`, the
174
+ **first** file in `Worker/Monitors/`) watches Rate's AIG warranty-contract creation
175
+ (TRUE-79129). It is the reference implementation for the **log-scan** monitor pattern:
176
+
177
+ 1. `Run()` self-registers the `Logs_Rate` connection (the guard + `_Config::databaseLogs`
178
+ pattern above), because the orchestrator never calls `initialize()`.
179
+ 2. Scans `Logs_Rate.Api` for **failed outbound** AIG contract POSTs in the last 30 minutes
180
+ — `direction = 'OUT'`, `method = 'POST'`, `route LIKE '%/contract'`, and
181
+ `responseCode` non-2xx **or NULL**. Returns `(object){isOk, message}`; read-only, so no
182
+ commit.
183
+ 3. The detection signal is the `_underscore` **outbound API log**, not the entitlement
184
+ itself — see [why a monitor is needed](#why-an-external-monitor) below.
185
+
186
+ The change-set `dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql` does
187
+ the registration: one `Core.Monitors` row (`phpClass = _Worker_Monitors_RateEntitlement`,
188
+ `peopleToNotify = ['devteam@goagilant.com']`, `requiredConsecutiveOks = 2`,
189
+ `reminderFrequencyMinutes = 60`) + one `Core.CronJobs` row (`action = 'Monitor/Run'`,
190
+ `parameters = JSON_OBJECT('monitorId', LAST_INSERT_ID())`, schedule `*/15`,
191
+ `maxExecutionTime = 60`). LAST_INSERT_ID() chains the cron row to the just-inserted monitor.
192
+
193
+ ### Detection via the _underscore outbound API log
194
+
195
+ `_underscore/ApiRequest.php` logs **every** outbound call to `_Model_Client_Logs_Api`
196
+ (table `Logs_<Client>.Api`) and **commits it immediately**, independent of whether the
197
+ caller later swallows the exception — which is exactly what makes log-scan monitoring
198
+ possible for silent-failure interceptors. Columns to scan: `dtStamp`
199
+ (`FIELD_DATETIME_CREATED`), `direction` (`OUT`), `method`, `hostname` (scheme://host),
200
+ `route` (path after host), `responseCode`. `setUrl()` splits a full URL into
201
+ `hostname` + `route`, so a call to `.../contract` is logged with `route` ending `/contract`
202
+ (and cancellation as `/contract/cancel`). **Pattern for new monitors:** scan the client's
203
+ `Logs_<Client>.Api` rather than trying to detect failure on the business object.
204
+
205
+ ### Why an external monitor
206
+
207
+ Some interceptors **silently swallow** their own failures (no persistent flag on the
208
+ business record), so the only durable failure signal is the committed outbound API log.
209
+ The Rate case is documented in
210
+ [Rate AIG contract creation](../../../clients/rate/features/aig-contract-creation.md).
211
+
153
212
  ## Client variations
154
213
 
155
214
  None — the framework is shared Core infrastructure. Individual monitors target specific
156
- clients' data flows (Compass, Prudential, AIG, …) but live as separate child classes.
215
+ clients' data flows (Compass, Prudential, AIG, Rate, …) but live as separate child classes.
157
216
 
158
217
  ## Gotchas / known issues
159
218
 
160
- - **First child monitor not yet built** — `worker2/Worker/Monitors/` does not exist yet
161
- (as of 2026-06-10).
162
- - **Migration not applied to staging/production Core** only local Core_2. Coordinate
163
- before merge.
219
+ - **Migration not applied to staging/production Core** — `Core.Monitors`
220
+ (`2026-05-21 - Monitors.sql`) is on local Core_2 only (as of 2026-06-10, re-confirmed
221
+ 2026-06-29). The new `2026-06-29a` Rate change-set will **hard-fail (table not found)** on
222
+ any environment where `2026-05-21 - Monitors.sql` has not yet been promoted — confirm the
223
+ Monitors table exists before deploying.
164
224
  - `worker2/MONITORING_PLAN.md` is referenced by the design doc but **missing on disk** —
165
225
  stale reference to resolve.
166
226
  - Child return value must be an **object** with both `isOk` and `message`; a bare array
@@ -177,6 +237,7 @@ clients' data flows (Compass, Prudential, AIG, …) but live as separate child c
177
237
  HTML email + dashboard deep-links · anti-flap on the alarm side.
178
238
 
179
239
  ## Change history
240
+ - 2026-06-29 — Built the first child monitor, `_Worker_Monitors_RateEntitlement` (TRUE-79129) + change-set `2026-06-29a`; added the log-scan worked example, the `_underscore` outbound-API-log detection pattern, and confirmed child details (orchestrator never calls `initialize()` so children self-register non-Core connections; no commit on read-only `Run()`; heredoc cannot interpolate `self::CONST`). Re-flagged that `Core.Monitors` is still local-only — the new change-set hard-fails where the table is absent. (mhammontree)
180
241
  - 2026-06-10 — Documented the v1.0 monitoring framework (orchestrator, `Core.Monitors` table, recovery-side anti-flap state machine, child contract). First child monitor + staging/prod migration still pending. (mhammontree)
181
242
 
182
243
  ## Related docs
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-06-25
9
+ updated: 2026-06-29
10
10
  owners: ["dfranks"]
11
11
  files:
12
12
  - worker2/Worker/Netsuite/SalesOrder.php
@@ -127,12 +127,51 @@ The table lives in the **`Forecast` schema on the core2 cluster** (reader
127
127
  sales-order FK column is **`netsuiteSalesOrderInternalId`** (`int unsigned`) — note this is *not* the
128
128
  same column name used by `Forecast.Sales` (see the column-name gotcha below).
129
129
 
130
+ ### Model/DB drift to know before backfilling location / backorder / amountDue (TRUE-79162)
131
+
132
+ A planned backfill of **location, backorder, and amount-due** data onto open orders runs into three
133
+ schema/model facts (all verified against the prod core2 reader; nothing built yet):
134
+
135
+ - **`_Model_Forecast_OpenOrderItem` (`_underscore/Model/Forecast/OpenOrderItem.php`) does NOT declare
136
+ `locationId` or `quantityBackordered` — even though both columns already exist in the prod
137
+ `Forecast.OpenOrderItems` table** (`locationId` `int unsigned NULL`, `quantityBackordered`
138
+ `decimal(15,4) NULL`). The model declares only `id, netsuiteSalesOrderInternalId, dateOrder,
139
+ orderNumber, customerId, salesRepEmployeeId, classificationId, accountId, itemId, lineNumber,
140
+ revenue, profit`. **Writing those two fields through the 2.0 importer requires adding the
141
+ properties to the model first** — the column existing in the DB is not sufficient.
142
+ - **`Forecast.Locations` is EMPTY in prod (0 rows).** Schema: `id, name, netsuiteInternalLocationId`
143
+ (UNIQUE, nullable), `typeId` — **no parent/hierarchy column.** Any open-order `locationId`
144
+ resolution returns NULL until this table is populated from NetSuite, so **populating `Locations` is
145
+ a hard prerequisite** for the location backfill. NetSuite returns a **leaf sub-location** but the
146
+ warehouse dashboard wants the **top-level location**, so a leaf→root rollup is needed (and there is
147
+ no parent column on `Locations` today to express it).
148
+ - **`Forecast.OpenOrderItems` has NO `amountDue` column** (neither does `Forecast.Sales` in prod —
149
+ see the [Sales import doc](../../_underscore/features/forecast-sale-import.md)), and `amountDue`
150
+ appears in no worker2 Netsuite handler or `_Model_Forecast_*`. Importing open-order amountDue
151
+ needs a **new column + model field**. The amountDue source semantics from the prior Sales work
152
+ (TRUE-78923) — REST `amountRemaining` / SuiteQL `foreignamountunpaid`, **anchor-line only, store
153
+ RAW POSITIVE** — carry over, **but `amountRemaining` is an AR/invoice field**, so for an *unbilled*
154
+ sales order it may return null/0. **Live-probe an open SO before finalizing** the open-order
155
+ amountDue source.
156
+
130
157
  ## Client variations
131
158
 
132
159
  None — uniform (platform-wide Forecast2 sync).
133
160
 
134
161
  ## Gotchas / known issues
135
162
 
163
+ - **A new synced `OpenOrderItems` column must be wired into BOTH `buildOpenLineRows()` AND the
164
+ `syncOpenLines()` change-detection diff — not just the write — or no-diff updates silently drop
165
+ it.** The forward-sync path is `_Worker_Netsuite_SalesOrder`'s private `buildOpenLineRows()`
166
+ (assembles each `OpenOrderItems` row array) and `syncOpenLines()` (change-detection + upsert).
167
+ This is the established forecast2 change-detection gotcha applied to open orders: a column added
168
+ only to the row build but not to the `syncOpenLines()` diff block is **discarded on a no-change
169
+ UPDATE** (the diff sees no difference, so the new value never persists). Any
170
+ locationId/quantityBackordered/amountDue addition (TRUE-79162) must touch both. Forward sync now
171
+ flows through this webhook importer; the legacy 1.0 cron
172
+ (`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php` open-order section) is
173
+ superseded for it. **Also mirror any new field into the reconciler `test/@dave/fixer.php` OOI path**
174
+ or it reverts the value on its next run (see the reconciliation doc).
136
175
  - **Billing an SO via invoice-transform flips it to "Billed" WITHOUT firing the SalesOrder afterSubmit UE → no edit/PUT webhook → `removeAll()` never runs → the OpenOrderItems rows linger (this is the real cause of the "billed order not removed" problem — NOT a removeAll-didn't-persist bug).** When a NetSuite Sales Order is billed by **creating an Invoice from it**, the SO's `status` transitions to `Billed` as a **side-effect of the invoice transform** — this does **not** trigger the SalesOrder `afterSubmit` User Event, so the AMQ enqueuer (`ue_api_msg_queue_enqueue`, deployed per-record-type) **never emits an edit/PUT webhook for the SO**. Consequently `_Worker_Netsuite_SalesOrder` is never invoked for the billing change, `importOpenOrder`'s status gate (which *would* call `removeAll` on a non-open status) never runs, and the order's `OpenOrderItems` rows survive until the nightly discrepancy-fix cron deletes them. **Confirmed live 2026-06-24/25 on SO 7190415** (tranId 280387): only a `create` webhook delivery exists in `Logs.Webhook` (2026-06-24 23:48, 16 rows inserted) — **NO** edit/delete delivery at billing time (~09:41 UTC 2026-06-25); `Core.WorkerJobs` shows only `Netsuite/Webhook` + `Netsuite/SalesOrder/post` (no put/delete); `Logs.Api` has only the original NS GET + a 200 `OPEN_ORDER_IMPORT` breadcrumb — **no 204 removeAll breadcrumb**. The invoice (7190621) that billed it **also produced no webhook** — its 16 `Forecast.Sales` rows came from the **legacy 5-min pull cron**, not the webhook. **This corrects the earlier suspicion that "removeAll fired but DB didn't persist"** — both `removeAll` paths *do* commit `DB_FORECAST`; `removeAll` was simply **never called**. **Diagnostic ladder for a lingering billed order:** `Logs.Webhook` (was an edit/delete even delivered? — here, no) → `Core.WorkerJobs` (was a put/delete processed? — no) → `Logs.Api` `source='OPEN_ORDER_IMPORT'` (any 204 removeAll breadcrumb? — no). **Fix direction (not yet implemented):** the SalesOrder-side enqueuer must also fire on the invoice-transform status change — either deploy the AMQ enqueuer on **Invoice** so the bill event drives an SO re-sync, or have the SalesOrder handler re-evaluate the SO when its child invoice arrives; until then the **stopgap is the continuous `loop_trueup_open_orders.php --by-lastmodified` runner** (see the reconciliation doc), which windows on `lastmodifieddate` so it catches an order billed today whose `tranDate` predates the window. The `removeAll` path in `SalesOrder.php` was re-confirmed correct in this session.
137
176
  - **Wrong FK column name → MySQL 1054 that masquerades as "order missing".** `OpenOrderItems`'s
138
177
  sales-order key is **`netsuiteSalesOrderInternalId`**. Do **NOT** query it with
@@ -297,6 +336,16 @@ test fixture (it surfaced the stale SO 7181316 above).
297
336
 
298
337
  ## Change history
299
338
 
339
+ - 2026-06-29 — **Recorded the model/DB drift + prerequisites for the open-order location/backorder/
340
+ amountDue backfill (TRUE-79162, planning only — no code written).** `_Model_Forecast_OpenOrderItem`
341
+ declares neither `locationId` nor `quantityBackordered` though both columns already exist in prod
342
+ `OpenOrderItems` (so the model needs the properties before the importer can write them);
343
+ `Forecast.Locations` is empty in prod with no parent column (hard prerequisite + leaf→root rollup
344
+ needed); `OpenOrderItems` has no `amountDue` column (new column + field required, and
345
+ `amountRemaining` is an AR field that may be null/0 on an unbilled SO — probe first). Added the
346
+ gotcha that any new synced column must be wired into **both** `buildOpenLineRows()` and the
347
+ `syncOpenLines()` change-detection diff (and mirrored in `fixer.php`'s OOI path) or it is dropped on
348
+ no-diff updates. (dfranks)
300
349
  - 2026-06-25 — **Root-caused "billed SO not removed from OpenOrderItems in real time" to a missing
301
350
  webhook, NOT a removeAll-didn't-persist bug.** Billing an SO via invoice-transform flips it to
302
351
  `Billed` as a side-effect that does **not** fire the SalesOrder `afterSubmit` UE → the AMQ enqueuer
@@ -16,7 +16,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
16
16
 
17
17
  ## 2.0 framework
18
18
 
19
- - **_underscore** (_Underscore) _(framework core)_ — 17 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
19
+ - **_underscore** (_Underscore) _(framework core)_ — 18 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
20
20
  - **worker2** (Worker) — 22 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
21
21
  - **api2** (API) — 7 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
22
22
  - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
@@ -2,6 +2,7 @@
2
2
 
3
3
  | Doc | Framework | Summary | Files |
4
4
  |-----|-----------|---------|-------|
5
+ | [Rate AIG Warranty Contract Creation (silent-failure interceptor)](features/aig-contract-creation.md) | 2.0 | When a Rate entitlement is created, `_Model_Rate_Entitlement::postPost` (`_underscore/Model/Rate/Entitlement.php`) creates an **AIG warranty contract** as a non | _underscore/Model/Rate/Entitlement.php, _underscore/ApiRequest.php, worker2/Worker/Monitors/RateEntitlement.php |
5
6
  | [Rate Monthly Reconciliation Report](features/monthly-reconciliation-report.md) | 1.0 | A monthly cron that emails an Excel reconciliation report covering all Rate subscription sales orders and their linked PayPal payments for the prior calendar mo | worker/crons/notifications/reports/rate/send_monthly_rate_purchases_report.php, worker/schedules/cron.worker.notification.json |
6
7
  | [Rate SalesOrder → NetSuite CashSale Export (postPost)](features/netsuite-cashsale-export.md) | 2.0 | Rate sells home-warranty / home-tech-support products. | _underscore/Model/Rate/SalesOrder.php, _underscore/Model/Rate/Item.php |
7
8
  | [Rate SAML SSO](features/saml-sso.md) | 2.0 | Rate uses Azure AD as its IdP (`login.rate.com`). | _underscore/Model/Rate/ClientAuthentication.php, saml/Controller/Index.php, toga2-view/src/hooks/useAuthenticationFlow.ts |
@@ -0,0 +1,97 @@
1
+ ---
2
+ title: "Rate AIG Warranty Contract Creation (silent-failure interceptor)"
3
+ framework: "2.0"
4
+ repo: _underscore
5
+ project: _Underscore
6
+ client: rate
7
+ type: client-feature
8
+ status: active
9
+ updated: 2026-06-29
10
+ owners: [mhammontree]
11
+ files:
12
+ - _underscore/Model/Rate/Entitlement.php
13
+ - _underscore/ApiRequest.php
14
+ - worker2/Worker/Monitors/RateEntitlement.php
15
+ related:
16
+ - clients/rate/profile.md
17
+ - ../../../2.0/apps/worker2/features/monitoring-framework.md
18
+ ---
19
+
20
+ ## Summary
21
+
22
+ When a Rate entitlement is created, `_Model_Rate_Entitlement::postPost`
23
+ (`_underscore/Model/Rate/Entitlement.php`) creates an **AIG warranty contract** as a
24
+ non-critical side effect: it authenticates against AIG, POSTs to `.../contract`, and on
25
+ success stamps `payload->c_aigContractNumber` and `payload->c_aigContractId` back onto the
26
+ entitlement. **It only attempts this when the entitlement payload has a primary contact
27
+ address.**
28
+
29
+ The catch: **every failure path is silent.** There is no persistent failure flag on the
30
+ entitlement, so a contract that never got created looks identical to one that did until a
31
+ customer or AIG reports it missing. That is why this flow is watched externally by the
32
+ `_Worker_Monitors_RateEntitlement` monitor rather than by any state on the entitlement.
33
+
34
+ This is a base-`_underscore` interceptor specific to Rate's product (it lives under
35
+ `Model/Rate/`); documented here as a Rate client-feature because the behavior and its
36
+ monitoring are Rate-scoped.
37
+
38
+ ## How it works
39
+
40
+ 1. `postPost` runs after an entitlement is saved. It proceeds **only if** the payload has a
41
+ primary contact address; otherwise it does nothing.
42
+ 2. Authenticate against AIG (POST `/authentication/login`) → obtain an access token.
43
+ 3. POST the contract payload to `.../contract`.
44
+ 4. On success: read `aigContractNumber` from the response and set
45
+ `payload->c_aigContractNumber` + `payload->c_aigContractId`.
46
+
47
+ ## Silent-failure paths (the gotcha)
48
+
49
+ All of the following fail **silently** — no exception surfaces to the caller and nothing is
50
+ persisted on the entitlement:
51
+
52
+ - A `try/catch` that only `error_log()`s with the note *"AIG contract creation is
53
+ non-critical"* and swallows the exception.
54
+ - **Auth response non-2xx** (no throw).
55
+ - **No access token** returned (no throw).
56
+ - **Contract POST non-2xx** (no throw).
57
+ - **Success response but missing `aigContractNumber`** (no throw).
58
+
59
+ Net effect: there is no durable signal on the entitlement that contract creation failed.
60
+
61
+ ## Detection / monitoring
62
+
63
+ Because the interceptor swallows failures, detection rides the **committed outbound API
64
+ log** instead. `_underscore/ApiRequest.php` logs every outbound call to
65
+ `_Model_Client_Logs_Api` (table `Logs_Rate.Api`) and commits it immediately, regardless of
66
+ whether the caller later swallows the exception. The AIG contract call lands with
67
+ `direction = OUT`, `method = POST`, and `route LIKE '%/contract'` (cancellation is
68
+ `/contract/cancel`).
69
+
70
+ `_Worker_Monitors_RateEntitlement` (worker2) scans `Logs_Rate.Api` for failed AIG contract
71
+ POSTs (`route LIKE '%/contract'`, `responseCode` non-2xx **or NULL**) in the last 30
72
+ minutes and alerts the dev team. See the
73
+ [Monitoring Framework worked example](../../../2.0/apps/worker2/features/monitoring-framework.md#worked-example--rateentitlement-the-first-child-monitor).
74
+
75
+ ### Known detection limitation (decided)
76
+
77
+ Detection matches **only** `POST .../contract` (`route LIKE '%/contract'`), which
78
+ deliberately **excludes** `/contract/cancel` and the shared `/authentication/login`.
79
+ Consequence: a creation failure caused by an **auth failure** is **not** caught, because
80
+ `/authentication/login` is shared with the cancellation flow and so cannot be attributed to
81
+ creation vs. cancellation. This is an accepted, documented limitation — not an oversight.
82
+
83
+ ## Gotchas / known issues
84
+
85
+ - **No persistent failure flag** — the only durable evidence of a failed contract creation
86
+ is the outbound API log row, not the entitlement.
87
+ - **Address-gated** — `postPost` does nothing when the payload lacks a primary contact
88
+ address, so an entitlement with no address never even attempts contract creation (and
89
+ never logs a `/contract` call).
90
+ - **Auth-induced failures are blind to the monitor** — see the detection limitation above.
91
+
92
+ ## Change history
93
+
94
+ - 2026-06-29 — Documented the silent-failure AIG warranty-contract interceptor and the
95
+ external log-scan monitor (`_Worker_Monitors_RateEntitlement`, TRUE-79129), including the
96
+ accepted detection limitation that auth-induced failures are not attributable (shared
97
+ `/authentication/login`). (mhammontree)
@@ -6,17 +6,21 @@ apps:
6
6
  - saml
7
7
  - toga2-view
8
8
  - worker
9
+ - worker2
10
+ - dbchanges2
9
11
  project: SAML SSO Gateway
10
12
  client: rate
11
13
  type: profile
12
14
  status: active
13
- updated: 2026-06-25
14
- owners: ["rgirish", "bala"]
15
+ updated: 2026-06-29
16
+ owners: ["rgirish", "bala", "mhammontree"]
15
17
  files: []
16
18
  related:
17
19
  - clients/rate/features/saml-sso.md
18
20
  - clients/rate/features/monthly-reconciliation-report.md
19
21
  - clients/rate/features/netsuite-cashsale-export.md
22
+ - clients/rate/features/service-card-entitlements.md
23
+ - clients/rate/features/aig-contract-creation.md
20
24
  ---
21
25
 
22
26
  ## Summary
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.231",
3
+ "version": "1.0.233",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",