toga-ai 1.0.231 → 1.0.233
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md +16 -0
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -1
- package/knowledge/2.0/apps/worker2/features/monitoring-framework.md +71 -10
- package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md +50 -1
- package/knowledge/INDEX.md +1 -1
- package/knowledge/clients/rate/INDEX.md +1 -0
- package/knowledge/clients/rate/features/aig-contract-creation.md +97 -0
- package/knowledge/clients/rate/profile.md +6 -2
- package/package.json +1 -1
|
@@ -405,11 +405,27 @@ None — Forecast2 is a single shared dataset.
|
|
|
405
405
|
longer open — so it **removes** billed/closed orders rather than resurrecting them. It is therefore
|
|
406
406
|
**not** a resurrection risk against the webhook `removeAll` path; it is in fact the current backstop
|
|
407
407
|
deleter for orders the webhook never sees billed (the invoice-transform gap).
|
|
408
|
+
- **`fixer.php`'s OpenOrderItems path does NOT carry `locationId` / `quantityBackordered` /
|
|
409
|
+
`amountDue` — mirror any new open-order column here or the reconciler reverts it.** `fixer.php`
|
|
410
|
+
already handles anchor-line **`amountDue` on the Sales path**, guarded by a column-existence check
|
|
411
|
+
`$hasAmountDueCol = forecastColumnExists('Sales','amountDue')` (so it's a no-op until the prod
|
|
412
|
+
column lands). Its **OOI** path — the lookup SELECT, change-detection, the
|
|
413
|
+
`buildOoiInsertSql`/`buildOoiUpdateSql` helpers, and the SuiteQL `transactionline` query — does
|
|
414
|
+
**not** include `locationId`/`quantityBackordered`/`amountDue`. Per the compute-identically
|
|
415
|
+
invariant, any field added to the open-order **importer** (TRUE-79162) must also be added to this
|
|
416
|
+
OOI path (and ideally behind the same `forecastColumnExists` guard pattern) or `fixer.php`/`looper`
|
|
417
|
+
will overwrite the importer's value to NULL on its next run.
|
|
408
418
|
- These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
|
|
409
419
|
against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
|
|
410
420
|
|
|
411
421
|
## Change history
|
|
412
422
|
|
|
423
|
+
- 2026-06-29 — **Recorded that `fixer.php`'s OOI path lacks `locationId`/`quantityBackordered`/
|
|
424
|
+
`amountDue`** (planning for TRUE-79162). The Sales path already handles anchor-line `amountDue`
|
|
425
|
+
behind a `forecastColumnExists('Sales','amountDue')` guard; the OpenOrderItems path (lookup SELECT,
|
|
426
|
+
change-detection, `buildOoiInsertSql`/`buildOoiUpdateSql`, SuiteQL `transactionline`) carries none
|
|
427
|
+
of the three new open-order fields. Per the compute-identically invariant, any field added to the
|
|
428
|
+
open-order importer must be mirrored here or the reconciler reverts it. (dfranks)
|
|
413
429
|
- 2026-06-29 — **Found/fixed the NULL-rate phantom open-order profit drift; extended PROFIT comparison to all
|
|
414
430
|
three categories.** Cost-only NS open-order lines (zero revenue) can have a **NULL `rate`**; the FIND
|
|
415
431
|
profit SQL did `tl.rate - unitCost` and `NULL - x = NULL` silently dropped the line from `SUM`, inflating
|
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
| [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
|
|
13
13
|
| [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
|
|
14
14
|
| [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
|
|
15
|
-
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Notification/Email.php, dbchanges2/Core/2026-05-21 - Monitors.sql |
|
|
15
|
+
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
|
|
16
16
|
| [NetSuite → TOGA Opportunity Sync (API Message Queue + worker2 webhook)](features/netsuite-opportunity-sync.md) | Outbound sync from NetSuite to TOGA for the record types the Forecast2 importer pulls (opportunities first; sales/items/etc. | worker2/Worker/Netsuite.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Controller/Index.php, _underscore/Worker.php, test/@dave/NetSuite/api-message-queue/lib_amq_queue.js, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/NetSuite/api-message-queue/ue_amq_drain.js, test/@dave/NetSuite/api-message-queue/ss_amq_drain.js, test/@dave/NetSuite/api-message-queue/DEPLOY_RUNBOOK.md, test/@dave/clickup/backfill_opportunity_numbers.php, test/@dave/clickup/probe_opportunity_fields.php, test/@dave/probe_clickup_desc_match.php, test/@dave/test_model_load_behavior.php, dbchanges2/Forecast/2026-06-25a - Add unique index on Opportunities netsuiteOpportunityInternalId.sql, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
|
|
17
17
|
| [NetSuite → Forecast Open-Orders Sync (salesOrder webhook → OpenOrderItems)](features/netsuite-salesorder-open-orders-sync.md) | Webhook-driven, single-record port of the legacy open-orders importer (TRUE-79142). | worker2/Worker/Netsuite/SalesOrder.php, worker2/Worker/Netsuite.php, test/@dave/probe_salesorder_rest_shape.php, test/@dave/probe_open_order_lines.php, test/@dave/check_so_status.php, test/@dave/check_so_history.php, test/@dave/probe_so_rest_lines.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_open_order_gating.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
|
|
18
18
|
| [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
|
|
@@ -6,17 +6,21 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-06-
|
|
9
|
+
updated: 2026-06-29
|
|
10
10
|
owners: [mhammontree]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Monitor.php
|
|
13
13
|
- worker2/Worker/Monitors/
|
|
14
|
+
- worker2/Worker/Monitors/RateEntitlement.php
|
|
14
15
|
- worker2/Worker/Notification/Email.php
|
|
16
|
+
- worker2/Worker/Rate.php
|
|
15
17
|
- dbchanges2/Core/2026-05-21 - Monitors.sql
|
|
18
|
+
- dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql
|
|
16
19
|
related:
|
|
17
20
|
- ../architecture.md
|
|
18
21
|
- ./creating-worker-actions.md
|
|
19
22
|
- ../../dbchanges2/architecture.md
|
|
23
|
+
- ../../../clients/rate/features/aig-contract-creation.md
|
|
20
24
|
---
|
|
21
25
|
|
|
22
26
|
## Summary
|
|
@@ -26,10 +30,11 @@ Prudential asset imports, AIG closed claims, …). One orchestrator applies cons
|
|
|
26
30
|
anti-flap logic and notification rules to any registered "is X healthy?" check. Replaces
|
|
27
31
|
scattered/ad-hoc monitoring where failures surfaced only when a client complained.
|
|
28
32
|
|
|
29
|
-
**Status (2026-06-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
+
**Status (2026-06-29):** v1.0 framework + orchestrator landed; **first child monitor built**
|
|
34
|
+
— `_Worker_Monitors_RateEntitlement` (TRUE-79129, see [worked example](#worked-example--rateentitlement-the-first-child-monitor)).
|
|
35
|
+
`Core.Monitors` migration applied to local Core_2 only — **still not on staging/production
|
|
36
|
+
Core** (coordinate before merge; the new Rate change-set hard-fails where the table is
|
|
37
|
+
absent). Dashboard/acknowledgment layer under discussion (see Pending scope).
|
|
33
38
|
|
|
34
39
|
Design principles: async isolation (one cron per monitor — a slow monitor can't block
|
|
35
40
|
others) · anti-flap on the recovery side only (N consecutive OKs before declaring
|
|
@@ -131,6 +136,19 @@ Index `Monitors_isActive_IDX (isActive)`. `Core.CronJobs` reused as-is — one r
|
|
|
131
136
|
(recipients see it verbatim); register any non-Core DB connection inline at the top of
|
|
132
137
|
`Run()` — the framework does **not** call `initialize()` on child monitors.
|
|
133
138
|
|
|
139
|
+
**Confirmed against the orchestrator (`Worker/Monitor.php`):**
|
|
140
|
+
- The orchestrator calls `$phpClass::Run()` with **no args** and does **not** call
|
|
141
|
+
`initialize()` on children. A child needing a non-Core connection (e.g. a `Logs_<Client>`
|
|
142
|
+
DB) must self-register it inside `Run()`, guarding against a double-register:
|
|
143
|
+
`if (!isset(_Database::$_registers[$alias])) { _Database::register(db, host, user, pass,
|
|
144
|
+
null, null, $alias); }`, with credentials from `_Config::databaseLogs(...)`. The
|
|
145
|
+
registration pattern mirrors `worker2/Worker/Rate.php::initialize()`.
|
|
146
|
+
- The orchestrator wraps the call in its own try/catch and turns any `Throwable` into an
|
|
147
|
+
alert — children must **not** catch (re-confirms the "fail loudly" rule above).
|
|
148
|
+
- A read-only `Run()` (e.g. a log scan) needs **no** `transactionCommit()`.
|
|
149
|
+
- **PHP heredoc gotcha:** `self::CONST` does **not** interpolate inside a heredoc. Bind
|
|
150
|
+
class constants to local variables before building a heredoc SQL string.
|
|
151
|
+
|
|
134
152
|
### Adding a new monitor (runbook)
|
|
135
153
|
|
|
136
154
|
1. Write the child class (one check, returns `{isOk, message}`).
|
|
@@ -150,17 +168,59 @@ No Lambda, SQS, or orchestrator changes needed per monitor.
|
|
|
150
168
|
- Pause a monitor: `isActive = 0` (cron row can stay active). Pause email only: `peopleToNotify = JSON_ARRAY()`.
|
|
151
169
|
- Manual state reset: clear `state`/`consecutiveOkCount`/`lastNotificationDt`/`lastNotificationType` — sparingly; the state machine self-corrects.
|
|
152
170
|
|
|
171
|
+
### Worked example — RateEntitlement (the first child monitor)
|
|
172
|
+
|
|
173
|
+
`_Worker_Monitors_RateEntitlement` (`worker2/Worker/Monitors/RateEntitlement.php`, the
|
|
174
|
+
**first** file in `Worker/Monitors/`) watches Rate's AIG warranty-contract creation
|
|
175
|
+
(TRUE-79129). It is the reference implementation for the **log-scan** monitor pattern:
|
|
176
|
+
|
|
177
|
+
1. `Run()` self-registers the `Logs_Rate` connection (the guard + `_Config::databaseLogs`
|
|
178
|
+
pattern above), because the orchestrator never calls `initialize()`.
|
|
179
|
+
2. Scans `Logs_Rate.Api` for **failed outbound** AIG contract POSTs in the last 30 minutes
|
|
180
|
+
— `direction = 'OUT'`, `method = 'POST'`, `route LIKE '%/contract'`, and
|
|
181
|
+
`responseCode` non-2xx **or NULL**. Returns `(object){isOk, message}`; read-only, so no
|
|
182
|
+
commit.
|
|
183
|
+
3. The detection signal is the `_underscore` **outbound API log**, not the entitlement
|
|
184
|
+
itself — see [why a monitor is needed](#why-an-external-monitor) below.
|
|
185
|
+
|
|
186
|
+
The change-set `dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql` does
|
|
187
|
+
the registration: one `Core.Monitors` row (`phpClass = _Worker_Monitors_RateEntitlement`,
|
|
188
|
+
`peopleToNotify = ['devteam@goagilant.com']`, `requiredConsecutiveOks = 2`,
|
|
189
|
+
`reminderFrequencyMinutes = 60`) + one `Core.CronJobs` row (`action = 'Monitor/Run'`,
|
|
190
|
+
`parameters = JSON_OBJECT('monitorId', LAST_INSERT_ID())`, schedule `*/15`,
|
|
191
|
+
`maxExecutionTime = 60`). LAST_INSERT_ID() chains the cron row to the just-inserted monitor.
|
|
192
|
+
|
|
193
|
+
### Detection via the _underscore outbound API log
|
|
194
|
+
|
|
195
|
+
`_underscore/ApiRequest.php` logs **every** outbound call to `_Model_Client_Logs_Api`
|
|
196
|
+
(table `Logs_<Client>.Api`) and **commits it immediately**, independent of whether the
|
|
197
|
+
caller later swallows the exception — which is exactly what makes log-scan monitoring
|
|
198
|
+
possible for silent-failure interceptors. Columns to scan: `dtStamp`
|
|
199
|
+
(`FIELD_DATETIME_CREATED`), `direction` (`OUT`), `method`, `hostname` (scheme://host),
|
|
200
|
+
`route` (path after host), `responseCode`. `setUrl()` splits a full URL into
|
|
201
|
+
`hostname` + `route`, so a call to `.../contract` is logged with `route` ending `/contract`
|
|
202
|
+
(and cancellation as `/contract/cancel`). **Pattern for new monitors:** scan the client's
|
|
203
|
+
`Logs_<Client>.Api` rather than trying to detect failure on the business object.
|
|
204
|
+
|
|
205
|
+
### Why an external monitor
|
|
206
|
+
|
|
207
|
+
Some interceptors **silently swallow** their own failures (no persistent flag on the
|
|
208
|
+
business record), so the only durable failure signal is the committed outbound API log.
|
|
209
|
+
The Rate case is documented in
|
|
210
|
+
[Rate AIG contract creation](../../../clients/rate/features/aig-contract-creation.md).
|
|
211
|
+
|
|
153
212
|
## Client variations
|
|
154
213
|
|
|
155
214
|
None — the framework is shared Core infrastructure. Individual monitors target specific
|
|
156
|
-
clients' data flows (Compass, Prudential, AIG, …) but live as separate child classes.
|
|
215
|
+
clients' data flows (Compass, Prudential, AIG, Rate, …) but live as separate child classes.
|
|
157
216
|
|
|
158
217
|
## Gotchas / known issues
|
|
159
218
|
|
|
160
|
-
- **
|
|
161
|
-
(as of 2026-06-10
|
|
162
|
-
-
|
|
163
|
-
|
|
219
|
+
- **Migration not applied to staging/production Core** — `Core.Monitors`
|
|
220
|
+
(`2026-05-21 - Monitors.sql`) is on local Core_2 only (as of 2026-06-10, re-confirmed
|
|
221
|
+
2026-06-29). The new `2026-06-29a` Rate change-set will **hard-fail (table not found)** on
|
|
222
|
+
any environment where `2026-05-21 - Monitors.sql` has not yet been promoted — confirm the
|
|
223
|
+
Monitors table exists before deploying.
|
|
164
224
|
- `worker2/MONITORING_PLAN.md` is referenced by the design doc but **missing on disk** —
|
|
165
225
|
stale reference to resolve.
|
|
166
226
|
- Child return value must be an **object** with both `isOk` and `message`; a bare array
|
|
@@ -177,6 +237,7 @@ clients' data flows (Compass, Prudential, AIG, …) but live as separate child c
|
|
|
177
237
|
HTML email + dashboard deep-links · anti-flap on the alarm side.
|
|
178
238
|
|
|
179
239
|
## Change history
|
|
240
|
+
- 2026-06-29 — Built the first child monitor, `_Worker_Monitors_RateEntitlement` (TRUE-79129) + change-set `2026-06-29a`; added the log-scan worked example, the `_underscore` outbound-API-log detection pattern, and confirmed child details (orchestrator never calls `initialize()` so children self-register non-Core connections; no commit on read-only `Run()`; heredoc cannot interpolate `self::CONST`). Re-flagged that `Core.Monitors` is still local-only — the new change-set hard-fails where the table is absent. (mhammontree)
|
|
180
241
|
- 2026-06-10 — Documented the v1.0 monitoring framework (orchestrator, `Core.Monitors` table, recovery-side anti-flap state machine, child contract). First child monitor + staging/prod migration still pending. (mhammontree)
|
|
181
242
|
|
|
182
243
|
## Related docs
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-06-
|
|
9
|
+
updated: 2026-06-29
|
|
10
10
|
owners: ["dfranks"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Netsuite/SalesOrder.php
|
|
@@ -127,12 +127,51 @@ The table lives in the **`Forecast` schema on the core2 cluster** (reader
|
|
|
127
127
|
sales-order FK column is **`netsuiteSalesOrderInternalId`** (`int unsigned`) — note this is *not* the
|
|
128
128
|
same column name used by `Forecast.Sales` (see the column-name gotcha below).
|
|
129
129
|
|
|
130
|
+
### Model/DB drift to know before backfilling location / backorder / amountDue (TRUE-79162)
|
|
131
|
+
|
|
132
|
+
A planned backfill of **location, backorder, and amount-due** data onto open orders runs into three
|
|
133
|
+
schema/model facts (all verified against the prod core2 reader; nothing built yet):
|
|
134
|
+
|
|
135
|
+
- **`_Model_Forecast_OpenOrderItem` (`_underscore/Model/Forecast/OpenOrderItem.php`) does NOT declare
|
|
136
|
+
`locationId` or `quantityBackordered` — even though both columns already exist in the prod
|
|
137
|
+
`Forecast.OpenOrderItems` table** (`locationId` `int unsigned NULL`, `quantityBackordered`
|
|
138
|
+
`decimal(15,4) NULL`). The model declares only `id, netsuiteSalesOrderInternalId, dateOrder,
|
|
139
|
+
orderNumber, customerId, salesRepEmployeeId, classificationId, accountId, itemId, lineNumber,
|
|
140
|
+
revenue, profit`. **Writing those two fields through the 2.0 importer requires adding the
|
|
141
|
+
properties to the model first** — the column existing in the DB is not sufficient.
|
|
142
|
+
- **`Forecast.Locations` is EMPTY in prod (0 rows).** Schema: `id, name, netsuiteInternalLocationId`
|
|
143
|
+
(UNIQUE, nullable), `typeId` — **no parent/hierarchy column.** Any open-order `locationId`
|
|
144
|
+
resolution returns NULL until this table is populated from NetSuite, so **populating `Locations` is
|
|
145
|
+
a hard prerequisite** for the location backfill. NetSuite returns a **leaf sub-location** but the
|
|
146
|
+
warehouse dashboard wants the **top-level location**, so a leaf→root rollup is needed (and there is
|
|
147
|
+
no parent column on `Locations` today to express it).
|
|
148
|
+
- **`Forecast.OpenOrderItems` has NO `amountDue` column** (neither does `Forecast.Sales` in prod —
|
|
149
|
+
see the [Sales import doc](../../_underscore/features/forecast-sale-import.md)), and `amountDue`
|
|
150
|
+
appears in no worker2 Netsuite handler or `_Model_Forecast_*`. Importing open-order amountDue
|
|
151
|
+
needs a **new column + model field**. The amountDue source semantics from the prior Sales work
|
|
152
|
+
(TRUE-78923) — REST `amountRemaining` / SuiteQL `foreignamountunpaid`, **anchor-line only, store
|
|
153
|
+
RAW POSITIVE** — carry over, **but `amountRemaining` is an AR/invoice field**, so for an *unbilled*
|
|
154
|
+
sales order it may return null/0. **Live-probe an open SO before finalizing** the open-order
|
|
155
|
+
amountDue source.
|
|
156
|
+
|
|
130
157
|
## Client variations
|
|
131
158
|
|
|
132
159
|
None — uniform (platform-wide Forecast2 sync).
|
|
133
160
|
|
|
134
161
|
## Gotchas / known issues
|
|
135
162
|
|
|
163
|
+
- **A new synced `OpenOrderItems` column must be wired into BOTH `buildOpenLineRows()` AND the
|
|
164
|
+
`syncOpenLines()` change-detection diff — not just the write — or no-diff updates silently drop
|
|
165
|
+
it.** The forward-sync path is `_Worker_Netsuite_SalesOrder`'s private `buildOpenLineRows()`
|
|
166
|
+
(assembles each `OpenOrderItems` row array) and `syncOpenLines()` (change-detection + upsert).
|
|
167
|
+
This is the established forecast2 change-detection gotcha applied to open orders: a column added
|
|
168
|
+
only to the row build but not to the `syncOpenLines()` diff block is **discarded on a no-change
|
|
169
|
+
UPDATE** (the diff sees no difference, so the new value never persists). Any
|
|
170
|
+
locationId/quantityBackordered/amountDue addition (TRUE-79162) must touch both. Forward sync now
|
|
171
|
+
flows through this webhook importer; the legacy 1.0 cron
|
|
172
|
+
(`worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php` open-order section) is
|
|
173
|
+
superseded for it. **Also mirror any new field into the reconciler `test/@dave/fixer.php` OOI path**
|
|
174
|
+
or it reverts the value on its next run (see the reconciliation doc).
|
|
136
175
|
- **Billing an SO via invoice-transform flips it to "Billed" WITHOUT firing the SalesOrder afterSubmit UE → no edit/PUT webhook → `removeAll()` never runs → the OpenOrderItems rows linger (this is the real cause of the "billed order not removed" problem — NOT a removeAll-didn't-persist bug).** When a NetSuite Sales Order is billed by **creating an Invoice from it**, the SO's `status` transitions to `Billed` as a **side-effect of the invoice transform** — this does **not** trigger the SalesOrder `afterSubmit` User Event, so the AMQ enqueuer (`ue_api_msg_queue_enqueue`, deployed per-record-type) **never emits an edit/PUT webhook for the SO**. Consequently `_Worker_Netsuite_SalesOrder` is never invoked for the billing change, `importOpenOrder`'s status gate (which *would* call `removeAll` on a non-open status) never runs, and the order's `OpenOrderItems` rows survive until the nightly discrepancy-fix cron deletes them. **Confirmed live 2026-06-24/25 on SO 7190415** (tranId 280387): only a `create` webhook delivery exists in `Logs.Webhook` (2026-06-24 23:48, 16 rows inserted) — **NO** edit/delete delivery at billing time (~09:41 UTC 2026-06-25); `Core.WorkerJobs` shows only `Netsuite/Webhook` + `Netsuite/SalesOrder/post` (no put/delete); `Logs.Api` has only the original NS GET + a 200 `OPEN_ORDER_IMPORT` breadcrumb — **no 204 removeAll breadcrumb**. The invoice (7190621) that billed it **also produced no webhook** — its 16 `Forecast.Sales` rows came from the **legacy 5-min pull cron**, not the webhook. **This corrects the earlier suspicion that "removeAll fired but DB didn't persist"** — both `removeAll` paths *do* commit `DB_FORECAST`; `removeAll` was simply **never called**. **Diagnostic ladder for a lingering billed order:** `Logs.Webhook` (was an edit/delete even delivered? — here, no) → `Core.WorkerJobs` (was a put/delete processed? — no) → `Logs.Api` `source='OPEN_ORDER_IMPORT'` (any 204 removeAll breadcrumb? — no). **Fix direction (not yet implemented):** the SalesOrder-side enqueuer must also fire on the invoice-transform status change — either deploy the AMQ enqueuer on **Invoice** so the bill event drives an SO re-sync, or have the SalesOrder handler re-evaluate the SO when its child invoice arrives; until then the **stopgap is the continuous `loop_trueup_open_orders.php --by-lastmodified` runner** (see the reconciliation doc), which windows on `lastmodifieddate` so it catches an order billed today whose `tranDate` predates the window. The `removeAll` path in `SalesOrder.php` was re-confirmed correct in this session.
|
|
137
176
|
- **Wrong FK column name → MySQL 1054 that masquerades as "order missing".** `OpenOrderItems`'s
|
|
138
177
|
sales-order key is **`netsuiteSalesOrderInternalId`**. Do **NOT** query it with
|
|
@@ -297,6 +336,16 @@ test fixture (it surfaced the stale SO 7181316 above).
|
|
|
297
336
|
|
|
298
337
|
## Change history
|
|
299
338
|
|
|
339
|
+
- 2026-06-29 — **Recorded the model/DB drift + prerequisites for the open-order location/backorder/
|
|
340
|
+
amountDue backfill (TRUE-79162, planning only — no code written).** `_Model_Forecast_OpenOrderItem`
|
|
341
|
+
declares neither `locationId` nor `quantityBackordered` though both columns already exist in prod
|
|
342
|
+
`OpenOrderItems` (so the model needs the properties before the importer can write them);
|
|
343
|
+
`Forecast.Locations` is empty in prod with no parent column (hard prerequisite + leaf→root rollup
|
|
344
|
+
needed); `OpenOrderItems` has no `amountDue` column (new column + field required, and
|
|
345
|
+
`amountRemaining` is an AR field that may be null/0 on an unbilled SO — probe first). Added the
|
|
346
|
+
gotcha that any new synced column must be wired into **both** `buildOpenLineRows()` and the
|
|
347
|
+
`syncOpenLines()` change-detection diff (and mirrored in `fixer.php`'s OOI path) or it is dropped on
|
|
348
|
+
no-diff updates. (dfranks)
|
|
300
349
|
- 2026-06-25 — **Root-caused "billed SO not removed from OpenOrderItems in real time" to a missing
|
|
301
350
|
webhook, NOT a removeAll-didn't-persist bug.** Billing an SO via invoice-transform flips it to
|
|
302
351
|
`Billed` as a side-effect that does **not** fire the SalesOrder `afterSubmit` UE → the AMQ enqueuer
|
package/knowledge/INDEX.md
CHANGED
|
@@ -16,7 +16,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
16
16
|
|
|
17
17
|
## 2.0 framework
|
|
18
18
|
|
|
19
|
-
- **_underscore** (_Underscore) _(framework core)_ —
|
|
19
|
+
- **_underscore** (_Underscore) _(framework core)_ — 18 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
20
20
|
- **worker2** (Worker) — 22 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
21
21
|
- **api2** (API) — 7 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
22
22
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Framework | Summary | Files |
|
|
4
4
|
|-----|-----------|---------|-------|
|
|
5
|
+
| [Rate AIG Warranty Contract Creation (silent-failure interceptor)](features/aig-contract-creation.md) | 2.0 | When a Rate entitlement is created, `_Model_Rate_Entitlement::postPost` (`_underscore/Model/Rate/Entitlement.php`) creates an **AIG warranty contract** as a non | _underscore/Model/Rate/Entitlement.php, _underscore/ApiRequest.php, worker2/Worker/Monitors/RateEntitlement.php |
|
|
5
6
|
| [Rate Monthly Reconciliation Report](features/monthly-reconciliation-report.md) | 1.0 | A monthly cron that emails an Excel reconciliation report covering all Rate subscription sales orders and their linked PayPal payments for the prior calendar mo | worker/crons/notifications/reports/rate/send_monthly_rate_purchases_report.php, worker/schedules/cron.worker.notification.json |
|
|
6
7
|
| [Rate SalesOrder → NetSuite CashSale Export (postPost)](features/netsuite-cashsale-export.md) | 2.0 | Rate sells home-warranty / home-tech-support products. | _underscore/Model/Rate/SalesOrder.php, _underscore/Model/Rate/Item.php |
|
|
7
8
|
| [Rate SAML SSO](features/saml-sso.md) | 2.0 | Rate uses Azure AD as its IdP (`login.rate.com`). | _underscore/Model/Rate/ClientAuthentication.php, saml/Controller/Index.php, toga2-view/src/hooks/useAuthenticationFlow.ts |
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Rate AIG Warranty Contract Creation (silent-failure interceptor)"
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: _underscore
|
|
5
|
+
project: _Underscore
|
|
6
|
+
client: rate
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-29
|
|
10
|
+
owners: [mhammontree]
|
|
11
|
+
files:
|
|
12
|
+
- _underscore/Model/Rate/Entitlement.php
|
|
13
|
+
- _underscore/ApiRequest.php
|
|
14
|
+
- worker2/Worker/Monitors/RateEntitlement.php
|
|
15
|
+
related:
|
|
16
|
+
- clients/rate/profile.md
|
|
17
|
+
- ../../../2.0/apps/worker2/features/monitoring-framework.md
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Summary
|
|
21
|
+
|
|
22
|
+
When a Rate entitlement is created, `_Model_Rate_Entitlement::postPost`
|
|
23
|
+
(`_underscore/Model/Rate/Entitlement.php`) creates an **AIG warranty contract** as a
|
|
24
|
+
non-critical side effect: it authenticates against AIG, POSTs to `.../contract`, and on
|
|
25
|
+
success stamps `payload->c_aigContractNumber` and `payload->c_aigContractId` back onto the
|
|
26
|
+
entitlement. **It only attempts this when the entitlement payload has a primary contact
|
|
27
|
+
address.**
|
|
28
|
+
|
|
29
|
+
The catch: **every failure path is silent.** There is no persistent failure flag on the
|
|
30
|
+
entitlement, so a contract that never got created looks identical to one that did until a
|
|
31
|
+
customer or AIG reports it missing. That is why this flow is watched externally by the
|
|
32
|
+
`_Worker_Monitors_RateEntitlement` monitor rather than by any state on the entitlement.
|
|
33
|
+
|
|
34
|
+
This is a base-`_underscore` interceptor specific to Rate's product (it lives under
|
|
35
|
+
`Model/Rate/`); documented here as a Rate client-feature because the behavior and its
|
|
36
|
+
monitoring are Rate-scoped.
|
|
37
|
+
|
|
38
|
+
## How it works
|
|
39
|
+
|
|
40
|
+
1. `postPost` runs after an entitlement is saved. It proceeds **only if** the payload has a
|
|
41
|
+
primary contact address; otherwise it does nothing.
|
|
42
|
+
2. Authenticate against AIG (POST `/authentication/login`) → obtain an access token.
|
|
43
|
+
3. POST the contract payload to `.../contract`.
|
|
44
|
+
4. On success: read `aigContractNumber` from the response and set
|
|
45
|
+
`payload->c_aigContractNumber` + `payload->c_aigContractId`.
|
|
46
|
+
|
|
47
|
+
## Silent-failure paths (the gotcha)
|
|
48
|
+
|
|
49
|
+
All of the following fail **silently** — no exception surfaces to the caller and nothing is
|
|
50
|
+
persisted on the entitlement:
|
|
51
|
+
|
|
52
|
+
- A `try/catch` that only `error_log()`s with the note *"AIG contract creation is
|
|
53
|
+
non-critical"* and swallows the exception.
|
|
54
|
+
- **Auth response non-2xx** (no throw).
|
|
55
|
+
- **No access token** returned (no throw).
|
|
56
|
+
- **Contract POST non-2xx** (no throw).
|
|
57
|
+
- **Success response but missing `aigContractNumber`** (no throw).
|
|
58
|
+
|
|
59
|
+
Net effect: there is no durable signal on the entitlement that contract creation failed.
|
|
60
|
+
|
|
61
|
+
## Detection / monitoring
|
|
62
|
+
|
|
63
|
+
Because the interceptor swallows failures, detection rides the **committed outbound API
|
|
64
|
+
log** instead. `_underscore/ApiRequest.php` logs every outbound call to
|
|
65
|
+
`_Model_Client_Logs_Api` (table `Logs_Rate.Api`) and commits it immediately, regardless of
|
|
66
|
+
whether the caller later swallows the exception. The AIG contract call lands with
|
|
67
|
+
`direction = OUT`, `method = POST`, and `route LIKE '%/contract'` (cancellation is
|
|
68
|
+
`/contract/cancel`).
|
|
69
|
+
|
|
70
|
+
`_Worker_Monitors_RateEntitlement` (worker2) scans `Logs_Rate.Api` for failed AIG contract
|
|
71
|
+
POSTs (`route LIKE '%/contract'`, `responseCode` non-2xx **or NULL**) in the last 30
|
|
72
|
+
minutes and alerts the dev team. See the
|
|
73
|
+
[Monitoring Framework worked example](../../../2.0/apps/worker2/features/monitoring-framework.md#worked-example--rateentitlement-the-first-child-monitor).
|
|
74
|
+
|
|
75
|
+
### Known detection limitation (decided)
|
|
76
|
+
|
|
77
|
+
Detection matches **only** `POST .../contract` (`route LIKE '%/contract'`), which
|
|
78
|
+
deliberately **excludes** `/contract/cancel` and the shared `/authentication/login`.
|
|
79
|
+
Consequence: a creation failure caused by an **auth failure** is **not** caught, because
|
|
80
|
+
`/authentication/login` is shared with the cancellation flow and so cannot be attributed to
|
|
81
|
+
creation vs. cancellation. This is an accepted, documented limitation — not an oversight.
|
|
82
|
+
|
|
83
|
+
## Gotchas / known issues
|
|
84
|
+
|
|
85
|
+
- **No persistent failure flag** — the only durable evidence of a failed contract creation
|
|
86
|
+
is the outbound API log row, not the entitlement.
|
|
87
|
+
- **Address-gated** — `postPost` does nothing when the payload lacks a primary contact
|
|
88
|
+
address, so an entitlement with no address never even attempts contract creation (and
|
|
89
|
+
never logs a `/contract` call).
|
|
90
|
+
- **Auth-induced failures are blind to the monitor** — see the detection limitation above.
|
|
91
|
+
|
|
92
|
+
## Change history
|
|
93
|
+
|
|
94
|
+
- 2026-06-29 — Documented the silent-failure AIG warranty-contract interceptor and the
|
|
95
|
+
external log-scan monitor (`_Worker_Monitors_RateEntitlement`, TRUE-79129), including the
|
|
96
|
+
accepted detection limitation that auth-induced failures are not attributable (shared
|
|
97
|
+
`/authentication/login`). (mhammontree)
|
|
@@ -6,17 +6,21 @@ apps:
|
|
|
6
6
|
- saml
|
|
7
7
|
- toga2-view
|
|
8
8
|
- worker
|
|
9
|
+
- worker2
|
|
10
|
+
- dbchanges2
|
|
9
11
|
project: SAML SSO Gateway
|
|
10
12
|
client: rate
|
|
11
13
|
type: profile
|
|
12
14
|
status: active
|
|
13
|
-
updated: 2026-06-
|
|
14
|
-
owners: ["rgirish", "bala"]
|
|
15
|
+
updated: 2026-06-29
|
|
16
|
+
owners: ["rgirish", "bala", "mhammontree"]
|
|
15
17
|
files: []
|
|
16
18
|
related:
|
|
17
19
|
- clients/rate/features/saml-sso.md
|
|
18
20
|
- clients/rate/features/monthly-reconciliation-report.md
|
|
19
21
|
- clients/rate/features/netsuite-cashsale-export.md
|
|
22
|
+
- clients/rate/features/service-card-entitlements.md
|
|
23
|
+
- clients/rate/features/aig-contract-creation.md
|
|
20
24
|
---
|
|
21
25
|
|
|
22
26
|
## Summary
|
package/package.json
CHANGED