toga-ai 1.0.197 → 1.0.199

package/knowledge/1.0/apps/worker/INDEX.md

@@ -5,7 +5,7 @@
 | [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
 | [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
 | [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
-| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php |
+| [Forecast2 ↔ NetSuite Reconciliation & Trueup Tooling](features/forecast2-netsuite-reconciliation.md) | CLI tools to **audit** and **repair** drift between the production `Forecast` DB (core2) and NetSuite. | test/@dave/reconcile_netsuite_totals.php, test/@dave/analyze_netsuite_forecast_diff.php, test/@dave/trueup_sales.php, test/@dave/trueup_open_orders.php, test/@dave/loop_trueup_open_orders.php, test/@dave/trueup_opportunities.php, test/@dave/probe_sales_gap_direct.php, test/@dave/probe_missing_oo_timing.php, test/@dave/probe_missing_oo_createdby.php, test/@dave/probe_drift_so_dates.php, test/@dave/probe_profit_invoices.php, test/@dave/probe_profit_gap.php, worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php, worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php, worker/crons/toga2/forecast2/import_open_orders.php, worker/schedules/cron.worker.infrastructure.json |
 | [NetSuite → TOGa Supply Per-Client Sync (thin wrappers)](features/netsuite-togasupply-per-client-sync.md) | Syncs NetSuite transactions (sales orders, purchase orders, invoices, item receipts, item fulfillments, inventory adjustments) into each TOGa Supply (2.0) clien | worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/crons/toga2/netsuite/sync_togasupply_canon.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |
 | [Prudential: Send Shipments for the Day report (daily cron)](features/send-shipments-for-the-day.md) | Daily cron (9:00 PM) that emails Prudential and Dell stakeholders an Excel report of all devices shipped that day, including tracking number, serial number, emp | worker/crons/notifications/reports/send_shipments_for_the_day.php |
 | [Onboarding a Client to the NetSuite TOGa Supply Sync](workflows/onboarding-client-to-netsuite-togasupply-sync.md) | How to add a new TOGa 2 client to the per-client NetSuite → TOGa Supply importer (`worker/crons/toga2/netsuite/`). | worker/crons/toga2/netsuite/sync_togasupply.php, worker/crons/toga2/netsuite/common_sync_togasupply.php, worker/schedules/cron.worker.sync.json, dbchanges2/_modules/netsuite/2026-04-01 - Parameters.sql |

package/knowledge/1.0/apps/worker/features/forecast2-netsuite-reconciliation.md

@@ -6,13 +6,14 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-24
+updated: 2026-06-25
 owners: [dfranks]
 files:
   - test/@dave/reconcile_netsuite_totals.php
   - test/@dave/analyze_netsuite_forecast_diff.php
   - test/@dave/trueup_sales.php
   - test/@dave/trueup_open_orders.php
+  - test/@dave/loop_trueup_open_orders.php
   - test/@dave/trueup_opportunities.php
   - test/@dave/probe_sales_gap_direct.php
   - test/@dave/probe_missing_oo_timing.php
@@ -21,6 +22,9 @@ files:
   - test/@dave/probe_profit_invoices.php
   - test/@dave/probe_profit_gap.php
   - worker/crons/toga2/forecast2/common_import_sales_from_netsuite.php
+  - worker/crons/toga2/forecast2/periodic_forecast_discrepancy_fix_open_orders.php
+  - worker/crons/toga2/forecast2/import_open_orders.php
+  - worker/schedules/cron.worker.infrastructure.json
 related:
   - ../architecture.md
   - ../../library/features/netsuite-suiteql-rest-shim.md
@@ -49,8 +53,29 @@ by reconciling a chosen tranDate range directly against NetSuite.
   NS_ONLY (missing from FC), FC_ONLY (stale/extra), DRIFT (value differs). Read-only.
 - `trueup_sales.php --from --to [--chunk-days N] [--prod] [--dry-run]` — makes `Forecast.Sales`
   match NetSuite for a tranDate range (insert/update/delete per line).
-- `trueup_open_orders.php --from= --to= [--prod] [--dry-run] [--verbose]` — same for
-  `Forecast.OpenOrderItems` (currently-open SOs whose tranDate falls in range).
+- `trueup_open_orders.php --from= --to= [--prod] [--dry-run] [--verbose] [--quiet] [--by-lastmodified]`
+  — same for `Forecast.OpenOrderItems` (currently-open SOs whose tranDate falls in range).
+  - **`--by-lastmodified`** windows **both** passes on `lastmodifieddate` instead of `tranDate`:
+    Step 1 (open-SO reconcile) and Step 3b (stale cleanup). `lastmodifieddate` is a TIMESTAMP →
+    bounded `TO_TIMESTAMP(from 00:00:00)..(to 23:59:59)` in the NetSuite **ET** session tz. Under this
+    mode Step 3b **inverts the lookup**: it asks NS for SOs **modified in the window** that are no longer
+    open (`status NOT IN B/D/E/F`) via id-only paged SuiteQL (**no per-id GET, no per-row request** —
+    cost scales with SOs modified, not with local row count), then deletes the local rows for any still
+    held. This catches an order **billed in the window whose `tranDate` predates it** — exactly the
+    invoice-transform billing case a `tranDate` window misses (see the open-orders sync doc).
+  - **`--quiet`** suppresses all progress (`out()` no-op) and prints the Step 4 results block **only**
+    when `insert+update+delete > 0` (a no-change pass prints nothing); transient NS failures + fatals
+    still go to stderr.
+  - **`--verbose`** was trimmed — removed per-line MATCH/EXCLUDE/NOOP and the always-on per-SO summary;
+    the per-SO header now prints lazily, once, only when the order has an INSERT/UPDATE/DELETE.
+- `loop_trueup_open_orders.php [--prod] [--dry-run] [--verbose] [--quiet] [--sleep=N] [--days-back=N]`
+  — continuous runner that recomputes a rolling `[today−N .. today]` window each pass and re-invokes
+  `trueup_open_orders.php --by-lastmodified` until Ctrl-C. Uses `PHP_BINARY` + `passthru` with
+  `escapeshellarg` over a **fixed flag allowlist** (`--prod`/`--dry-run`/`--verbose`/`--quiet`); survives
+  a failed pass. With `--quiet` it also suppresses its own per-pass banner + the sleeping line, so the
+  only output is a trueup Step 4 block when rows actually change. **This is the stopgap for the real-time
+  open-order removal gap** (an SO billed via invoice-transform fires no webhook → `removeAll` never runs —
+  see the open-orders sync doc).
 - `trueup_opportunities.php --from --to [--prod] [--dry-run]` — same for `Forecast.Opportunities`
   by tranDate range. Pure SuiteQL (no per-id GET), so it's fast: a 2026-YTD prod run reconciled
   ~4,000 opps in ~13s to a $0 delta.
@@ -206,11 +231,39 @@ None — Forecast2 is a single shared dataset.
   revenue. `trueup_sales.php` repairs them: it compares against NS by trandate, detects the sign
   mismatch, and delete+re-inserts with the correct sign. See the shim/reference docs for why a
   uniform `-foreignamount` is correct and an extra per-type `$factor` double-negates.
+- **`OpenOrderItems.dtPendingBilling` does not exist in prod and NO prod code writes it.** A `SHOW
+  COLUMNS` on the core2 reader confirms the column is **present locally, absent in prod** — the
+  TRUE-79081 migration was never deployed. `periodic_forecast_discrepancy_fix_open_orders.php` **intentionally
+  does not write it** (explicit guard comments; writing it would raise MySQL 1054) and worker2's
+  `_Worker_Netsuite_SalesOrder` never references it. So **nothing currently sets
+  `OpenOrderItems.dtPendingBilling` in prod.** (This **corrects** the earlier note that it was "fixed in
+  both writers" on 2026-06-15 — the writers were since changed to **skip** it pending the prod migration.)
+- **Of the legacy open-orders crons, only the discrepancy-fix is active — and it CLEANS billed orders,
+  it does not resurrect them.** `import_open_orders.php` is **`"active": 0` (DISABLED)** in
+  `worker/schedules/cron.worker.infrastructure.json`, but its flag `SHOULD_SYNC_OPEN_ORDERS = true` is
+  still set — a **latent re-enable footgun** (flipping `active` back on would resume the cron with its
+  known cascade/date-window quirks). The **only active legacy writer** of `OpenOrderItems` is
+  `periodic_forecast_discrepancy_fix_open_orders.php` (`active:1`, daily 3:00 AM): it INSERTs only for
+  currently-**open** SOs (status B/D/E/F) and its **Step-3b stale-cleanup DELETEs** rows whose SO is no
+  longer open — so it **removes** billed/closed orders rather than resurrecting them. It is therefore
+  **not** a resurrection risk against the webhook `removeAll` path; it is in fact the current backstop
+  deleter for orders the webhook never sees billed (the invoice-transform gap).
 - These tools live in `test/@dave/` (developer tooling), but `trueup_open_orders` has been run
   against production. The `Defaults`/checkpoint mechanics of the scheduled sync are separate.
 
 ## Change history
 
+- 2026-06-25 — **`trueup_open_orders` gained `--by-lastmodified` / `--quiet` / leaner `--verbose`; added
+  `loop_trueup_open_orders.php` as the stopgap for the real-time billed-order removal gap.**
+  `--by-lastmodified` windows both Step 1 and Step 3b on `lastmodifieddate` (ET TIMESTAMP bounds) and
+  inverts Step 3b to an id-only paged SuiteQL "modified-and-no-longer-open" lookup (no per-id GET), so it
+  catches an order billed in-window whose `tranDate` predates it. `--quiet` prints only a non-zero Step 4
+  block. `loop_trueup_open_orders.php` is a continuous rolling-window runner (PHP_BINARY + passthru over a
+  fixed flag allowlist). Also recorded that **`OpenOrderItems.dtPendingBilling` is local-schema-only and no
+  prod code writes it** (corrects the earlier "fixed in both writers" note — both writers now skip it), and
+  that of the legacy open-orders crons **only `periodic_forecast_discrepancy_fix_open_orders.php` is active**
+  and it cleans up billed/closed orders rather than resurrecting them (while `import_open_orders.php` is
+  `active:0` but still flags `SHOULD_SYNC_OPEN_ORDERS=true` — a latent re-enable footgun). (dfranks)
 - 2026-06-24 — **Sales audit correctness pass.** Fixed a phantom Sales delta: `reconcile`'s NS Sales
   SUM/diff now applies the sync's status exclusions (Unapproved Payment / Voided / Rejected) so it can't
   count rows the sync never stores (closed revenue delta to $0). Added a Sales per-transaction presence

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-06-24
+updated: 2026-06-25
 owners: [dfranks]
 files:
   - _underscore/Component/Forecast/SaleImport/SaleImport.php
@@ -154,4 +154,11 @@ would be needed only for a future real-time item webhook, **not** for this Sales
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-06-25 — **Verified `sync()` end-to-end across all four sale types** (prod NS reads, local Forecast
+  writes; 40+ records spanning invoice/cashSale/creditMemo/cashRefund): revenue reconciled **to the penny**
+  against an independent SuiteQL `SUM(-foreignamount)` oracle on every record; the uniform sign factor held
+  (invoice/cashSale positive, creditMemo/cashRefund negative); kit handling correct (itemGroup line and the
+  no-item-id summary line dropped, the real component line kept); tax-reversal credit memos correctly net to
+  $0 line revenue; the guarded `INSERT … WHERE NOT EXISTS` proved idempotent (re-sync → 0 duplicates); and
+  item self-heal worked for old records. No code change — confirms the existing behavior. (dfranks)
 - 2026-06-24 — Built the Forecast.Sales real-time webhook importer (shared `_Component_Forecast_SaleImport` engine + four thin worker2 handlers), replacing the legacy cron SALES section; decided the uniform-factor sign convention against the raw REST record; deduped `fetchRecord`/sublist pagination onto `_Component_Forecast_Db`. (dfranks)

package/knowledge/2.0/apps/ai-bdr/features/call-orchestration.md

@@ -17,6 +17,7 @@ files:
 related:
   - ../architecture.md
   - vapi-integration.md
+  - ../../worker2/features/vapi-webhook-handler.md
 ---
 
 ## Summary
@@ -204,8 +205,23 @@ gracefully handles `"Information not available"`.
 
 ## Where the PHP code lives
 
-**Not in this repo.** The handler lives in the TOGA worker repo (`worker`
-for 1.0 or `worker2` for 2.0). Likely class shape:
+**Now committed in `worker2`** as `_Worker_Vapi` (`worker2/Worker/Vapi.php`,
+the webhook) alongside the outbound dialer `_Worker_Ai_Bdr_Vapi`
+(`worker2/Worker/Ai/Bdr/Vapi.php`). Full detail:
+**`2.0/apps/worker2/features/vapi-webhook-handler.md`**.
+
+> **Corrections confirmed against the live `_Worker_Vapi` code (2026-06-24):**
+> 1. **Structured output is in `artifact.structuredOutputs`** (keyed by a random
+>    UUID; first entry's `result`), **not** `analysis.structuredData` —
+>    `analysis{}` is always empty in practice.
+> 2. **The webhook join key is `assistantOverrides.metadata.contactAttemptUuid`**
+>    (the dialer pre-creates the `ContactAttempt` and passes its UUID). `call.id`
+>    is stored only as `c_vapiCallIdentifier`, not used as the join key.
+> 3. **Filter on `message.type`** — only `end-of-call-report` is processed;
+>    `status-update`/`hang`/etc. are acknowledged and skipped. Inbound and test
+>    calls carry no `contactAttemptUuid` and must be skipped gracefully.
+
+Historical (pre-commit) shape guess:
 
 - 1.0: an `App_Controller_*` for the webhook (under the framework's HTTP
   surface), with an `App_Worker_*` or scheduled cron for cadence.

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -15,4 +15,5 @@
 | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |
 | [Team Sprint Management & Reporting](features/team-sprint-management.md) | `_Worker_Team_Sprint` (file `Worker/Team/Sprint.php`) is the engine behind TOGA's internal **development-sprint process and reporting**. | worker2/Worker/Team/Sprint.php |
 | [Teams Meeting Transcript Export](features/teams-transcript-export.md) | `_Worker_Team_Transcripts` (action `Team/Transcripts/Export`) polls Microsoft Graph for Teams meeting transcripts produced by a set of organizers, classifies ea | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Core/2026-06-18a - Teams Transcript Export schedule.sql |
+| [VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)](features/vapi-webhook-handler.md) | `_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the AI-BDR call loop** — the webhook that receives VAPI's end-of-call r | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php |
 | [WJE Freshservice Sync (worker2)](features/wje-freshservice-sync.md) | WJE ("WJE IT", helpdesk `wje.freshservice.com`) is a **Freshservice**-based help-desk client whose tickets, contacts, assets, groups, categories, and canned res | worker2/Worker/Wje.php, _underscore/Component/Api/Wje/Wje.php, _underscore/Model/Wje/Ticket.php, _underscore/Model/Wje/TicketNote.php, _underscore/Model/Wje/Contact.php, _underscore/Model/Wje/Unit.php, _underscore/Model/Wje/TicketTeam.php, _underscore/Model/Wje/TicketCategory.php, _underscore/Model/Wje/AssetType.php, _underscore/Model/Wje/PredefinedReply.php, library/app/api/wje.php, worker/crons/toga2/wje/import_supporting_records.php, worker/crons/toga2/wje/sync_togasupply_wje.php, worker/crons/notifications/reports/wje/wje_common.php, library/app/systemmonitor/wje.php, dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql |

package/knowledge/2.0/apps/worker2/features/netsuite-opportunity-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-18
+updated: 2026-06-25
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite.php
@@ -321,6 +321,25 @@ None — platform-wide Forecast sync.
   HTTP call so a concurrent drainer pass skips it (`Sending` ∉ sendable), narrowing the double-send
   window to the load→claim gap. `findSendableIds` must list every sendable status
   (`Created`,`Pending`,`Retry`,`Sending`) — omitting `Created` silently matches nothing (`candidates:0`).
+- **The enqueuer is generic over all record types but RELEASED only on Opportunity + Sales Order.**
+  `ue_api_msg_queue_enqueue` (`customscript_ue_amq_enqueue`) is generic via `RECORD_TYPE_MAP`, fires
+  `afterSubmit` for create/edit/xedit/delete, has **no execution-context filter**, and runs with
+  `DEV_OVERRIDE.enabled=false` in prod (posts to `webhook.togahub.com/netsuite`). Probe-confirmed it is
+  **`Released` on only two record types**: Opportunity (`customdeploy1`) and Sales Order
+  (`customdeploy2`). So Invoice/CashSale/CreditMemo/CashRefund (the Forecast.Sales types) are **not yet
+  enqueued via webhook** — which is why a billing invoice fires no webhook (see the open-orders doc's
+  invoice-transform gap). Adding a real-time sale-import webhook requires releasing this same enqueuer on
+  those four types.
+- **`lib_amq_queue.js` is a LIBRARY module (no `@NScriptType`) → it has NO deployment of its own, and its
+  `N/log` output surfaces in the CALLING script's execution log.** Its `transmit()` wraps `https.request`
+  in a try, but the surrounding `record.load`/claim and `markFailure`'s `submitFields` are **outside** that
+  try — so a throw there escapes to `ss_amq_drain`'s per-row catch. A live drainer error
+  `{"queueId":40529,"error":"UNEXPECTED_ERROR: An unexpected SuiteScript error has occurred"}` (a TypeError,
+  observed 2026-06-25 1:10am) originates in this transmit path and shows up under **`ss_amq_drain`'s**
+  execution log (not a library log of its own). **Durable error detail is also persisted on the API Message
+  Queue custom record** fields `custrecordcustrecord_amq_errormsg` / `_resp_status` / `_resp_body`
+  (doubled-prefix ids — see the doubled-id gotcha), which are **SuiteQL-queryable** for after-the-fact
+  diagnosis even though the calling-script log rolls off.
 
 ## CU→NS direction (future — not yet built): reverse mapping
 
@@ -345,6 +364,15 @@ same **skip-if-unchanged** compare on the extracted values, and **actor-identity
 trigger a CU→NS write. The NS→CU change-detection above is the complementary backstop, not a substitute.
 
 ## Change history
+- 2026-06-25 — **Characterized the AMQ enqueuer scope + a live drainer TypeError.** The enqueuer is
+  generic over all record types via `RECORD_TYPE_MAP` (no context filter, prod `DEV_OVERRIDE.enabled=false`)
+  but probe-confirmed `Released` on **only Opportunity (`customdeploy1`) + Sales Order (`customdeploy2`)** —
+  so the four Forecast.Sales record types are not yet webhook-enqueued. Recorded that `lib_amq_queue.js` is a
+  **library module** (no `@NScriptType`/own deployment) whose `N/log` surfaces in the **calling** script's
+  (`ss_amq_drain`) log, and that an `UNEXPECTED_ERROR` (TypeError, queueId 40529, 2026-06-25 1:10am)
+  originates in its `transmit()` path where `record.load`/claim + `markFailure.submitFields` sit **outside**
+  the `https.request` try. Durable error detail also persists on the queue record's
+  `custrecordcustrecord_amq_errormsg`/`_resp_status`/`_resp_body` fields (SuiteQL-queryable). (dfranks)
 - 2026-06-24 — **Default presales-lead assignee fallback.** `buildCustomFields()` now wires the
   previously-unused `DEFAULT_PRESALES_LEAD_USER_ID = '87374309'` (Cory Martin, comartin@togatech.com —
   verified active via ClickUp `GET /team`) into the else-branch when `resolvePresalesUserId()` returns

package/knowledge/2.0/apps/worker2/features/netsuite-salesorder-open-orders-sync.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-06-23
+updated: 2026-06-25
 owners: ["dfranks"]
 files:
   - worker2/Worker/Netsuite/SalesOrder.php
@@ -133,6 +133,7 @@ None — uniform (platform-wide Forecast2 sync).
 
 ## Gotchas / known issues
 
+- **Billing an SO via invoice-transform flips it to "Billed" WITHOUT firing the SalesOrder afterSubmit UE → no edit/PUT webhook → `removeAll()` never runs → the OpenOrderItems rows linger (this is the real cause of the "billed order not removed" problem — NOT a removeAll-didn't-persist bug).** When a NetSuite Sales Order is billed by **creating an Invoice from it**, the SO's `status` transitions to `Billed` as a **side-effect of the invoice transform** — this does **not** trigger the SalesOrder `afterSubmit` User Event, so the AMQ enqueuer (`ue_api_msg_queue_enqueue`, deployed per-record-type) **never emits an edit/PUT webhook for the SO**. Consequently `_Worker_Netsuite_SalesOrder` is never invoked for the billing change, `importOpenOrder`'s status gate (which *would* call `removeAll` on a non-open status) never runs, and the order's `OpenOrderItems` rows survive until the nightly discrepancy-fix cron deletes them. **Confirmed live 2026-06-24/25 on SO 7190415** (tranId 280387): only a `create` webhook delivery exists in `Logs.Webhook` (2026-06-24 23:48, 16 rows inserted) — **NO** edit/delete delivery at billing time (~09:41 UTC 2026-06-25); `Core.WorkerJobs` shows only `Netsuite/Webhook` + `Netsuite/SalesOrder/post` (no put/delete); `Logs.Api` has only the original NS GET + a 200 `OPEN_ORDER_IMPORT` breadcrumb — **no 204 removeAll breadcrumb**. The invoice (7190621) that billed it **also produced no webhook** — its 16 `Forecast.Sales` rows came from the **legacy 5-min pull cron**, not the webhook. **This corrects the earlier suspicion that "removeAll fired but DB didn't persist"** — both `removeAll` paths *do* commit `DB_FORECAST`; `removeAll` was simply **never called**. **Diagnostic ladder for a lingering billed order:** `Logs.Webhook` (was an edit/delete even delivered? — here, no) → `Core.WorkerJobs` (was a put/delete processed? — no) → `Logs.Api` `source='OPEN_ORDER_IMPORT'` (any 204 removeAll breadcrumb? — no). **Fix direction (not yet implemented):** the SalesOrder-side enqueuer must also fire on the invoice-transform status change — either deploy the AMQ enqueuer on **Invoice** so the bill event drives an SO re-sync, or have the SalesOrder handler re-evaluate the SO when its child invoice arrives; until then the **stopgap is the continuous `loop_trueup_open_orders.php --by-lastmodified` runner** (see the reconciliation doc), which windows on `lastmodifieddate` so it catches an order billed today whose `tranDate` predates the window. The `removeAll` path in `SalesOrder.php` was re-confirmed correct in this session.
 - **Wrong FK column name → MySQL 1054 that masquerades as "order missing".** `OpenOrderItems`'s
   sales-order key is **`netsuiteSalesOrderInternalId`**. Do **NOT** query it with
   `netsuiteTransactionInternalId` — that is the **`Forecast.Sales`** column. On `OpenOrderItems` it
@@ -296,6 +297,16 @@ test fixture (it surfaced the stale SO 7181316 above).
 
 ## Change history
 
+- 2026-06-25 — **Root-caused "billed SO not removed from OpenOrderItems in real time" to a missing
+  webhook, NOT a removeAll-didn't-persist bug.** Billing an SO via invoice-transform flips it to
+  `Billed` as a side-effect that does **not** fire the SalesOrder `afterSubmit` UE → the AMQ enqueuer
+  emits no edit/PUT webhook → `_Worker_Netsuite_SalesOrder` never runs → `removeAll()` is never called
+  (both removeAll paths confirmed to commit `DB_FORECAST` correctly — the call simply never happens).
+  Confirmed live on SO 7190415: only a `create` delivery in `Logs.Webhook`, no put/delete in
+  `Core.WorkerJobs`, no 204 removeAll breadcrumb in `Logs.Api`; the billing invoice produced no webhook
+  either (its Sales rows came from the legacy pull cron). Recorded the `Logs.Webhook`→`WorkerJobs`→
+  `OPEN_ORDER_IMPORT`-breadcrumb diagnostic ladder and the stopgap (`loop_trueup_open_orders.php
+  --by-lastmodified`). (dfranks)
 - 2026-06-24 — **Outbound push (`CreateNetSuite`/`UpdateNetSuite`/`Sync`) converted SOAP →
   REST-only** via `_Component_Api_Netsuite`: item + customer lookups now SuiteQL
   (`resolveNetSuiteCustomerId()` throws on 0/>1 name matches), `buildNetSuiteOrder()` emits a REST

package/knowledge/2.0/apps/worker2/features/vapi-webhook-handler.md

@@ -0,0 +1,156 @@
+---
+title: VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-06-24
+owners: [snaredla]
+files:
+  - worker2/Worker/Vapi.php
+  - worker2/Worker/Ai/Bdr/Vapi.php
+related:
+  - ../../ai-bdr/features/call-orchestration.md
+  - ../../ai-bdr/features/vapi-integration.md
+---
+
+## Summary
+
+`_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the
+AI-BDR call loop** — the webhook that receives VAPI's end-of-call report, writes the call
+results back onto the originating `ContactAttempt`, and dispatches the post-call action
+(book meeting / schedule callback / nurture / DNC). This is the concrete handler that the
+`ai-bdr` repo's `call-orchestration.md` referred to as "lives in worker/worker2 — add a
+doc when committed."
+
+It is the **counterpart** to the outbound dialer `_Worker_Ai_Bdr_Vapi`
+([worker2/Worker/Ai/Bdr/Vapi.php](worker2/Worker/Ai/Bdr/Vapi.php)), which places the calls.
+
+## Key files / entry points
+
+| File | Role |
+|---|---|
+| `worker2/Worker/Vapi.php` (`_Worker_Vapi`) | Inbound **webhook** — processes `end-of-call-report`, writes back, dispatches actions |
+| `worker2/Worker/Ai/Bdr/Vapi.php` (`_Worker_Ai_Bdr_Vapi`) | Outbound **dialer** — creates the `ContactAttempt`, places the VAPI call |
+
+- Action route: `Vapi/Webhook` → `webhook.togahub.com/vapi` (VAPI server messages POST here).
+- `initialize()` registers the `Client_True` DB connection (`_underscore::DB_CLIENT`).
+
+## How it works
+
+1. **`Webhook($payload, $headers)`** — JSON-decodes the body, takes `message`, hands off to
+   `processVapiEndOfCallPayload()`.
+2. **`processVapiEndOfCallPayload()`**:
+   - Reads `call.assistantOverrides.metadata.contactAttemptUuid` (falls back to
+     `call.metadata`).
+   - Loads `_Model_True_ContactAttempt` by that UUID. This is the **join key** back to our DB.
+   - Extracts structured output via `extractVapiStructuredOutput()`.
+   - `updateContactAttempt()` writes the results, then `executePostCallActions()` dispatches.
+3. **`extractVapiStructuredOutput($artifact)`** — returns the first
+   `artifact.structuredOutputs[*].result` array. **`analysis{}` is always empty — the data
+   lives in `artifact.structuredOutputs`, keyed by a random UUID.**
+4. **`updateContactAttempt()`** — maps `structured.phpWorkerAction` → `c_callOutcome` /
+   `c_actionToTake` via `$actionMap`; builds a payload of only non-empty `c_*` fields (so
+   blanks never overwrite existing data); converts VAPI ISO timestamps to **Central Time**;
+   persists via `_Component_Api_Toga::send('PUT', '/contact-attempts/{uuid}', …)` (Toga 2.0
+   REST API, **not** a direct DB write); also clears the contact's `dtNextContactRequested`.
+5. **`executePostCallActions()`** — `switch ($contactAttempt->c_actionToTake)`:
+   - `BOOK_CALCOM_MEETING` → `actionCalComBookMeeting` (Cal.com booking + queue
+     `Ai/Bdr/Netsuite/createLeadFromBookedMeeting` + mark COMPLETED)
+   - `SCHEDULE_CALLBACK` → `actionScheduleCallback` (sets `dtNextContactRequested` +
+     `contactCallTypeId=CALLBACK`)
+   - `SEND_NURTURE_EMAIL` → `actionSendNurtureEmail` (queue
+     `Ai/Bdr/Netsuite/createLeadFromNurtureEmail` + mark COMPLETED)
+   - `DO_NOT_CALL` → `actionFlagDnc` (`isOkayToCall=0` + EXCLUDE from PENDING/ACTIVE campaigns)
+   - `NO_ACTION` / default → reset the campaign-contact to PENDING
+
+### phpWorkerAction → outcome/action map
+
+| `phpWorkerAction` | `c_callOutcome` | `c_actionToTake` |
+|---|---|---|
+| `BOOK_CALCOM_MEETING` | MEETING_BOOKED | BOOK_CALCOM_MEETING |
+| `SCHEDULE_CALLBACK` | CALLBACK_SCHEDULED | SCHEDULE_CALLBACK |
+| `SEND_NURTURE_EMAIL` | NURTURE_REQUESTED | SEND_NURTURE_EMAIL |
+| `FLAG_DNC_REMOVE_FROM_CAMPAIGNS` | DNC_REQUESTED | DO_NOT_CALL |
+| `MARK_NOT_INTERESTED` | NOT_INTERESTED | DO_NOT_CALL |
+| `MARK_WRONG_PERSON` | WRONG_PERSON | DO_NOT_CALL |
+| `MARK_VOICEMAIL_RETRY` | VOICEMAIL | NO_ACTION |
+| `MARK_NO_ANSWER_RETRY` | NO_ANSWER | NO_ACTION |
+| `MARK_GATEKEEPER_BLOCKED_RETRY` | GATEKEEPER_BLOCKED | NO_ACTION |
+
+## The contactAttemptUuid write-back mechanism
+
+`contactAttemptUuid` is the **sole linkage** between a VAPI call and our DB:
+
+1. The dialer (`_Worker_Ai_Bdr_Vapi`) creates the `ContactAttempt` row **before** placing the
+   call ([Ai/Bdr/Vapi.php:438](worker2/Worker/Ai/Bdr/Vapi.php#L438),
+   [:694](worker2/Worker/Ai/Bdr/Vapi.php#L694)) and passes its UUID in
+   `assistantOverrides.metadata.contactAttemptUuid`
+   ([Ai/Bdr/Vapi.php:628](worker2/Worker/Ai/Bdr/Vapi.php#L628)).
+2. VAPI echoes that same metadata onto **every** server event for the call.
+3. The webhook loads the row by that UUID and writes results back to it.
+
+`contactId` and `campaignId` are read from the **loaded `ContactAttempt` row**, never from the
+webhook payload. `call.id` is stored only as `c_vapiCallIdentifier`.
+
+## Message-type filtering (the bug this doc was born from)
+
+VAPI sends the webhook **many** server-message types: `status-update`, `hang`,
+`speech-update`, `end-of-call-report`, etc. **Only `end-of-call-report` carries the call data
+(`artifact` / structured output) and should be processed.** The handler must guard on
+`message.type === 'end-of-call-report'` in `Webhook()` and acknowledge (`return 'ok'`) all
+other types before they reach `processVapiEndOfCallPayload()`.
+
+- **Regression origin:** commit `583f6be` "Added remaining webhooks" (2026-05-14) created this
+  file and broadened the VAPI subscription to all server-message types. Without a type guard,
+  every event was processed as an end-of-call report. First Sentry errors appeared ~late May.
+- **Symptom:** `Warning: Undefined property: stdClass::$contactAttemptUuid` (promoted to an
+  `ErrorException` by the framework error handler, captured by Sentry).
+
+## Inbound / test calls vs worker-placed outbound calls
+
+The metadata shape depends on **who placed the call**, not the event type:
+
+- **Outbound (our dialer)** → metadata carries `contactAttemptUuid` (every event type,
+  including status-updates).
+- **Inbound calls and dashboard test calls** → **no `contactAttemptUuid`** (they never went
+  through our dialer, so no `ContactAttempt` was pre-created). Test calls may instead carry
+  hand-typed metadata like `{contactId, campaignId, attemptNumber}`.
+
+Therefore an `end-of-call-report` can legitimately arrive **without** a `contactAttemptUuid`
+(inbound/test). Those have no attempt to write back to and should be **skipped gracefully**
+(`return 'ok'`), not thrown — otherwise they raise `Contact Attempt not found` and fail the
+job. `Contact Attempt not found` should be reserved for a report that *has* a
+`contactAttemptUuid` but no matching row (a real data-integrity anomaly).
+
+## Data model
+
+Writes to `ContactAttempts` (via Toga 2.0 API) — `dtStarted`, `dtEnded`, `transcript`,
+`contactCallTypeId`, and many `c_*` fields (`c_callOutcome`, `c_actionToTake`,
+`c_prospectSentiment`, `c_qualificationScore`, `c_callSummary`, `c_meetingTime`,
+`c_meetingType`, `c_contactEmail`, `c_callbackTime`, `c_recordingUrl`, `c_callCost`,
+`c_vapiCallIdentifier`, `c_dncRequested`, etc.). Also updates `Contacts`
+(`dtNextContactRequested`, `contactCallTypeId`, `isOkayToCall`) and `Campaigns_Contacts`
+status. See the `ai-bdr` skill / `_Model_True_ContactAttempt` for the full field list.
+
+## Gotchas / known issues
+
+- **Structured data is in `artifact.structuredOutputs`, not `analysis.structuredData`.**
+  `analysis{}` is always empty. (Corrects the older `ai-bdr/call-orchestration.md`.)
+- **Join key is `contactAttemptUuid`, not `call.id`.** `call.id` is only stored as
+  `c_vapiCallIdentifier`.
+- **No message-type guard = errors.** Always filter to `end-of-call-report` first.
+- **Persistence is via the Toga REST API**, not a direct model `save()` — failures are
+  caught and logged, not re-thrown.
+- **Timestamps are stored in Central Time**; Cal.com booking converts `c_meetingTime` back to
+  UTC.
+
+## Related docs
+- [AI-BDR Call Orchestration (the PHP↔VAPI contract)](../../ai-bdr/features/call-orchestration.md)
+- [VAPI Integration — assistants, tools, structured output](../../ai-bdr/features/vapi-integration.md)
+
+## Change history
+- 2026-06-24 — Initial doc: webhook handler, contactAttemptUuid write-back, message-type
+  filtering bug, inbound/test vs outbound metadata. (snaredla)

package/knowledge/INDEX.md

@@ -16,7 +16,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 14 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 14 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 15 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 6 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 2 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.197",
+  "version": "1.0.199",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",