toga-ai 1.0.291 → 1.0.293

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -10,7 +10,7 @@
 | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind TOGa Supply's Fulfill & Ship: buying a carrier label (UPS/FedEx), persisting it, and creating the NetSuite Item Fulfillment with tracki | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ItemFulfillments/TrackingNumber.php, _underscore/Component/Library/LabelPdf/LabelPdf.php, _underscore/Component/Library/Carriers/ShipmentRequest/ShipmentRequest.php, _underscore/Component/Library/Carriers/Ups/Ups.php, _underscore/Component/Library/Carriers/Fedex/Fedex.php, _underscore/Trait/Netsuite/ItemFulfillment.php, _underscore/Trait/Netsuite/SalesOrder.php, _underscore/Component/Library/NetSuite/NetSuite.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/ShippingMethod.php, _underscore/Model.php, _underscore/Cloud.php |
 | [_Component_*/_Model_* project-namespace registration (autoloader) & backslash-qualify traps](features/component-model-namespace-registration.md) | Every **project-local** `_Component_*` and `_Model_*` class in a 2.0 app **must declare the project namespace** at the top of the file: ```php namespace <NAMESP | _underscore/Loader.php, worker2/_.php, api2/_.php, worker2/Component/Forecast/Db/Db.php, worker2/Component/Forecast/SaleImport/SaleImport.php, api2/Component/Api/Netsuite/Netsuite.php |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
-| [Error Reporting — Issue/Event Aggregation (exceptionHandler)](features/error-reporting-issue-event.md) | `_underscore`'s global exception handler persists every uncaught exception into a two-table **Issue / Event** model in the **shared Core Logs DB** (`_underscore | _underscore/Error.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, dbchanges2/Logs/2026-07-06 - Issue and Event tables.sql |
+| [Error Reporting — Issue/Event Aggregation (agreed POST-to-receiver design)](features/error-reporting-issue-event.md) | Platform-wide error-reporting infrastructure for TOGA 2.0, built around a two-table **Issue / Event** aggregation model in the shared **Core Logs DB**. | _underscore/Error.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, dbchanges2/Logs/2026-07-06 - Issue and Event tables.sql |
 | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
 | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | _underscore/Component/Forecast/SaleImport/SaleImport.php, _underscore/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
 | [Item-Fulfillment Stage Lifecycle (picked/packed/shipped) & Order Status](features/item-fulfillment-stage-lifecycle-and-order-status.md) | Every ItemFulfillment (IF) now carries an explicit **stage** — picked → packed → shipped — resolved through `ItemFulfillmentStages → ItemFulfillmentStatuses` (m | _underscore/Model/Client/SalesOrder.php, _underscore/Model/Quad/SalesOrder.php, _underscore/Model/Compass/SalesOrder.php, _underscore/Model/Compass/SalesOrderStatus.php, _underscore/Model/Client/SalesOrderItem.php, _underscore/Model/Client/Item.php, _underscore/Model/Client/PurchaseOrderItem.php, library/app/api/toga2.php, dbchanges2/Client/2026-06-30a - BackfillNullStageItemFulfillmentsToShipped.sql, dbchanges2/Client/2026-06-30b - SalesOrderStatusesPickedPacked.sql, dbchanges2/Client/2026-06-30c - ItemFulfillmentStageIdNotNull.sql, dbchanges2/Client_CompassCanada/2026-06-30a - ItemFulfillmentLifecycleAndShippedBackfill.sql |

package/knowledge/2.0/apps/_underscore/features/component-model-namespace-registration.md

@@ -92,6 +92,15 @@ project-local `_Component_*`/`_Model_*` must still declare the project namespace
   unqualified `catch (Exception $e)` fails **silently** — the more insidious of the two.
 
 ## Change history
+- 2026-07-08 — **Production recurrence** of this exact fatal (concrete fingerprint, no doc change
+  to the mechanism): a worker2 deploy declared `_Component_Forecast_SaleImport`
+  (`Worker/Netsuite/Invoice.php` autoload) and its sibling `_Component_Forecast_Db`
+  (`Component/Forecast/Db/Db.php`, opportunity jobs) in the global namespace → every
+  Forecast-writing NetSuite webhook type threw at autoload the moment the deploy landed, flipping
+  a Forecast-wide ~98%-success window to ~72%-failure at a single timestamp. Confirms the
+  invisible-to-`php -l`, blocks-all-writes-before-DB behavior in prod. Diagnostic path (how the
+  before/after `isSuccess` split is read out of `Core.WorkerJobs`) is captured in the
+  [Forecast.Sales import doc's diagnostic playbook](./forecast-sale-import.md). (dfranks)
 - 2026-07-08 — Documented the `_Component_*`/`_Model_*` project-namespace registration
   requirement (Loader.php eval-bridge mechanism, ~71-83), the runtime-only failure mode (not
   caught by `php -l`), and the three constructs that must be backslash-qualified inside a

package/knowledge/2.0/apps/_underscore/features/error-reporting-issue-event.md

@@ -1,12 +1,12 @@
 ---
-title: Error Reporting — Issue/Event Aggregation (exceptionHandler)
+title: Error Reporting — Issue/Event Aggregation (agreed POST-to-receiver design)
 framework: "2.0"
 repo: _underscore
 project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-07-06
+updated: 2026-07-08
 owners: ["dfranks"]
 files:
   - _underscore/Error.php
@@ -21,39 +21,51 @@ related:
 
 ## Summary
 
-`_underscore`'s global exception handler persists every uncaught exception into a
-two-table **Issue / Event** model in the **shared Core Logs DB** (`_underscore::DB_LOGS`,
-models under `Model/Core/Logs/`) — **not** the per-client Logs DB. An **Issue** is the
-deduplicated record for a distinct error (keyed by a hash of the backtrace) carrying an
-aggregate `occurrences` counter; an **Event** is one row per individual occurrence with
-its own stack trace and context. This is platform-wide error-reporting infrastructure,
-not client-specific.
-
-## Key files / entry points
-
-- **`_underscore/Error.php`** — `exceptionHandler()` (registered via
-  `set_exception_handler`). Computes `$issueHash = md5(serialize($backtrace))`, then, only
-  when the Core-Logs register is present (`isset(\_Database::$_registers[_underscore::DB_LOGS])`),
-  upserts an `_Model_Core_Logs_Issue` and inserts an `_Model_Core_Logs_Event`, committing
-  the `DB_LOGS` transaction.
+Platform-wide error-reporting infrastructure for TOGA 2.0, built around a two-table
+**Issue / Event** aggregation model in the shared **Core Logs DB**. An **Issue** is the
+deduplicated record for a distinct error (keyed by a hash of the stack trace) carrying an
+aggregate `occurrences` counter and an escalatable `urgency`; an **Event** is one row per
+individual occurrence with its own trace and context.
+
+**The agreed architecture (see below) decouples *reporting* from *persistence*:**
+application error handlers **POST** error payloads to a centralized `/errors` receiver
+endpoint; a **worker2 cron** consumes them and does all the hashing, upserting, aggregation,
+escalation, and ClickUp sync into the Core Logs DB. Handlers do **not** write Issue/Event
+rows to the database directly. This is the canonical target design for the whole
+error-monitoring epic (TRUE-781xx) — the handler ticket, the dashboard ticket, and the
+escalation-cron ticket must all conform to it.
+
+## Agreed error-monitoring architecture
+
+Source: the **"2026-05-21 - Errors, Monitoring and Alerts"** meeting design (approved;
+cited via Talos DevCore meeting notes). Pipeline:
+
+1. **Report (application side).** An application's error/exception handler serializes the
+   error (message, stack trace, context) and **POSTs it to a centralized `/errors` receiver
+   endpoint** (the meeting referenced `webhook.hub.com/error`). The handler's only job is to
+   transmit — it performs **no direct database persistence**.
+2. **Ingest (worker2 cron).** A worker2 cron consumes the received error payloads and:
+   - builds `issueHash` from the stack trace to identify the distinct error,
+   - **upserts an Issue** and **inserts an Event** into the **Core** Logs DB,
+   - aggregates occurrence counts within a time window,
+   - auto-escalates / de-escalates the Issue `urgency` based on occurrence frequency,
+   - syncs the Issue to ClickUp.
+3. **Scope.** API 2.0 handlers first; 1.0 is deferred.
+
+The critical rule: **handlers POST; the cron persists (to Core).** Any approach where the
+handler writes Issue/Event rows inline to a database contradicts this design.
+
+## Data model (Core Logs DB)
+
 - **`_underscore/Model/Core/Logs/Issue.php`** — `const DATABASE = _underscore::DB_LOGS;
   const TABLE = 'Issues';`. Fields: `id, uuid, dtCreated, dtLastOccurred, urgency, subject,
-  description, clickupIdentifier, hash, errorMessage, occurrences, trace`.
+  description, clickupIdentifier, hash, errorMessage, occurrences, trace`. One row per
+  distinct error (dedup key = `hash`); `occurrences` is the rolling count, `dtLastOccurred`
+  the most-recent hit, `urgency` a list field, `clickupIdentifier` links to an escalation
+  ticket. There is **no `type` column**.
 - **`_underscore/Model/Core/Logs/Event.php`** — `const DATABASE = _underscore::DB_LOGS;
   const TABLE = 'Events';`. Fields: `id, uuid, issueId (FK → Issue), errorMessage, trace,
-  context, dtCreated`.
-
-## How it works
-
-1. On an uncaught exception, hash the backtrace to identify the distinct error.
-2. Load the existing `Issue` by `hash`. On a miss, create it with `errorMessage`, `trace`,
-   `urgency = 'LOW'`, and `occurrences = 0`.
-3. **Increment `occurrences`** (`($issue->occurrences ?? 0) + 1`) and set
-   `dtLastOccurred`, then save — so the counter advances on every occurrence, hit or new.
-4. Insert one `Event` for this occurrence: `issueId`, `errorMessage`, per-occurrence
-   `trace`, and `context` (`print_r($GLOBALS, true)`).
-5. Commit the `DB_LOGS` transaction; surface `Error {issueId}, Event {eventId}` in the
-   debug body.
+  context, dtCreated`. One row per occurrence.
 
 The `Issues`/`Events` tables live in the shared **Core Logs** DB, provisioned from
 `dbchanges2/Logs/` (the `Logs/` folder targets the framework-level, non-tenant `Logs` DB
@@ -61,35 +73,55 @@ with unqualified table names — see the dbchanges2 architecture folder→DB map
 with per-client API/error logging in `Logs_<Tenant>` via `Model/Client/Logs/*`
 (`DB_CLIENT_LOGS`, `dbchanges2/Logs_Client/`).
 
-## Data model
+## Current state of `_underscore/Error.php` (important — reads differently than you'd expect)
+
+`_underscore/Error.php` registers `exceptionHandler()` via `set_exception_handler`, but in
+**`_production` it does NOT persist any Issue/Event rows.** It builds a debug HTML body,
+writes it to an S3 error-details file, and (when a Logs register is present) records a
+single `_Model_Client_Logs_Error` / `_Model_Core_Logs_Error` row — the older flat error log.
+There is no Issue/Event aggregation, no `issueHash` upsert, and no ClickUp escalation in the
+handler on `_production`.
 
-- **Issue** = one row per distinct error (dedup key = `hash`); `occurrences` is the rolling
-  count, `dtLastOccurred` the most-recent hit, `urgency` a list field, `clickupIdentifier`
-  links to an escalation ticket. There is **no `type` column** — it was dropped.
-- **Event** = one row per occurrence, FK `issueId` → Issue, carrying the occurrence's own
-  `trace` and `context`.
+**An inline-DB-persist approach exists on an in-flight branch, but it conflicts with the
+agreed design and must not be treated as the model to follow:** that approach has the
+handler compute `$issueHash = md5(serialize($backtrace))` and directly upsert
+`_Model_Client_Logs_Issue` + insert `_Model_Client_Logs_Event`, committing a transaction —
+i.e. handler-side direct persistence, and to the **Client** Logs DB rather than **Core**.
+That contradicts the approved architecture on two counts: (a) handlers should POST to the
+`/errors` receiver, not persist; and (b) aggregation belongs in the Core Logs DB, driven by
+the worker2 cron. New work (TRUE-78178 handlers) should implement the POST-to-receiver path,
+not extend the inline-persist branch.
 
 ## Gotchas / known issues
 
-- **`occurrences` must be incremented, not just seeded.** It was previously set to `0` only
-  at creation and never advanced, so the aggregate counter never moved. Fixed 2026-07-06 to
+- **Do not assume error persistence "already works."** On `_production` the handler writes
+  only an S3 debug body and a flat `*_Logs_Error` row — no Issue/Event aggregation exists in
+  production yet. The Issue/Event model is the *target*, delivered by the POST→cron pipeline.
+- **`occurrences` must be incremented, not just seeded** — a naive upsert that sets
+  `occurrences = 0` only at creation never advances the aggregate counter. The cron must
   increment on every occurrence.
-- **`Event.trace` must be set per occurrence.** The schema has a per-Event `trace` column
-  that was never populated; now recorded on every Event.
-- **Persistence is conditional on the Core-Logs register.** If
-  `\_Database::$_registers[_underscore::DB_LOGS]` isn't set, the whole Issue/Event block is
-  skipped silently — no error row is written.
+- **`Event.trace` must be populated per occurrence** — the per-Event `trace` column is easy
+  to leave empty; record it on every Event.
+- **Persistence must be conditional on the Core-Logs register.** If the Core Logs register
+  (`\_Database::$_registers[_underscore::DB_LOGS]`) isn't set, an Issue/Event write must be
+  skipped rather than fatal — but this belongs in the cron, not the handler.
+- **Client vs Core Logs DB mix-up.** The Issue/Event tables belong in the **Core** Logs DB
+  (`DB_LOGS`, `Model/Core/Logs/*`). Writing them to a per-client Logs DB
+  (`DB_CLIENT_LOGS`, `_Model_Client_Logs_*`) is the wrong target and does not match the
+  agreed design.
 - **Shared-schema migration collision.** Because `Issues`/`Events` live in the *shared* Core
-  Logs DB, any two dbchanges2 `Logs/` migrations that both `CREATE TABLE Issues`/`Events`
-  will collide — the second fails with "table already exists". Two separate error-reporting
-  workstreams introduced these same tables (one adding the escalation cron + an
-  `IssueEmailAddresses` table, one adding this error-class persistence). Only **one**
-  migration may create the shared tables; reconcile overlapping `Logs/` create-scripts at
-  merge so the tables are created exactly once and later migrations only `ALTER`.
+  Logs DB, two dbchanges2 `Logs/` migrations that both `CREATE TABLE Issues`/`Events` will
+  collide ("table already exists"). Only **one** migration may create the shared tables;
+  later migrations only `ALTER`.
 
 ## Change history
 
-- 2026-07-06 — Moved Issue/Event tables and exceptionHandler persistence from the per-client
-  Logs DB to the shared Core Logs DB and dropped the `type` column; fixed `occurrences` to
-  increment each occurrence and set `Event.trace` per occurrence; removed dead commented-out
-  per-client-logs persistence block. (dfranks)
+- 2026-07-08 — Corrected KB drift: documented the approved 2026-05-21 error-monitoring
+  architecture (handlers POST to a centralized `/errors` receiver; a worker2 cron builds
+  issueHash, upserts Issues + inserts Events into the Core Logs DB, aggregates/escalates,
+  syncs ClickUp — handlers do not persist). Clarified that `_production` Error.php performs
+  no Issue/Event persistence (only an S3 debug body + flat `*_Logs_Error` row), and that the
+  inline-DB-persist approach (handler-side upsert to the *Client* Logs DB) conflicts with the
+  agreed design and is not the pattern to follow. (dfranks)
+- 2026-07-06 — Prior doc described the exceptionHandler as directly persisting Issue/Event
+  rows to the Core Logs DB with dedup; superseded by the 2026-07-08 correction above. (dfranks)

package/knowledge/2.0/apps/_underscore/features/forecast-sale-import.md

@@ -447,6 +447,46 @@ Keep **create→delete windows short** so most records slip between cron runs; o
 `import_sales` during a batch. (The daily discrepancy-fix also cleans stragglers since the
 record is deleted in NetSuite.)
 
+## Diagnostic playbook — proving whether inbound webhooks actually wrote Forecast.Sales (run cold)
+Durable procedure for answering "did the real-time webhook path work, or is a `fixer.php`
+sweep masking a broken pipeline?" It spans **two DB clusters** — correlate them; never infer
+success from a `Forecast.Sales` row alone.
+1. **Receipt (prod-logs `Logs.Webhook`)** — proves the POST *arrived*, nothing more. Columns:
+   `id, uuid, dtStamp, sourceIp, method, route, requestPayload` (JSON with
+   `recordType`/`internalId`/`eventType`/`action`). NetSuite hits land on route `/netsuite`.
+   **There is NO status/error column** — a row here means "received", not "succeeded".
+2. **Success/failure (prod-core `Core.WorkerJobs`)** — the authoritative outcome:
+   `isSuccess` (NULL=pending/running, 0=fail, 1=success), `failureReason` (mediumtext, full
+   stack trace), `action`, `dtCreated`.
+3. **Two-stage fan-out.** The router job `action='Netsuite/Webhook'` fans out to per-record-type
+   **child** jobs (`Netsuite/Invoice/post`, `Netsuite/SalesOrder/put`, `Netsuite/Opportunity/*`,
+   etc.). The **router almost always succeeds**; the Forecast writes — and their failures — live
+   in the **child** jobs. Filter WorkerJobs on the child `action`, not the router, to judge the
+   pipeline.
+4. **CRITICAL — a `Forecast.Sales` row does NOT prove the webhook worked.** `fixer.php` (the
+   reconciliation sweep) independently writes/repairs rows, so a present row can mask a fully
+   broken webhook path. **Always correlate `Logs.Webhook.dtStamp` against
+   `WorkerJobs.isSuccess`/`dtCreated` by uuid/timestamp** — never conclude success from row
+   presence.
+5. **Bucket errors by class.** `failureReason` embeds a per-row `[uuid] ` prefix; strip it with
+   `CASE`/`REGEXP_REPLACE` to collapse per-row UUIDs into error classes and get clean counts —
+   the fast way to distinguish a deployment-wide fatal from long-tail noise.
+
+**Known WorkerJobs failure classes on this pipeline (durable):**
+- **Autoload fatal blocks ALL Forecast writes before any DB work** — a global-namespace
+  `_Component_Forecast_SaleImport`/`_Component_Forecast_Db` throws at autoload (see the
+  [namespace-registration doc](./component-model-namespace-registration.md)); every
+  Forecast-writing child type (Invoice/SalesOrder/Opportunity/JournalEntry/CashSale/CreditMemo)
+  fails identically and the fatal is **invisible to `php -l`**. A clean before/after `isSuccess`
+  split at a single deploy timestamp is the fingerprint (a bad push flips a Forecast-wide
+  ~98%-success window to ~72%-failure). Recurred in production 2026-07-08.
+- **`Netsuite/InventoryItem/*` / `NonInventoryItem/post` — `watchdog: exceeded maxExecutionTime
+  without completing`**: item syncs timing out at the 300s watchdog.
+- **`Netsuite/Invoice/post` — MySQL error 1213 `Deadlock found when trying to get lock`** on the
+  Forecast DB: pre-existing, low-volume, ~1.6–3.9s each.
+- **`Netsuite/Webhook` router — `Unsupported NetSuite eventType: approve`** (`Worker/Netsuite.php`
+  ~line 53): salesOrder `approve` events aren't handled by the router and fail at fan-out.
+
 ## Gotchas / known issues
 - **Testing locally pollutes PROD unless you UNDEPLOY the prod AMQ enqueuer first.** Each sale
   transaction type has **TWO** NetSuite UE enqueuer deployments: **"AMQ — Enqueuer"**
@@ -499,6 +539,19 @@ record is deleted in NetSuite.)
 - The cron's sign handling is not portable here — see Sign convention.
 
 ## Change history
+- 2026-07-08 — **Added the webhook-vs-fixer diagnostic playbook** for proving whether inbound
+  NetSuite webhooks actually wrote `Forecast.Sales`: correlate `Logs.Webhook` (prod-logs;
+  receipt-only, NO status column) against `Core.WorkerJobs` (prod-core; `isSuccess`/
+  `failureReason`), judge the **child** `Netsuite/<Type>/<verb>` jobs (not the almost-always-OK
+  `Netsuite/Webhook` router), and **never infer success from a `Forecast.Sales` row** because
+  `fixer.php` independently writes/masks rows — correlate timestamps. Bucket `failureReason` with
+  `CASE`/`REGEXP_REPLACE` (strip the `[uuid] ` prefix) into error classes. Recorded a concrete
+  **production recurrence (2026-07-08)** of the global-namespace autoload fatal on
+  `_Component_Forecast_SaleImport`/`_Component_Forecast_Db` (Forecast-wide, invisible to `php -l`,
+  clean before/after `isSuccess` split at a single deploy timestamp), plus three lower-volume
+  WorkerJobs failure classes on this pipeline (item-sync watchdog timeout; Invoice deadlock 1213;
+  router `Unsupported NetSuite eventType: approve`). Read-only diagnostic session, no code change.
+  (dfranks)
 - 2026-07-08 — **Clarified the revenue sign convention is NOT the `amountDue` "store raw
   positive" rule.** The TRUE-78923 "don't sign-flip credit memos/refunds" guidance applies
   only to `amountDue` (which this importer does not write); the line `amount`/revenue sign

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.291",
+  "version": "1.0.293",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",