toga-ai 1.0.762 → 1.0.764

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -21,7 +21,7 @@
 | [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
 | [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, _underscore/String.php, worker2/Worker/Infrastructure/Email/Send.php, _underscore/Model/Client/Logs/Email.php, _underscore/Model/Client/Logs/EmailAttachment.php |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Quad/ApprovalDecision.php, _underscore/Model/Quad/SalesOrder.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
-| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Database.php, _underscore/Exception/Business.php, api2/Controller/Index.php, worker2/Controller/Index.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, worker2/Worker/Infrastructure/Errors.php, tools/mvc/errors/issue/get.php, tools/mvc/errors/post.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Logs/2026-08-01a - Rename tables to singular.sql, dbchanges2/Logs_Client/2026-08-01a - Drop Error table.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
+| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Query.php, _underscore/Database.php, _underscore/Exception/Business.php, _underscore/Exception/Database.php, _underscore/Test/DatabaseFingerprint.php, api2/Controller/Index.php, worker2/Controller/Index.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, worker2/Worker/Infrastructure/Errors.php, tools/mvc/errors/issue/get.php, tools/mvc/errors/post.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Logs/2026-08-01a - Rename tables to singular.sql, dbchanges2/Logs_Client/2026-08-01a - Drop Error table.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
 | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
 | [FIELD_STORAGE fields — per-row lazy hydration and the platform-wide missing-column 500](features/field-storage-row-hydration.md) | `FIELD_STORAGE` is the 2.0 field type for blob-backed columns (S3 or local folder). | _underscore/Model.php, _underscore/Model/Client/TrackingNumber.php, _underscore/Model/Client/Invoice.php, dbchanges2/Core/HISTORIC/2024/2024-11b - item-fulfillments.sql |
 | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |

package/knowledge/2.0/apps/_underscore/features/error-reporting-issue-event.md

@@ -6,12 +6,15 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-08-19
+updated: 2026-09-03
 owners: ["dfranks", "jcardinal", "mhammontree", "ajean", "bala", "apeterson"]
 files:
   - _underscore/Error.php
+  - _underscore/Query.php
   - _underscore/Database.php
   - _underscore/Exception/Business.php
+  - _underscore/Exception/Database.php
+  - _underscore/Test/DatabaseFingerprint.php
   - api2/Controller/Index.php
   - worker2/Controller/Index.php
   - _underscore/Model/Core/Logs/Issue.php
@@ -132,6 +135,61 @@ produced a different hash on every request — dedup never actually worked.
 cannot orphan a curated Issue: an admin repoints the new fingerprint at the existing Issue
 from the Tools console to merge them.
 
+#### Database errors fingerprint on the MESSAGE, not the trace (added 2026-09-03)
+
+The trace-based scheme above **over-merges database errors**, because they all surface from a
+few shared query methods. Every DB failure is thrown as a plain `Exception` from `_Query`, so
+their top ~5 frames are nearly identical (e.g. `_Query->getFoundRows()` → `processRoutePairs()`
+→ `execute()`). The one thing that told two DB bugs apart — the message (errno + column/table) —
+was exactly what the fingerprint **discards**. Result: production Issue `3L` (id 64) bundled a
+`1146` "table doesn't exist" **plus four different `1054` "unknown column"** bugs, across
+clients, into one Issue with ~2,514 occurrences — genuinely different defects hidden as one.
+
+The fix routes DB errors down their own fingerprint path:
+
+- **`_Query` now throws a typed `_Exception_Database`** (`_underscore/Exception/Database.php`,
+  `class _Exception_Database extends Exception`) instead of a plain `Exception`, carrying typed
+  `errorNumber` / `errorText` (getters `getErrorNumber()` / `getErrorText()`). It **extends plain
+  `Exception`, NOT `_Exception_Validation`**, so a DB error stays **HTTP 500 + creates a
+  `Logs.Issue`** (api2 maps only `_Exception_Validation` to 400). Every existing
+  `catch (Exception|Throwable)` and the deadlock-retry (which reads the stored errno, not the
+  type) are unaffected. It **deliberately does NOT keep the SQL** as a property/getter — that
+  avoids an easy leak of a query with literal values.
+- **`_Error::buildFingerprint()` gains an early `instanceof _Exception_Database` branch** →
+  `buildDatabaseFingerprint()`, which keys the fingerprint on
+  **`errno + normalizeDatabaseErrorText(errorText)`** instead of stack frames. So DB errors
+  split by exact table/column, while everything else keeps the trace scheme.
+
+`normalizeDatabaseErrorText()` masks the error text, **in this order**, so no per-row value or
+echoed SQL fragment reaches the shared `Logs.IssueFingerprint` table:
+
+1. Cap the input at 2000 chars.
+2. **Strip the tenant DB prefix** `(Client|Archive|Logs)_<Tenant>.` from identifiers — so the
+   **same bug on two clients is ONE Issue** (which tenant is still recorded on `Event.clientId`).
+3. Mask the value in `Duplicate entry '<value>' for key '<key>'` — **keep the key**, drop the
+   value (groups across values, still splits on a different key).
+4. **DEFAULT-DENY:** keep a quoted token only if it matches a schema-identifier shape
+   `^[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*$`; otherwise mask it to `?`. This keeps
+   row values, password hashes, and echoed SQL fragments from a `1064` **out** of the shared
+   fingerprint table.
+5. Digits → `N`.
+
+`firstApplicationArea()` derives the context `areaPath` from the first application frame.
+
+> **Forward-only, one-time re-grouping (expected).** Existing Issues keep their old fingerprint
+> hashes; new events form new fingerprints. On deploy, Issue `3L` fans out into ~5 fresh Issues
+> — fresh notifications and up to 5 new ClickUp tasks, **no carryover of any muting**. This is a
+> one-time, deliberate effect. **No schema change.** (CTO-reviewed: AGREE-WITH-ALTERNATIVE —
+> scope message-based grouping to DB errors ONLY, because arbitrary exception messages have no
+> known shape to mask safely.)
+
+> **No PHPUnit harness in `_underscore`.** There is only the vendored `library` test suite. The
+> masking is covered by a **standalone CLI script**, `_underscore/Test/DatabaseFingerprint.php`
+> (`php Test/DatabaseFingerprint.php`, 16 checks): the 5 errors that merged into `3L` now split
+> into 5; same bug on two clients = one Issue; duplicate-entry groups across values but splits on
+> the key; default-deny masks non-identifier values / hashes / SQL fragments; a different errno
+> is a different Issue.
+
 ### Where each exception class routes (the class you throw IS the routing decision)
 
 | Thrown class | HTTP | Sentry | `Logs.Issue` | Fingerprint / routing |
@@ -139,6 +197,7 @@ from the Tools console to merge them.
 | `_Exception_Validation` | **400** | no | **no** | n/a — bad client input. Still leaves a full `Logs.Api` row (see [api2 request logging](../../api2/features/request-logging.md)) |
 | `_Exception_Business` | **4xx** (was 500 before 2026-08-04) | yes | **yes** | stable `issueKey` (refactor-immune) → business recipients configured in the Tools console. **Now a subclass of `_Exception_Validation` — must be caught FIRST, see below** |
 | plain global `Exception` | **500** | yes | **yes** | **trace**-based fingerprint (moves when code moves) → developer ClickUp task |
+| `_Exception_Database` | **500** | yes | **yes** | **not thrown by hand** — `_Query` throws it for every DB error. **Fingerprinted by `errno + masked error text`, not trace** (below). Subclass of plain `Exception`, so it routes exactly like one → developer ClickUp task |
 | `_Exception` | — | — | — | **never throw it deliberately** (below) |
 
 Verified against `api2/Controller/Index.php` (≈L205-245, L369-377) and `_Error.php`
@@ -567,6 +626,16 @@ gracefully — it silently deletes all observability for the whole tier. Any cha
   redactor (in `_underscore/Error.php` **and** `library/app/error/capture.php` together, per the
   gotcha above), and until then **never paste a `Logs.Event.context` or `Logs_<Client>.Api` row
   into a ticket, chat, or knowledge doc** — those rows carry live tokens and customer PII.
+- **⚠ OPEN (pre-existing, not fixed 2026-09-03): the raw DB-error message — including ~120 chars
+  of SQL with literal values and the tenant DB name — is stored into `Issue.subject` /
+  `Event.errorMessage`.** `_Error.php` (~L413 and ~L487) does `substr(..., 255)` of the assembled
+  message ("Database Error Occurred… Query:\n<sql>") into those columns on the **shared** `Logs`
+  cluster. This predates the `_Exception_Database` work (the old plain `Exception` carried the
+  same message), so it did not block that change. **Recommended follow-up:** for
+  `_Exception_Database`, store the **masked signature**
+  (`'DB #' . errno . ': ' . normalizeDatabaseErrorText(errorText)`) as subject/errorMessage
+  instead of the raw SQL. The 2026-09-03 fix shipped **without** this — treat it as a known
+  open item.
 - **Hand-written SQL is what a table rename actually breaks — and it fails *after* the model
   succeeds.** The singular rename left plural literals in hand-written statements across four
   files, so `$issue->save()` succeeded and then `UPDATE Issues …` threw **1146** before the
@@ -696,6 +765,26 @@ read you feed it — `_Error` promotes a *"read property on int"* warning into a
 
 ## Change history
 
+- 2026-09-03 — **Database errors now fingerprint on the message, not the trace.** Root cause of
+  a real prod problem: Issue `3L` (id 64, ~2,514 occurrences) bundled a `1146` plus four
+  different `1054` bugs across clients into one Issue, because every DB error is thrown from a
+  few shared `_Query` methods with near-identical top frames and the trace scheme drops the
+  message — the only thing telling them apart. Fix: `_Query` (~L356) now throws a typed
+  **`_Exception_Database`** (`_underscore/Exception/Database.php`, extends **plain `Exception`**
+  so it stays 500 + `Logs.Issue`; typed `errorNumber`/`errorText`; deliberately keeps NO SQL to
+  avoid a value leak). `_Error::buildFingerprint()` gets an early `instanceof _Exception_Database`
+  branch → `buildDatabaseFingerprint()`, keyed on `errno + normalizeDatabaseErrorText(errorText)`.
+  The masker: cap 2000 chars; strip the tenant DB prefix so the same bug on two clients is ONE
+  Issue (tenant still on `Event.clientId`); mask the duplicate-entry value but keep the key;
+  **default-deny** — keep a quoted token only if it matches a schema-identifier shape, else `?`,
+  so no row values/hashes/SQL fragments reach the shared `Logs.IssueFingerprint`; digits → `N`.
+  Forward-only, one-time re-grouping on deploy (`3L` fans into ~5 fresh Issues, no muting
+  carryover, no schema change). CTO AGREE-WITH-ALTERNATIVE: scope message-grouping to DB errors
+  only. No PHPUnit harness in `_underscore`, so covered by a standalone CLI script
+  `_underscore/Test/DatabaseFingerprint.php` (16 checks). **Left OPEN (pre-existing, cso-flagged):**
+  the raw DB-error message with ~120 chars of literal-value SQL + the tenant DB name is still
+  `substr(...,255)` into `Issue.subject`/`Event.errorMessage` on the shared `Logs` cluster —
+  recommended follow-up is to store the masked signature there instead. (jcardinal)
 - 2026-08-19 — Recorded that **email recipients are configured per `issueId`, not per `issueKey`**,
   so they cannot be pre-configured before a condition first fires — the `Logs.Issue` row must exist
   first, which means the **first occurrence of any new business exception always routes to a

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-31
+updated: 2026-09-03
 owners: ["jcardinal", "mhammontree", "bala"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -69,9 +69,17 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
   `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (3h), so a single order stuck too
   long pages even under the count threshold — this catches the 1.0 cron's
   [poison-order starvation](../../../clients/compass-usa/features/mits-sales-order-transmission-alerting.md).
-  **Age is measured from the FINAL (stage-2) approval decision** (`ApprovalDecisions` stage 2
-  `dtDecision` where `isApproved = 1` — not stage 1, not order date; that is the "eligible to
-  transmit" clock). The query **excludes canceled orders** (`salesOrderStageId IS NULL OR
+  **Age is measured from the LATER of the two approval decisions —
+  `GREATEST(stage-1, stage-2) dtDecision`** — because an order is eligible to transmit only once
+  **both** stages are approved. Measuring from **stage-2 alone is wrong** (it was the source of a
+  false HIGH — see the 2026-09-03 change-history): Compass **VIP auto-approve stamps the stage-2
+  (manager) decision at order creation** (`postPost`), so stage 1 (admin) can land days later; a
+  stage-2-only clock reads a huge age the instant stage-1 approves on a perfectly healthy order.
+  **NULL-guarded:** an `isApproved = 1` decision can still carry a **NULL `dtDecision`**, and a bare
+  `GREATEST` returns NULL — so `MAX()` would silently drop that order and hide a genuinely stuck one.
+  Each stage is therefore `COALESCE`d to the other stage's decision, then to `SalesOrders.dtCreated`,
+  so the age clock is never NULL and a stuck order always surfaces (fail loud). The query
+  **excludes canceled orders** (`salesOrderStageId IS NULL OR
   <> 8`), mirroring the transmit cron's WHERE, so an order canceled after approval (e.g.
   SA136567, now stage 8) neither counts nor false-alarms forever — this also fixed a
   pre-existing count mismatch vs. the cron. Config: `$ALARM_AGE_MINUTES = 180`; body echoes
@@ -430,6 +438,26 @@ check for another client.
   monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
 
 ## Change history
+- 2026-09-03 — **Fixed a false HIGH age alarm** in `CompassSalesOrderTransmission()`.
+  `oldestPendingAgeMinutes` was measured from **stage-2 approval only**
+  (`TIMESTAMPDIFF(MINUTE, ApprovalDecisionsStage2.dtDecision, NOW())`), but stage 2 is **not**
+  always the final approval: Compass **VIP auto-approve stamps the stage-2 (manager) decision at
+  order creation** (`postPost`), so stage-1 (admin) can land days later. An order transmits only
+  once **both** stages approve, so the correct clock is the **later of the two** decisions.
+  **Incident 2026-09-03:** OneUptime HIGH, `pendingCount=1`, `oldestPendingAgeMinutes=2360`
+  (threshold 180) — order SA136844 (`Client_Compass` id 120481): stage-2 VIP auto-approved at
+  creation 09-01 19:55, stage-1 admin approved 09-03 11:15, transmitted 09-03 11:18 (3 min after
+  becoming eligible). The alarm fired the exact minute stage-1 approved — a **monitor** bug, the
+  `dtDecision` values were correct. Scale (prod, last 30 days both-approved = 1927): 439 had
+  stage-2 before stage-1, **133 with that gap > 180 min** (each a false HIGH if pending at run
+  time). **Fix:** age now `GREATEST(stage-1, stage-2) dtDecision`, and because an `isApproved=1`
+  row can carry a **NULL `dtDecision`** (1,415 such rows in prod) where a bare `GREATEST` returns
+  NULL and `MAX()` would silently drop a stuck order, each stage is `COALESCE`d to the other
+  stage's decision then to `SalesOrders.dtCreated` — the clock is never NULL, a stuck order always
+  surfaces (fail loud). Summary string changed from "since stage-2 approval" to "since eligible
+  (later of both approvals)". Timezone confirmed **not** an issue (DB `NOW()` and `dtDecision` both
+  Central). No OneUptime/SQL/config change — the age alarm still rides the existing `"alarm":"HIGH"`
+  token. sql-reviewer SAFE TO MERGE, php -l clean; worker2-only, not deployed. (jcardinal)
 - 2026-08-31 — Added **age-based detection** to `CompassSalesOrderTransmission()`: it now alarms
   on `pendingCount > 10` **OR** `oldestPendingAgeMinutes > 180` (new `$ALARM_AGE_MINUTES = 180`),
   so one order stuck too long pages even under the count threshold — catching the 1.0 transmit

package/knowledge/2.0/standards/backend-php.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: shared
 type: standard
 status: active
-updated: 2026-09-02
+updated: 2026-09-03
 owners: [jcardinal, mhammontree, dfranks, bala, ajean]
 files: []
 related:
@@ -741,6 +741,15 @@ notified**. Pick deliberately:
 > throw `_Exception_Validation` on an empty recipient.) Note `_Exception_Business` **extends**
 > `_Exception_Validation`, so it also satisfies this `instanceof` test.
 
+> **DB errors are their own subclass, and you never throw it.** `_Query` throws
+> `_Exception_Database` (a subclass of plain `Exception`, so it routes identically — 500 +
+> `Logs.Issue` → developer task) for every database failure. The one difference is identity:
+> it is **fingerprinted by `errno + a masked error message`, not by the trace**, because all
+> DB errors share the same few `_Query` frames and the trace scheme over-merged genuinely
+> different bugs into one Issue. Do not "upgrade" a DB failure to another class to change its
+> routing — the class and its message-based grouping are set in `_Query`/`_Error`. Detail:
+> `2.0/apps/_underscore/features/error-reporting-issue-event.md`.
+
 `_Exception_Business`'s constructor is `(issueKey, message, minimumUrgency)`. The `issueKey` is
 the fingerprint, so the Issue's identity survives refactoring; a plain `Exception`'s
 trace-based fingerprint changes when the code moves.
@@ -1024,6 +1033,13 @@ See: 2.0/apps/worker2/features/cross-account-aws-access.md
 
 ## Change history
 
+- 2026-09-03 — Error Handling: added a note under the exception-routing table that DB errors are
+  their own subclass — `_Query` throws `_Exception_Database` (subclass of plain `Exception`, so it
+  routes identically: 500 + `Logs.Issue` → developer task), but it is **fingerprinted by
+  `errno` + a masked error message, not by the trace**, because all DB errors share the same few
+  `_Query` frames and the trace scheme over-merged different bugs into one Issue. Do not "upgrade" a
+  DB failure to another class to change its routing; the class and its message-based grouping live in
+  `_Query`/`_Error`. Detail: `2.0/apps/_underscore/features/error-reporting-issue-event.md`. (jcardinal)
 - 2026-09-02 — Database/SQL: any **cross-process counter or shared rollup needs its own DB alias,
   committed immediately**. `Route.php` holds a transaction open on the default connection until the
   request/job ends, so a counter written there is invisible to every other process (a fleet-wide

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.762",
+  "version": "1.0.764",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",