toga-ai 1.0.484 → 1.0.486

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -7,6 +7,7 @@
7
7
  | [CloudFront Client Setup](features/cloudfront-client-setup.md) | An SSO-gated admin tool at **`/devops/cloudfront-clients`** in the Tools 1.0 app that onboards a client onto **CloudFront + Route 53 across multiple AWS account | tools/_/app/devops/cloudfront.php, tools/mvc/devops/cloudfront-clients/get.php, tools/mvc/devops/cloudfront-clients/post.php, tools/assets/js/cloudfront-clients.js, tools/assets/css/cloudfront-clients.css, tools/_/app/nav.php, tools/_/app/frameworkindex.php, tools/config.production.ini |
8
8
  | [Design Demo Admin](features/design-demo-admin.md) | A self-serve admin UI at **`/design`** in the SSO-protected **Tools** app that lets the design team publish self-contained "Claude Design" HTML exports as **ver | tools/_/app/design/github.php, tools/mvc/design/get.php, tools/mvc/design/post.php, tools/assets/css/design.css, tools/assets/js/design.js, tools/_/app/frameworkindex.php, tools/_/app/nav.php, tools/composer.json |
9
9
  | [Tools — Developers Folder (UUID & Password Generators)](features/developer-tools.md) | The first two tools shipped in the Tools app, both under the **Developers** folder and gated to personas **Development Team** / **TOGa Technology**. | tools/mvc/developers/uuid/get.php, tools/mvc/developers/password/get.php |
10
+ | [/errors Curation Console (Tools → shared Core Logs DB)](features/errors-curation-console.md) | Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as a 1.0 Tools MVC page reading the **shared Core Logs DB** through | tools/mvc/errors/get.php, tools/mvc/errors/post.php, tools/_/app/nav.php, tools/config.production.ini |
10
11
  | [Tools MVC — Routing, CSRF & App_Database Access Patterns](features/mvc-data-access-patterns.md) | The load-bearing 1.0 (`App_`) framework conventions a developer needs when adding a page to the Tools app — URL routing, CSRF, and DB access through `App_Databa | tools/_/app/nav.php, tools/mvc/get.php |
11
12
  | [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php, tools/mvc/get.php |
12
13
  | [Tools SAML SSO Consumer & Persona-Gated Auth (App_Auth)](features/saml-sso-auth.md) | `App_Auth` is the Tools app's authentication layer: it consumes the SAML gateway `?saml=` handoff (see the 2.0 SAML downstream integration contract), establishe | tools/_/app/auth.php, tools/mvc/sso/initiate/get.php, tools/mvc/sso/get.php, tools/mvc/login/get.php, tools/mvc/login/post.php, tools/mvc/logout/get.php, tools/mvc/get.php, tools/config.production.ini, tools/config.local.ini |
@@ -0,0 +1,81 @@
1
+ ---
2
+ title: /errors Curation Console (Tools → shared Core Logs DB)
3
+ framework: "1.0"
4
+ repo: tools
5
+ project: Tools
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-07-30
10
+ owners: ["jcardinal"]
11
+ files:
12
+ - tools/mvc/errors/get.php
13
+ - tools/mvc/errors/post.php
14
+ - tools/_/app/nav.php
15
+ - tools/config.production.ini
16
+ related:
17
+ - ../architecture.md
18
+ - ./mvc-data-access-patterns.md
19
+ - ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
20
+ - ../../../2.0/apps/worker2/features/error-escalation-cron.md
21
+ ---
22
+
23
+ ## Summary
24
+
25
+ Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as
26
+ a 1.0 Tools MVC page reading the **shared Core Logs DB** through a new **`db_toga2logs`**
27
+ alias. It curates; it never notifies.
28
+
29
+ The highest-value view is the list of issues **below the ClickUp threshold** — errors that are
30
+ happening and were deliberately never ticketed. Nobody had that view before.
31
+
32
+ ## How it works
33
+
34
+ - **Curate** subject, description, `minimumUrgency`, assignee, and mute window.
35
+ - **Manage per-client email recipients** (`IssueEmailAddresses`). `clientId = 0` means **all
36
+ clients** and requires an explicit confirmation flag; the list view renders it as
37
+ **"ALL CLIENTS"**.
38
+ - **Merge fingerprints** — repoint an `IssueFingerprints` row at an existing Issue. Without this,
39
+ the fingerprint/Issue split is inert: a refactor that shifts line numbers produces a new Issue
40
+ and the curated one is orphaned.
41
+ - **Any save sets `isManaged`**, which does double duty: it protects the curated text from being
42
+ overwritten by later occurrences, **and** it exempts the Issue from the cron's GC sweep.
43
+
44
+ **This page notifies nobody.** Escalation, ClickUp ticketing, and email are owned entirely by
45
+ `_Worker_Infrastructure_Errors::Escalate`. Adding a notify action here would create a second,
46
+ untracked alerting path.
47
+
48
+ ### Why Tools and not TOGa Hub
49
+
50
+ The 2026-06-12 meeting had proposed TOGa Hub, but Hub is far off. The tables are the same either
51
+ way, so the choice is not load-bearing — a Hub port later reads the same schema.
52
+
53
+ ## Configuration
54
+
55
+ `[database_toga2logs]` was added to `tools/config.production.ini`.
56
+
57
+ **`dbname` is NOT necessarily `Logs`.** 2.0 resolves the Core Logs database name at **runtime**
58
+ from `Core.Databases` id **11** (`worker2` `CORE_LOGS_DATABASE_ID`); `Logs` is only the
59
+ `_underscore` *register alias*. Read the real name from `Core.Databases` before configuring a new
60
+ environment, or the console silently points at a database that may not exist.
61
+
62
+ ## Gotchas / known issues
63
+
64
+ - **This is a 1.0 app reading a 2.0 shared database.** Access is plain `App_Database` via the
65
+ `db_toga2logs` alias, not the `_Model` layer — so none of the 2.0 model protections apply.
66
+ Escape every interpolated value and allowlist every identifier (see the 1.0 back-end standard).
67
+ - **Cross-tenant surface.** The Core Logs DB is shared by every client, so this console can show
68
+ one client's captured context to a viewer looking at another client's problem. It is
69
+ internal-only and persona-gated for that reason — do not expose any part of it to a client.
70
+ - **Curated text is authoritative.** Once `isManaged` is set, later occurrences no longer update
71
+ subject/description, so a stale curated subject stays stale until someone edits it again.
72
+
73
+ ## Change history
74
+
75
+ - 2026-07-30 — Built as part of TRUE-78188: new `/errors` triage/curation console over the
76
+ shared Core Logs DB via a new `db_toga2logs` alias; curation sets `isManaged` (protects text
77
+ and exempts from GC); fingerprint merging; per-client recipient management with an explicit
78
+ confirmation for `clientId = 0` ("ALL CLIENTS"). Chosen over TOGa Hub because Hub is far off
79
+ and the schema is the same either way. Documented that the Core Logs `dbname` resolves at
80
+ runtime from `Core.Databases` id 11 and is not necessarily "Logs". (jcardinal)
81
+ </content>
@@ -15,7 +15,7 @@
15
15
  | [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
16
16
  | [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, worker2/Worker/Infrastructure/Email/Send.php |
17
17
  | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
18
- | [Error Reporting — Issue/Event Aggregation (agreed POST-to-receiver design)](features/error-reporting-issue-event.md) | Platform-wide error-reporting infrastructure for TOGA 2.0, built around a two-table **Issue / Event** aggregation model in the shared **Core Logs DB**. | _underscore/Error.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, dbchanges2/Logs/2026-07-06 - Issue and Event tables.sql |
18
+ | [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Exception/Business.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
19
19
  | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
20
20
  | [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
21
21
  | [isFulfillable Propagation Up the SO↔PO Chain](features/fulfillable-item-propagation.md) | `Items.isFulfillable` is a boolean that gates whether a storefront line's **Qty Fulfilled** cell is actionable. | _underscore/Model/Client/Item.php, _underscore/Model/Compass/Item.php, dbchanges2/Core/2026-07-17 - Items isFulfillable RecordField.sql, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, dbchanges2/Client/2026-07-17 - ItemsisFulfillable.sql |
@@ -1,148 +1,271 @@
1
1
  ---
2
- title: Error Reporting — Issue/Event Aggregation (agreed POST-to-receiver design)
2
+ title: Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation
3
3
  framework: "2.0"
4
4
  repo: _underscore
5
5
  project: _Underscore
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-15
10
- owners: ["dfranks"]
9
+ updated: 2026-07-30
10
+ owners: ["dfranks", "jcardinal"]
11
11
  files:
12
12
  - _underscore/Error.php
13
+ - _underscore/Exception/Business.php
13
14
  - _underscore/Model/Core/Logs/Issue.php
14
15
  - _underscore/Model/Core/Logs/Event.php
15
- - dbchanges2/Logs/2026-07-06 - Issue and Event tables.sql
16
+ - _underscore/Model/Core/Logs/IssueFingerprint.php
17
+ - _underscore/Model/Core/Logs/IssueClickupTask.php
18
+ - _underscore/Model/Core/Logs/IssueEmailAddress.php
19
+ - _underscore/Model/Core/Logs/IssueAreaOwner.php
20
+ - dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql
21
+ - dbchanges2/Core/2026-07-30a - Error escalation cron job.sql
16
22
  related:
17
23
  - ../architecture.md
18
24
  - ./per-client-database-connections.md
25
+ - ./email-send-pipeline.md
26
+ - ../../worker2/features/error-escalation-cron.md
27
+ - ../../../1.0/apps/tools/features/errors-curation-console.md
19
28
  - ../../dbchanges2/architecture.md
20
29
  ---
21
30
 
22
31
  ## Summary
23
32
 
24
- Platform-wide error-reporting infrastructure for TOGA 2.0, built around a two-table
25
- **Issue / Event** aggregation model in the shared **Core Logs DB**. An **Issue** is the
26
- deduplicated record for a distinct error (keyed by a hash of the stack trace) carrying an
27
- aggregate `occurrences` counter and an escalatable `urgency`; an **Event** is one row per
28
- individual occurrence with its own trace and context.
29
-
30
- **The agreed architecture (see below) decouples *reporting* from *persistence*:**
31
- application error handlers **POST** error payloads to a centralized `/errors` receiver
32
- endpoint; a **worker2 cron** consumes them and does all the hashing, upserting, aggregation,
33
- escalation, and ClickUp sync into the Core Logs DB. Handlers do **not** write Issue/Event
34
- rows to the database directly. This is the canonical target design for the whole
35
- error-monitoring epic (TRUE-781xx) — the handler ticket, the dashboard ticket, and the
36
- escalation-cron ticket must all conform to it.
37
-
38
- ## Agreed error-monitoring architecture
39
-
40
- Source: the **"2026-05-21 - Errors, Monitoring and Alerts"** meeting design (approved;
41
- cited via Talos DevCore meeting notes). Pipeline:
42
-
43
- 1. **Report (application side).** An application's error/exception handler serializes the
44
- error (message, stack trace, context) and **POSTs it to a centralized `/errors` receiver
45
- endpoint** (the meeting referenced `webhook.hub.com/error`). The handler's only job is to
46
- transmit it performs **no direct database persistence**.
47
- 2. **Ingest (worker2 cron).** A worker2 cron consumes the received error payloads and:
48
- - builds `issueHash` from the stack trace to identify the distinct error,
49
- - **upserts an Issue** and **inserts an Event** into the **Core** Logs DB,
50
- - aggregates occurrence counts within a time window,
51
- - auto-escalates / de-escalates the Issue `urgency` based on occurrence frequency,
52
- - syncs the Issue to ClickUp.
53
- 3. **Scope.** API 2.0 handlers first; 1.0 is deferred.
54
-
55
- The critical rule: **handlers POST; the cron persists (to Core).** Any approach where the
56
- handler writes Issue/Event rows inline to a database contradicts this design.
57
-
58
- ## Data model (Core Logs DB)
59
-
60
- - **`_underscore/Model/Core/Logs/Issue.php`** — `const DATABASE = _underscore::DB_LOGS;
61
- const TABLE = 'Issues';`. Fields: `id, uuid, dtCreated, dtLastOccurred, urgency, subject,
62
- description, clickupIdentifier, hash, errorMessage, occurrences, trace`. One row per
63
- distinct error (dedup key = `hash`); `occurrences` is the rolling count, `dtLastOccurred`
64
- the most-recent hit, `urgency` a list field, `clickupIdentifier` links to an escalation
65
- ticket. There is **no `type` column**.
66
- - **`_underscore/Model/Core/Logs/Event.php`** — `const DATABASE = _underscore::DB_LOGS;
67
- const TABLE = 'Events';`. Fields: `id, uuid, issueId (FK → Issue), errorMessage, trace,
68
- context, dtCreated`. One row per occurrence.
69
-
70
- The `Issues`/`Events` tables live in the shared **Core Logs** DB, provisioned from
71
- `dbchanges2/Logs/` (the `Logs/` folder targets the framework-level, non-tenant `Logs` DB
72
- with unqualified table names see the dbchanges2 architecture folder→DB mapping). Contrast
73
- with per-client API/error logging in `Logs_<Tenant>` via `Model/Client/Logs/*`
74
- (`DB_CLIENT_LOGS`, `dbchanges2/Logs_Client/`).
75
-
76
- ## Current state of `_underscore/Error.php` (important — reads differently than you'd expect)
77
-
78
- `_underscore/Error.php` registers `exceptionHandler()` via `set_exception_handler`, but in
79
- **`_production` it does NOT persist any Issue/Event rows.** It builds a debug HTML body,
80
- writes it to an S3 error-details file, and (when a Logs register is present) records a
81
- single `_Model_Client_Logs_Error` / `_Model_Core_Logs_Error` row the older flat error log.
82
- There is no Issue/Event aggregation, no `issueHash` upsert, and no ClickUp escalation in the
83
- handler on `_production`.
84
-
85
- **An inline-DB-persist approach exists on an in-flight branch, but it conflicts with the
86
- agreed design and must not be treated as the model to follow:** that approach has the
87
- handler compute `$issueHash = md5(serialize($backtrace))` and directly upsert
88
- `_Model_Client_Logs_Issue` + insert `_Model_Client_Logs_Event`, committing a transaction —
89
- i.e. handler-side direct persistence, and to the **Client** Logs DB rather than **Core**.
90
- That contradicts the approved architecture on two counts: (a) handlers should POST to the
91
- `/errors` receiver, not persist; and (b) aggregation belongs in the Core Logs DB, driven by
92
- the worker2 cron. New work (TRUE-78178 handlers) should implement the POST-to-receiver path,
93
- not extend the inline-persist branch.
33
+ Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in
34
+ the **shared Core Logs DB**. An **Issue** is the deduplicated record for a distinct problem
35
+ (carrying curated text, an urgency, and a quotable reference); an **Event** is one row per
36
+ occurrence. Capture lives in `_underscore/Error.php`; escalation and ClickUp/email
37
+ notification live entirely in the worker2 cron (see
38
+ [error-escalation-cron](../../worker2/features/error-escalation-cron.md)); curation lives in
39
+ the 1.0 Tools `/errors` console.
40
+
41
+ `_Error::captureException(Throwable $e, ?int $clientId): ?string` is the **single capture
42
+ entry point** the exception handler, the shutdown handler, and api2 all funnel through it,
43
+ and it returns the quotable reference. Add new capture paths by calling it, never by
44
+ re-implementing persistence.
45
+
46
+ ## Data model (shared Core Logs DB)
47
+
48
+ Six tables, created by `dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql`:
49
+ `Issues`, `IssueFingerprints`, `Events`, `IssueEmailAddresses`, `IssueClickupTasks`,
50
+ `IssueAreaOwners`.
51
+
52
+ **An Issue is never client-scoped.** An Issue is identified by *code*, and code is global — the
53
+ same bug hitting six clients is **one** Issue. Per-client scoping lives on the occurrence
54
+ (`Events.clientId`) and on notification routing (`IssueEmailAddresses.clientId`), never on the
55
+ Issue itself. Anything that reads like "this client's issue" is a modelling error.
56
+
57
+ `IssueEmailAddresses.clientId = 0` means **all clients**.
58
+
59
+ ## How it works
60
+
61
+ ### Fingerprinting (identity)
62
+
63
+ The fingerprint is the normalized top-N **application** frames (repo-relative `path:line`) plus
64
+ the exception class; framework/vendor plumbing is filtered out. The exception **message is
65
+ deliberately excluded** it carries per-occurrence data (ids, values) and would defeat dedup.
66
+
67
+ This replaced `md5(serialize(debug_backtrace()))`, which serialized **call arguments** and so
68
+ produced a different hash on every request — dedup never actually worked.
69
+
70
+ `IssueFingerprints` is many-to-one against `Issues`, so a refactor that shifts line numbers
71
+ cannot orphan a curated Issue: an admin repoints the new fingerprint at the existing Issue
72
+ from the Tools console to merge them.
73
+
74
+ ### `_Exception_Business` identity immune to refactoring
75
+
76
+ `_underscore/Exception/Business.php` carries a stable `issueKey` plus a `minimumUrgency` floor.
77
+ When an `issueKey` is declared, **the key is the fingerprint**, so the Issue's identity does not
78
+ depend on line numbers at all. It extends `Exception` (not `_Exception`), matching
79
+ `_Exception_Validation`.
80
+
81
+ It **deliberately never throws on a bad argument** it normalizes and `error_log()`s instead.
82
+ Replacing the caller's business error with an argument error destroys the real signal.
83
+
84
+ ### The quotable reference
85
+
86
+ `Issues.reference` is a Sentry-style base-26 encoding of `Issues.id`: 1 → `1A`, 26 → `1Z`,
87
+ 27 → `2A`. Because `AUTO_INCREMENT` never reuses ids, gaps are expected and permanent — a
88
+ reference can never be recycled onto a different problem, so it is safe to quote to a customer.
89
+
90
+ The reference is written by PHP in a second statement (see MySQL constraints below), which is
91
+ why the column is nullable even though it is always populated.
92
+
93
+ ### Shutdown handler
94
+
95
+ `register_shutdown_function` is registered with a **pre-allocated memory reserve**. Before this,
96
+ `_underscore` had **no shutdown handler at all**, so fatals, OOM, and timeouts were completely
97
+ uncaptured.
98
+
99
+ ## MySQL constraints that shaped the schema (verified on 8.0.46)
100
+
101
+ - **A stored generated column blocks `ON DELETE CASCADE`.** A foreign key with
102
+ `ON DELETE CASCADE` is rejected with error **1215** when the constrained column is the base
103
+ column of a **STORED** generated column. `IssueClickupTasks.issueId` is the base of
104
+ `openEpisodeIssueId`, so that generated column must be **VIRTUAL**. VIRTUAL has no such
105
+ restriction and still supports a UNIQUE secondary index. Proved both ways locally.
106
+ - **Error 3109: a generated column cannot refer to an auto-increment column.** This is why
107
+ `Issues.reference` cannot be generated and must be written by PHP.
108
+
109
+ ## One-open-episode guard (UNIQUE on a generated column)
110
+
111
+ `INSERT ... SELECT ... WHERE NOT EXISTS` does **not** prevent a double insert under
112
+ REPEATABLE READ: two overlapping cron runs each evaluate the `NOT EXISTS` against their own
113
+ snapshot and both pass. The guard is instead a VIRTUAL generated column
114
+ `openEpisodeIssueId = issueId while dtResolved IS NULL, else NULL` with a **UNIQUE** key —
115
+ MySQL permits duplicate NULLs, so many resolved episodes coexist while at most one is open.
116
+
117
+ The code **claims the episode before calling ClickUp**, so a losing run fails (error 1062)
118
+ before it can create a duplicate external task; the claim is **released on API failure** so a
119
+ stuck placeholder row cannot permanently block the Issue. Verified: second open episode → 1062;
120
+ resolve-then-reopen allowed; many resolved episodes allowed; cascade delete works.
121
+
122
+ ## Sentry coexistence (removal is a separate ticket)
123
+
124
+ Sentry stays running **in parallel**. Shipping an unproven replacement for the only working
125
+ error reporting with no overlap means going blind. Three findings make removal its own ticket:
126
+
127
+ 1. `_Worker_Sentry::Webhook` is an **existing Sentry→ClickUp pipeline writing into the same
128
+ list `901110669877`** this work replaces — retiring it also means turning off the Sentry-side
129
+ issue alerts.
130
+ 2. `\Sentry\configureScope()` calls `http_response_code(500)` as a blanket default, and api2
131
+ documents working *around* it. Remove Sentry without fixing every early-exit branch and
132
+ validation 400s ship as HTTP 200 — which changes whether apiproxy retries across regions.
133
+ 3. `\Sentry\init()` registers PHP error handlers. `_Error::errorHandler()` also throws on
134
+ warnings, but that promotion must be verified **empirically** — if it does not survive,
135
+ code that currently 500s would silently continue with a null value, which is worse.
136
+
137
+ Also: this pipeline is **2.0-only**. Sentry must remain for the 1.0 apps (desk1, retail1,
138
+ forecast1, worker1, view1, commerce1) until a `library` / `App_Error` port lands, so the
139
+ Sentry-removal ticket cannot complete until the 1.0 port does.
94
140
 
95
141
  ## Gotchas / known issues
96
142
 
97
- - **Do not assume error persistence "already works."** On `_production` the handler writes
98
- only an S3 debug body and a flat `*_Logs_Error` row no Issue/Event aggregation exists in
99
- production yet. The Issue/Event model is the *target*, delivered by the POST→cron pipeline.
100
- - **`occurrences` must be incremented, not just seeded** — a naive upsert that sets
101
- `occurrences = 0` only at creation never advances the aggregate counter. The cron must
102
- increment on every occurrence.
103
- - **`Event.trace` must be populated per occurrence** the per-Event `trace` column is easy
104
- to leave empty; record it on every Event.
105
- - **Persistence must be conditional on the Core-Logs register.** If the Core Logs register
106
- (`\_Database::$_registers[_underscore::DB_LOGS]`) isn't set, an Issue/Event write must be
107
- skipped rather than fatal but this belongs in the cron, not the handler.
108
- - **Client vs Core Logs DB mix-up.** The Issue/Event tables belong in the **Core** Logs DB
109
- (`DB_LOGS`, `Model/Core/Logs/*`). Writing them to a per-client Logs DB
110
- (`DB_CLIENT_LOGS`, `_Model_Client_Logs_*`) is the wrong target and does not match the
111
- agreed design.
112
- - **Shared-schema migration collision.** Because `Issues`/`Events` live in the *shared* Core
113
- Logs DB, two dbchanges2 `Logs/` migrations that both `CREATE TABLE Issues`/`Events` will
114
- collide ("table already exists"). Only **one** migration may create the shared tables;
115
- later migrations only `ALTER`.
116
- - **Cross-tenant secret exposure when captured context lands in the shared Core Logs DB.**
117
- Moving the Issue/Event sink from a per-client Logs DB to the *shared* Core Logs DB widens
118
- the blast radius of anything captured in `context`. A pre-migration `print_r($GLOBALS, true)`
119
- dump (session tokens, auth headers, credentials from superglobals) that was acceptable while
120
- per-tenant becomes a cross-tenant leak once every tenant can read the shared table.
121
- `_underscore/Error.php::redactSensitiveContext()` recursively redacts superglobal keys
122
- matching sensitive fragments (`password/secret/token/authorization/auth/apikey/credential/`
123
- `cookie/session/jwt/private`) **before** persisting. Rule: sanitize captured context at the
124
- point you move a log sink from per-client to shared.
125
- - **Strict-mysqli varchar overflow silently drops the Issue/Event row.** `_Error::initialize()`
126
- sets `mysqli_report(MYSQLI_REPORT_STRICT)`, so assigning an over-255-char value to a
127
- `varchar(255)` column (`errorMessage`, `subject`) **throws** on `->save()`. Because the write
128
- happens inside the error handler's `catch(Throwable)`, the throw is swallowed and the row is
129
- silently dropped the error pipeline fails precisely when a long/detailed message matters
130
- most. Truncate any raw exception message / user string to the column width before `save()`
131
- on any `_underscore` model write under strict mysqli.
143
+ - **Read-your-writes: reads go to the READ host and cannot see an uncommitted write.** A
144
+ second `$issue->save()` re-reads the row via `_Model::initialize()`, and that SELECT goes to
145
+ the read host a separate connection that cannot see the still-uncommitted INSERT on the
146
+ write connection. Symptom: `Issues` rows with `reference` NULL, **zero** Events, **zero**
147
+ IssueFingerprints, no reference in the API response, and the logged cause *"Error during
148
+ Model build for primary key id '1' … Exactly 1 row was expected to be returned but 0 were."*
149
+ Capture aborted half-written. Fix: force reads to the writer for the whole capture block via
150
+ `_Database::getIsReadHostEnabled()` / `setIsReadHostEnabled(false)` with a `finally` restore
151
+ the same TRUE-80448 pattern `_Email::send()` uses.
152
+ - **Never rebuild a model inside an error handler.** `_Model`'s build **throws** when the row
153
+ is missing, and nothing inside an error handler should throw over a value derivable from the
154
+ primary key. `new _Model_Core_Logs_Issue($id)` rebuilds were replaced with a light SELECT
155
+ helper, and `reference` is written with a plain UPDATE rather than a second `save()`.
156
+ - **`Issues.issueKey` is UNIQUE handle the first-occurrence race.** Two concurrent first
157
+ occurrences of the same declared business condition raced on the INSERT and the loser's
158
+ occurrence was **silently dropped**. The loser must adopt the winner's Issue.
159
+ - **`LAST_INSERT_ID(totalOccurrences + 1)` is the atomic event-number allocator** (a `_Model`
160
+ dirty-check read-modify-write loses increments). But `LAST_INSERT_ID` is only *assigned* when
161
+ the UPDATE matches a row — proven locally: against a missing row, `affected = 0` while
162
+ `LAST_INSERT_ID()` still returned the **stale** previous value `2`. Verify affected rows
163
+ before trusting it, or the event number silently collides with `UNIQUE(issueId, eventNumber)`.
164
+ - **A new Issue must be seeded at its declared `minimumUrgency`**, not at LOW — otherwise the
165
+ floor is ignored until the next cron run.
166
+ - **MySQL `UUID()` is forbidden for inserted records** by the 2.0 standard (time-based, not
167
+ random) — use `_String::generateUuid()` in PHP. It remains acceptable inside `.sql`
168
+ migrations, where no PHP exists (65 existing migrations do this).
169
+ - **`VALUES()` in `ON DUPLICATE KEY UPDATE` is deprecated in MySQL 8.0.20+** — use the row-alias
170
+ form. **Evaluation order matters:** in `IssueAreaOwners`, `consecutiveCount` is assigned
171
+ **before** `clickupUserId` so the `IF()` compares against the *old* owner. Reversing them
172
+ makes the condition always true and the counter never resets.
173
+ - **`Events.context` is an allowlist, never a dump.** It replaced `print_r($GLOBALS)` filtered
174
+ by a key-name blocklist that blocklist recursed **arrays only** and walked straight past
175
+ objects whose *properties* hold DB credentials, into a table shared by every client.
176
+ `requestUri` and `referer` have query strings stripped (a `?token=abc` was `[REDACTED]` under
177
+ `get` and stored in clear in `requestUri`). The sensitive-fragment list covers **PII**, not
178
+ just credentials. Persisted traces are rebuilt **argument-free**.
179
+ - **Keep the debug `print_r($GLOBALS)` dump inside the `isDebugMode()` branch.** It was being
180
+ built on every exception — including the OOM path the memory reserve exists to protect.
181
+ - **Strict-mysqli varchar overflow silently drops the row.** `_Error::initialize()` sets
182
+ `mysqli_report(MYSQLI_REPORT_STRICT)`, so assigning an over-255-char value to a
183
+ `varchar(255)` (`errorMessage`, `subject`) **throws** on `->save()`. Inside the handler's
184
+ `catch(Throwable)` the throw is swallowed and the row is silently dropped — precisely when a
185
+ long message matters most. Truncate to column width before `save()`.
186
+ - **Shared-schema migration collision.** Because these tables live in the *shared* Core Logs
187
+ DB, only **one** migration may `CREATE` them; later migrations only `ALTER`.
188
+ - **Pre-existing committed secrets in this area (not fixed — out of scope, report only).**
189
+ A ClickUp API token at `_underscore/Component/Api/Clickup/Clickup.php:5`; SES SMTP
190
+ credentials at `_underscore/Email.php:11-12`; a named individual's Sentry personal access
191
+ token at `worker2/Worker/Sentry.php:4`. Also unfixed: api2 writes `apache_request_headers()`
192
+ verbatim into `Logs.Api` (bearer tokens in plaintext) and ships `$_SERVER` to Sentry. Treat
193
+ all of these as compromised whenever this area is touched again.
194
+
195
+ ## Decisions superseding the 2026-05-21 design
196
+
197
+ ### The POST-to-a-central-`/errors`-receiver hop is DROPPED (jcardinal, 2026-07-30)
198
+
199
+ The **"2026-05-21 – Errors, Monitoring and Alerts"** meeting had approved a *POST-to-receiver*
200
+ pipeline: application handlers POST a serialized error to a centralized `/errors` receiver
201
+ endpoint (`webhook.hub.com/error`) and perform **no** database persistence; a worker2 cron
202
+ consumes the payloads and does all hashing, upserting, aggregation, escalation, and ClickUp
203
+ sync into the Core Logs DB.
204
+
205
+ **That hop is dropped.** Error capture writes **directly** to the Core Logs DB from
206
+ `_Error::captureException()`. Inline persistence is the model to follow; do not reintroduce an
207
+ HTTP hop.
208
+
209
+ Rationale: an error is frequently caused **by something being down**, so routing error capture
210
+ through an HTTP call to api2 would lose exactly the errors you most need. Inline capture also
211
+ buys the synchronous `reference` the api2 response envelope needs, which a fire-and-forget POST
212
+ cannot. The agreed design still holds on the *sink* (shared Core Logs DB, not per-client) and on
213
+ *who escalates* (the worker2 cron owns escalation and notification only).
214
+
215
+ The cost of a direct write is that **each app must have the Core Logs connection registered**.
216
+ The capture path already guards this with
217
+ `isset(_Database::$_registers[_underscore::DB_LOGS])` — it **skips** capture rather than
218
+ fatalling when the register is absent.
219
+
220
+ ### Why api2 is in scope at all
221
+
222
+ api2 catches `Throwable` in its own front controller and therefore **never reaches
223
+ `_Error::exceptionHandler`**. `captureException()` had to become a shared entry point that
224
+ api2's catch blocks call directly — otherwise the entire inbound-API 500 surface would go
225
+ unrecorded.
226
+
227
+ ### `Issues.type` enum and `BusinessIssues_Users` were proposed and not built (jcardinal, 2026-07-30)
228
+
229
+ The **2026-06-12 "Monitoring Business Exceptions Sync"** meeting proposed an `Issues.type` enum
230
+ (`'technical'`/`'business'`) and a `BusinessIssues_Users` bridge table (clientId +
231
+ clientUserId). **Neither was built.** As built instead:
232
+
233
+ - **No `type` column.** `issueKey` (non-null = a developer declared it a business condition in
234
+ code) plus `isManaged` (a human curated it) carry the same information while also being
235
+ useful on their own.
236
+ - **`IssueEmailAddresses` uses varchar email addresses scoped by `clientId`** (`0` = all
237
+ clients) rather than client-user ids, so recipients who have **no login** can still be
238
+ addressed.
132
239
 
133
240
  ## Change history
134
241
 
242
+ - 2026-07-30 — Reworked TRUE-78188: six-table schema in the shared Core Logs DB (Issue is
243
+ global by code; client scope lives on Events/IssueEmailAddresses); `_Error::captureException()`
244
+ as the single capture entry point; trace-frame fingerprinting with the message excluded
245
+ (replacing `md5(serialize(debug_backtrace()))`, which never deduped); many fingerprints → one
246
+ Issue for refactor safety; `_Exception_Business` with a stable `issueKey` + `minimumUrgency`;
247
+ base-26 quotable reference; a shutdown handler with a memory reserve (there was none).
248
+ Documented two MySQL constraints (1215 stored-generated-column vs. ON DELETE CASCADE; 3109
249
+ generated column cannot refer to auto-increment), the UNIQUE-on-VIRTUAL-generated-column
250
+ open-episode guard (`WHERE NOT EXISTS` is not safe under REPEATABLE READ), the read-host
251
+ read-your-writes failure that broke capture entirely, the issueKey first-occurrence race, the
252
+ `LAST_INSERT_ID` stale-value trap, the `Events.context` allowlist replacing the
253
+ `print_r($GLOBALS)` blocklist, and the Sentry-parallel decision. (jcardinal)
254
+ - 2026-07-30 — **Decided: the 2026-05-21 POST-to-a-central-`/errors`-receiver hop is dropped.**
255
+ Capture writes directly from `_Error::captureException()`, guarded by
256
+ `isset(_Database::$_registers[_underscore::DB_LOGS])`; an HTTP hop would lose errors caused by
257
+ something being down. Recorded that api2 is in scope because its front controller catches
258
+ `Throwable` and never reaches `_Error::exceptionHandler`. Also recorded that the 2026-06-12
259
+ meeting's proposed `Issues.type` enum and `BusinessIssues_Users` bridge table were **not
260
+ built** — `issueKey` + `isManaged` and `clientId`-scoped varchar `IssueEmailAddresses` are the
261
+ as-built equivalents. Supersedes the 2026-07-08 correction. (jcardinal)
135
262
  - 2026-07-15 — Added two framework gotchas from aligning the `_underscore` side of the
136
- Issue/Event pipeline to the shared Core Logs DB: (1) cross-tenant secret exposure when
137
- captured context lands in the shared DB, mitigated by `redactSensitiveContext()` in
138
- Error.php; (2) strict-mysqli varchar overflow throwing inside the handler's `catch(Throwable)`
139
- silently drops the Issue/Event row — truncate to column width before `save()`. (dfranks)
263
+ Issue/Event pipeline to the shared Core Logs DB: cross-tenant secret exposure in captured
264
+ context, and strict-mysqli varchar overflow throwing inside `catch(Throwable)`. (dfranks)
140
265
  - 2026-07-08 — Corrected KB drift: documented the approved 2026-05-21 error-monitoring
141
- architecture (handlers POST to a centralized `/errors` receiver; a worker2 cron builds
142
- issueHash, upserts Issues + inserts Events into the Core Logs DB, aggregates/escalates,
143
- syncs ClickUp — handlers do not persist). Clarified that `_production` Error.php performs
144
- no Issue/Event persistence (only an S3 debug body + flat `*_Logs_Error` row), and that the
145
- inline-DB-persist approach (handler-side upsert to the *Client* Logs DB) conflicts with the
146
- agreed design and is not the pattern to follow. (dfranks)
266
+ architecture (handlers POST to a centralized `/errors` receiver; a worker2 cron aggregates
267
+ into the Core Logs DB). **Superseded 2026-07-30 the receiver hop was dropped.** (dfranks)
147
268
  - 2026-07-06 — Prior doc described the exceptionHandler as directly persisting Issue/Event
148
- rows to the Core Logs DB with dedup; superseded by the 2026-07-08 correction above. (dfranks)
269
+ rows with dedup; superseded by the 2026-07-08 correction. (dfranks)
270
+ </content>
271
+ </invoke>
@@ -17,7 +17,7 @@
17
17
  | [TableView row-filtering via apiWhereClause (options.where grammar, end to end)](features/tableview-apiwhereclause-row-filtering.md) | `TableViews.apiWhereClause` (TEXT, nullable) is the sanctioned, code-free way to restrict or exclude rows from a 2.0 table view. | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TableView.php, toga2-supply/src/api/toga.ts |
18
18
  | [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql |
19
19
  | [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
20
- | [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TrackingNumber.php |
20
+ | [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, api2/Component/Api/V2/Response/Oauth/Oauth.php, api2/Controller/Index.php, toga2-supply/src/globalTypes.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, _underscore/Model/Client/TrackingNumber.php |
21
21
  | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
22
22
  | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
23
23
  | [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
@@ -6,12 +6,18 @@ project: API
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-29
9
+ updated: 2026-07-30
10
10
  owners: [mhammontree, tcox, jcardinal]
11
11
  files:
12
12
  - api2/Component/Api/V2/V2.php
13
+ - api2/Component/Api/V2/Response/Response.php
14
+ - api2/Component/Api/V2/Response/Oauth/Oauth.php
15
+ - api2/Controller/Index.php
16
+ - toga2-supply/src/globalTypes.ts
17
+ - toga2-supply/src/pages/Orders/api/OrdersApi.ts
13
18
  - _underscore/Model/Client/TrackingNumber.php
14
19
  related:
20
+ - ../../_underscore/features/error-reporting-issue-event.md
15
21
  - ../../_underscore/features/acl-permission-chain.md
16
22
  - scripted-api-post-body-args.md
17
23
  - v2-rest-query-contract.md
@@ -91,6 +97,61 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
91
97
  DB fully migrated. On a local dev box with PHP opcache disabled a branch switch takes effect
92
98
  immediately (files are read fresh per request) — no Apache restart needed.
93
99
 
100
+ ## BREAKING (2026-07-30): `error` is now an object, not a bare string
101
+
102
+ The V2 envelope's `error` changed from a bare string (`"EO-1"`) to an **object**:
103
+
104
+ ```json
105
+ "error": { "code": "EO-1", "id": "1C-1" }
106
+ ```
107
+
108
+ - **`code`** — the same short message code documented above.
109
+ - **`id`** — the quotable **Core Logs reference** for a *captured exception* (see
110
+ [error reporting](../../_underscore/features/error-reporting-issue-event.md)), and **`null`**
111
+ for a graceful rejection. `id` is **always present** so the shape is predictable — consumers
112
+ must not branch on its absence.
113
+
114
+ The object is assembled from **private** properties (`json_encode` emits public only) via
115
+ `setErrorReference()` + `refreshError()`, so it works whether the reference arrives **before or
116
+ after** the code — the controller sets the reference first. All six orderings were verified.
117
+
118
+ **The OAuth response subclass deliberately keeps the flat string form**
119
+ (`const USES_STRUCTURED_ERROR = false`). RFC 6749 §5.2 defines `error` as a single ASCII string,
120
+ and both classes share `addDefinedMessage()`, so emitting the object there would break every
121
+ conforming OAuth client library. Do not "unify" this.
122
+
123
+ **Also fixed here (CRITICAL): api2 returned `$e->getTrace()` to external callers on every 500.**
124
+ Unlike `getTraceAsString()`, `getTrace()` returns raw frame **arguments** — and the DB-bootstrap
125
+ catch has `_Database::register()` on the stack, so the **database username and password were
126
+ reachable by anyone who could force a 500**. `message` and `trace` are now gated behind
127
+ `_Environment::isDebugMode()`.
128
+
129
+ ### Consumer audit
130
+
131
+ - **`library/app/api/toga2.php`** (the 1.0↔2.0 bridge) branches on `isSuccess` only, never
132
+ `error` — safe.
133
+ - **`toga2-supply`** — `ApiResponse<T>.error` is now `ApiError | null`; 30 envelope reads in
134
+ `OrdersApi.ts` became `.error?.code` (all 21 declarations there are
135
+ `await togaApiRequest(...)`, verified).
136
+ - **`toga-blox-npm` is not affected** — no `isSuccess` anywhere; it reads axios'
137
+ `response.data.data.…`.
138
+ - **Outstanding:** the external `API-DOCUMENTATION.md` lives outside the checkout and still
139
+ documents `error` as a string.
140
+
141
+ ### Gotcha — do NOT regex-sweep `.error` → `.error?.code`
142
+
143
+ Only direct `togaApiRequest()` results are envelopes. These look identical and are **local
144
+ `{success, error}` shapes whose `error` is a plain string** — adding `?.code` breaks the failure
145
+ guard and swaps the real backend message for a generic fallback:
146
+
147
+ - `payload` in `UpdateShipmentApi.ts` (`response?.data?.itemFulfillments?.generateReturnLabel`)
148
+ - `shipmentResponse` from `fulfillShipment()`
149
+ - `response` in `EditShipmentForm.tsx` from `updateShipment()` / `saveShipment()`
150
+ - `messages[0].identifiers.error` — that is the message-identifiers map, not the envelope.
151
+
152
+ Verification gap: `toga2-supply` had no `node_modules`, so `tsc` could not be run.
153
+ **`npm run build` is the definitive check** — the type change will surface anything missed.
154
+
94
155
  ## Related recipes
95
156
 
96
157
  - **Add a writable field to a V2 record (3-file migration)** and **expose a scripted API**
@@ -103,6 +164,17 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
103
164
 
104
165
  ## Change history
105
166
 
167
+ - 2026-07-30 — **BREAKING:** envelope `error` changed from a bare string to
168
+ `{ code, id }`, where `id` is the quotable Core Logs reference for a captured exception and
169
+ `null` for a graceful rejection (always present, for a predictable shape); assembled from
170
+ private properties via `setErrorReference()`/`refreshError()` so code and reference can arrive
171
+ in any order. The OAuth subclass keeps the flat string form
172
+ (`USES_STRUCTURED_ERROR = false`) per RFC 6749 §5.2. Fixed a CRITICAL disclosure: api2
173
+ returned `$e->getTrace()` (raw frame **arguments**, including `_Database::register()`'s DB
174
+ username/password) to external callers on every 500 — `message`/`trace` are now gated behind
175
+ `_Environment::isDebugMode()`. Recorded the consumer audit (library bridge and toga-blox-npm
176
+ unaffected; toga2-supply migrated) and the regex-sweep gotcha for look-alike local
177
+ `{success, error}` shapes. (jcardinal)
106
178
  - 2026-07-29 — Added **EV-13** (`recordsPerPage` over the 10000 max) and **diagnosis note 6**:
107
179
  what a denial looks like from the client side — `EZ-1` = record-level (`data.<route>` null,
108
180
  and a bare `allowRead=1` with no logic group still returns it); `EZ-2` = field-level,
@@ -18,6 +18,7 @@
18
18
  | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
19
19
  | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
20
20
  | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
21
+ | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Config/production.ini, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
21
22
  | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
22
23
  | [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
23
24
  | [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
@@ -0,0 +1,157 @@
1
+ ---
2
+ title: Error Escalation Cron (Errors::Escalate → ClickUp / email)
3
+ framework: "2.0"
4
+ repo: worker2
5
+ project: Worker
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-07-30
10
+ owners: ["jcardinal"]
11
+ files:
12
+ - worker2/Worker/Infrastructure/Errors.php
13
+ - worker2/Worker/Clickup/ErrorTask.php
14
+ - worker2/Worker/Clickup.php
15
+ - worker2/Config/production.ini
16
+ - dbchanges2/Core/2026-07-30a - Error escalation cron job.sql
17
+ related:
18
+ - ../../_underscore/features/error-reporting-issue-event.md
19
+ - ../../../1.0/apps/tools/features/errors-curation-console.md
20
+ - ./creating-worker-actions.md
21
+ - ./clickup-project-routing.md
22
+ ---
23
+
24
+ ## Summary
25
+
26
+ `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of
27
+ **escalation, de-escalation, ClickUp ticketing, reminders, business email, owner learning, and
28
+ GC** for the Issue/Event error-reporting pipeline. Capture is `_underscore`'s job (see
29
+ [error-reporting-issue-event](../../_underscore/features/error-reporting-issue-event.md)); the
30
+ Tools `/errors` console curates but **notifies nobody**. All outbound notification funnels
31
+ through this one action, so there is exactly one place to mute or debug alerting.
32
+
33
+ ## How it works
34
+
35
+ ### Urgency = max of two independent axes, plus a floor
36
+
37
+ - **Volume** — occurrences in a 10-minute window: **20 / 50 / 100**.
38
+ - **Neglect** — hours an *unacknowledged* ClickUp task has sat: **1 / 4 / 24**.
39
+ - **Floor** — the Issue's declared `minimumUrgency` applies under *both* axes.
40
+
41
+ The neglect axis exists because volume alone cannot see a rare-but-serious error, nor a serious
42
+ error that everyone is ignoring.
43
+
44
+ ### Fast up, slow down
45
+
46
+ Escalation applies **the same run** a band is crossed. De-escalation requires the count to sit
47
+ under a **lower release band (12 / 35 / 70)** for **3 consecutive windows**. Without that gap,
48
+ an issue hovering near a threshold flaps every minute and burns a ClickUp write each time.
49
+
50
+ ### Task-creation gate
51
+
52
+ A ClickUp task is created only when urgency ≥ NORMAL, **or** a floor is set, **or** the Issue is
53
+ curated, **or** it is still rare. Creating a task for every distinct error on first sight is what
54
+ buries a team.
55
+
56
+ ### Reminders
57
+
58
+ Backed off **30m → 1h → 4h → daily**. Never sent on de-escalation. Suppressed entirely while
59
+ `dtMutedUntil` is in the future.
60
+
61
+ ### Sprint promotion
62
+
63
+ Uses the ClickUp **v2 multi-list** "Add Task To List" endpoint
64
+ (`POST /list/{id}/task/{taskId}`) so the errors list stays the task's **permanent home**. The v3
65
+ `home_list` move endpoint was rejected: v3 is not reliably deployed, it needs a workspace id
66
+ that exists nowhere in the codebase, and it requires `status_mappings` maintenance.
67
+
68
+ The target sprint list is resolved from `[clickup] sprint_folder_id` because `Team.Sprints`
69
+ carries no ClickUp list id — `GET /folder/{id}/list`, then the list whose `start_date`/`due_date`
70
+ range contains today.
71
+
72
+ ### Recurrence (episodes)
73
+
74
+ One `IssueClickupTasks` row per ClickUp **episode**. Closing the task fires a webhook that stamps
75
+ `dtResolved` and increments `recurrenceCount`. A later occurrence opens a **new** task linked to
76
+ its predecessors via `POST /task/{id}/link/{links_to}` — a **link, not a dependency**; a
77
+ dependency would *block* the task. A cooldown prevents recreating a task 60 seconds after
78
+ someone closed it while the error was still firing.
79
+
80
+ ### Acknowledgement
81
+
82
+ ClickUp is the acknowledge/resolve system — developers already live there, and a second inbox in
83
+ Tools would just be ignored. **Any** status change on an error task counts as acknowledgement,
84
+ which is safe only because the automation never sets status itself.
85
+
86
+ ### Area-level owner learning (observation only)
87
+
88
+ `IssueAreaOwners` learns an owner after **5 in a row** and ships **observation-only** — a hint in
89
+ the task body, never an auto-assignment. Misassignment is expensive: a few wrong tickets and
90
+ developers stop trusting the queue. Learning is per **area**, not per issue, because one issue
91
+ rarely recurs enough to learn from before it is fixed.
92
+
93
+ ### Garbage collection (split, deliberately)
94
+
95
+ - **Events** purge at **7 days**.
96
+ - **Issues** purge only when nothing curated them: not `isManaged`, no `issueKey`, no floor, no
97
+ recipients, no ClickUp history, no events.
98
+
99
+ The original blanket 30-day delete would have **cascaded a configured business issue's email
100
+ recipients away** after a quiet month.
101
+
102
+ ## Configuration
103
+
104
+ - **`[_underscore] error_escalation_enabled`** — added to **all 8** worker2 configs: `1` in
105
+ `production.ini` **only**, `0` elsewhere. It is the master switch for **all outbound
106
+ notification**; errors are still **captured** regardless, so non-production records everything
107
+ and notifies nobody.
108
+ - Gated on a config key rather than string-matching `_Environment::$name` because
109
+ `_Environment` has **no `isProduction()`** — only a raw name string and an ad-hoc
110
+ `substr($name, 0, 4) == 'dev-'` idiom. Guessing env names would silently spam, or silently
111
+ stay quiet, in the wrong place.
112
+ - Read via `_Config::_underscore(key, false)`. **The second falsy argument matters:** it makes
113
+ `_Config` return `false` for a missing key/group **instead of throwing**, and an uncaught
114
+ throw would fail the whole cron.
115
+ - **`[clickup] sprint_folder_id`** — added to `worker2/Config/production.ini`, left **empty with
116
+ instructions**. Note `[clickup]` exists in only **2 of the 8** worker2 configs.
117
+
118
+ ## Gotchas / known issues
119
+
120
+ - **The working-set query must be a UNION, not an OR.** It is a UNION of three separately
121
+ indexable branches. MySQL will **not** `index_merge` across an `OR` when one branch is a
122
+ correlated `EXISTS`, so the OR form **full-scans `Issues` every 60 seconds, forever**.
123
+ - **`INSERT ... SELECT ... WHERE NOT EXISTS` is not a concurrency guard** under REPEATABLE READ.
124
+ The open-episode guard is a UNIQUE key on a VIRTUAL generated column, and the code **claims
125
+ the episode before calling ClickUp** so a losing run fails before creating a duplicate
126
+ external task — see the `_underscore` doc for the full mechanics.
127
+ - **Business email must be scoped per client.** Recipients were resolved from the **most recent
128
+ event's** client, so with two clients firing in the same window **client B's business users
129
+ received email about client A's data**. It now sends one email **per distinct client that
130
+ actually fired in the window**.
131
+ - **Business email bodies/subjects must not carry `errorMessage` or `trace`.** Both are seeded
132
+ once from whichever client created the Issue and are never re-scoped. An uncurated business
133
+ issue gets a **neutral** subject rather than another client's raw error text.
134
+ - **`IssueEmailAddresses.clientId = 0` means "all clients"** and requires an explicit
135
+ confirmation flag in the Tools console; list views render it as **"ALL CLIENTS"**.
136
+ - **The ClickUp webhook has no HMAC signature verification** (pre-existing, unfixed). A forged
137
+ POST can mark issues acknowledged — freezing the neglect axis — or resolve episodes. That is
138
+ alert suppression against the error-reporting system itself. Fix before relying on this
139
+ pipeline as the only alerting path.
140
+ - **`_Worker_Sentry::Webhook` writes into the same ClickUp list (`901110669877`)** this cron
141
+ targets. Both pipelines run in parallel by design until the Sentry-removal ticket lands, so
142
+ expect duplicate-looking tasks from two sources.
143
+
144
+ ## Change history
145
+
146
+ - 2026-07-30 — Built as part of TRUE-78188: action renamed `SyncWithClickup` → `Escalate`; new
147
+ `Worker/Clickup/ErrorTask.php`. Two-axis urgency (volume 20/50/100 + neglect 1/4/24) with a
148
+ `minimumUrgency` floor; fast-up/slow-down with release bands 12/35/70 over 3 windows; a
149
+ task-creation gate; 30m→1h→4h→daily reminders honouring `dtMutedUntil`; sprint promotion via
150
+ the v2 multi-list endpoint (v3 `home_list` rejected); episode recurrence via task *links* not
151
+ dependencies; a UNION (not OR) working-set query; and split GC (Events 7d, Issues only when
152
+ uncurated). Fixed from review: per-client business email scoping (client B was receiving
153
+ client A's data), neutral subjects with no errorMessage/trace, seeding new issues at their
154
+ declared floor, `_String::generateUuid()` over MySQL `UUID()`, and the row-alias form of
155
+ `ON DUPLICATE KEY UPDATE`. Added `error_escalation_enabled` (all 8 configs) and
156
+ `[clickup] sprint_folder_id`. (jcardinal)
157
+ </content>
@@ -393,6 +393,9 @@ Signed integers can reference positive and negative numbers, but unsigned can on
393
393
  1. **Use the `_Model` layer.** The metadata-driven ORM builds and escapes queries for you — prefer it over hand-written SQL whenever possible.
394
394
  2. **When you must write SQL by hand, escape every interpolated value with `_Database::escape()`** before placing it in the query string.
395
395
  * `_Database::escape()` adds MySQL escape characters (`\`, `'`, `"`, newlines, etc.) and is the correct helper for query interpolation. Do **not** confuse it with `_Database::protect()`, which *strips* `% \ / * " '` and the literal ` or` — that is lossy filtering, not escaping, and will corrupt data you intend to store intact.
396
+ * **Prepared statements are not available.** `_Query` accepts only a raw SQL string, and `_Database_Driver` declares no `prepare`/`bind` (verified 2026-07-30 — neither the Mysql nor the Pgsql driver implements one). Escaping and casting *are* the parameterization mechanism in 2.0; do not go looking for a bind API, and do not trust any guidance that names a `_Db` class — there isn't one.
397
+ * **Escaping protects values, not identifiers.** Table names, column names, sort directions, and `LIMIT` expressions must come from a **hardcoded allowlist** — never from a request, and never merely escaped. `ORDER BY " . _Database::escape($_GET['sort'])` is still injectable.
398
+ * **Validate enum values in PHP.** `FIELD_LIST` performs no enum validation and converts any falsy value to SQL `NULL`; MySQL coerces rather than rejects an unknown member. Check against the allowed set before the value reaches the query — a typo'd urgency becomes `NULL`, not an error.
396
399
 
397
400
  ```php
398
401
  // Bad — raw interpolation
@@ -14,12 +14,12 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
14
14
  - **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
15
15
  - **test** (Test) — 13 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
16
16
  - **toga** (TOGa) — 2 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
17
- - **tools** (Tools) — 11 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
17
+ - **tools** (Tools) — 12 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
18
18
 
19
19
  ## 2.0 framework
20
20
 
21
21
  - **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
22
- - **worker2** (Worker) — 37 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
22
+ - **worker2** (Worker) — 38 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
23
23
  - **api2** (API) — 20 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
24
24
  - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
25
25
  - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
@@ -0,0 +1,97 @@
1
+ ---
2
+ type: session
3
+ slug: TRUE-80487-Debug-Fix-Rate-WH-Warranty
4
+ title: Rate Whole Home Warranty per-address dedup — root-caused "works in beta, fails in prod"
5
+ author: mhammontree
6
+ repos: [_underscore, toga2-view, dbchanges2, test]
7
+ framework: "2.0"
8
+ client: rate
9
+ status: active
10
+ created: 2026-07-30
11
+ updated: 2026-07-30
12
+ ---
13
+
14
+ # Session: TRUE-80487-Debug-Fix-Rate-WH-Warranty
15
+ **Date:** 2026-07-30
16
+ **Project/Repo:** _underscore / toga2-view / dbchanges2 / test (2.0)
17
+ **Task:** Production troubleshooting of the Rate Whole Home Warranty per-address duplicate-purchase guard (TRUE-79533, already deployed), reported as "works in beta, not in production" — root-caused to three independent code defects, none of them a deployment gap.
18
+
19
+ ---
20
+
21
+ ## What WORKED
22
+
23
+ - **Eliminated every deployment-gap hypothesis with hard evidence.** `_underscore` `_production` HEAD is `ba7eff66` (TRUE-79533 PR #661 merged 2026-07-29); `dbchanges2` `_main` is `138df5e` (PR #418). `git diff origin/_beta -- Model/Rate/Entitlement.php` returns **zero diff** — prod and beta run identical code.
24
+ - **Confirmed all prod DB artifacts present.** `Core.RecordFields` id **2482** = `entitlements.serviceAddressId` registered; `Client_Rate.Entitlements.serviceAddressId` column + index exist; ACL grants present (`2482` → role 1 `isWritable=0`, mirroring `saleItemId` 1265; `addresses.id` field 41 → roles 1,2,3). `ApiPayloadInterceptors` in prod is **byte-identical to dev-sandbox** (id 4 = rec 191 PRE/POST `minDepth NULL`; id 2 = rec 191 POST/POST `minDepth 5`).
25
+ - **ROOT CAUSE #1 — the backend guard has never executed, in any environment.** `isWholeHomeWarranty()` at `_underscore/Model/Rate/Entitlement.php:116-118` reads `$payload->saleItem->title`, but the real checkout payload sends `"saleItem":{"uuid":"3e14effa-0f78-4708-b338-60872a39aff3"}` with **no `title`**. `stripos('', 'warranty')` → false → `prePost` returns at line 58 before the guard runs. Evidence: beta payload for entitlement 106, `Logs_Rate.Api` id 48823 (dev-sandbox, 2026-07-27 11:41:15), verbatim.
26
+ - **Proved it with beta's own data.** dev-sandbox has **three active WH entitlements at the same address** `1864 HIGH GROVE LN 60540`: `ET100057` (id 90, 2026-07-24), `ET100067` (id 100, **2026-07-27 10:10**), `ET100073` (id 106, **2026-07-27 11:41**) — two created on the very day the feature was marked "beta-verified". Plus two at `121 N LA SALLE ST 60602` (`ET100071`, `ET100072`). All same borrower `5686507f-a7b2-4fca-b007-dd3ae875e4bb`, all pinned, all active.
27
+ - **Confirmed the guard's rejection message appears in NO log, in either environment, ever.** All 8 beta `POST /v2/entitlements` 400s are `EV-12` (reference errors) and one `EV-8` (`contractFulfillmentPreference`, 2026-07-15, pre-dating the PRE interceptor registration).
28
+ - **ROOT CAUSE #2 — `state.code` vs `state.uuid`.** `prePost:66` reads `$address->state->code`, but the payload sends `"state":{"uuid":"94417b26-..."}`. Fixing #1 alone would make every WH purchase throw *"A valid service address is required"* — a false rejection on valid addresses. **Both must be fixed together.**
29
+ - **ROOT CAUSE #3 — frontend availability check fails open AND is borrower-scoped.** `toga2-view/src/pages/ZipValidation/viewModels/useZipValidationViewModel.ts:135-139` — `const address = entitlement.serviceAddress; if (!address?.line1) return false;` and `serviceAddress` is derived only from the pin (`useActiveServices.ts:108`). Worse, `useActiveServices.ts:69-71` scopes the query `where: { and: [{ "Customers.c_borrowerId": { "=": borrowerId } }] }` — so it can **never** see another borrower's warranty, while the backend guard is explicitly global. Its code comment claims "same source as the backend guard's dedup join" — that comment is wrong; the backend uses `COALESCE(serviceAddressId, ContactAddresses.addressId)`, the frontend has no fallback.
30
+ - **Verified the backend WOULD block, without paying.** Replicated `hasActiveWarrantyAtAddress()` SQL read-only against prod for `1864 HIGH GROVE LN / Naperville / 60540 / IL` → returns entitlement **43 (`ET100017`)**. So a real purchase reaches PayPal, the customer is charged, and *then* the entitlement POST is rejected.
31
+ - **Confirmed the apartment-building exposure is real.** Property type lives only in router state / `sessionStorage` (`useCheckoutPageViewModel.ts:178-205`) and is **never sent to the API** — the purchase payload contains no `propertyType` / `isEligible`. `prePost:68-70` requires line1/city/state/zip but **not** `line2`. The `requireUnitNumber` flow (`ZipValidation.tsx:62-75`, `198-199`) is client-side only and bypassable. Live proof: beta `ET100073` carries `line2: "STE 500"` on a single-family home, while ids 90/100 at the same `line1` have no unit.
32
+ - **Production data repaired by the developer and verified clean.** Three active WH warranties, all pinned + validated, both-or-neither invariant holds on all nine rows: 43 (`ET100017`) → addr 9 Naperville, 40 (`ET100014`) → addr 6 Nashua, 38 (`ET100012`) → addr 4 Wailuku. Subscription 22 (`ET100015`, Daniel Moran) closed out `isActive=0, dateCancelled='2026-06-03'`; address 7 `city` corrected `60657` → `Chicago`.
33
+
34
+ ## What did NOT work — DO NOT RETRY THESE
35
+
36
+ - **"A migration is missing in production."** DISPROVEN. `Core.RecordFields` 2482 exists; `Client_Rate.Entitlements.serviceAddressId` column + index exist; both ACL grants landed. Do not re-audit `Core/2026-07-22a`, `Client/2026-07-22a`, `Client/2026-07-23a`, `Client_Rate/2026-07-23a`, `Client_Rate/2026-07-24a` — all applied.
37
+ - **"TRUE-79251 merge damage deleted the WH helper methods."** DISPROVEN. `git diff origin/_beta -- Model/Rate/Entitlement.php` → zero diff. `PASSTHROUGH_KEY_FULFILLMENT_PREFERENCE` (line 32) and all three private helpers (`normalizedField` 107, `isWholeHomeWarranty` 116, `hasActiveWarrantyAtAddress` 137) present.
38
+ - **"`_underscore` prod wasn't redeployed past the uuid-clobber fix `9c2e4b1c`."** DISPROVEN by the same zero diff + PR #661 merged.
39
+ - **"`toga2-view` prod still sends `{depth:-1}` and starves `postPost`."** DISPROVEN. `genericApi.ts:54,68` only append `?depth=-1` when `returnPayloadDepth` is explicitly set; no checkout override. Diff vs `_beta` touches only GetSupport files.
40
+ - **"Cross-cluster `Core.RecordFields` references from `Client*/` migration folders fail in prod."** DISPROVEN for this case. Despite prod splitting `prod-core` / `prod-client` / `prod-logs` / `prod-archive` onto separate clusters, the ACL grants landed correctly. ~20 long-standing ACL migrations use this same pattern. (Note: the `dbchanges2-cluster-isolation` hook was NOT found under `.claude/hooks/toga/` — separate follow-up, not this bug.)
41
+ - **"The interceptor row is missing, inactive, or duplicated in prod."** DISPROVEN — prod rows are identical to dev-sandbox, same ids and uuids.
42
+ - **"`POST /v2/entitlement-sales-orders` is the prod purchase path (record 257, no interceptor)."** DISPROVEN. It is only a bridge link: payload is `{"entitlement":{"uuid"},"salesOrder":{"uuid"}}`. It creates no entitlement and fires no interceptor.
43
+ - **"Production simply hasn't processed a purchase since deploy, so there's nothing to debug."** TRUE as a fact (last `POST /v2/entitlements` in prod = 2026-07-10, two × 400 `EV-8`; newest `Entitlements` row 2026-06-15) but WRONG as a conclusion — the developer then reproduced the failure live through the UI. Do not stop at this observation.
44
+ - **Query that hangs — do not run.** `SELECT ... FROM Logs_Rate.Api WHERE route LIKE '%address%'` timed out after **300s** (unindexed full scan on a ~12.7M-row table). Always constrain with the `dtStamp` index: `WHERE dtStamp >= '...' AND dtStamp < '...'`.
45
+ - **Columns that do not exist (queries will error).** `Client_Rate.ApiPayloadInterceptors.phpMethod` — absent in prod (yet `Client_Rate/2025-12-10 - PostInterceptor.sql` inserts it, so that file is not re-runnable). `Client_Rate.Contacts.emailAddress` — does not exist. `Logs_Rate.Api.uri` — the column is `route`.
46
+ - **Widening the frontend `useActiveServices` query to all borrowers.** REJECTED, do not do this — it would let any portal user read other customers' entitlements and addresses. A data-leak fix worse than the bug.
47
+
48
+ ## Not tried yet (candidates for next session)
49
+
50
+ - Fix `isWholeHomeWarranty()` to resolve `saleItem.uuid` → item (fall back to payload `name` and/or `WH_SALE_ITEM_ID = 3`), matching the authority `hasActiveWarrantyAtAddress()` already uses in SQL.
51
+ - Fix state resolution to accept `state.uuid` with `state.code` as fallback.
52
+ - Normalize zip to first 5 digits in the dedup comparison (`Entitlement.php:161` is exact-match; frontend already truncates at `addressCompare.ts:22`). Untested risk: USPS returning `60540-9233` vs stored `60540`.
53
+ - Normalize the unit designator before comparison so `STE 500` / `Suite 500` / `#500` / `Unit 500` collapse to one dwelling.
54
+ - Build the server-side global pre-check: boolean-only scripted endpoint reusing `hasActiveWarrantyAtAddress()`, called from the availability step, deleting the frontend's local dedup. Needs a `dbchanges2` migration registering it in `Core.RecordScripts` **plus** the client ACL grant — note `validateAddress` (RecordScripts id 17) exists only in the live DB and NOT in `dbchanges2`, the exact parity trap to avoid.
55
+ - Log one raw USPS / FedEx / UPS validation response for a known apartment vs a known single-family address, to settle field names empirically. `Fedex.php:417-421` already receives the full `attributes` object and reads only `Matched` / `Resolved`; `Ups.php:801-807` reads only `NoCandidatesIndicator` / `Candidate`.
56
+ - Capture the USPS DPV "valid but secondary unit missing" signal into `validateAddress`'s normalized return with an explicit unknown state, and enforce in `prePost`.
57
+ - Send and persist `propertyType` with the purchase so eligibility is enforced server-side rather than trusted from the browser.
58
+ - Update `test/@Mark/Rate/verify_wholehome_per_address_guard.php` to use the REAL payload shape (uuid-only `saleItem`, uuid-only `state`). It currently passes 18/18 against a shape production never sends — this is why all three defects survived review.
59
+ - Clean up beta duplicates: dev-sandbox entitlements 100, 105, 106.
60
+ - Controlled prod WH purchase to exercise the untested `minDepth = 5` on the POST interceptor, and to test USPS "Suggested" (`60540-9233`) vs "You entered" (`60540`).
61
+
62
+ ## Current file state
63
+
64
+ | File | Status | Notes |
65
+ |------|--------|-------|
66
+ | `_underscore/Model/Rate/Entitlement.php` | **investigated, NOT modified** | Holds root causes #1 (line 116-118) and #2 (line 66). Also: raw SQL string concatenation at lines 371/375 violates the parameterized-query standard; accepted TOCTOU noted at 133-135. |
67
+ | `toga2-view/src/pages/ZipValidation/viewModels/useZipValidationViewModel.ts` | **investigated, NOT modified** | Root cause #3 fail-open at 135-139. Misleading comment claims parity with backend join. |
68
+ | `toga2-view/src/hooks/useActiveServices.ts` | **investigated, NOT modified** | Borrower-scoped `where` at 69-71; pin-only `serviceAddress` resolution at 108; silent EZ-2 degrade at 90. |
69
+ | `toga2-view/src/pages/CheckOut/viewModel/useCheckoutPageViewModel.ts` | **investigated, NOT modified** | `propertyType` read from router state / sessionStorage (178-205), never sent to API. |
70
+ | `_underscore/Model/Client/Address.php` | **investigated, NOT modified** | `validateAddress` waterfall USPS→FedEx→UPS (31-46); returns success + normalized fields only, no unit-required signal. |
71
+ | `_underscore/Component/Library/Carriers/{Usps,Fedex,Ups}` | **investigated, NOT modified** | Signals available but discarded. NOTE: this is a namespace inside `_underscore`, NOT the 1.0 `library` repo. |
72
+ | `test/@Mark/Rate/verify_wholehome_per_address_guard.php` | **not opened** | Passes 18/18 against the wrong payload shape. Needs rewrite. |
73
+ | **Production `Client_Rate` data** | **CHANGED by developer, verified** | Ents 43→addr 9, 40→addr 6, 38→addr 4 pinned; those 3 addresses `isValidated=1`; subscription 22 closed `isActive=0, dateCancelled='2026-06-03'`; address 7 `city` → `Chicago`. No repo files changed this session. |
74
+
75
+ ## Decisions made
76
+
77
+ - **Keep `dateEnd` OUT of the dedup predicate.** Rationale: the guard's contract is `isActive = 1 AND dateCancelled IS NULL`; adding `dateEnd` would let real duplicates through when a renewal hasn't rolled `dateEnd` forward yet. Rejected alternative: filter on `dateEnd`. Confirmed instance of the underlying problem: Kimberly Stearns (`ET100014`) has `dateEnd = 2026-06-01` yet was charged 2026-07-01 — renewals are not advancing `dateEnd`. **Separate ticket.**
78
+ - **Do NOT widen the frontend query to all borrowers; move the check server-side instead.** Rationale: the frontend fundamentally cannot answer "does anyone have a warranty here?" without exposing other customers' data. A boolean-only endpoint reusing the existing helper makes availability and purchase the same code path, so they cannot drift.
79
+ - **Skip the legacy-entitlement backfill migration.** Rationale: only two rows mattered and they were fixed by hand; once the server-side check ships, pinning is no longer required for dedup correctness. Migration would be pure data churn.
80
+ - **Do not chase "dwelling type" from shipping carriers.** Rationale: postal data models deliverability, not property characteristics — no carrier returns single-family vs multi-unit. UPS `AddressClassification` / FedEx `classification` only give Residential vs Commercial, and an apartment building is "Residential". The actionable signal is USPS DPV "secondary unit required but missing". True dwelling type needs property data (assessor/parcel, or Melissa/Smarty-class vendor) — a procurement decision for Paulina + underwriting.
81
+ - **Did NOT complete the PayPal test payment.** Rationale: the backend rejects *after* the charge; the read-only SQL replication already proved what the guard decides.
82
+ - **Branch naming = bare ticket ID `TRUE-80487`.** Rationale: matches live convention in all four repos (`TRUE-78188`, `TRUE-78314`, `TRUE-79191`, `TRUE-79868`). The documented `fix/short-description` rule in the git-workflow standard is stale — flag for correction.
83
+ - **Repo scope:** minimum `_underscore` + `test` (makes the global rejection actually work); add `dbchanges2` + `toga2-view` for the server-side pre-check. `api2` NOT needed (scripted endpoints are served by the V2 engine from `_underscore` models; registration is DB-side). Recommendation: branch all four — an unused branch is free, discovering a needed migration mid-sprint is not.
84
+ - **Reframed the ticket premise.** "Works in beta, fails in prod" is false. Same code, same migrations, same interceptor rows. Beta only *appeared* to work because its test data satisfied both frontend preconditions by accident — a single test borrower, and data created after the pin shipped. Production satisfied neither.
85
+
86
+ ## Blockers
87
+
88
+ - **Need the current dev branch name** to base the four `TRUE-80487` branches on. `_underscore` shows `_beta`; `toga2-view` shows both `_beta` and `_stage`. Developer said they are on "a different dev branch now" — unresolved.
89
+ - **Need a product decision from Paulina: per-dwelling or per-building coverage?** Currently `123 Main St` and `123 Main St APT 2` are distinct addresses, so both can hold a warranty. This changes what the fix does. Sharpened by the finding that a *no-unit* purchase at a multi-unit address is indistinguishable from a single-family purchase and carries building-scale liability at single-home pricing.
90
+ - **Need a scope decision:** does property-type / unit-required enforcement land in TRUE-80487, or a separate ticket? Materially larger than "one warranty per address" — it is "what counts as an insurable address". The gate fixes are independent of both blockers and can proceed regardless.
91
+
92
+ ## Exact next step
93
+
94
+ > Create the `TRUE-80487` branch on `_underscore` (base = the dev branch the developer names), then fix `isWholeHomeWarranty()` at `_underscore/Model/Rate/Entitlement.php:116-118` to identify a WH purchase by resolving `saleItem.uuid` → `Items` (with payload `name` / `WH_SALE_ITEM_ID = 3` as fallbacks) **together with** the `state.uuid` → `code` resolution at line 66 — never one without the other, or every WH purchase throws "A valid service address is required". Then run `test/@Mark/Rate/verify_wholehome_per_address_guard.php` rewritten to the real payload shape (uuid-only `saleItem` and `state`), and confirm it FAILS on the current code before the fix and passes after.
95
+
96
+ ---
97
+ _Saved by /session-save on 2026-07-30_
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.484",
3
+ "version": "1.0.486",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",
@@ -4,24 +4,74 @@ These rules are critical. Violations create exploitable vulnerabilities. Follow
4
4
 
5
5
  ## SQL injection prevention
6
6
 
7
- Never build SQL strings by concatenating or interpolating user input.
7
+ Never build SQL strings by concatenating or interpolating **unescaped** values.
8
+
9
+ > **Neither framework supports prepared statements.** This rule previously demanded them and
10
+ > gave `_Db::select()` as the correct pattern. **`_Db` does not exist.** Verified 2026-07-30:
11
+ > in 2.0, `_Query` accepts only a raw SQL string and `_Database_Driver` declares no
12
+ > `prepare`/`bind` (neither the Mysql nor the Pgsql driver implements one); 1.0 is the same
13
+ > through `App_Database`. There are zero uses of `bind_param`, `mysqli_stmt`, `PDO::`, or
14
+ > `::prepare(` anywhere in the non-vendor codebase. Escaping **is** the parameterization
15
+ > mechanism here, so that is what this rule requires.
16
+
17
+ **Prefer the model layer.** `_Model` (2.0) and `App_Model` (1.0) build and escape queries for
18
+ you. Hand-written SQL is the fallback, not the default.
19
+
20
+ **When you write SQL by hand, every interpolated value must be escaped or cast — no exceptions
21
+ for "internal" or "trusted" sources.**
8
22
 
9
23
  ```php
10
24
  // CRITICAL VIOLATION — never do this
11
- $sql = "SELECT * FROM users WHERE id = " . $_GET['id'];
12
- $sql = "SELECT * FROM orders WHERE email = '$email'";
25
+ $sql = "SELECT * FROM Users WHERE id = " . $_GET['id'];
26
+ $sql = "SELECT * FROM Orders WHERE email = '$email'";
27
+
28
+ // CORRECT (2.0) — cast numerics, escape strings
29
+ $sql = "
30
+ SELECT *
31
+ FROM Orders
32
+ WHERE
33
+ id = " . ((int) $_GET['id']) . " AND
34
+ email = '" . _Database::escape($email) . "'
35
+ ";
36
+
37
+ // CORRECT (1.0) — same shape, different helper
38
+ $sql = "
39
+ SELECT *
40
+ FROM Orders
41
+ WHERE
42
+ id = " . ((int) $_GET['id']) . " AND
43
+ email = '" . App_Database::sqlEscape($email) . "'
44
+ ";
45
+ ```
13
46
 
14
- // CORRECT always use prepared statements
15
- $stmt = $pdo->prepare('SELECT * FROM users WHERE id = ?');
16
- $stmt->execute([$_GET['id']]);
47
+ Applies to every operation: SELECT, INSERT, UPDATE, DELETE, and DDL.
48
+
49
+ **Cast numerics with `(int)`/`(float)`; escape strings.** A cast is sufficient on its own for a
50
+ numeric — there is nothing left to inject once the value is an integer. A string always needs the
51
+ escape helper.
52
+
53
+ **Identifiers are not covered by escaping.** `_Database::escape()` and
54
+ `App_Database::sqlEscape()` protect *values*, not table names, column names, sort directions, or
55
+ `LIMIT` expressions. Anything used as an identifier must come from a **hardcoded allowlist** —
56
+ never from a request, and never merely escaped.
57
+
58
+ ```php
59
+ // CRITICAL VIOLATION — escaping does not make an identifier safe
60
+ $sql = "SELECT * FROM Orders ORDER BY " . _Database::escape($_GET['sort']);
17
61
 
18
- // CORRECT — framework parameterized query
19
- _Db::select('orders', ['email' => $email]);
62
+ // CORRECT — allowlist
63
+ $allowedSortColumns = ['dtCreated', 'total', 'status'];
64
+ $sortColumn = in_array($_GET['sort'], $allowedSortColumns, true) ? $_GET['sort'] : 'dtCreated';
65
+ $sql = "SELECT * FROM Orders ORDER BY $sortColumn";
20
66
  ```
21
67
 
22
- This applies to every SQL operation: SELECT, INSERT, UPDATE, DELETE, and DDL. No exceptions for "internal" data or "trusted" sources — use parameterized queries everywhere.
68
+ **Enum columns need validating in PHP.** 2.0's `FIELD_LIST` performs no enum validation and
69
+ silently converts a falsy value to SQL `NULL`; MySQL will coerce rather than reject an unknown
70
+ member. Check the value against the allowed set before it reaches the query.
23
71
 
24
- Integer casting (`(int) $_GET['id']`) reduces risk but is not a substitute for prepared statements. Use both.
72
+ One caveat on 2.0's helpers: `_Database::escape()` is the correct one. Do **not** confuse it with
73
+ `_Database::protect()`, which *strips* `% \ / * " '` and the literal ` or` — lossy filtering, not
74
+ escaping, and it will corrupt data you meant to store intact.
25
75
 
26
76
  ## XSS prevention
27
77