toga-ai 1.0.484 → 1.0.486
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/tools/INDEX.md +1 -0
- package/knowledge/1.0/apps/tools/features/errors-curation-console.md +81 -0
- package/knowledge/2.0/apps/_underscore/INDEX.md +1 -1
- package/knowledge/2.0/apps/_underscore/features/error-reporting-issue-event.md +243 -120
- package/knowledge/2.0/apps/api2/INDEX.md +1 -1
- package/knowledge/2.0/apps/api2/features/v2-api-error-codes.md +73 -1
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
- package/knowledge/2.0/apps/worker2/features/error-escalation-cron.md +157 -0
- package/knowledge/2.0/standards/backend-php.md +3 -0
- package/knowledge/INDEX.md +2 -2
- package/knowledge/sessions/2026-07-30-TRUE-80487-Debug-Fix-Rate-WH-Warranty-mhammontree.md +97 -0
- package/package.json +1 -1
- package/rules/common/security.md +60 -10
|
@@ -7,6 +7,7 @@
|
|
|
7
7
|
| [CloudFront Client Setup](features/cloudfront-client-setup.md) | An SSO-gated admin tool at **`/devops/cloudfront-clients`** in the Tools 1.0 app that onboards a client onto **CloudFront + Route 53 across multiple AWS account | tools/_/app/devops/cloudfront.php, tools/mvc/devops/cloudfront-clients/get.php, tools/mvc/devops/cloudfront-clients/post.php, tools/assets/js/cloudfront-clients.js, tools/assets/css/cloudfront-clients.css, tools/_/app/nav.php, tools/_/app/frameworkindex.php, tools/config.production.ini |
|
|
8
8
|
| [Design Demo Admin](features/design-demo-admin.md) | A self-serve admin UI at **`/design`** in the SSO-protected **Tools** app that lets the design team publish self-contained "Claude Design" HTML exports as **ver | tools/_/app/design/github.php, tools/mvc/design/get.php, tools/mvc/design/post.php, tools/assets/css/design.css, tools/assets/js/design.js, tools/_/app/frameworkindex.php, tools/_/app/nav.php, tools/composer.json |
|
|
9
9
|
| [Tools — Developers Folder (UUID & Password Generators)](features/developer-tools.md) | The first two tools shipped in the Tools app, both under the **Developers** folder and gated to personas **Development Team** / **TOGa Technology**. | tools/mvc/developers/uuid/get.php, tools/mvc/developers/password/get.php |
|
|
10
|
+
| [/errors Curation Console (Tools → shared Core Logs DB)](features/errors-curation-console.md) | Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as a 1.0 Tools MVC page reading the **shared Core Logs DB** through | tools/mvc/errors/get.php, tools/mvc/errors/post.php, tools/_/app/nav.php, tools/config.production.ini |
|
|
10
11
|
| [Tools MVC — Routing, CSRF & App_Database Access Patterns](features/mvc-data-access-patterns.md) | The load-bearing 1.0 (`App_`) framework conventions a developer needs when adding a page to the Tools app — URL routing, CSRF, and DB access through `App_Databa | tools/_/app/nav.php, tools/mvc/get.php |
|
|
11
12
|
| [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php, tools/mvc/get.php |
|
|
12
13
|
| [Tools SAML SSO Consumer & Persona-Gated Auth (App_Auth)](features/saml-sso-auth.md) | `App_Auth` is the Tools app's authentication layer: it consumes the SAML gateway `?saml=` handoff (see the 2.0 SAML downstream integration contract), establishe | tools/_/app/auth.php, tools/mvc/sso/initiate/get.php, tools/mvc/sso/get.php, tools/mvc/login/get.php, tools/mvc/login/post.php, tools/mvc/logout/get.php, tools/mvc/get.php, tools/config.production.ini, tools/config.local.ini |
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: /errors Curation Console (Tools → shared Core Logs DB)
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: tools
|
|
5
|
+
project: Tools
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-30
|
|
10
|
+
owners: ["jcardinal"]
|
|
11
|
+
files:
|
|
12
|
+
- tools/mvc/errors/get.php
|
|
13
|
+
- tools/mvc/errors/post.php
|
|
14
|
+
- tools/_/app/nav.php
|
|
15
|
+
- tools/config.production.ini
|
|
16
|
+
related:
|
|
17
|
+
- ../architecture.md
|
|
18
|
+
- ./mvc-data-access-patterns.md
|
|
19
|
+
- ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
|
|
20
|
+
- ../../../2.0/apps/worker2/features/error-escalation-cron.md
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as
|
|
26
|
+
a 1.0 Tools MVC page reading the **shared Core Logs DB** through a new **`db_toga2logs`**
|
|
27
|
+
alias. It curates; it never notifies.
|
|
28
|
+
|
|
29
|
+
The highest-value view is the list of issues **below the ClickUp threshold** — errors that are
|
|
30
|
+
happening and were deliberately never ticketed. Nobody had that view before.
|
|
31
|
+
|
|
32
|
+
## How it works
|
|
33
|
+
|
|
34
|
+
- **Curate** subject, description, `minimumUrgency`, assignee, and mute window.
|
|
35
|
+
- **Manage per-client email recipients** (`IssueEmailAddresses`). `clientId = 0` means **all
|
|
36
|
+
clients** and requires an explicit confirmation flag; the list view renders it as
|
|
37
|
+
**"ALL CLIENTS"**.
|
|
38
|
+
- **Merge fingerprints** — repoint an `IssueFingerprints` row at an existing Issue. Without this,
|
|
39
|
+
the fingerprint/Issue split is inert: a refactor that shifts line numbers produces a new Issue
|
|
40
|
+
and the curated one is orphaned.
|
|
41
|
+
- **Any save sets `isManaged`**, which does double duty: it protects the curated text from being
|
|
42
|
+
overwritten by later occurrences, **and** it exempts the Issue from the cron's GC sweep.
|
|
43
|
+
|
|
44
|
+
**This page notifies nobody.** Escalation, ClickUp ticketing, and email are owned entirely by
|
|
45
|
+
`_Worker_Infrastructure_Errors::Escalate`. Adding a notify action here would create a second,
|
|
46
|
+
untracked alerting path.
|
|
47
|
+
|
|
48
|
+
### Why Tools and not TOGa Hub
|
|
49
|
+
|
|
50
|
+
The 2026-06-12 meeting had proposed TOGa Hub, but Hub is far off. The tables are the same either
|
|
51
|
+
way, so the choice is not load-bearing — a Hub port later reads the same schema.
|
|
52
|
+
|
|
53
|
+
## Configuration
|
|
54
|
+
|
|
55
|
+
`[database_toga2logs]` was added to `tools/config.production.ini`.
|
|
56
|
+
|
|
57
|
+
**`dbname` is NOT necessarily `Logs`.** 2.0 resolves the Core Logs database name at **runtime**
|
|
58
|
+
from `Core.Databases` id **11** (`worker2` `CORE_LOGS_DATABASE_ID`); `Logs` is only the
|
|
59
|
+
`_underscore` *register alias*. Read the real name from `Core.Databases` before configuring a new
|
|
60
|
+
environment, or the console silently points at a database that may not exist.
|
|
61
|
+
|
|
62
|
+
## Gotchas / known issues
|
|
63
|
+
|
|
64
|
+
- **This is a 1.0 app reading a 2.0 shared database.** Access is plain `App_Database` via the
|
|
65
|
+
`db_toga2logs` alias, not the `_Model` layer — so none of the 2.0 model protections apply.
|
|
66
|
+
Escape every interpolated value and allowlist every identifier (see the 1.0 back-end standard).
|
|
67
|
+
- **Cross-tenant surface.** The Core Logs DB is shared by every client, so this console can show
|
|
68
|
+
one client's captured context to a viewer looking at another client's problem. It is
|
|
69
|
+
internal-only and persona-gated for that reason — do not expose any part of it to a client.
|
|
70
|
+
- **Curated text is authoritative.** Once `isManaged` is set, later occurrences no longer update
|
|
71
|
+
subject/description, so a stale curated subject stays stale until someone edits it again.
|
|
72
|
+
|
|
73
|
+
## Change history
|
|
74
|
+
|
|
75
|
+
- 2026-07-30 — Built as part of TRUE-78188: new `/errors` triage/curation console over the
|
|
76
|
+
shared Core Logs DB via a new `db_toga2logs` alias; curation sets `isManaged` (protects text
|
|
77
|
+
and exempts from GC); fingerprint merging; per-client recipient management with an explicit
|
|
78
|
+
confirmation for `clientId = 0` ("ALL CLIENTS"). Chosen over TOGa Hub because Hub is far off
|
|
79
|
+
and the schema is the same either way. Documented that the Core Logs `dbname` resolves at
|
|
80
|
+
runtime from `Core.Databases` id 11 and is not necessarily "Logs". (jcardinal)
|
|
81
|
+
</content>
|
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
| [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
|
|
16
16
|
| [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, worker2/Worker/Infrastructure/Email/Send.php |
|
|
17
17
|
| [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
|
|
18
|
-
| [Error Reporting — Issue/Event
|
|
18
|
+
| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Exception/Business.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
|
|
19
19
|
| [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
|
|
20
20
|
| [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
|
|
21
21
|
| [isFulfillable Propagation Up the SO↔PO Chain](features/fulfillable-item-propagation.md) | `Items.isFulfillable` is a boolean that gates whether a storefront line's **Qty Fulfilled** cell is actionable. | _underscore/Model/Client/Item.php, _underscore/Model/Compass/Item.php, dbchanges2/Core/2026-07-17 - Items isFulfillable RecordField.sql, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, dbchanges2/Client/2026-07-17 - ItemsisFulfillable.sql |
|
|
@@ -1,148 +1,271 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: Error Reporting — Issue/Event
|
|
2
|
+
title: Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
repo: _underscore
|
|
5
5
|
project: _Underscore
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
10
|
-
owners: ["dfranks"]
|
|
9
|
+
updated: 2026-07-30
|
|
10
|
+
owners: ["dfranks", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Error.php
|
|
13
|
+
- _underscore/Exception/Business.php
|
|
13
14
|
- _underscore/Model/Core/Logs/Issue.php
|
|
14
15
|
- _underscore/Model/Core/Logs/Event.php
|
|
15
|
-
-
|
|
16
|
+
- _underscore/Model/Core/Logs/IssueFingerprint.php
|
|
17
|
+
- _underscore/Model/Core/Logs/IssueClickupTask.php
|
|
18
|
+
- _underscore/Model/Core/Logs/IssueEmailAddress.php
|
|
19
|
+
- _underscore/Model/Core/Logs/IssueAreaOwner.php
|
|
20
|
+
- dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql
|
|
21
|
+
- dbchanges2/Core/2026-07-30a - Error escalation cron job.sql
|
|
16
22
|
related:
|
|
17
23
|
- ../architecture.md
|
|
18
24
|
- ./per-client-database-connections.md
|
|
25
|
+
- ./email-send-pipeline.md
|
|
26
|
+
- ../../worker2/features/error-escalation-cron.md
|
|
27
|
+
- ../../../1.0/apps/tools/features/errors-curation-console.md
|
|
19
28
|
- ../../dbchanges2/architecture.md
|
|
20
29
|
---
|
|
21
30
|
|
|
22
31
|
## Summary
|
|
23
32
|
|
|
24
|
-
Platform-wide error
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
with
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
`
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
handler
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
33
|
+
Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in
|
|
34
|
+
the **shared Core Logs DB**. An **Issue** is the deduplicated record for a distinct problem
|
|
35
|
+
(carrying curated text, an urgency, and a quotable reference); an **Event** is one row per
|
|
36
|
+
occurrence. Capture lives in `_underscore/Error.php`; escalation and ClickUp/email
|
|
37
|
+
notification live entirely in the worker2 cron (see
|
|
38
|
+
[error-escalation-cron](../../worker2/features/error-escalation-cron.md)); curation lives in
|
|
39
|
+
the 1.0 Tools `/errors` console.
|
|
40
|
+
|
|
41
|
+
`_Error::captureException(Throwable $e, ?int $clientId): ?string` is the **single capture
|
|
42
|
+
entry point** — the exception handler, the shutdown handler, and api2 all funnel through it,
|
|
43
|
+
and it returns the quotable reference. Add new capture paths by calling it, never by
|
|
44
|
+
re-implementing persistence.
|
|
45
|
+
|
|
46
|
+
## Data model (shared Core Logs DB)
|
|
47
|
+
|
|
48
|
+
Six tables, created by `dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql`:
|
|
49
|
+
`Issues`, `IssueFingerprints`, `Events`, `IssueEmailAddresses`, `IssueClickupTasks`,
|
|
50
|
+
`IssueAreaOwners`.
|
|
51
|
+
|
|
52
|
+
**An Issue is never client-scoped.** An Issue is identified by *code*, and code is global — the
|
|
53
|
+
same bug hitting six clients is **one** Issue. Per-client scoping lives on the occurrence
|
|
54
|
+
(`Events.clientId`) and on notification routing (`IssueEmailAddresses.clientId`), never on the
|
|
55
|
+
Issue itself. Anything that reads like "this client's issue" is a modelling error.
|
|
56
|
+
|
|
57
|
+
`IssueEmailAddresses.clientId = 0` means **all clients**.
|
|
58
|
+
|
|
59
|
+
## How it works
|
|
60
|
+
|
|
61
|
+
### Fingerprinting (identity)
|
|
62
|
+
|
|
63
|
+
The fingerprint is the normalized top-N **application** frames (repo-relative `path:line`) plus
|
|
64
|
+
the exception class; framework/vendor plumbing is filtered out. The exception **message is
|
|
65
|
+
deliberately excluded** — it carries per-occurrence data (ids, values) and would defeat dedup.
|
|
66
|
+
|
|
67
|
+
This replaced `md5(serialize(debug_backtrace()))`, which serialized **call arguments** and so
|
|
68
|
+
produced a different hash on every request — dedup never actually worked.
|
|
69
|
+
|
|
70
|
+
`IssueFingerprints` is many-to-one against `Issues`, so a refactor that shifts line numbers
|
|
71
|
+
cannot orphan a curated Issue: an admin repoints the new fingerprint at the existing Issue
|
|
72
|
+
from the Tools console to merge them.
|
|
73
|
+
|
|
74
|
+
### `_Exception_Business` — identity immune to refactoring
|
|
75
|
+
|
|
76
|
+
`_underscore/Exception/Business.php` carries a stable `issueKey` plus a `minimumUrgency` floor.
|
|
77
|
+
When an `issueKey` is declared, **the key is the fingerprint**, so the Issue's identity does not
|
|
78
|
+
depend on line numbers at all. It extends `Exception` (not `_Exception`), matching
|
|
79
|
+
`_Exception_Validation`.
|
|
80
|
+
|
|
81
|
+
It **deliberately never throws on a bad argument** — it normalizes and `error_log()`s instead.
|
|
82
|
+
Replacing the caller's business error with an argument error destroys the real signal.
|
|
83
|
+
|
|
84
|
+
### The quotable reference
|
|
85
|
+
|
|
86
|
+
`Issues.reference` is a Sentry-style base-26 encoding of `Issues.id`: 1 → `1A`, 26 → `1Z`,
|
|
87
|
+
27 → `2A`. Because `AUTO_INCREMENT` never reuses ids, gaps are expected and permanent — a
|
|
88
|
+
reference can never be recycled onto a different problem, so it is safe to quote to a customer.
|
|
89
|
+
|
|
90
|
+
The reference is written by PHP in a second statement (see MySQL constraints below), which is
|
|
91
|
+
why the column is nullable even though it is always populated.
|
|
92
|
+
|
|
93
|
+
### Shutdown handler
|
|
94
|
+
|
|
95
|
+
`register_shutdown_function` is registered with a **pre-allocated memory reserve**. Before this,
|
|
96
|
+
`_underscore` had **no shutdown handler at all**, so fatals, OOM, and timeouts were completely
|
|
97
|
+
uncaptured.
|
|
98
|
+
|
|
99
|
+
## MySQL constraints that shaped the schema (verified on 8.0.46)
|
|
100
|
+
|
|
101
|
+
- **A stored generated column blocks `ON DELETE CASCADE`.** A foreign key with
|
|
102
|
+
`ON DELETE CASCADE` is rejected with error **1215** when the constrained column is the base
|
|
103
|
+
column of a **STORED** generated column. `IssueClickupTasks.issueId` is the base of
|
|
104
|
+
`openEpisodeIssueId`, so that generated column must be **VIRTUAL**. VIRTUAL has no such
|
|
105
|
+
restriction and still supports a UNIQUE secondary index. Proved both ways locally.
|
|
106
|
+
- **Error 3109: a generated column cannot refer to an auto-increment column.** This is why
|
|
107
|
+
`Issues.reference` cannot be generated and must be written by PHP.
|
|
108
|
+
|
|
109
|
+
## One-open-episode guard (UNIQUE on a generated column)
|
|
110
|
+
|
|
111
|
+
`INSERT ... SELECT ... WHERE NOT EXISTS` does **not** prevent a double insert under
|
|
112
|
+
REPEATABLE READ: two overlapping cron runs each evaluate the `NOT EXISTS` against their own
|
|
113
|
+
snapshot and both pass. The guard is instead a VIRTUAL generated column
|
|
114
|
+
`openEpisodeIssueId = issueId while dtResolved IS NULL, else NULL` with a **UNIQUE** key —
|
|
115
|
+
MySQL permits duplicate NULLs, so many resolved episodes coexist while at most one is open.
|
|
116
|
+
|
|
117
|
+
The code **claims the episode before calling ClickUp**, so a losing run fails (error 1062)
|
|
118
|
+
before it can create a duplicate external task; the claim is **released on API failure** so a
|
|
119
|
+
stuck placeholder row cannot permanently block the Issue. Verified: second open episode → 1062;
|
|
120
|
+
resolve-then-reopen allowed; many resolved episodes allowed; cascade delete works.
|
|
121
|
+
|
|
122
|
+
## Sentry coexistence (removal is a separate ticket)
|
|
123
|
+
|
|
124
|
+
Sentry stays running **in parallel**. Shipping an unproven replacement for the only working
|
|
125
|
+
error reporting with no overlap means going blind. Three findings make removal its own ticket:
|
|
126
|
+
|
|
127
|
+
1. `_Worker_Sentry::Webhook` is an **existing Sentry→ClickUp pipeline writing into the same
|
|
128
|
+
list `901110669877`** this work replaces — retiring it also means turning off the Sentry-side
|
|
129
|
+
issue alerts.
|
|
130
|
+
2. `\Sentry\configureScope()` calls `http_response_code(500)` as a blanket default, and api2
|
|
131
|
+
documents working *around* it. Remove Sentry without fixing every early-exit branch and
|
|
132
|
+
validation 400s ship as HTTP 200 — which changes whether apiproxy retries across regions.
|
|
133
|
+
3. `\Sentry\init()` registers PHP error handlers. `_Error::errorHandler()` also throws on
|
|
134
|
+
warnings, but that promotion must be verified **empirically** — if it does not survive,
|
|
135
|
+
code that currently 500s would silently continue with a null value, which is worse.
|
|
136
|
+
|
|
137
|
+
Also: this pipeline is **2.0-only**. Sentry must remain for the 1.0 apps (desk1, retail1,
|
|
138
|
+
forecast1, worker1, view1, commerce1) until a `library` / `App_Error` port lands, so the
|
|
139
|
+
Sentry-removal ticket cannot complete until the 1.0 port does.
|
|
94
140
|
|
|
95
141
|
## Gotchas / known issues
|
|
96
142
|
|
|
97
|
-
- **
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
`
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
143
|
+
- **Read-your-writes: reads go to the READ host and cannot see an uncommitted write.** A
|
|
144
|
+
second `$issue->save()` re-reads the row via `_Model::initialize()`, and that SELECT goes to
|
|
145
|
+
the read host — a separate connection that cannot see the still-uncommitted INSERT on the
|
|
146
|
+
write connection. Symptom: `Issues` rows with `reference` NULL, **zero** Events, **zero**
|
|
147
|
+
IssueFingerprints, no reference in the API response, and the logged cause *"Error during
|
|
148
|
+
Model build for primary key id '1' … Exactly 1 row was expected to be returned but 0 were."*
|
|
149
|
+
Capture aborted half-written. Fix: force reads to the writer for the whole capture block via
|
|
150
|
+
`_Database::getIsReadHostEnabled()` / `setIsReadHostEnabled(false)` with a `finally` restore
|
|
151
|
+
— the same TRUE-80448 pattern `_Email::send()` uses.
|
|
152
|
+
- **Never rebuild a model inside an error handler.** `_Model`'s build **throws** when the row
|
|
153
|
+
is missing, and nothing inside an error handler should throw over a value derivable from the
|
|
154
|
+
primary key. `new _Model_Core_Logs_Issue($id)` rebuilds were replaced with a light SELECT
|
|
155
|
+
helper, and `reference` is written with a plain UPDATE rather than a second `save()`.
|
|
156
|
+
- **`Issues.issueKey` is UNIQUE — handle the first-occurrence race.** Two concurrent first
|
|
157
|
+
occurrences of the same declared business condition raced on the INSERT and the loser's
|
|
158
|
+
occurrence was **silently dropped**. The loser must adopt the winner's Issue.
|
|
159
|
+
- **`LAST_INSERT_ID(totalOccurrences + 1)` is the atomic event-number allocator** (a `_Model`
|
|
160
|
+
dirty-check read-modify-write loses increments). But `LAST_INSERT_ID` is only *assigned* when
|
|
161
|
+
the UPDATE matches a row — proven locally: against a missing row, `affected = 0` while
|
|
162
|
+
`LAST_INSERT_ID()` still returned the **stale** previous value `2`. Verify affected rows
|
|
163
|
+
before trusting it, or the event number silently collides with `UNIQUE(issueId, eventNumber)`.
|
|
164
|
+
- **A new Issue must be seeded at its declared `minimumUrgency`**, not at LOW — otherwise the
|
|
165
|
+
floor is ignored until the next cron run.
|
|
166
|
+
- **MySQL `UUID()` is forbidden for inserted records** by the 2.0 standard (time-based, not
|
|
167
|
+
random) — use `_String::generateUuid()` in PHP. It remains acceptable inside `.sql`
|
|
168
|
+
migrations, where no PHP exists (65 existing migrations do this).
|
|
169
|
+
- **`VALUES()` in `ON DUPLICATE KEY UPDATE` is deprecated in MySQL 8.0.20+** — use the row-alias
|
|
170
|
+
form. **Evaluation order matters:** in `IssueAreaOwners`, `consecutiveCount` is assigned
|
|
171
|
+
**before** `clickupUserId` so the `IF()` compares against the *old* owner. Reversing them
|
|
172
|
+
makes the condition always true and the counter never resets.
|
|
173
|
+
- **`Events.context` is an allowlist, never a dump.** It replaced `print_r($GLOBALS)` filtered
|
|
174
|
+
by a key-name blocklist — that blocklist recursed **arrays only** and walked straight past
|
|
175
|
+
objects whose *properties* hold DB credentials, into a table shared by every client.
|
|
176
|
+
`requestUri` and `referer` have query strings stripped (a `?token=abc` was `[REDACTED]` under
|
|
177
|
+
`get` and stored in clear in `requestUri`). The sensitive-fragment list covers **PII**, not
|
|
178
|
+
just credentials. Persisted traces are rebuilt **argument-free**.
|
|
179
|
+
- **Keep the debug `print_r($GLOBALS)` dump inside the `isDebugMode()` branch.** It was being
|
|
180
|
+
built on every exception — including the OOM path the memory reserve exists to protect.
|
|
181
|
+
- **Strict-mysqli varchar overflow silently drops the row.** `_Error::initialize()` sets
|
|
182
|
+
`mysqli_report(MYSQLI_REPORT_STRICT)`, so assigning an over-255-char value to a
|
|
183
|
+
`varchar(255)` (`errorMessage`, `subject`) **throws** on `->save()`. Inside the handler's
|
|
184
|
+
`catch(Throwable)` the throw is swallowed and the row is silently dropped — precisely when a
|
|
185
|
+
long message matters most. Truncate to column width before `save()`.
|
|
186
|
+
- **Shared-schema migration collision.** Because these tables live in the *shared* Core Logs
|
|
187
|
+
DB, only **one** migration may `CREATE` them; later migrations only `ALTER`.
|
|
188
|
+
- **Pre-existing committed secrets in this area (not fixed — out of scope, report only).**
|
|
189
|
+
A ClickUp API token at `_underscore/Component/Api/Clickup/Clickup.php:5`; SES SMTP
|
|
190
|
+
credentials at `_underscore/Email.php:11-12`; a named individual's Sentry personal access
|
|
191
|
+
token at `worker2/Worker/Sentry.php:4`. Also unfixed: api2 writes `apache_request_headers()`
|
|
192
|
+
verbatim into `Logs.Api` (bearer tokens in plaintext) and ships `$_SERVER` to Sentry. Treat
|
|
193
|
+
all of these as compromised whenever this area is touched again.
|
|
194
|
+
|
|
195
|
+
## Decisions superseding the 2026-05-21 design
|
|
196
|
+
|
|
197
|
+
### The POST-to-a-central-`/errors`-receiver hop is DROPPED (jcardinal, 2026-07-30)
|
|
198
|
+
|
|
199
|
+
The **"2026-05-21 – Errors, Monitoring and Alerts"** meeting had approved a *POST-to-receiver*
|
|
200
|
+
pipeline: application handlers POST a serialized error to a centralized `/errors` receiver
|
|
201
|
+
endpoint (`webhook.hub.com/error`) and perform **no** database persistence; a worker2 cron
|
|
202
|
+
consumes the payloads and does all hashing, upserting, aggregation, escalation, and ClickUp
|
|
203
|
+
sync into the Core Logs DB.
|
|
204
|
+
|
|
205
|
+
**That hop is dropped.** Error capture writes **directly** to the Core Logs DB from
|
|
206
|
+
`_Error::captureException()`. Inline persistence is the model to follow; do not reintroduce an
|
|
207
|
+
HTTP hop.
|
|
208
|
+
|
|
209
|
+
Rationale: an error is frequently caused **by something being down**, so routing error capture
|
|
210
|
+
through an HTTP call to api2 would lose exactly the errors you most need. Inline capture also
|
|
211
|
+
buys the synchronous `reference` the api2 response envelope needs, which a fire-and-forget POST
|
|
212
|
+
cannot. The agreed design still holds on the *sink* (shared Core Logs DB, not per-client) and on
|
|
213
|
+
*who escalates* (the worker2 cron owns escalation and notification only).
|
|
214
|
+
|
|
215
|
+
The cost of a direct write is that **each app must have the Core Logs connection registered**.
|
|
216
|
+
The capture path already guards this with
|
|
217
|
+
`isset(_Database::$_registers[_underscore::DB_LOGS])` — it **skips** capture rather than
|
|
218
|
+
fatalling when the register is absent.
|
|
219
|
+
|
|
220
|
+
### Why api2 is in scope at all
|
|
221
|
+
|
|
222
|
+
api2 catches `Throwable` in its own front controller and therefore **never reaches
|
|
223
|
+
`_Error::exceptionHandler`**. `captureException()` had to become a shared entry point that
|
|
224
|
+
api2's catch blocks call directly — otherwise the entire inbound-API 500 surface would go
|
|
225
|
+
unrecorded.
|
|
226
|
+
|
|
227
|
+
### `Issues.type` enum and `BusinessIssues_Users` were proposed and not built (jcardinal, 2026-07-30)
|
|
228
|
+
|
|
229
|
+
The **2026-06-12 "Monitoring Business Exceptions Sync"** meeting proposed an `Issues.type` enum
|
|
230
|
+
(`'technical'`/`'business'`) and a `BusinessIssues_Users` bridge table (clientId +
|
|
231
|
+
clientUserId). **Neither was built.** As built instead:
|
|
232
|
+
|
|
233
|
+
- **No `type` column.** `issueKey` (non-null = a developer declared it a business condition in
|
|
234
|
+
code) plus `isManaged` (a human curated it) carry the same information while also being
|
|
235
|
+
useful on their own.
|
|
236
|
+
- **`IssueEmailAddresses` uses varchar email addresses scoped by `clientId`** (`0` = all
|
|
237
|
+
clients) rather than client-user ids, so recipients who have **no login** can still be
|
|
238
|
+
addressed.
|
|
132
239
|
|
|
133
240
|
## Change history
|
|
134
241
|
|
|
242
|
+
- 2026-07-30 — Reworked TRUE-78188: six-table schema in the shared Core Logs DB (Issue is
|
|
243
|
+
global by code; client scope lives on Events/IssueEmailAddresses); `_Error::captureException()`
|
|
244
|
+
as the single capture entry point; trace-frame fingerprinting with the message excluded
|
|
245
|
+
(replacing `md5(serialize(debug_backtrace()))`, which never deduped); many fingerprints → one
|
|
246
|
+
Issue for refactor safety; `_Exception_Business` with a stable `issueKey` + `minimumUrgency`;
|
|
247
|
+
base-26 quotable reference; a shutdown handler with a memory reserve (there was none).
|
|
248
|
+
Documented two MySQL constraints (1215 stored-generated-column vs. ON DELETE CASCADE; 3109
|
|
249
|
+
generated column cannot refer to auto-increment), the UNIQUE-on-VIRTUAL-generated-column
|
|
250
|
+
open-episode guard (`WHERE NOT EXISTS` is not safe under REPEATABLE READ), the read-host
|
|
251
|
+
read-your-writes failure that broke capture entirely, the issueKey first-occurrence race, the
|
|
252
|
+
`LAST_INSERT_ID` stale-value trap, the `Events.context` allowlist replacing the
|
|
253
|
+
`print_r($GLOBALS)` blocklist, and the Sentry-parallel decision. (jcardinal)
|
|
254
|
+
- 2026-07-30 — **Decided: the 2026-05-21 POST-to-a-central-`/errors`-receiver hop is dropped.**
|
|
255
|
+
Capture writes directly from `_Error::captureException()`, guarded by
|
|
256
|
+
`isset(_Database::$_registers[_underscore::DB_LOGS])`; an HTTP hop would lose errors caused by
|
|
257
|
+
something being down. Recorded that api2 is in scope because its front controller catches
|
|
258
|
+
`Throwable` and never reaches `_Error::exceptionHandler`. Also recorded that the 2026-06-12
|
|
259
|
+
meeting's proposed `Issues.type` enum and `BusinessIssues_Users` bridge table were **not
|
|
260
|
+
built** — `issueKey` + `isManaged` and `clientId`-scoped varchar `IssueEmailAddresses` are the
|
|
261
|
+
as-built equivalents. Supersedes the 2026-07-08 correction. (jcardinal)
|
|
135
262
|
- 2026-07-15 — Added two framework gotchas from aligning the `_underscore` side of the
|
|
136
|
-
Issue/Event pipeline to the shared Core Logs DB:
|
|
137
|
-
|
|
138
|
-
Error.php; (2) strict-mysqli varchar overflow throwing inside the handler's `catch(Throwable)`
|
|
139
|
-
silently drops the Issue/Event row — truncate to column width before `save()`. (dfranks)
|
|
263
|
+
Issue/Event pipeline to the shared Core Logs DB: cross-tenant secret exposure in captured
|
|
264
|
+
context, and strict-mysqli varchar overflow throwing inside `catch(Throwable)`. (dfranks)
|
|
140
265
|
- 2026-07-08 — Corrected KB drift: documented the approved 2026-05-21 error-monitoring
|
|
141
|
-
architecture (handlers POST to a centralized `/errors` receiver; a worker2 cron
|
|
142
|
-
|
|
143
|
-
syncs ClickUp — handlers do not persist). Clarified that `_production` Error.php performs
|
|
144
|
-
no Issue/Event persistence (only an S3 debug body + flat `*_Logs_Error` row), and that the
|
|
145
|
-
inline-DB-persist approach (handler-side upsert to the *Client* Logs DB) conflicts with the
|
|
146
|
-
agreed design and is not the pattern to follow. (dfranks)
|
|
266
|
+
architecture (handlers POST to a centralized `/errors` receiver; a worker2 cron aggregates
|
|
267
|
+
into the Core Logs DB). **Superseded 2026-07-30 — the receiver hop was dropped.** (dfranks)
|
|
147
268
|
- 2026-07-06 — Prior doc described the exceptionHandler as directly persisting Issue/Event
|
|
148
|
-
rows
|
|
269
|
+
rows with dedup; superseded by the 2026-07-08 correction. (dfranks)
|
|
270
|
+
</content>
|
|
271
|
+
</invoke>
|
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
| [TableView row-filtering via apiWhereClause (options.where grammar, end to end)](features/tableview-apiwhereclause-row-filtering.md) | `TableViews.apiWhereClause` (TEXT, nullable) is the sanctioned, code-free way to restrict or exclude rows from a 2.0 table view. | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TableView.php, toga2-supply/src/api/toga.ts |
|
|
18
18
|
| [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql |
|
|
19
19
|
| [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
|
|
20
|
-
| [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TrackingNumber.php |
|
|
20
|
+
| [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, api2/Component/Api/V2/Response/Oauth/Oauth.php, api2/Controller/Index.php, toga2-supply/src/globalTypes.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, _underscore/Model/Client/TrackingNumber.php |
|
|
21
21
|
| [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
|
|
22
22
|
| [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
|
|
23
23
|
| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
|
|
@@ -6,12 +6,18 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-30
|
|
10
10
|
owners: [mhammontree, tcox, jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
13
|
+
- api2/Component/Api/V2/Response/Response.php
|
|
14
|
+
- api2/Component/Api/V2/Response/Oauth/Oauth.php
|
|
15
|
+
- api2/Controller/Index.php
|
|
16
|
+
- toga2-supply/src/globalTypes.ts
|
|
17
|
+
- toga2-supply/src/pages/Orders/api/OrdersApi.ts
|
|
13
18
|
- _underscore/Model/Client/TrackingNumber.php
|
|
14
19
|
related:
|
|
20
|
+
- ../../_underscore/features/error-reporting-issue-event.md
|
|
15
21
|
- ../../_underscore/features/acl-permission-chain.md
|
|
16
22
|
- scripted-api-post-body-args.md
|
|
17
23
|
- v2-rest-query-contract.md
|
|
@@ -91,6 +97,61 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
|
|
|
91
97
|
DB fully migrated. On a local dev box with PHP opcache disabled a branch switch takes effect
|
|
92
98
|
immediately (files are read fresh per request) — no Apache restart needed.
|
|
93
99
|
|
|
100
|
+
## BREAKING (2026-07-30): `error` is now an object, not a bare string
|
|
101
|
+
|
|
102
|
+
The V2 envelope's `error` changed from a bare string (`"EO-1"`) to an **object**:
|
|
103
|
+
|
|
104
|
+
```json
|
|
105
|
+
"error": { "code": "EO-1", "id": "1C-1" }
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
- **`code`** — the same short message code documented above.
|
|
109
|
+
- **`id`** — the quotable **Core Logs reference** for a *captured exception* (see
|
|
110
|
+
[error reporting](../../_underscore/features/error-reporting-issue-event.md)), and **`null`**
|
|
111
|
+
for a graceful rejection. `id` is **always present** so the shape is predictable — consumers
|
|
112
|
+
must not branch on its absence.
|
|
113
|
+
|
|
114
|
+
The object is assembled from **private** properties (`json_encode` emits public only) via
|
|
115
|
+
`setErrorReference()` + `refreshError()`, so it works whether the reference arrives **before or
|
|
116
|
+
after** the code — the controller sets the reference first. All six orderings were verified.
|
|
117
|
+
|
|
118
|
+
**The OAuth response subclass deliberately keeps the flat string form**
|
|
119
|
+
(`const USES_STRUCTURED_ERROR = false`). RFC 6749 §5.2 defines `error` as a single ASCII string,
|
|
120
|
+
and both classes share `addDefinedMessage()`, so emitting the object there would break every
|
|
121
|
+
conforming OAuth client library. Do not "unify" this.
|
|
122
|
+
|
|
123
|
+
**Also fixed here (CRITICAL): api2 returned `$e->getTrace()` to external callers on every 500.**
|
|
124
|
+
Unlike `getTraceAsString()`, `getTrace()` returns raw frame **arguments** — and the DB-bootstrap
|
|
125
|
+
catch has `_Database::register()` on the stack, so the **database username and password were
|
|
126
|
+
reachable by anyone who could force a 500**. `message` and `trace` are now gated behind
|
|
127
|
+
`_Environment::isDebugMode()`.
|
|
128
|
+
|
|
129
|
+
### Consumer audit
|
|
130
|
+
|
|
131
|
+
- **`library/app/api/toga2.php`** (the 1.0↔2.0 bridge) branches on `isSuccess` only, never
|
|
132
|
+
`error` — safe.
|
|
133
|
+
- **`toga2-supply`** — `ApiResponse<T>.error` is now `ApiError | null`; 30 envelope reads in
|
|
134
|
+
`OrdersApi.ts` became `.error?.code` (all 21 declarations there are
|
|
135
|
+
`await togaApiRequest(...)`, verified).
|
|
136
|
+
- **`toga-blox-npm` is not affected** — no `isSuccess` anywhere; it reads axios'
|
|
137
|
+
`response.data.data.…`.
|
|
138
|
+
- **Outstanding:** the external `API-DOCUMENTATION.md` lives outside the checkout and still
|
|
139
|
+
documents `error` as a string.
|
|
140
|
+
|
|
141
|
+
### Gotcha — do NOT regex-sweep `.error` → `.error?.code`
|
|
142
|
+
|
|
143
|
+
Only direct `togaApiRequest()` results are envelopes. These look identical and are **local
|
|
144
|
+
`{success, error}` shapes whose `error` is a plain string** — adding `?.code` breaks the failure
|
|
145
|
+
guard and swaps the real backend message for a generic fallback:
|
|
146
|
+
|
|
147
|
+
- `payload` in `UpdateShipmentApi.ts` (`response?.data?.itemFulfillments?.generateReturnLabel`)
|
|
148
|
+
- `shipmentResponse` from `fulfillShipment()`
|
|
149
|
+
- `response` in `EditShipmentForm.tsx` from `updateShipment()` / `saveShipment()`
|
|
150
|
+
- `messages[0].identifiers.error` — that is the message-identifiers map, not the envelope.
|
|
151
|
+
|
|
152
|
+
Verification gap: `toga2-supply` had no `node_modules`, so `tsc` could not be run.
|
|
153
|
+
**`npm run build` is the definitive check** — the type change will surface anything missed.
|
|
154
|
+
|
|
94
155
|
## Related recipes
|
|
95
156
|
|
|
96
157
|
- **Add a writable field to a V2 record (3-file migration)** and **expose a scripted API**
|
|
@@ -103,6 +164,17 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
|
|
|
103
164
|
|
|
104
165
|
## Change history
|
|
105
166
|
|
|
167
|
+
- 2026-07-30 — **BREAKING:** envelope `error` changed from a bare string to
|
|
168
|
+
`{ code, id }`, where `id` is the quotable Core Logs reference for a captured exception and
|
|
169
|
+
`null` for a graceful rejection (always present, for a predictable shape); assembled from
|
|
170
|
+
private properties via `setErrorReference()`/`refreshError()` so code and reference can arrive
|
|
171
|
+
in any order. The OAuth subclass keeps the flat string form
|
|
172
|
+
(`USES_STRUCTURED_ERROR = false`) per RFC 6749 §5.2. Fixed a CRITICAL disclosure: api2
|
|
173
|
+
returned `$e->getTrace()` (raw frame **arguments**, including `_Database::register()`'s DB
|
|
174
|
+
username/password) to external callers on every 500 — `message`/`trace` are now gated behind
|
|
175
|
+
`_Environment::isDebugMode()`. Recorded the consumer audit (library bridge and toga-blox-npm
|
|
176
|
+
unaffected; toga2-supply migrated) and the regex-sweep gotcha for look-alike local
|
|
177
|
+
`{success, error}` shapes. (jcardinal)
|
|
106
178
|
- 2026-07-29 — Added **EV-13** (`recordsPerPage` over the 10000 max) and **diagnosis note 6**:
|
|
107
179
|
what a denial looks like from the client side — `EZ-1` = record-level (`data.<route>` null,
|
|
108
180
|
and a bare `allowRead=1` with no logic group still returns it); `EZ-2` = field-level,
|
|
@@ -18,6 +18,7 @@
|
|
|
18
18
|
| [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
|
|
19
19
|
| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
|
|
20
20
|
| [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
|
|
21
|
+
| [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Config/production.ini, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
|
|
21
22
|
| [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
|
|
22
23
|
| [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
|
|
23
24
|
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Error Escalation Cron (Errors::Escalate → ClickUp / email)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-30
|
|
10
|
+
owners: ["jcardinal"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Infrastructure/Errors.php
|
|
13
|
+
- worker2/Worker/Clickup/ErrorTask.php
|
|
14
|
+
- worker2/Worker/Clickup.php
|
|
15
|
+
- worker2/Config/production.ini
|
|
16
|
+
- dbchanges2/Core/2026-07-30a - Error escalation cron job.sql
|
|
17
|
+
related:
|
|
18
|
+
- ../../_underscore/features/error-reporting-issue-event.md
|
|
19
|
+
- ../../../1.0/apps/tools/features/errors-curation-console.md
|
|
20
|
+
- ./creating-worker-actions.md
|
|
21
|
+
- ./clickup-project-routing.md
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Summary
|
|
25
|
+
|
|
26
|
+
`_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of
|
|
27
|
+
**escalation, de-escalation, ClickUp ticketing, reminders, business email, owner learning, and
|
|
28
|
+
GC** for the Issue/Event error-reporting pipeline. Capture is `_underscore`'s job (see
|
|
29
|
+
[error-reporting-issue-event](../../_underscore/features/error-reporting-issue-event.md)); the
|
|
30
|
+
Tools `/errors` console curates but **notifies nobody**. All outbound notification funnels
|
|
31
|
+
through this one action, so there is exactly one place to mute or debug alerting.
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
### Urgency = max of two independent axes, plus a floor
|
|
36
|
+
|
|
37
|
+
- **Volume** — occurrences in a 10-minute window: **20 / 50 / 100**.
|
|
38
|
+
- **Neglect** — hours an *unacknowledged* ClickUp task has sat: **1 / 4 / 24**.
|
|
39
|
+
- **Floor** — the Issue's declared `minimumUrgency` applies under *both* axes.
|
|
40
|
+
|
|
41
|
+
The neglect axis exists because volume alone cannot see a rare-but-serious error, nor a serious
|
|
42
|
+
error that everyone is ignoring.
|
|
43
|
+
|
|
44
|
+
### Fast up, slow down
|
|
45
|
+
|
|
46
|
+
Escalation applies **the same run** a band is crossed. De-escalation requires the count to sit
|
|
47
|
+
under a **lower release band (12 / 35 / 70)** for **3 consecutive windows**. Without that gap,
|
|
48
|
+
an issue hovering near a threshold flaps every minute and burns a ClickUp write each time.
|
|
49
|
+
|
|
50
|
+
### Task-creation gate
|
|
51
|
+
|
|
52
|
+
A ClickUp task is created only when urgency ≥ NORMAL, **or** a floor is set, **or** the Issue is
|
|
53
|
+
curated, **or** it is still rare. Creating a task for every distinct error on first sight is what
|
|
54
|
+
buries a team.
|
|
55
|
+
|
|
56
|
+
### Reminders
|
|
57
|
+
|
|
58
|
+
Backed off **30m → 1h → 4h → daily**. Never sent on de-escalation. Suppressed entirely while
|
|
59
|
+
`dtMutedUntil` is in the future.
|
|
60
|
+
|
|
61
|
+
### Sprint promotion
|
|
62
|
+
|
|
63
|
+
Uses the ClickUp **v2 multi-list** "Add Task To List" endpoint
|
|
64
|
+
(`POST /list/{id}/task/{taskId}`) so the errors list stays the task's **permanent home**. The v3
|
|
65
|
+
`home_list` move endpoint was rejected: v3 is not reliably deployed, it needs a workspace id
|
|
66
|
+
that exists nowhere in the codebase, and it requires `status_mappings` maintenance.
|
|
67
|
+
|
|
68
|
+
The target sprint list is resolved from `[clickup] sprint_folder_id` because `Team.Sprints`
|
|
69
|
+
carries no ClickUp list id — `GET /folder/{id}/list`, then the list whose `start_date`/`due_date`
|
|
70
|
+
range contains today.
|
|
71
|
+
|
|
72
|
+
### Recurrence (episodes)
|
|
73
|
+
|
|
74
|
+
One `IssueClickupTasks` row per ClickUp **episode**. Closing the task fires a webhook that stamps
|
|
75
|
+
`dtResolved` and increments `recurrenceCount`. A later occurrence opens a **new** task linked to
|
|
76
|
+
its predecessors via `POST /task/{id}/link/{links_to}` — a **link, not a dependency**; a
|
|
77
|
+
dependency would *block* the task. A cooldown prevents recreating a task 60 seconds after
|
|
78
|
+
someone closed it while the error was still firing.
|
|
79
|
+
|
|
80
|
+
### Acknowledgement
|
|
81
|
+
|
|
82
|
+
ClickUp is the acknowledge/resolve system — developers already live there, and a second inbox in
|
|
83
|
+
Tools would just be ignored. **Any** status change on an error task counts as acknowledgement,
|
|
84
|
+
which is safe only because the automation never sets status itself.
|
|
85
|
+
|
|
86
|
+
### Area-level owner learning (observation only)
|
|
87
|
+
|
|
88
|
+
`IssueAreaOwners` learns an owner after **5 in a row** and ships **observation-only** — a hint in
|
|
89
|
+
the task body, never an auto-assignment. Misassignment is expensive: a few wrong tickets and
|
|
90
|
+
developers stop trusting the queue. Learning is per **area**, not per issue, because one issue
|
|
91
|
+
rarely recurs enough to learn from before it is fixed.
|
|
92
|
+
|
|
93
|
+
### Garbage collection (split, deliberately)
|
|
94
|
+
|
|
95
|
+
- **Events** purge at **7 days**.
|
|
96
|
+
- **Issues** purge only when nothing curated them: not `isManaged`, no `issueKey`, no floor, no
|
|
97
|
+
recipients, no ClickUp history, no events.
|
|
98
|
+
|
|
99
|
+
The original blanket 30-day delete would have **cascaded a configured business issue's email
|
|
100
|
+
recipients away** after a quiet month.
|
|
101
|
+
|
|
102
|
+
## Configuration
|
|
103
|
+
|
|
104
|
+
- **`[_underscore] error_escalation_enabled`** — added to **all 8** worker2 configs: `1` in
|
|
105
|
+
`production.ini` **only**, `0` elsewhere. It is the master switch for **all outbound
|
|
106
|
+
notification**; errors are still **captured** regardless, so non-production records everything
|
|
107
|
+
and notifies nobody.
|
|
108
|
+
- Gated on a config key rather than string-matching `_Environment::$name` because
|
|
109
|
+
`_Environment` has **no `isProduction()`** — only a raw name string and an ad-hoc
|
|
110
|
+
`substr($name, 0, 4) == 'dev-'` idiom. Guessing env names would silently spam, or silently
|
|
111
|
+
stay quiet, in the wrong place.
|
|
112
|
+
- Read via `_Config::_underscore(key, false)`. **The second falsy argument matters:** it makes
|
|
113
|
+
`_Config` return `false` for a missing key/group **instead of throwing**, and an uncaught
|
|
114
|
+
throw would fail the whole cron.
|
|
115
|
+
- **`[clickup] sprint_folder_id`** — added to `worker2/Config/production.ini`, left **empty with
|
|
116
|
+
instructions**. Note `[clickup]` exists in only **2 of the 8** worker2 configs.
|
|
117
|
+
|
|
118
|
+
## Gotchas / known issues
|
|
119
|
+
|
|
120
|
+
- **The working-set query must be a UNION, not an OR.** It is a UNION of three separately
|
|
121
|
+
indexable branches. MySQL will **not** `index_merge` across an `OR` when one branch is a
|
|
122
|
+
correlated `EXISTS`, so the OR form **full-scans `Issues` every 60 seconds, forever**.
|
|
123
|
+
- **`INSERT ... SELECT ... WHERE NOT EXISTS` is not a concurrency guard** under REPEATABLE READ.
|
|
124
|
+
The open-episode guard is a UNIQUE key on a VIRTUAL generated column, and the code **claims
|
|
125
|
+
the episode before calling ClickUp** so a losing run fails before creating a duplicate
|
|
126
|
+
external task — see the `_underscore` doc for the full mechanics.
|
|
127
|
+
- **Business email must be scoped per client.** Recipients were resolved from the **most recent
|
|
128
|
+
event's** client, so with two clients firing in the same window **client B's business users
|
|
129
|
+
received email about client A's data**. It now sends one email **per distinct client that
|
|
130
|
+
actually fired in the window**.
|
|
131
|
+
- **Business email bodies/subjects must not carry `errorMessage` or `trace`.** Both are seeded
|
|
132
|
+
once from whichever client created the Issue and are never re-scoped. An uncurated business
|
|
133
|
+
issue gets a **neutral** subject rather than another client's raw error text.
|
|
134
|
+
- **`IssueEmailAddresses.clientId = 0` means "all clients"** and requires an explicit
|
|
135
|
+
confirmation flag in the Tools console; list views render it as **"ALL CLIENTS"**.
|
|
136
|
+
- **The ClickUp webhook has no HMAC signature verification** (pre-existing, unfixed). A forged
|
|
137
|
+
POST can mark issues acknowledged — freezing the neglect axis — or resolve episodes. That is
|
|
138
|
+
alert suppression against the error-reporting system itself. Fix before relying on this
|
|
139
|
+
pipeline as the only alerting path.
|
|
140
|
+
- **`_Worker_Sentry::Webhook` writes into the same ClickUp list (`901110669877`)** this cron
|
|
141
|
+
targets. Both pipelines run in parallel by design until the Sentry-removal ticket lands, so
|
|
142
|
+
expect duplicate-looking tasks from two sources.
|
|
143
|
+
|
|
144
|
+
## Change history
|
|
145
|
+
|
|
146
|
+
- 2026-07-30 — Built as part of TRUE-78188: action renamed `SyncWithClickup` → `Escalate`; new
|
|
147
|
+
`Worker/Clickup/ErrorTask.php`. Two-axis urgency (volume 20/50/100 + neglect 1/4/24) with a
|
|
148
|
+
`minimumUrgency` floor; fast-up/slow-down with release bands 12/35/70 over 3 windows; a
|
|
149
|
+
task-creation gate; 30m→1h→4h→daily reminders honouring `dtMutedUntil`; sprint promotion via
|
|
150
|
+
the v2 multi-list endpoint (v3 `home_list` rejected); episode recurrence via task *links* not
|
|
151
|
+
dependencies; a UNION (not OR) working-set query; and split GC (Events 7d, Issues only when
|
|
152
|
+
uncurated). Fixed from review: per-client business email scoping (client B was receiving
|
|
153
|
+
client A's data), neutral subjects with no errorMessage/trace, seeding new issues at their
|
|
154
|
+
declared floor, `_String::generateUuid()` over MySQL `UUID()`, and the row-alias form of
|
|
155
|
+
`ON DUPLICATE KEY UPDATE`. Added `error_escalation_enabled` (all 8 configs) and
|
|
156
|
+
`[clickup] sprint_folder_id`. (jcardinal)
|
|
157
|
+
</content>
|
|
@@ -393,6 +393,9 @@ Signed integers can reference positive and negative numbers, but unsigned can on
|
|
|
393
393
|
1. **Use the `_Model` layer.** The metadata-driven ORM builds and escapes queries for you — prefer it over hand-written SQL whenever possible.
|
|
394
394
|
2. **When you must write SQL by hand, escape every interpolated value with `_Database::escape()`** before placing it in the query string.
|
|
395
395
|
* `_Database::escape()` adds MySQL escape characters (`\`, `'`, `"`, newlines, etc.) and is the correct helper for query interpolation. Do **not** confuse it with `_Database::protect()`, which *strips* `% \ / * " '` and the literal ` or` — that is lossy filtering, not escaping, and will corrupt data you intend to store intact.
|
|
396
|
+
* **Prepared statements are not available.** `_Query` accepts only a raw SQL string, and `_Database_Driver` declares no `prepare`/`bind` (verified 2026-07-30 — neither the Mysql nor the Pgsql driver implements one). Escaping and casting *are* the parameterization mechanism in 2.0; do not go looking for a bind API, and do not trust any guidance that names a `_Db` class — there isn't one.
|
|
397
|
+
* **Escaping protects values, not identifiers.** Table names, column names, sort directions, and `LIMIT` expressions must come from a **hardcoded allowlist** — never from a request, and never merely escaped. `ORDER BY " . _Database::escape($_GET['sort'])` is still injectable.
|
|
398
|
+
* **Validate enum values in PHP.** `FIELD_LIST` performs no enum validation and converts any falsy value to SQL `NULL`; MySQL coerces rather than rejects an unknown member. Check against the allowed set before the value reaches the query — a typo'd urgency becomes `NULL`, not an error.
|
|
396
399
|
|
|
397
400
|
```php
|
|
398
401
|
// Bad — raw interpolation
|
package/knowledge/INDEX.md
CHANGED
|
@@ -14,12 +14,12 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
14
14
|
- **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
|
|
15
15
|
- **test** (Test) — 13 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
|
|
16
16
|
- **toga** (TOGa) — 2 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
|
|
17
|
-
- **tools** (Tools) —
|
|
17
|
+
- **tools** (Tools) — 12 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
|
|
18
18
|
|
|
19
19
|
## 2.0 framework
|
|
20
20
|
|
|
21
21
|
- **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
22
|
-
- **worker2** (Worker) —
|
|
22
|
+
- **worker2** (Worker) — 38 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
23
23
|
- **api2** (API) — 20 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
24
24
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
25
25
|
- **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: TRUE-80487-Debug-Fix-Rate-WH-Warranty
|
|
4
|
+
title: Rate Whole Home Warranty per-address dedup — root-caused "works in beta, fails in prod"
|
|
5
|
+
author: mhammontree
|
|
6
|
+
repos: [_underscore, toga2-view, dbchanges2, test]
|
|
7
|
+
framework: "2.0"
|
|
8
|
+
client: rate
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-07-30
|
|
11
|
+
updated: 2026-07-30
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: TRUE-80487-Debug-Fix-Rate-WH-Warranty
|
|
15
|
+
**Date:** 2026-07-30
|
|
16
|
+
**Project/Repo:** _underscore / toga2-view / dbchanges2 / test (2.0)
|
|
17
|
+
**Task:** Production troubleshooting of the Rate Whole Home Warranty per-address duplicate-purchase guard (TRUE-79533, already deployed), reported as "works in beta, not in production" — root-caused to three independent code defects, none of them a deployment gap.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
|
|
23
|
+
- **Eliminated every deployment-gap hypothesis with hard evidence.** `_underscore` `_production` HEAD is `ba7eff66` (TRUE-79533 PR #661 merged 2026-07-29); `dbchanges2` `_main` is `138df5e` (PR #418). `git diff origin/_beta -- Model/Rate/Entitlement.php` returns **zero diff** — prod and beta run identical code.
|
|
24
|
+
- **Confirmed all prod DB artifacts present.** `Core.RecordFields` id **2482** = `entitlements.serviceAddressId` registered; `Client_Rate.Entitlements.serviceAddressId` column + index exist; ACL grants present (`2482` → role 1 `isWritable=0`, mirroring `saleItemId` 1265; `addresses.id` field 41 → roles 1,2,3). `ApiPayloadInterceptors` in prod is **byte-identical to dev-sandbox** (id 4 = rec 191 PRE/POST `minDepth NULL`; id 2 = rec 191 POST/POST `minDepth 5`).
|
|
25
|
+
- **ROOT CAUSE #1 — the backend guard has never executed, in any environment.** `isWholeHomeWarranty()` at `_underscore/Model/Rate/Entitlement.php:116-118` reads `$payload->saleItem->title`, but the real checkout payload sends `"saleItem":{"uuid":"3e14effa-0f78-4708-b338-60872a39aff3"}` with **no `title`**. `stripos('', 'warranty')` → false → `prePost` returns at line 58 before the guard runs. Evidence: beta payload for entitlement 106, `Logs_Rate.Api` id 48823 (dev-sandbox, 2026-07-27 11:41:15), verbatim.
|
|
26
|
+
- **Proved it with beta's own data.** dev-sandbox has **three active WH entitlements at the same address** `1864 HIGH GROVE LN 60540`: `ET100057` (id 90, 2026-07-24), `ET100067` (id 100, **2026-07-27 10:10**), `ET100073` (id 106, **2026-07-27 11:41**) — two created on the very day the feature was marked "beta-verified". Plus two at `121 N LA SALLE ST 60602` (`ET100071`, `ET100072`). All same borrower `5686507f-a7b2-4fca-b007-dd3ae875e4bb`, all pinned, all active.
|
|
27
|
+
- **Confirmed the guard's rejection message appears in NO log, in either environment, ever.** All 8 beta `POST /v2/entitlements` 400s are `EV-12` (reference errors) and one `EV-8` (`contractFulfillmentPreference`, 2026-07-15, pre-dating the PRE interceptor registration).
|
|
28
|
+
- **ROOT CAUSE #2 — `state.code` vs `state.uuid`.** `prePost:66` reads `$address->state->code`, but the payload sends `"state":{"uuid":"94417b26-..."}`. Fixing #1 alone would make every WH purchase throw *"A valid service address is required"* — a false rejection on valid addresses. **Both must be fixed together.**
|
|
29
|
+
- **ROOT CAUSE #3 — frontend availability check fails open AND is borrower-scoped.** `toga2-view/src/pages/ZipValidation/viewModels/useZipValidationViewModel.ts:135-139` — `const address = entitlement.serviceAddress; if (!address?.line1) return false;` and `serviceAddress` is derived only from the pin (`useActiveServices.ts:108`). Worse, `useActiveServices.ts:69-71` scopes the query `where: { and: [{ "Customers.c_borrowerId": { "=": borrowerId } }] }` — so it can **never** see another borrower's warranty, while the backend guard is explicitly global. Its code comment claims "same source as the backend guard's dedup join" — that comment is wrong; the backend uses `COALESCE(serviceAddressId, ContactAddresses.addressId)`, the frontend has no fallback.
|
|
30
|
+
- **Verified the backend WOULD block, without paying.** Replicated `hasActiveWarrantyAtAddress()` SQL read-only against prod for `1864 HIGH GROVE LN / Naperville / 60540 / IL` → returns entitlement **43 (`ET100017`)**. So a real purchase reaches PayPal, the customer is charged, and *then* the entitlement POST is rejected.
|
|
31
|
+
- **Confirmed the apartment-building exposure is real.** Property type lives only in router state / `sessionStorage` (`useCheckoutPageViewModel.ts:178-205`) and is **never sent to the API** — the purchase payload contains no `propertyType` / `isEligible`. `prePost:68-70` requires line1/city/state/zip but **not** `line2`. The `requireUnitNumber` flow (`ZipValidation.tsx:62-75`, `198-199`) is client-side only and bypassable. Live proof: beta `ET100073` carries `line2: "STE 500"` on a single-family home, while ids 90/100 at the same `line1` have no unit.
|
|
32
|
+
- **Production data repaired by the developer and verified clean.** Three active WH warranties, all pinned + validated, both-or-neither invariant holds on all nine rows: 43 (`ET100017`) → addr 9 Naperville, 40 (`ET100014`) → addr 6 Nashua, 38 (`ET100012`) → addr 4 Wailuku. Subscription 22 (`ET100015`, Daniel Moran) closed out `isActive=0, dateCancelled='2026-06-03'`; address 7 `city` corrected `60657` → `Chicago`.
|
|
33
|
+
|
|
34
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
35
|
+
|
|
36
|
+
- **"A migration is missing in production."** DISPROVEN. `Core.RecordFields` 2482 exists; `Client_Rate.Entitlements.serviceAddressId` column + index exist; both ACL grants landed. Do not re-audit `Core/2026-07-22a`, `Client/2026-07-22a`, `Client/2026-07-23a`, `Client_Rate/2026-07-23a`, `Client_Rate/2026-07-24a` — all applied.
|
|
37
|
+
- **"TRUE-79251 merge damage deleted the WH helper methods."** DISPROVEN. `git diff origin/_beta -- Model/Rate/Entitlement.php` → zero diff. `PASSTHROUGH_KEY_FULFILLMENT_PREFERENCE` (line 32) and all three private helpers (`normalizedField` 107, `isWholeHomeWarranty` 116, `hasActiveWarrantyAtAddress` 137) present.
|
|
38
|
+
- **"`_underscore` prod wasn't redeployed past the uuid-clobber fix `9c2e4b1c`."** DISPROVEN by the same zero diff + PR #661 merged.
|
|
39
|
+
- **"`toga2-view` prod still sends `{depth:-1}` and starves `postPost`."** DISPROVEN. `genericApi.ts:54,68` only append `?depth=-1` when `returnPayloadDepth` is explicitly set; no checkout override. Diff vs `_beta` touches only GetSupport files.
|
|
40
|
+
- **"Cross-cluster `Core.RecordFields` references from `Client*/` migration folders fail in prod."** DISPROVEN for this case. Despite prod splitting `prod-core` / `prod-client` / `prod-logs` / `prod-archive` onto separate clusters, the ACL grants landed correctly. ~20 long-standing ACL migrations use this same pattern. (Note: the `dbchanges2-cluster-isolation` hook was NOT found under `.claude/hooks/toga/` — separate follow-up, not this bug.)
|
|
41
|
+
- **"The interceptor row is missing, inactive, or duplicated in prod."** DISPROVEN — prod rows are identical to dev-sandbox, same ids and uuids.
|
|
42
|
+
- **"`POST /v2/entitlement-sales-orders` is the prod purchase path (record 257, no interceptor)."** DISPROVEN. It is only a bridge link: payload is `{"entitlement":{"uuid"},"salesOrder":{"uuid"}}`. It creates no entitlement and fires no interceptor.
|
|
43
|
+
- **"Production simply hasn't processed a purchase since deploy, so there's nothing to debug."** TRUE as a fact (last `POST /v2/entitlements` in prod = 2026-07-10, two × 400 `EV-8`; newest `Entitlements` row 2026-06-15) but WRONG as a conclusion — the developer then reproduced the failure live through the UI. Do not stop at this observation.
|
|
44
|
+
- **Query that hangs — do not run.** `SELECT ... FROM Logs_Rate.Api WHERE route LIKE '%address%'` timed out after **300s** (unindexed full scan on a ~12.7M-row table). Always constrain with the `dtStamp` index: `WHERE dtStamp >= '...' AND dtStamp < '...'`.
|
|
45
|
+
- **Columns that do not exist (queries will error).** `Client_Rate.ApiPayloadInterceptors.phpMethod` — absent in prod (yet `Client_Rate/2025-12-10 - PostInterceptor.sql` inserts it, so that file is not re-runnable). `Client_Rate.Contacts.emailAddress` — does not exist. `Logs_Rate.Api.uri` — the column is `route`.
|
|
46
|
+
- **Widening the frontend `useActiveServices` query to all borrowers.** REJECTED, do not do this — it would let any portal user read other customers' entitlements and addresses. A data-leak fix worse than the bug.
|
|
47
|
+
|
|
48
|
+
## Not tried yet (candidates for next session)
|
|
49
|
+
|
|
50
|
+
- Fix `isWholeHomeWarranty()` to resolve `saleItem.uuid` → item (fall back to payload `name` and/or `WH_SALE_ITEM_ID = 3`), matching the authority `hasActiveWarrantyAtAddress()` already uses in SQL.
|
|
51
|
+
- Fix state resolution to accept `state.uuid` with `state.code` as fallback.
|
|
52
|
+
- Normalize zip to first 5 digits in the dedup comparison (`Entitlement.php:161` is exact-match; frontend already truncates at `addressCompare.ts:22`). Untested risk: USPS returning `60540-9233` vs stored `60540`.
|
|
53
|
+
- Normalize the unit designator before comparison so `STE 500` / `Suite 500` / `#500` / `Unit 500` collapse to one dwelling.
|
|
54
|
+
- Build the server-side global pre-check: boolean-only scripted endpoint reusing `hasActiveWarrantyAtAddress()`, called from the availability step, deleting the frontend's local dedup. Needs a `dbchanges2` migration registering it in `Core.RecordScripts` **plus** the client ACL grant — note `validateAddress` (RecordScripts id 17) exists only in the live DB and NOT in `dbchanges2`, the exact parity trap to avoid.
|
|
55
|
+
- Log one raw USPS / FedEx / UPS validation response for a known apartment vs a known single-family address, to settle field names empirically. `Fedex.php:417-421` already receives the full `attributes` object and reads only `Matched` / `Resolved`; `Ups.php:801-807` reads only `NoCandidatesIndicator` / `Candidate`.
|
|
56
|
+
- Capture the USPS DPV "valid but secondary unit missing" signal into `validateAddress`'s normalized return with an explicit unknown state, and enforce in `prePost`.
|
|
57
|
+
- Send and persist `propertyType` with the purchase so eligibility is enforced server-side rather than trusted from the browser.
|
|
58
|
+
- Update `test/@Mark/Rate/verify_wholehome_per_address_guard.php` to use the REAL payload shape (uuid-only `saleItem`, uuid-only `state`). It currently passes 18/18 against a shape production never sends — this is why all three defects survived review.
|
|
59
|
+
- Clean up beta duplicates: dev-sandbox entitlements 100, 105, 106.
|
|
60
|
+
- Controlled prod WH purchase to exercise the untested `minDepth = 5` on the POST interceptor, and to test USPS "Suggested" (`60540-9233`) vs "You entered" (`60540`).
|
|
61
|
+
|
|
62
|
+
## Current file state
|
|
63
|
+
|
|
64
|
+
| File | Status | Notes |
|
|
65
|
+
|------|--------|-------|
|
|
66
|
+
| `_underscore/Model/Rate/Entitlement.php` | **investigated, NOT modified** | Holds root causes #1 (line 116-118) and #2 (line 66). Also: raw SQL string concatenation at lines 371/375 violates the parameterized-query standard; accepted TOCTOU noted at 133-135. |
|
|
67
|
+
| `toga2-view/src/pages/ZipValidation/viewModels/useZipValidationViewModel.ts` | **investigated, NOT modified** | Root cause #3 fail-open at 135-139. Misleading comment claims parity with backend join. |
|
|
68
|
+
| `toga2-view/src/hooks/useActiveServices.ts` | **investigated, NOT modified** | Borrower-scoped `where` at 69-71; pin-only `serviceAddress` resolution at 108; silent EZ-2 degrade at 90. |
|
|
69
|
+
| `toga2-view/src/pages/CheckOut/viewModel/useCheckoutPageViewModel.ts` | **investigated, NOT modified** | `propertyType` read from router state / sessionStorage (178-205), never sent to API. |
|
|
70
|
+
| `_underscore/Model/Client/Address.php` | **investigated, NOT modified** | `validateAddress` waterfall USPS→FedEx→UPS (31-46); returns success + normalized fields only, no unit-required signal. |
|
|
71
|
+
| `_underscore/Component/Library/Carriers/{Usps,Fedex,Ups}` | **investigated, NOT modified** | Signals available but discarded. NOTE: this is a namespace inside `_underscore`, NOT the 1.0 `library` repo. |
|
|
72
|
+
| `test/@Mark/Rate/verify_wholehome_per_address_guard.php` | **not opened** | Passes 18/18 against the wrong payload shape. Needs rewrite. |
|
|
73
|
+
| **Production `Client_Rate` data** | **CHANGED by developer, verified** | Ents 43→addr 9, 40→addr 6, 38→addr 4 pinned; those 3 addresses `isValidated=1`; subscription 22 closed `isActive=0, dateCancelled='2026-06-03'`; address 7 `city` → `Chicago`. No repo files changed this session. |
|
|
74
|
+
|
|
75
|
+
## Decisions made
|
|
76
|
+
|
|
77
|
+
- **Keep `dateEnd` OUT of the dedup predicate.** Rationale: the guard's contract is `isActive = 1 AND dateCancelled IS NULL`; adding `dateEnd` would let real duplicates through when a renewal hasn't rolled `dateEnd` forward yet. Rejected alternative: filter on `dateEnd`. Confirmed instance of the underlying problem: Kimberly Stearns (`ET100014`) has `dateEnd = 2026-06-01` yet was charged 2026-07-01 — renewals are not advancing `dateEnd`. **Separate ticket.**
|
|
78
|
+
- **Do NOT widen the frontend query to all borrowers; move the check server-side instead.** Rationale: the frontend fundamentally cannot answer "does anyone have a warranty here?" without exposing other customers' data. A boolean-only endpoint reusing the existing helper makes availability and purchase the same code path, so they cannot drift.
|
|
79
|
+
- **Skip the legacy-entitlement backfill migration.** Rationale: only two rows mattered and they were fixed by hand; once the server-side check ships, pinning is no longer required for dedup correctness. Migration would be pure data churn.
|
|
80
|
+
- **Do not chase "dwelling type" from shipping carriers.** Rationale: postal data models deliverability, not property characteristics — no carrier returns single-family vs multi-unit. UPS `AddressClassification` / FedEx `classification` only give Residential vs Commercial, and an apartment building is "Residential". The actionable signal is USPS DPV "secondary unit required but missing". True dwelling type needs property data (assessor/parcel, or Melissa/Smarty-class vendor) — a procurement decision for Paulina + underwriting.
|
|
81
|
+
- **Did NOT complete the PayPal test payment.** Rationale: the backend rejects *after* the charge; the read-only SQL replication already proved what the guard decides.
|
|
82
|
+
- **Branch naming = bare ticket ID `TRUE-80487`.** Rationale: matches live convention in all four repos (`TRUE-78188`, `TRUE-78314`, `TRUE-79191`, `TRUE-79868`). The documented `fix/short-description` rule in the git-workflow standard is stale — flag for correction.
|
|
83
|
+
- **Repo scope:** minimum `_underscore` + `test` (makes the global rejection actually work); add `dbchanges2` + `toga2-view` for the server-side pre-check. `api2` NOT needed (scripted endpoints are served by the V2 engine from `_underscore` models; registration is DB-side). Recommendation: branch all four — an unused branch is free, discovering a needed migration mid-sprint is not.
|
|
84
|
+
- **Reframed the ticket premise.** "Works in beta, fails in prod" is false. Same code, same migrations, same interceptor rows. Beta only *appeared* to work because its test data satisfied both frontend preconditions by accident — a single test borrower, and data created after the pin shipped. Production satisfied neither.
|
|
85
|
+
|
|
86
|
+
## Blockers
|
|
87
|
+
|
|
88
|
+
- **Need the current dev branch name** to base the four `TRUE-80487` branches on. `_underscore` shows `_beta`; `toga2-view` shows both `_beta` and `_stage`. Developer said they are on "a different dev branch now" — unresolved.
|
|
89
|
+
- **Need a product decision from Paulina: per-dwelling or per-building coverage?** Currently `123 Main St` and `123 Main St APT 2` are distinct addresses, so both can hold a warranty. This changes what the fix does. Sharpened by the finding that a *no-unit* purchase at a multi-unit address is indistinguishable from a single-family purchase and carries building-scale liability at single-home pricing.
|
|
90
|
+
- **Need a scope decision:** does property-type / unit-required enforcement land in TRUE-80487, or a separate ticket? Materially larger than "one warranty per address" — it is "what counts as an insurable address". The gate fixes are independent of both blockers and can proceed regardless.
|
|
91
|
+
|
|
92
|
+
## Exact next step
|
|
93
|
+
|
|
94
|
+
> Create the `TRUE-80487` branch on `_underscore` (base = the dev branch the developer names), then fix `isWholeHomeWarranty()` at `_underscore/Model/Rate/Entitlement.php:116-118` to identify a WH purchase by resolving `saleItem.uuid` → `Items` (with payload `name` / `WH_SALE_ITEM_ID = 3` as fallbacks) **together with** the `state.uuid` → `code` resolution at line 66 — never one without the other, or every WH purchase throws "A valid service address is required". Then run `test/@Mark/Rate/verify_wholehome_per_address_guard.php` rewritten to the real payload shape (uuid-only `saleItem` and `state`), and confirm it FAILS on the current code before the fix and passes after.
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
_Saved by /session-save on 2026-07-30_
|
package/package.json
CHANGED
package/rules/common/security.md
CHANGED
|
@@ -4,24 +4,74 @@ These rules are critical. Violations create exploitable vulnerabilities. Follow
|
|
|
4
4
|
|
|
5
5
|
## SQL injection prevention
|
|
6
6
|
|
|
7
|
-
Never build SQL strings by concatenating or interpolating
|
|
7
|
+
Never build SQL strings by concatenating or interpolating **unescaped** values.
|
|
8
|
+
|
|
9
|
+
> **Neither framework supports prepared statements.** This rule previously demanded them and
|
|
10
|
+
> gave `_Db::select()` as the correct pattern. **`_Db` does not exist.** Verified 2026-07-30:
|
|
11
|
+
> in 2.0, `_Query` accepts only a raw SQL string and `_Database_Driver` declares no
|
|
12
|
+
> `prepare`/`bind` (neither the Mysql nor the Pgsql driver implements one); 1.0 is the same
|
|
13
|
+
> through `App_Database`. There are zero uses of `bind_param`, `mysqli_stmt`, `PDO::`, or
|
|
14
|
+
> `::prepare(` anywhere in the non-vendor codebase. Escaping **is** the parameterization
|
|
15
|
+
> mechanism here, so that is what this rule requires.
|
|
16
|
+
|
|
17
|
+
**Prefer the model layer.** `_Model` (2.0) and `App_Model` (1.0) build and escape queries for
|
|
18
|
+
you. Hand-written SQL is the fallback, not the default.
|
|
19
|
+
|
|
20
|
+
**When you write SQL by hand, every interpolated value must be escaped or cast — no exceptions
|
|
21
|
+
for "internal" or "trusted" sources.**
|
|
8
22
|
|
|
9
23
|
```php
|
|
10
24
|
// CRITICAL VIOLATION — never do this
|
|
11
|
-
$sql = "SELECT * FROM
|
|
12
|
-
$sql = "SELECT * FROM
|
|
25
|
+
$sql = "SELECT * FROM Users WHERE id = " . $_GET['id'];
|
|
26
|
+
$sql = "SELECT * FROM Orders WHERE email = '$email'";
|
|
27
|
+
|
|
28
|
+
// CORRECT (2.0) — cast numerics, escape strings
|
|
29
|
+
$sql = "
|
|
30
|
+
SELECT *
|
|
31
|
+
FROM Orders
|
|
32
|
+
WHERE
|
|
33
|
+
id = " . ((int) $_GET['id']) . " AND
|
|
34
|
+
email = '" . _Database::escape($email) . "'
|
|
35
|
+
";
|
|
36
|
+
|
|
37
|
+
// CORRECT (1.0) — same shape, different helper
|
|
38
|
+
$sql = "
|
|
39
|
+
SELECT *
|
|
40
|
+
FROM Orders
|
|
41
|
+
WHERE
|
|
42
|
+
id = " . ((int) $_GET['id']) . " AND
|
|
43
|
+
email = '" . App_Database::sqlEscape($email) . "'
|
|
44
|
+
";
|
|
45
|
+
```
|
|
13
46
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
47
|
+
Applies to every operation: SELECT, INSERT, UPDATE, DELETE, and DDL.
|
|
48
|
+
|
|
49
|
+
**Cast numerics with `(int)`/`(float)`; escape strings.** A cast is sufficient on its own for a
|
|
50
|
+
numeric — there is nothing left to inject once the value is an integer. A string always needs the
|
|
51
|
+
escape helper.
|
|
52
|
+
|
|
53
|
+
**Identifiers are not covered by escaping.** `_Database::escape()` and
|
|
54
|
+
`App_Database::sqlEscape()` protect *values*, not table names, column names, sort directions, or
|
|
55
|
+
`LIMIT` expressions. Anything used as an identifier must come from a **hardcoded allowlist** —
|
|
56
|
+
never from a request, and never merely escaped.
|
|
57
|
+
|
|
58
|
+
```php
|
|
59
|
+
// CRITICAL VIOLATION — escaping does not make an identifier safe
|
|
60
|
+
$sql = "SELECT * FROM Orders ORDER BY " . _Database::escape($_GET['sort']);
|
|
17
61
|
|
|
18
|
-
// CORRECT —
|
|
19
|
-
|
|
62
|
+
// CORRECT — allowlist
|
|
63
|
+
$allowedSortColumns = ['dtCreated', 'total', 'status'];
|
|
64
|
+
$sortColumn = in_array($_GET['sort'], $allowedSortColumns, true) ? $_GET['sort'] : 'dtCreated';
|
|
65
|
+
$sql = "SELECT * FROM Orders ORDER BY $sortColumn";
|
|
20
66
|
```
|
|
21
67
|
|
|
22
|
-
|
|
68
|
+
**Enum columns need validating in PHP.** 2.0's `FIELD_LIST` performs no enum validation and
|
|
69
|
+
silently converts a falsy value to SQL `NULL`; MySQL will coerce rather than reject an unknown
|
|
70
|
+
member. Check the value against the allowed set before it reaches the query.
|
|
23
71
|
|
|
24
|
-
|
|
72
|
+
One caveat on 2.0's helpers: `_Database::escape()` is the correct one. Do **not** confuse it with
|
|
73
|
+
`_Database::protect()`, which *strips* `% \ / * " '` and the literal ` or` — lossy filtering, not
|
|
74
|
+
escaping, and it will corrupt data you meant to store intact.
|
|
25
75
|
|
|
26
76
|
## XSS prevention
|
|
27
77
|
|