toga-ai 1.0.485 → 1.0.486
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/tools/INDEX.md +1 -0
- package/knowledge/1.0/apps/tools/features/errors-curation-console.md +81 -0
- package/knowledge/2.0/apps/_underscore/INDEX.md +1 -1
- package/knowledge/2.0/apps/_underscore/features/error-reporting-issue-event.md +243 -120
- package/knowledge/2.0/apps/api2/INDEX.md +1 -1
- package/knowledge/2.0/apps/api2/features/v2-api-error-codes.md +73 -1
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
- package/knowledge/2.0/apps/worker2/features/error-escalation-cron.md +157 -0
- package/knowledge/2.0/standards/backend-php.md +3 -0
- package/knowledge/INDEX.md +2 -2
- package/package.json +1 -1
- package/rules/common/security.md +60 -10
|
@@ -7,6 +7,7 @@
|
|
|
7
7
|
| [CloudFront Client Setup](features/cloudfront-client-setup.md) | An SSO-gated admin tool at **`/devops/cloudfront-clients`** in the Tools 1.0 app that onboards a client onto **CloudFront + Route 53 across multiple AWS account | tools/_/app/devops/cloudfront.php, tools/mvc/devops/cloudfront-clients/get.php, tools/mvc/devops/cloudfront-clients/post.php, tools/assets/js/cloudfront-clients.js, tools/assets/css/cloudfront-clients.css, tools/_/app/nav.php, tools/_/app/frameworkindex.php, tools/config.production.ini |
|
|
8
8
|
| [Design Demo Admin](features/design-demo-admin.md) | A self-serve admin UI at **`/design`** in the SSO-protected **Tools** app that lets the design team publish self-contained "Claude Design" HTML exports as **ver | tools/_/app/design/github.php, tools/mvc/design/get.php, tools/mvc/design/post.php, tools/assets/css/design.css, tools/assets/js/design.js, tools/_/app/frameworkindex.php, tools/_/app/nav.php, tools/composer.json |
|
|
9
9
|
| [Tools — Developers Folder (UUID & Password Generators)](features/developer-tools.md) | The first two tools shipped in the Tools app, both under the **Developers** folder and gated to personas **Development Team** / **TOGa Technology**. | tools/mvc/developers/uuid/get.php, tools/mvc/developers/password/get.php |
|
|
10
|
+
| [/errors Curation Console (Tools → shared Core Logs DB)](features/errors-curation-console.md) | Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as a 1.0 Tools MVC page reading the **shared Core Logs DB** through | tools/mvc/errors/get.php, tools/mvc/errors/post.php, tools/_/app/nav.php, tools/config.production.ini |
|
|
10
11
|
| [Tools MVC — Routing, CSRF & App_Database Access Patterns](features/mvc-data-access-patterns.md) | The load-bearing 1.0 (`App_`) framework conventions a developer needs when adding a page to the Tools app — URL routing, CSRF, and DB access through `App_Databa | tools/_/app/nav.php, tools/mvc/get.php |
|
|
11
12
|
| [Tools Persona-Gated Navigation (App_Nav)](features/persona-gated-navigation.md) | `App_Nav` is the Tools app's two-level, **persona-gated** navigation. | tools/_/app/nav.php, tools/mvc/get.php |
|
|
12
13
|
| [Tools SAML SSO Consumer & Persona-Gated Auth (App_Auth)](features/saml-sso-auth.md) | `App_Auth` is the Tools app's authentication layer: it consumes the SAML gateway `?saml=` handoff (see the 2.0 SAML downstream integration contract), establishe | tools/_/app/auth.php, tools/mvc/sso/initiate/get.php, tools/mvc/sso/get.php, tools/mvc/login/get.php, tools/mvc/login/post.php, tools/mvc/logout/get.php, tools/mvc/get.php, tools/config.production.ini, tools/config.local.ini |
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: /errors Curation Console (Tools → shared Core Logs DB)
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: tools
|
|
5
|
+
project: Tools
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-30
|
|
10
|
+
owners: ["jcardinal"]
|
|
11
|
+
files:
|
|
12
|
+
- tools/mvc/errors/get.php
|
|
13
|
+
- tools/mvc/errors/post.php
|
|
14
|
+
- tools/_/app/nav.php
|
|
15
|
+
- tools/config.production.ini
|
|
16
|
+
related:
|
|
17
|
+
- ../architecture.md
|
|
18
|
+
- ./mvc-data-access-patterns.md
|
|
19
|
+
- ../../../2.0/apps/_underscore/features/error-reporting-issue-event.md
|
|
20
|
+
- ../../../2.0/apps/worker2/features/error-escalation-cron.md
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
Internal-only triage/curation screen for the 2.0 Issue/Event error-reporting pipeline, built as
|
|
26
|
+
a 1.0 Tools MVC page reading the **shared Core Logs DB** through a new **`db_toga2logs`**
|
|
27
|
+
alias. It curates; it never notifies.
|
|
28
|
+
|
|
29
|
+
The highest-value view is the list of issues **below the ClickUp threshold** — errors that are
|
|
30
|
+
happening and were deliberately never ticketed. Nobody had that view before.
|
|
31
|
+
|
|
32
|
+
## How it works
|
|
33
|
+
|
|
34
|
+
- **Curate** subject, description, `minimumUrgency`, assignee, and mute window.
|
|
35
|
+
- **Manage per-client email recipients** (`IssueEmailAddresses`). `clientId = 0` means **all
|
|
36
|
+
clients** and requires an explicit confirmation flag; the list view renders it as
|
|
37
|
+
**"ALL CLIENTS"**.
|
|
38
|
+
- **Merge fingerprints** — repoint an `IssueFingerprints` row at an existing Issue. Without this,
|
|
39
|
+
the fingerprint/Issue split is inert: a refactor that shifts line numbers produces a new Issue
|
|
40
|
+
and the curated one is orphaned.
|
|
41
|
+
- **Any save sets `isManaged`**, which does double duty: it protects the curated text from being
|
|
42
|
+
overwritten by later occurrences, **and** it exempts the Issue from the cron's GC sweep.
|
|
43
|
+
|
|
44
|
+
**This page notifies nobody.** Escalation, ClickUp ticketing, and email are owned entirely by
|
|
45
|
+
`_Worker_Infrastructure_Errors::Escalate`. Adding a notify action here would create a second,
|
|
46
|
+
untracked alerting path.
|
|
47
|
+
|
|
48
|
+
### Why Tools and not TOGa Hub
|
|
49
|
+
|
|
50
|
+
The 2026-06-12 meeting had proposed TOGa Hub, but Hub is far off. The tables are the same either
|
|
51
|
+
way, so the choice is not load-bearing — a Hub port later reads the same schema.
|
|
52
|
+
|
|
53
|
+
## Configuration
|
|
54
|
+
|
|
55
|
+
`[database_toga2logs]` was added to `tools/config.production.ini`.
|
|
56
|
+
|
|
57
|
+
**`dbname` is NOT necessarily `Logs`.** 2.0 resolves the Core Logs database name at **runtime**
|
|
58
|
+
from `Core.Databases` id **11** (`worker2` `CORE_LOGS_DATABASE_ID`); `Logs` is only the
|
|
59
|
+
`_underscore` *register alias*. Read the real name from `Core.Databases` before configuring a new
|
|
60
|
+
environment, or the console silently points at a database that may not exist.
|
|
61
|
+
|
|
62
|
+
## Gotchas / known issues
|
|
63
|
+
|
|
64
|
+
- **This is a 1.0 app reading a 2.0 shared database.** Access is plain `App_Database` via the
|
|
65
|
+
`db_toga2logs` alias, not the `_Model` layer — so none of the 2.0 model protections apply.
|
|
66
|
+
Escape every interpolated value and allowlist every identifier (see the 1.0 back-end standard).
|
|
67
|
+
- **Cross-tenant surface.** The Core Logs DB is shared by every client, so this console can show
|
|
68
|
+
one client's captured context to a viewer looking at another client's problem. It is
|
|
69
|
+
internal-only and persona-gated for that reason — do not expose any part of it to a client.
|
|
70
|
+
- **Curated text is authoritative.** Once `isManaged` is set, later occurrences no longer update
|
|
71
|
+
subject/description, so a stale curated subject stays stale until someone edits it again.
|
|
72
|
+
|
|
73
|
+
## Change history
|
|
74
|
+
|
|
75
|
+
- 2026-07-30 — Built as part of TRUE-78188: new `/errors` triage/curation console over the
|
|
76
|
+
shared Core Logs DB via a new `db_toga2logs` alias; curation sets `isManaged` (protects text
|
|
77
|
+
and exempts from GC); fingerprint merging; per-client recipient management with an explicit
|
|
78
|
+
confirmation for `clientId = 0` ("ALL CLIENTS"). Chosen over TOGa Hub because Hub is far off
|
|
79
|
+
and the schema is the same either way. Documented that the Core Logs `dbname` resolves at
|
|
80
|
+
runtime from `Core.Databases` id 11 and is not necessarily "Logs". (jcardinal)
|
|
81
|
+
</content>
|
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
| [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
|
|
16
16
|
| [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, worker2/Worker/Infrastructure/Email/Send.php |
|
|
17
17
|
| [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
|
|
18
|
-
| [Error Reporting — Issue/Event
|
|
18
|
+
| [Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation](features/error-reporting-issue-event.md) | Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in the **shared Core Logs DB**. | _underscore/Error.php, _underscore/Exception/Business.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, _underscore/Model/Core/Logs/IssueFingerprint.php, _underscore/Model/Core/Logs/IssueClickupTask.php, _underscore/Model/Core/Logs/IssueEmailAddress.php, _underscore/Model/Core/Logs/IssueAreaOwner.php, dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
|
|
19
19
|
| [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |
|
|
20
20
|
| [Forecast.Sales NetSuite import engine (real-time webhook)](features/forecast-sale-import.md) | Real-time importer that takes a NetSuite **sale** record and writes its lines into `Forecast.Sales` (the Forecast2 revenue table). | worker2/Component/Forecast/SaleImport/SaleImport.php, worker2/Component/Forecast/Db/Db.php, _underscore/Component/Api/Netsuite/Netsuite.php, worker2/Worker/Netsuite/Invoice.php, worker2/Worker/Netsuite/CashSale.php, worker2/Worker/Netsuite/CreditMemo.php, worker2/Worker/Netsuite/CashRefund.php, worker2/Worker/Netsuite/JournalEntry.php, worker2/Worker/Netsuite/Opportunity.php, worker2/Worker/Netsuite/SalesOrder.php, dbchanges2/Forecast/2026-06-26a - Add journalEntry to Sales transaction type enum.sql, test/@dave/test_invoice_lifecycle.php, test/@dave/test_je_lifecycle.php, test/@dave/test_creditmemo_lifecycle.php, test/@dave/test_cashsale_lifecycle.php, test/@dave/test_cashrefund_lifecycle.php, test/@dave/test_fetchrecord_routes.php, test/@dave/verify_je_classification.php, test/@dave/probe_je_accounts.php, test/@dave/probe_je_shape.php, test/@dave/fixer.php, test/@dave/Junk Drawer/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, test/@dave/Junk Drawer/NetSuite/api-message-queue/dev_ue_api_msg_queue_enqueue.js |
|
|
21
21
|
| [isFulfillable Propagation Up the SO↔PO Chain](features/fulfillable-item-propagation.md) | `Items.isFulfillable` is a boolean that gates whether a storefront line's **Qty Fulfilled** cell is actionable. | _underscore/Model/Client/Item.php, _underscore/Model/Compass/Item.php, dbchanges2/Core/2026-07-17 - Items isFulfillable RecordField.sql, dbchanges2/Core/2026-07-17 - RegisterItemIsFulfillableInterceptors.sql, dbchanges2/Client/2026-07-17 - ItemsisFulfillable.sql |
|
|
@@ -1,148 +1,271 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: Error Reporting — Issue/Event
|
|
2
|
+
title: Error Reporting — Issue/Event Capture, Fingerprinting & Aggregation
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
repo: _underscore
|
|
5
5
|
project: _Underscore
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
10
|
-
owners: ["dfranks"]
|
|
9
|
+
updated: 2026-07-30
|
|
10
|
+
owners: ["dfranks", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Error.php
|
|
13
|
+
- _underscore/Exception/Business.php
|
|
13
14
|
- _underscore/Model/Core/Logs/Issue.php
|
|
14
15
|
- _underscore/Model/Core/Logs/Event.php
|
|
15
|
-
-
|
|
16
|
+
- _underscore/Model/Core/Logs/IssueFingerprint.php
|
|
17
|
+
- _underscore/Model/Core/Logs/IssueClickupTask.php
|
|
18
|
+
- _underscore/Model/Core/Logs/IssueEmailAddress.php
|
|
19
|
+
- _underscore/Model/Core/Logs/IssueAreaOwner.php
|
|
20
|
+
- dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql
|
|
21
|
+
- dbchanges2/Core/2026-07-30a - Error escalation cron job.sql
|
|
16
22
|
related:
|
|
17
23
|
- ../architecture.md
|
|
18
24
|
- ./per-client-database-connections.md
|
|
25
|
+
- ./email-send-pipeline.md
|
|
26
|
+
- ../../worker2/features/error-escalation-cron.md
|
|
27
|
+
- ../../../1.0/apps/tools/features/errors-curation-console.md
|
|
19
28
|
- ../../dbchanges2/architecture.md
|
|
20
29
|
---
|
|
21
30
|
|
|
22
31
|
## Summary
|
|
23
32
|
|
|
24
|
-
Platform-wide error
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
with
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
`
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
handler
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
33
|
+
Platform-wide error reporting for TOGA 2.0, built on an **Issue / Event** aggregation model in
|
|
34
|
+
the **shared Core Logs DB**. An **Issue** is the deduplicated record for a distinct problem
|
|
35
|
+
(carrying curated text, an urgency, and a quotable reference); an **Event** is one row per
|
|
36
|
+
occurrence. Capture lives in `_underscore/Error.php`; escalation and ClickUp/email
|
|
37
|
+
notification live entirely in the worker2 cron (see
|
|
38
|
+
[error-escalation-cron](../../worker2/features/error-escalation-cron.md)); curation lives in
|
|
39
|
+
the 1.0 Tools `/errors` console.
|
|
40
|
+
|
|
41
|
+
`_Error::captureException(Throwable $e, ?int $clientId): ?string` is the **single capture
|
|
42
|
+
entry point** — the exception handler, the shutdown handler, and api2 all funnel through it,
|
|
43
|
+
and it returns the quotable reference. Add new capture paths by calling it, never by
|
|
44
|
+
re-implementing persistence.
|
|
45
|
+
|
|
46
|
+
## Data model (shared Core Logs DB)
|
|
47
|
+
|
|
48
|
+
Six tables, created by `dbchanges2/Logs/2026-07-30a - Error reporting Issues and Events.sql`:
|
|
49
|
+
`Issues`, `IssueFingerprints`, `Events`, `IssueEmailAddresses`, `IssueClickupTasks`,
|
|
50
|
+
`IssueAreaOwners`.
|
|
51
|
+
|
|
52
|
+
**An Issue is never client-scoped.** An Issue is identified by *code*, and code is global — the
|
|
53
|
+
same bug hitting six clients is **one** Issue. Per-client scoping lives on the occurrence
|
|
54
|
+
(`Events.clientId`) and on notification routing (`IssueEmailAddresses.clientId`), never on the
|
|
55
|
+
Issue itself. Anything that reads like "this client's issue" is a modelling error.
|
|
56
|
+
|
|
57
|
+
`IssueEmailAddresses.clientId = 0` means **all clients**.
|
|
58
|
+
|
|
59
|
+
## How it works
|
|
60
|
+
|
|
61
|
+
### Fingerprinting (identity)
|
|
62
|
+
|
|
63
|
+
The fingerprint is the normalized top-N **application** frames (repo-relative `path:line`) plus
|
|
64
|
+
the exception class; framework/vendor plumbing is filtered out. The exception **message is
|
|
65
|
+
deliberately excluded** — it carries per-occurrence data (ids, values) and would defeat dedup.
|
|
66
|
+
|
|
67
|
+
This replaced `md5(serialize(debug_backtrace()))`, which serialized **call arguments** and so
|
|
68
|
+
produced a different hash on every request — dedup never actually worked.
|
|
69
|
+
|
|
70
|
+
`IssueFingerprints` is many-to-one against `Issues`, so a refactor that shifts line numbers
|
|
71
|
+
cannot orphan a curated Issue: an admin repoints the new fingerprint at the existing Issue
|
|
72
|
+
from the Tools console to merge them.
|
|
73
|
+
|
|
74
|
+
### `_Exception_Business` — identity immune to refactoring
|
|
75
|
+
|
|
76
|
+
`_underscore/Exception/Business.php` carries a stable `issueKey` plus a `minimumUrgency` floor.
|
|
77
|
+
When an `issueKey` is declared, **the key is the fingerprint**, so the Issue's identity does not
|
|
78
|
+
depend on line numbers at all. It extends `Exception` (not `_Exception`), matching
|
|
79
|
+
`_Exception_Validation`.
|
|
80
|
+
|
|
81
|
+
It **deliberately never throws on a bad argument** — it normalizes and `error_log()`s instead.
|
|
82
|
+
Replacing the caller's business error with an argument error destroys the real signal.
|
|
83
|
+
|
|
84
|
+
### The quotable reference
|
|
85
|
+
|
|
86
|
+
`Issues.reference` is a Sentry-style base-26 encoding of `Issues.id`: 1 → `1A`, 26 → `1Z`,
|
|
87
|
+
27 → `2A`. Because `AUTO_INCREMENT` never reuses ids, gaps are expected and permanent — a
|
|
88
|
+
reference can never be recycled onto a different problem, so it is safe to quote to a customer.
|
|
89
|
+
|
|
90
|
+
The reference is written by PHP in a second statement (see MySQL constraints below), which is
|
|
91
|
+
why the column is nullable even though it is always populated.
|
|
92
|
+
|
|
93
|
+
### Shutdown handler
|
|
94
|
+
|
|
95
|
+
`register_shutdown_function` is registered with a **pre-allocated memory reserve**. Before this,
|
|
96
|
+
`_underscore` had **no shutdown handler at all**, so fatals, OOM, and timeouts were completely
|
|
97
|
+
uncaptured.
|
|
98
|
+
|
|
99
|
+
## MySQL constraints that shaped the schema (verified on 8.0.46)
|
|
100
|
+
|
|
101
|
+
- **A stored generated column blocks `ON DELETE CASCADE`.** A foreign key with
|
|
102
|
+
`ON DELETE CASCADE` is rejected with error **1215** when the constrained column is the base
|
|
103
|
+
column of a **STORED** generated column. `IssueClickupTasks.issueId` is the base of
|
|
104
|
+
`openEpisodeIssueId`, so that generated column must be **VIRTUAL**. VIRTUAL has no such
|
|
105
|
+
restriction and still supports a UNIQUE secondary index. Proved both ways locally.
|
|
106
|
+
- **Error 3109: a generated column cannot refer to an auto-increment column.** This is why
|
|
107
|
+
`Issues.reference` cannot be generated and must be written by PHP.
|
|
108
|
+
|
|
109
|
+
## One-open-episode guard (UNIQUE on a generated column)
|
|
110
|
+
|
|
111
|
+
`INSERT ... SELECT ... WHERE NOT EXISTS` does **not** prevent a double insert under
|
|
112
|
+
REPEATABLE READ: two overlapping cron runs each evaluate the `NOT EXISTS` against their own
|
|
113
|
+
snapshot and both pass. The guard is instead a VIRTUAL generated column
|
|
114
|
+
`openEpisodeIssueId = issueId while dtResolved IS NULL, else NULL` with a **UNIQUE** key —
|
|
115
|
+
MySQL permits duplicate NULLs, so many resolved episodes coexist while at most one is open.
|
|
116
|
+
|
|
117
|
+
The code **claims the episode before calling ClickUp**, so a losing run fails (error 1062)
|
|
118
|
+
before it can create a duplicate external task; the claim is **released on API failure** so a
|
|
119
|
+
stuck placeholder row cannot permanently block the Issue. Verified: second open episode → 1062;
|
|
120
|
+
resolve-then-reopen allowed; many resolved episodes allowed; cascade delete works.
|
|
121
|
+
|
|
122
|
+
## Sentry coexistence (removal is a separate ticket)
|
|
123
|
+
|
|
124
|
+
Sentry stays running **in parallel**. Shipping an unproven replacement for the only working
|
|
125
|
+
error reporting with no overlap means going blind. Three findings make removal its own ticket:
|
|
126
|
+
|
|
127
|
+
1. `_Worker_Sentry::Webhook` is an **existing Sentry→ClickUp pipeline writing into the same
|
|
128
|
+
list `901110669877`** this work replaces — retiring it also means turning off the Sentry-side
|
|
129
|
+
issue alerts.
|
|
130
|
+
2. `\Sentry\configureScope()` calls `http_response_code(500)` as a blanket default, and api2
|
|
131
|
+
documents working *around* it. Remove Sentry without fixing every early-exit branch and
|
|
132
|
+
validation 400s ship as HTTP 200 — which changes whether apiproxy retries across regions.
|
|
133
|
+
3. `\Sentry\init()` registers PHP error handlers. `_Error::errorHandler()` also throws on
|
|
134
|
+
warnings, but that promotion must be verified **empirically** — if it does not survive,
|
|
135
|
+
code that currently 500s would silently continue with a null value, which is worse.
|
|
136
|
+
|
|
137
|
+
Also: this pipeline is **2.0-only**. Sentry must remain for the 1.0 apps (desk1, retail1,
|
|
138
|
+
forecast1, worker1, view1, commerce1) until a `library` / `App_Error` port lands, so the
|
|
139
|
+
Sentry-removal ticket cannot complete until the 1.0 port does.
|
|
94
140
|
|
|
95
141
|
## Gotchas / known issues
|
|
96
142
|
|
|
97
|
-
- **
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
`
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
143
|
+
- **Read-your-writes: reads go to the READ host and cannot see an uncommitted write.** A
|
|
144
|
+
second `$issue->save()` re-reads the row via `_Model::initialize()`, and that SELECT goes to
|
|
145
|
+
the read host — a separate connection that cannot see the still-uncommitted INSERT on the
|
|
146
|
+
write connection. Symptom: `Issues` rows with `reference` NULL, **zero** Events, **zero**
|
|
147
|
+
IssueFingerprints, no reference in the API response, and the logged cause *"Error during
|
|
148
|
+
Model build for primary key id '1' … Exactly 1 row was expected to be returned but 0 were."*
|
|
149
|
+
Capture aborted half-written. Fix: force reads to the writer for the whole capture block via
|
|
150
|
+
`_Database::getIsReadHostEnabled()` / `setIsReadHostEnabled(false)` with a `finally` restore
|
|
151
|
+
— the same TRUE-80448 pattern `_Email::send()` uses.
|
|
152
|
+
- **Never rebuild a model inside an error handler.** `_Model`'s build **throws** when the row
|
|
153
|
+
is missing, and nothing inside an error handler should throw over a value derivable from the
|
|
154
|
+
primary key. `new _Model_Core_Logs_Issue($id)` rebuilds were replaced with a light SELECT
|
|
155
|
+
helper, and `reference` is written with a plain UPDATE rather than a second `save()`.
|
|
156
|
+
- **`Issues.issueKey` is UNIQUE — handle the first-occurrence race.** Two concurrent first
|
|
157
|
+
occurrences of the same declared business condition raced on the INSERT and the loser's
|
|
158
|
+
occurrence was **silently dropped**. The loser must adopt the winner's Issue.
|
|
159
|
+
- **`LAST_INSERT_ID(totalOccurrences + 1)` is the atomic event-number allocator** (a `_Model`
|
|
160
|
+
dirty-check read-modify-write loses increments). But `LAST_INSERT_ID` is only *assigned* when
|
|
161
|
+
the UPDATE matches a row — proven locally: against a missing row, `affected = 0` while
|
|
162
|
+
`LAST_INSERT_ID()` still returned the **stale** previous value `2`. Verify affected rows
|
|
163
|
+
before trusting it, or the event number silently collides with `UNIQUE(issueId, eventNumber)`.
|
|
164
|
+
- **A new Issue must be seeded at its declared `minimumUrgency`**, not at LOW — otherwise the
|
|
165
|
+
floor is ignored until the next cron run.
|
|
166
|
+
- **MySQL `UUID()` is forbidden for inserted records** by the 2.0 standard (time-based, not
|
|
167
|
+
random) — use `_String::generateUuid()` in PHP. It remains acceptable inside `.sql`
|
|
168
|
+
migrations, where no PHP exists (65 existing migrations do this).
|
|
169
|
+
- **`VALUES()` in `ON DUPLICATE KEY UPDATE` is deprecated in MySQL 8.0.20+** — use the row-alias
|
|
170
|
+
form. **Evaluation order matters:** in `IssueAreaOwners`, `consecutiveCount` is assigned
|
|
171
|
+
**before** `clickupUserId` so the `IF()` compares against the *old* owner. Reversing them
|
|
172
|
+
makes the condition always true and the counter never resets.
|
|
173
|
+
- **`Events.context` is an allowlist, never a dump.** It replaced `print_r($GLOBALS)` filtered
|
|
174
|
+
by a key-name blocklist — that blocklist recursed **arrays only** and walked straight past
|
|
175
|
+
objects whose *properties* hold DB credentials, into a table shared by every client.
|
|
176
|
+
`requestUri` and `referer` have query strings stripped (a `?token=abc` was `[REDACTED]` under
|
|
177
|
+
`get` and stored in clear in `requestUri`). The sensitive-fragment list covers **PII**, not
|
|
178
|
+
just credentials. Persisted traces are rebuilt **argument-free**.
|
|
179
|
+
- **Keep the debug `print_r($GLOBALS)` dump inside the `isDebugMode()` branch.** It was being
|
|
180
|
+
built on every exception — including the OOM path the memory reserve exists to protect.
|
|
181
|
+
- **Strict-mysqli varchar overflow silently drops the row.** `_Error::initialize()` sets
|
|
182
|
+
`mysqli_report(MYSQLI_REPORT_STRICT)`, so assigning an over-255-char value to a
|
|
183
|
+
`varchar(255)` (`errorMessage`, `subject`) **throws** on `->save()`. Inside the handler's
|
|
184
|
+
`catch(Throwable)` the throw is swallowed and the row is silently dropped — precisely when a
|
|
185
|
+
long message matters most. Truncate to column width before `save()`.
|
|
186
|
+
- **Shared-schema migration collision.** Because these tables live in the *shared* Core Logs
|
|
187
|
+
DB, only **one** migration may `CREATE` them; later migrations only `ALTER`.
|
|
188
|
+
- **Pre-existing committed secrets in this area (not fixed — out of scope, report only).**
|
|
189
|
+
A ClickUp API token at `_underscore/Component/Api/Clickup/Clickup.php:5`; SES SMTP
|
|
190
|
+
credentials at `_underscore/Email.php:11-12`; a named individual's Sentry personal access
|
|
191
|
+
token at `worker2/Worker/Sentry.php:4`. Also unfixed: api2 writes `apache_request_headers()`
|
|
192
|
+
verbatim into `Logs.Api` (bearer tokens in plaintext) and ships `$_SERVER` to Sentry. Treat
|
|
193
|
+
all of these as compromised whenever this area is touched again.
|
|
194
|
+
|
|
195
|
+
## Decisions superseding the 2026-05-21 design
|
|
196
|
+
|
|
197
|
+
### The POST-to-a-central-`/errors`-receiver hop is DROPPED (jcardinal, 2026-07-30)
|
|
198
|
+
|
|
199
|
+
The **"2026-05-21 – Errors, Monitoring and Alerts"** meeting had approved a *POST-to-receiver*
|
|
200
|
+
pipeline: application handlers POST a serialized error to a centralized `/errors` receiver
|
|
201
|
+
endpoint (`webhook.hub.com/error`) and perform **no** database persistence; a worker2 cron
|
|
202
|
+
consumes the payloads and does all hashing, upserting, aggregation, escalation, and ClickUp
|
|
203
|
+
sync into the Core Logs DB.
|
|
204
|
+
|
|
205
|
+
**That hop is dropped.** Error capture writes **directly** to the Core Logs DB from
|
|
206
|
+
`_Error::captureException()`. Inline persistence is the model to follow; do not reintroduce an
|
|
207
|
+
HTTP hop.
|
|
208
|
+
|
|
209
|
+
Rationale: an error is frequently caused **by something being down**, so routing error capture
|
|
210
|
+
through an HTTP call to api2 would lose exactly the errors you most need. Inline capture also
|
|
211
|
+
buys the synchronous `reference` the api2 response envelope needs, which a fire-and-forget POST
|
|
212
|
+
cannot. The agreed design still holds on the *sink* (shared Core Logs DB, not per-client) and on
|
|
213
|
+
*who escalates* (the worker2 cron owns escalation and notification only).
|
|
214
|
+
|
|
215
|
+
The cost of a direct write is that **each app must have the Core Logs connection registered**.
|
|
216
|
+
The capture path already guards this with
|
|
217
|
+
`isset(_Database::$_registers[_underscore::DB_LOGS])` — it **skips** capture rather than
|
|
218
|
+
fatalling when the register is absent.
|
|
219
|
+
|
|
220
|
+
### Why api2 is in scope at all
|
|
221
|
+
|
|
222
|
+
api2 catches `Throwable` in its own front controller and therefore **never reaches
|
|
223
|
+
`_Error::exceptionHandler`**. `captureException()` had to become a shared entry point that
|
|
224
|
+
api2's catch blocks call directly — otherwise the entire inbound-API 500 surface would go
|
|
225
|
+
unrecorded.
|
|
226
|
+
|
|
227
|
+
### `Issues.type` enum and `BusinessIssues_Users` were proposed and not built (jcardinal, 2026-07-30)
|
|
228
|
+
|
|
229
|
+
The **2026-06-12 "Monitoring Business Exceptions Sync"** meeting proposed an `Issues.type` enum
|
|
230
|
+
(`'technical'`/`'business'`) and a `BusinessIssues_Users` bridge table (clientId +
|
|
231
|
+
clientUserId). **Neither was built.** As built instead:
|
|
232
|
+
|
|
233
|
+
- **No `type` column.** `issueKey` (non-null = a developer declared it a business condition in
|
|
234
|
+
code) plus `isManaged` (a human curated it) carry the same information while also being
|
|
235
|
+
useful on their own.
|
|
236
|
+
- **`IssueEmailAddresses` uses varchar email addresses scoped by `clientId`** (`0` = all
|
|
237
|
+
clients) rather than client-user ids, so recipients who have **no login** can still be
|
|
238
|
+
addressed.
|
|
132
239
|
|
|
133
240
|
## Change history
|
|
134
241
|
|
|
242
|
+
- 2026-07-30 — Reworked TRUE-78188: six-table schema in the shared Core Logs DB (Issue is
|
|
243
|
+
global by code; client scope lives on Events/IssueEmailAddresses); `_Error::captureException()`
|
|
244
|
+
as the single capture entry point; trace-frame fingerprinting with the message excluded
|
|
245
|
+
(replacing `md5(serialize(debug_backtrace()))`, which never deduped); many fingerprints → one
|
|
246
|
+
Issue for refactor safety; `_Exception_Business` with a stable `issueKey` + `minimumUrgency`;
|
|
247
|
+
base-26 quotable reference; a shutdown handler with a memory reserve (there was none).
|
|
248
|
+
Documented two MySQL constraints (1215 stored-generated-column vs. ON DELETE CASCADE; 3109
|
|
249
|
+
generated column cannot refer to auto-increment), the UNIQUE-on-VIRTUAL-generated-column
|
|
250
|
+
open-episode guard (`WHERE NOT EXISTS` is not safe under REPEATABLE READ), the read-host
|
|
251
|
+
read-your-writes failure that broke capture entirely, the issueKey first-occurrence race, the
|
|
252
|
+
`LAST_INSERT_ID` stale-value trap, the `Events.context` allowlist replacing the
|
|
253
|
+
`print_r($GLOBALS)` blocklist, and the Sentry-parallel decision. (jcardinal)
|
|
254
|
+
- 2026-07-30 — **Decided: the 2026-05-21 POST-to-a-central-`/errors`-receiver hop is dropped.**
|
|
255
|
+
Capture writes directly from `_Error::captureException()`, guarded by
|
|
256
|
+
`isset(_Database::$_registers[_underscore::DB_LOGS])`; an HTTP hop would lose errors caused by
|
|
257
|
+
something being down. Recorded that api2 is in scope because its front controller catches
|
|
258
|
+
`Throwable` and never reaches `_Error::exceptionHandler`. Also recorded that the 2026-06-12
|
|
259
|
+
meeting's proposed `Issues.type` enum and `BusinessIssues_Users` bridge table were **not
|
|
260
|
+
built** — `issueKey` + `isManaged` and `clientId`-scoped varchar `IssueEmailAddresses` are the
|
|
261
|
+
as-built equivalents. Supersedes the 2026-07-08 correction. (jcardinal)
|
|
135
262
|
- 2026-07-15 — Added two framework gotchas from aligning the `_underscore` side of the
|
|
136
|
-
Issue/Event pipeline to the shared Core Logs DB:
|
|
137
|
-
|
|
138
|
-
Error.php; (2) strict-mysqli varchar overflow throwing inside the handler's `catch(Throwable)`
|
|
139
|
-
silently drops the Issue/Event row — truncate to column width before `save()`. (dfranks)
|
|
263
|
+
Issue/Event pipeline to the shared Core Logs DB: cross-tenant secret exposure in captured
|
|
264
|
+
context, and strict-mysqli varchar overflow throwing inside `catch(Throwable)`. (dfranks)
|
|
140
265
|
- 2026-07-08 — Corrected KB drift: documented the approved 2026-05-21 error-monitoring
|
|
141
|
-
architecture (handlers POST to a centralized `/errors` receiver; a worker2 cron
|
|
142
|
-
|
|
143
|
-
syncs ClickUp — handlers do not persist). Clarified that `_production` Error.php performs
|
|
144
|
-
no Issue/Event persistence (only an S3 debug body + flat `*_Logs_Error` row), and that the
|
|
145
|
-
inline-DB-persist approach (handler-side upsert to the *Client* Logs DB) conflicts with the
|
|
146
|
-
agreed design and is not the pattern to follow. (dfranks)
|
|
266
|
+
architecture (handlers POST to a centralized `/errors` receiver; a worker2 cron aggregates
|
|
267
|
+
into the Core Logs DB). **Superseded 2026-07-30 — the receiver hop was dropped.** (dfranks)
|
|
147
268
|
- 2026-07-06 — Prior doc described the exceptionHandler as directly persisting Issue/Event
|
|
148
|
-
rows
|
|
269
|
+
rows with dedup; superseded by the 2026-07-08 correction. (dfranks)
|
|
270
|
+
</content>
|
|
271
|
+
</invoke>
|
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
| [TableView row-filtering via apiWhereClause (options.where grammar, end to end)](features/tableview-apiwhereclause-row-filtering.md) | `TableViews.apiWhereClause` (TEXT, nullable) is the sanctioned, code-free way to restrict or exclude rows from a 2.0 table view. | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TableView.php, toga2-supply/src/api/toga.ts |
|
|
18
18
|
| [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql |
|
|
19
19
|
| [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
|
|
20
|
-
| [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TrackingNumber.php |
|
|
20
|
+
| [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, api2/Component/Api/V2/Response/Response.php, api2/Component/Api/V2/Response/Oauth/Oauth.php, api2/Controller/Index.php, toga2-supply/src/globalTypes.ts, toga2-supply/src/pages/Orders/api/OrdersApi.ts, _underscore/Model/Client/TrackingNumber.php |
|
|
21
21
|
| [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
|
|
22
22
|
| [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
|
|
23
23
|
| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
|
|
@@ -6,12 +6,18 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-30
|
|
10
10
|
owners: [mhammontree, tcox, jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
13
|
+
- api2/Component/Api/V2/Response/Response.php
|
|
14
|
+
- api2/Component/Api/V2/Response/Oauth/Oauth.php
|
|
15
|
+
- api2/Controller/Index.php
|
|
16
|
+
- toga2-supply/src/globalTypes.ts
|
|
17
|
+
- toga2-supply/src/pages/Orders/api/OrdersApi.ts
|
|
13
18
|
- _underscore/Model/Client/TrackingNumber.php
|
|
14
19
|
related:
|
|
20
|
+
- ../../_underscore/features/error-reporting-issue-event.md
|
|
15
21
|
- ../../_underscore/features/acl-permission-chain.md
|
|
16
22
|
- scripted-api-post-body-args.md
|
|
17
23
|
- v2-rest-query-contract.md
|
|
@@ -91,6 +97,61 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
|
|
|
91
97
|
DB fully migrated. On a local dev box with PHP opcache disabled a branch switch takes effect
|
|
92
98
|
immediately (files are read fresh per request) — no Apache restart needed.
|
|
93
99
|
|
|
100
|
+
## BREAKING (2026-07-30): `error` is now an object, not a bare string
|
|
101
|
+
|
|
102
|
+
The V2 envelope's `error` changed from a bare string (`"EO-1"`) to an **object**:
|
|
103
|
+
|
|
104
|
+
```json
|
|
105
|
+
"error": { "code": "EO-1", "id": "1C-1" }
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
- **`code`** — the same short message code documented above.
|
|
109
|
+
- **`id`** — the quotable **Core Logs reference** for a *captured exception* (see
|
|
110
|
+
[error reporting](../../_underscore/features/error-reporting-issue-event.md)), and **`null`**
|
|
111
|
+
for a graceful rejection. `id` is **always present** so the shape is predictable — consumers
|
|
112
|
+
must not branch on its absence.
|
|
113
|
+
|
|
114
|
+
The object is assembled from **private** properties (`json_encode` emits public only) via
|
|
115
|
+
`setErrorReference()` + `refreshError()`, so it works whether the reference arrives **before or
|
|
116
|
+
after** the code — the controller sets the reference first. All six orderings were verified.
|
|
117
|
+
|
|
118
|
+
**The OAuth response subclass deliberately keeps the flat string form**
|
|
119
|
+
(`const USES_STRUCTURED_ERROR = false`). RFC 6749 §5.2 defines `error` as a single ASCII string,
|
|
120
|
+
and both classes share `addDefinedMessage()`, so emitting the object there would break every
|
|
121
|
+
conforming OAuth client library. Do not "unify" this.
|
|
122
|
+
|
|
123
|
+
**Also fixed here (CRITICAL): api2 returned `$e->getTrace()` to external callers on every 500.**
|
|
124
|
+
Unlike `getTraceAsString()`, `getTrace()` returns raw frame **arguments** — and the DB-bootstrap
|
|
125
|
+
catch has `_Database::register()` on the stack, so the **database username and password were
|
|
126
|
+
reachable by anyone who could force a 500**. `message` and `trace` are now gated behind
|
|
127
|
+
`_Environment::isDebugMode()`.
|
|
128
|
+
|
|
129
|
+
### Consumer audit
|
|
130
|
+
|
|
131
|
+
- **`library/app/api/toga2.php`** (the 1.0↔2.0 bridge) branches on `isSuccess` only, never
|
|
132
|
+
`error` — safe.
|
|
133
|
+
- **`toga2-supply`** — `ApiResponse<T>.error` is now `ApiError | null`; 30 envelope reads in
|
|
134
|
+
`OrdersApi.ts` became `.error?.code` (all 21 declarations there are
|
|
135
|
+
`await togaApiRequest(...)`, verified).
|
|
136
|
+
- **`toga-blox-npm` is not affected** — no `isSuccess` anywhere; it reads axios'
|
|
137
|
+
`response.data.data.…`.
|
|
138
|
+
- **Outstanding:** the external `API-DOCUMENTATION.md` lives outside the checkout and still
|
|
139
|
+
documents `error` as a string.
|
|
140
|
+
|
|
141
|
+
### Gotcha — do NOT regex-sweep `.error` → `.error?.code`
|
|
142
|
+
|
|
143
|
+
Only direct `togaApiRequest()` results are envelopes. These look identical and are **local
|
|
144
|
+
`{success, error}` shapes whose `error` is a plain string** — adding `?.code` breaks the failure
|
|
145
|
+
guard and swaps the real backend message for a generic fallback:
|
|
146
|
+
|
|
147
|
+
- `payload` in `UpdateShipmentApi.ts` (`response?.data?.itemFulfillments?.generateReturnLabel`)
|
|
148
|
+
- `shipmentResponse` from `fulfillShipment()`
|
|
149
|
+
- `response` in `EditShipmentForm.tsx` from `updateShipment()` / `saveShipment()`
|
|
150
|
+
- `messages[0].identifiers.error` — that is the message-identifiers map, not the envelope.
|
|
151
|
+
|
|
152
|
+
Verification gap: `toga2-supply` had no `node_modules`, so `tsc` could not be run.
|
|
153
|
+
**`npm run build` is the definitive check** — the type change will surface anything missed.
|
|
154
|
+
|
|
94
155
|
## Related recipes
|
|
95
156
|
|
|
96
157
|
- **Add a writable field to a V2 record (3-file migration)** and **expose a scripted API**
|
|
@@ -103,6 +164,17 @@ migration instead of re-deriving it. All field/script ACL rows live in the **CLI
|
|
|
103
164
|
|
|
104
165
|
## Change history
|
|
105
166
|
|
|
167
|
+
- 2026-07-30 — **BREAKING:** envelope `error` changed from a bare string to
|
|
168
|
+
`{ code, id }`, where `id` is the quotable Core Logs reference for a captured exception and
|
|
169
|
+
`null` for a graceful rejection (always present, for a predictable shape); assembled from
|
|
170
|
+
private properties via `setErrorReference()`/`refreshError()` so code and reference can arrive
|
|
171
|
+
in any order. The OAuth subclass keeps the flat string form
|
|
172
|
+
(`USES_STRUCTURED_ERROR = false`) per RFC 6749 §5.2. Fixed a CRITICAL disclosure: api2
|
|
173
|
+
returned `$e->getTrace()` (raw frame **arguments**, including `_Database::register()`'s DB
|
|
174
|
+
username/password) to external callers on every 500 — `message`/`trace` are now gated behind
|
|
175
|
+
`_Environment::isDebugMode()`. Recorded the consumer audit (library bridge and toga-blox-npm
|
|
176
|
+
unaffected; toga2-supply migrated) and the regex-sweep gotcha for look-alike local
|
|
177
|
+
`{success, error}` shapes. (jcardinal)
|
|
106
178
|
- 2026-07-29 — Added **EV-13** (`recordsPerPage` over the 10000 max) and **diagnosis note 6**:
|
|
107
179
|
what a denial looks like from the client side — `EZ-1` = record-level (`data.<route>` null,
|
|
108
180
|
and a bare `allowRead=1` with no logic group still returns it); `EZ-2` = field-level,
|
|
@@ -18,6 +18,7 @@
|
|
|
18
18
|
| [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
|
|
19
19
|
| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
|
|
20
20
|
| [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
|
|
21
|
+
| [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Config/production.ini, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql |
|
|
21
22
|
| [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |
|
|
22
23
|
| [Etilize Item Translation Import](features/etilize-item-translation-import.md) | The abstract worker class `_Worker_Etilize_ItemTranslations` imports **non-English** item text from Etilize into the client's `ItemTranslations` table. | worker2/Worker/Etilize/ItemTranslations.php |
|
|
23
24
|
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | A unified, DB-driven monitoring framework for business-critical data flows (Compass POs, Prudential asset imports, AIG closed claims, …). | worker2/Worker/Monitor.php, worker2/Worker/Monitors/, worker2/Worker/Monitors/RateEntitlement.php, worker2/Worker/Notification/Email.php, worker2/Worker/Rate.php, dbchanges2/Core/2026-05-21 - Monitors.sql, dbchanges2/Core/2026-06-29a - Rate Entitlement Contract Monitor.sql |
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Error Escalation Cron (Errors::Escalate → ClickUp / email)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-30
|
|
10
|
+
owners: ["jcardinal"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Infrastructure/Errors.php
|
|
13
|
+
- worker2/Worker/Clickup/ErrorTask.php
|
|
14
|
+
- worker2/Worker/Clickup.php
|
|
15
|
+
- worker2/Config/production.ini
|
|
16
|
+
- dbchanges2/Core/2026-07-30a - Error escalation cron job.sql
|
|
17
|
+
related:
|
|
18
|
+
- ../../_underscore/features/error-reporting-issue-event.md
|
|
19
|
+
- ../../../1.0/apps/tools/features/errors-curation-console.md
|
|
20
|
+
- ./creating-worker-actions.md
|
|
21
|
+
- ./clickup-project-routing.md
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Summary
|
|
25
|
+
|
|
26
|
+
`_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of
|
|
27
|
+
**escalation, de-escalation, ClickUp ticketing, reminders, business email, owner learning, and
|
|
28
|
+
GC** for the Issue/Event error-reporting pipeline. Capture is `_underscore`'s job (see
|
|
29
|
+
[error-reporting-issue-event](../../_underscore/features/error-reporting-issue-event.md)); the
|
|
30
|
+
Tools `/errors` console curates but **notifies nobody**. All outbound notification funnels
|
|
31
|
+
through this one action, so there is exactly one place to mute or debug alerting.
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
### Urgency = max of two independent axes, plus a floor
|
|
36
|
+
|
|
37
|
+
- **Volume** — occurrences in a 10-minute window: **20 / 50 / 100**.
|
|
38
|
+
- **Neglect** — hours an *unacknowledged* ClickUp task has sat: **1 / 4 / 24**.
|
|
39
|
+
- **Floor** — the Issue's declared `minimumUrgency` applies under *both* axes.
|
|
40
|
+
|
|
41
|
+
The neglect axis exists because volume alone cannot see a rare-but-serious error, nor a serious
|
|
42
|
+
error that everyone is ignoring.
|
|
43
|
+
|
|
44
|
+
### Fast up, slow down
|
|
45
|
+
|
|
46
|
+
Escalation applies **the same run** a band is crossed. De-escalation requires the count to sit
|
|
47
|
+
under a **lower release band (12 / 35 / 70)** for **3 consecutive windows**. Without that gap,
|
|
48
|
+
an issue hovering near a threshold flaps every minute and burns a ClickUp write each time.
|
|
49
|
+
|
|
50
|
+
### Task-creation gate
|
|
51
|
+
|
|
52
|
+
A ClickUp task is created only when urgency ≥ NORMAL, **or** a floor is set, **or** the Issue is
|
|
53
|
+
curated, **or** it is still rare. Creating a task for every distinct error on first sight is what
|
|
54
|
+
buries a team.
|
|
55
|
+
|
|
56
|
+
### Reminders
|
|
57
|
+
|
|
58
|
+
Backed off **30m → 1h → 4h → daily**. Never sent on de-escalation. Suppressed entirely while
|
|
59
|
+
`dtMutedUntil` is in the future.
|
|
60
|
+
|
|
61
|
+
### Sprint promotion
|
|
62
|
+
|
|
63
|
+
Uses the ClickUp **v2 multi-list** "Add Task To List" endpoint
|
|
64
|
+
(`POST /list/{id}/task/{taskId}`) so the errors list stays the task's **permanent home**. The v3
|
|
65
|
+
`home_list` move endpoint was rejected: v3 is not reliably deployed, it needs a workspace id
|
|
66
|
+
that exists nowhere in the codebase, and it requires `status_mappings` maintenance.
|
|
67
|
+
|
|
68
|
+
The target sprint list is resolved from `[clickup] sprint_folder_id` because `Team.Sprints`
|
|
69
|
+
carries no ClickUp list id — `GET /folder/{id}/list`, then the list whose `start_date`/`due_date`
|
|
70
|
+
range contains today.
|
|
71
|
+
|
|
72
|
+
### Recurrence (episodes)
|
|
73
|
+
|
|
74
|
+
One `IssueClickupTasks` row per ClickUp **episode**. Closing the task fires a webhook that stamps
|
|
75
|
+
`dtResolved` and increments `recurrenceCount`. A later occurrence opens a **new** task linked to
|
|
76
|
+
its predecessors via `POST /task/{id}/link/{links_to}` — a **link, not a dependency**; a
|
|
77
|
+
dependency would *block* the task. A cooldown prevents recreating a task 60 seconds after
|
|
78
|
+
someone closed it while the error was still firing.
|
|
79
|
+
|
|
80
|
+
### Acknowledgement
|
|
81
|
+
|
|
82
|
+
ClickUp is the acknowledge/resolve system — developers already live there, and a second inbox in
|
|
83
|
+
Tools would just be ignored. **Any** status change on an error task counts as acknowledgement,
|
|
84
|
+
which is safe only because the automation never sets status itself.
|
|
85
|
+
|
|
86
|
+
### Area-level owner learning (observation only)
|
|
87
|
+
|
|
88
|
+
`IssueAreaOwners` learns an owner after **5 in a row** and ships **observation-only** — a hint in
|
|
89
|
+
the task body, never an auto-assignment. Misassignment is expensive: a few wrong tickets and
|
|
90
|
+
developers stop trusting the queue. Learning is per **area**, not per issue, because one issue
|
|
91
|
+
rarely recurs enough to learn from before it is fixed.
|
|
92
|
+
|
|
93
|
+
### Garbage collection (split, deliberately)
|
|
94
|
+
|
|
95
|
+
- **Events** purge at **7 days**.
|
|
96
|
+
- **Issues** purge only when nothing curated them: not `isManaged`, no `issueKey`, no floor, no
|
|
97
|
+
recipients, no ClickUp history, no events.
|
|
98
|
+
|
|
99
|
+
The original blanket 30-day delete would have **cascaded a configured business issue's email
|
|
100
|
+
recipients away** after a quiet month.
|
|
101
|
+
|
|
102
|
+
## Configuration
|
|
103
|
+
|
|
104
|
+
- **`[_underscore] error_escalation_enabled`** — added to **all 8** worker2 configs: `1` in
|
|
105
|
+
`production.ini` **only**, `0` elsewhere. It is the master switch for **all outbound
|
|
106
|
+
notification**; errors are still **captured** regardless, so non-production records everything
|
|
107
|
+
and notifies nobody.
|
|
108
|
+
- Gated on a config key rather than string-matching `_Environment::$name` because
|
|
109
|
+
`_Environment` has **no `isProduction()`** — only a raw name string and an ad-hoc
|
|
110
|
+
`substr($name, 0, 4) == 'dev-'` idiom. Guessing env names would silently spam, or silently
|
|
111
|
+
stay quiet, in the wrong place.
|
|
112
|
+
- Read via `_Config::_underscore(key, false)`. **The second falsy argument matters:** it makes
|
|
113
|
+
`_Config` return `false` for a missing key/group **instead of throwing**, and an uncaught
|
|
114
|
+
throw would fail the whole cron.
|
|
115
|
+
- **`[clickup] sprint_folder_id`** — added to `worker2/Config/production.ini`, left **empty with
|
|
116
|
+
instructions**. Note `[clickup]` exists in only **2 of the 8** worker2 configs.
|
|
117
|
+
|
|
118
|
+
## Gotchas / known issues
|
|
119
|
+
|
|
120
|
+
- **The working-set query must be a UNION, not an OR.** It is a UNION of three separately
|
|
121
|
+
indexable branches. MySQL will **not** `index_merge` across an `OR` when one branch is a
|
|
122
|
+
correlated `EXISTS`, so the OR form **full-scans `Issues` every 60 seconds, forever**.
|
|
123
|
+
- **`INSERT ... SELECT ... WHERE NOT EXISTS` is not a concurrency guard** under REPEATABLE READ.
|
|
124
|
+
The open-episode guard is a UNIQUE key on a VIRTUAL generated column, and the code **claims
|
|
125
|
+
the episode before calling ClickUp** so a losing run fails before creating a duplicate
|
|
126
|
+
external task — see the `_underscore` doc for the full mechanics.
|
|
127
|
+
- **Business email must be scoped per client.** Recipients were resolved from the **most recent
|
|
128
|
+
event's** client, so with two clients firing in the same window **client B's business users
|
|
129
|
+
received email about client A's data**. It now sends one email **per distinct client that
|
|
130
|
+
actually fired in the window**.
|
|
131
|
+
- **Business email bodies/subjects must not carry `errorMessage` or `trace`.** Both are seeded
|
|
132
|
+
once from whichever client created the Issue and are never re-scoped. An uncurated business
|
|
133
|
+
issue gets a **neutral** subject rather than another client's raw error text.
|
|
134
|
+
- **`IssueEmailAddresses.clientId = 0` means "all clients"** and requires an explicit
|
|
135
|
+
confirmation flag in the Tools console; list views render it as **"ALL CLIENTS"**.
|
|
136
|
+
- **The ClickUp webhook has no HMAC signature verification** (pre-existing, unfixed). A forged
|
|
137
|
+
POST can mark issues acknowledged — freezing the neglect axis — or resolve episodes. That is
|
|
138
|
+
alert suppression against the error-reporting system itself. Fix before relying on this
|
|
139
|
+
pipeline as the only alerting path.
|
|
140
|
+
- **`_Worker_Sentry::Webhook` writes into the same ClickUp list (`901110669877`)** this cron
|
|
141
|
+
targets. Both pipelines run in parallel by design until the Sentry-removal ticket lands, so
|
|
142
|
+
expect duplicate-looking tasks from two sources.
|
|
143
|
+
|
|
144
|
+
## Change history
|
|
145
|
+
|
|
146
|
+
- 2026-07-30 — Built as part of TRUE-78188: action renamed `SyncWithClickup` → `Escalate`; new
|
|
147
|
+
`Worker/Clickup/ErrorTask.php`. Two-axis urgency (volume 20/50/100 + neglect 1/4/24) with a
|
|
148
|
+
`minimumUrgency` floor; fast-up/slow-down with release bands 12/35/70 over 3 windows; a
|
|
149
|
+
task-creation gate; 30m→1h→4h→daily reminders honouring `dtMutedUntil`; sprint promotion via
|
|
150
|
+
the v2 multi-list endpoint (v3 `home_list` rejected); episode recurrence via task *links* not
|
|
151
|
+
dependencies; a UNION (not OR) working-set query; and split GC (Events 7d, Issues only when
|
|
152
|
+
uncurated). Fixed from review: per-client business email scoping (client B was receiving
|
|
153
|
+
client A's data), neutral subjects with no errorMessage/trace, seeding new issues at their
|
|
154
|
+
declared floor, `_String::generateUuid()` over MySQL `UUID()`, and the row-alias form of
|
|
155
|
+
`ON DUPLICATE KEY UPDATE`. Added `error_escalation_enabled` (all 8 configs) and
|
|
156
|
+
`[clickup] sprint_folder_id`. (jcardinal)
|
|
157
|
+
</content>
|
|
@@ -393,6 +393,9 @@ Signed integers can reference positive and negative numbers, but unsigned can on
|
|
|
393
393
|
1. **Use the `_Model` layer.** The metadata-driven ORM builds and escapes queries for you — prefer it over hand-written SQL whenever possible.
|
|
394
394
|
2. **When you must write SQL by hand, escape every interpolated value with `_Database::escape()`** before placing it in the query string.
|
|
395
395
|
* `_Database::escape()` adds MySQL escape characters (`\`, `'`, `"`, newlines, etc.) and is the correct helper for query interpolation. Do **not** confuse it with `_Database::protect()`, which *strips* `% \ / * " '` and the literal ` or` — that is lossy filtering, not escaping, and will corrupt data you intend to store intact.
|
|
396
|
+
* **Prepared statements are not available.** `_Query` accepts only a raw SQL string, and `_Database_Driver` declares no `prepare`/`bind` (verified 2026-07-30 — neither the Mysql nor the Pgsql driver implements one). Escaping and casting *are* the parameterization mechanism in 2.0; do not go looking for a bind API, and do not trust any guidance that names a `_Db` class — there isn't one.
|
|
397
|
+
* **Escaping protects values, not identifiers.** Table names, column names, sort directions, and `LIMIT` expressions must come from a **hardcoded allowlist** — never from a request, and never merely escaped. `ORDER BY " . _Database::escape($_GET['sort'])` is still injectable.
|
|
398
|
+
* **Validate enum values in PHP.** `FIELD_LIST` performs no enum validation and converts any falsy value to SQL `NULL`; MySQL coerces rather than rejects an unknown member. Check against the allowed set before the value reaches the query — a typo'd urgency becomes `NULL`, not an error.
|
|
396
399
|
|
|
397
400
|
```php
|
|
398
401
|
// Bad — raw interpolation
|
package/knowledge/INDEX.md
CHANGED
|
@@ -14,12 +14,12 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
14
14
|
- **walmarttechservices** (Walmart Tech Services) — 1 doc(s) → [1.0/apps/walmarttechservices/INDEX.md](1.0/apps/walmarttechservices/INDEX.md)
|
|
15
15
|
- **test** (Test) — 13 doc(s) → [1.0/apps/test/INDEX.md](1.0/apps/test/INDEX.md)
|
|
16
16
|
- **toga** (TOGa) — 2 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
|
|
17
|
-
- **tools** (Tools) —
|
|
17
|
+
- **tools** (Tools) — 12 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
|
|
18
18
|
|
|
19
19
|
## 2.0 framework
|
|
20
20
|
|
|
21
21
|
- **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
22
|
-
- **worker2** (Worker) —
|
|
22
|
+
- **worker2** (Worker) — 38 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
23
23
|
- **api2** (API) — 20 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
24
24
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
25
25
|
- **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
package/package.json
CHANGED
package/rules/common/security.md
CHANGED
|
@@ -4,24 +4,74 @@ These rules are critical. Violations create exploitable vulnerabilities. Follow
|
|
|
4
4
|
|
|
5
5
|
## SQL injection prevention
|
|
6
6
|
|
|
7
|
-
Never build SQL strings by concatenating or interpolating
|
|
7
|
+
Never build SQL strings by concatenating or interpolating **unescaped** values.
|
|
8
|
+
|
|
9
|
+
> **Neither framework supports prepared statements.** This rule previously demanded them and
|
|
10
|
+
> gave `_Db::select()` as the correct pattern. **`_Db` does not exist.** Verified 2026-07-30:
|
|
11
|
+
> in 2.0, `_Query` accepts only a raw SQL string and `_Database_Driver` declares no
|
|
12
|
+
> `prepare`/`bind` (neither the Mysql nor the Pgsql driver implements one); 1.0 is the same
|
|
13
|
+
> through `App_Database`. There are zero uses of `bind_param`, `mysqli_stmt`, `PDO::`, or
|
|
14
|
+
> `::prepare(` anywhere in the non-vendor codebase. Escaping **is** the parameterization
|
|
15
|
+
> mechanism here, so that is what this rule requires.
|
|
16
|
+
|
|
17
|
+
**Prefer the model layer.** `_Model` (2.0) and `App_Model` (1.0) build and escape queries for
|
|
18
|
+
you. Hand-written SQL is the fallback, not the default.
|
|
19
|
+
|
|
20
|
+
**When you write SQL by hand, every interpolated value must be escaped or cast — no exceptions
|
|
21
|
+
for "internal" or "trusted" sources.**
|
|
8
22
|
|
|
9
23
|
```php
|
|
10
24
|
// CRITICAL VIOLATION — never do this
|
|
11
|
-
$sql = "SELECT * FROM
|
|
12
|
-
$sql = "SELECT * FROM
|
|
25
|
+
$sql = "SELECT * FROM Users WHERE id = " . $_GET['id'];
|
|
26
|
+
$sql = "SELECT * FROM Orders WHERE email = '$email'";
|
|
27
|
+
|
|
28
|
+
// CORRECT (2.0) — cast numerics, escape strings
|
|
29
|
+
$sql = "
|
|
30
|
+
SELECT *
|
|
31
|
+
FROM Orders
|
|
32
|
+
WHERE
|
|
33
|
+
id = " . ((int) $_GET['id']) . " AND
|
|
34
|
+
email = '" . _Database::escape($email) . "'
|
|
35
|
+
";
|
|
36
|
+
|
|
37
|
+
// CORRECT (1.0) — same shape, different helper
|
|
38
|
+
$sql = "
|
|
39
|
+
SELECT *
|
|
40
|
+
FROM Orders
|
|
41
|
+
WHERE
|
|
42
|
+
id = " . ((int) $_GET['id']) . " AND
|
|
43
|
+
email = '" . App_Database::sqlEscape($email) . "'
|
|
44
|
+
";
|
|
45
|
+
```
|
|
13
46
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
47
|
+
Applies to every operation: SELECT, INSERT, UPDATE, DELETE, and DDL.
|
|
48
|
+
|
|
49
|
+
**Cast numerics with `(int)`/`(float)`; escape strings.** A cast is sufficient on its own for a
|
|
50
|
+
numeric — there is nothing left to inject once the value is an integer. A string always needs the
|
|
51
|
+
escape helper.
|
|
52
|
+
|
|
53
|
+
**Identifiers are not covered by escaping.** `_Database::escape()` and
|
|
54
|
+
`App_Database::sqlEscape()` protect *values*, not table names, column names, sort directions, or
|
|
55
|
+
`LIMIT` expressions. Anything used as an identifier must come from a **hardcoded allowlist** —
|
|
56
|
+
never from a request, and never merely escaped.
|
|
57
|
+
|
|
58
|
+
```php
|
|
59
|
+
// CRITICAL VIOLATION — escaping does not make an identifier safe
|
|
60
|
+
$sql = "SELECT * FROM Orders ORDER BY " . _Database::escape($_GET['sort']);
|
|
17
61
|
|
|
18
|
-
// CORRECT —
|
|
19
|
-
|
|
62
|
+
// CORRECT — allowlist
|
|
63
|
+
$allowedSortColumns = ['dtCreated', 'total', 'status'];
|
|
64
|
+
$sortColumn = in_array($_GET['sort'], $allowedSortColumns, true) ? $_GET['sort'] : 'dtCreated';
|
|
65
|
+
$sql = "SELECT * FROM Orders ORDER BY $sortColumn";
|
|
20
66
|
```
|
|
21
67
|
|
|
22
|
-
|
|
68
|
+
**Enum columns need validating in PHP.** 2.0's `FIELD_LIST` performs no enum validation and
|
|
69
|
+
silently converts a falsy value to SQL `NULL`; MySQL will coerce rather than reject an unknown
|
|
70
|
+
member. Check the value against the allowed set before it reaches the query.
|
|
23
71
|
|
|
24
|
-
|
|
72
|
+
One caveat on 2.0's helpers: `_Database::escape()` is the correct one. Do **not** confuse it with
|
|
73
|
+
`_Database::protect()`, which *strips* `% \ / * " '` and the literal ` or` — lossy filtering, not
|
|
74
|
+
escaping, and it will corrupt data you meant to store intact.
|
|
25
75
|
|
|
26
76
|
## XSS prevention
|
|
27
77
|
|