toga-ai 1.0.477 → 1.0.479
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/tools/features/design-demo-admin.md +42 -1
- package/knowledge/1.0/standards/frontend.md +75 -0
- package/knowledge/2.0/apps/_underscore/INDEX.md +1 -1
- package/knowledge/2.0/apps/_underscore/architecture.md +17 -0
- package/knowledge/2.0/apps/_underscore/features/email-send-pipeline.md +1 -0
- package/knowledge/2.0/apps/_underscore/features/per-client-database-connections.md +36 -1
- package/knowledge/2.0/apps/dbchanges2/architecture.md +117 -0
- package/knowledge/2.0/apps/worker2/INDEX.md +2 -1
- package/knowledge/2.0/apps/worker2/features/all-client-email-queue-monitor.md +170 -0
- package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md +19 -1
- package/knowledge/2.0/standards/framework-rules.md +24 -0
- package/knowledge/INDEX.md +1 -1
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Tools
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-30
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- tools/_/app/design/github.php
|
|
@@ -19,6 +19,7 @@ files:
|
|
|
19
19
|
- tools/composer.json
|
|
20
20
|
related:
|
|
21
21
|
- ../architecture.md
|
|
22
|
+
- ../../../standards/frontend.md
|
|
22
23
|
- ../features/persona-gated-navigation.md
|
|
23
24
|
- ../features/saml-sso-auth.md
|
|
24
25
|
- ../workflows/deploy-to-elastic-beanstalk-al2023.md
|
|
@@ -92,6 +93,29 @@ on the next write of that project.json. Tab management via a per-pill "⋯" menu
|
|
|
92
93
|
- **Reorder** — a single-file `tabs.json` commit, validated as a **permutation** of the
|
|
93
94
|
existing list.
|
|
94
95
|
|
|
96
|
+
### Text-storage invariant: JSON holds plain text, escape once at display
|
|
97
|
+
|
|
98
|
+
`tabs.json` tab names and `project.json` `title` values are stored as **plain text**. The
|
|
99
|
+
single correct escape happens at display time in `assets/js/design.js` `esc()` (and in
|
|
100
|
+
`buildStub()`, which `htmlspecialchars()` the title into the generated `index.html`).
|
|
101
|
+
|
|
102
|
+
`App_Design_Github::decodeEntities()` (one pass of `html_entity_decode` with
|
|
103
|
+
`ENT_QUOTES | UTF-8`) enforces that invariant at two chokepoints:
|
|
104
|
+
|
|
105
|
+
- **On read out of storage** — `loadTabs()`, `normalizeTabs()` (including the legacy scalar
|
|
106
|
+
`tab`), and the project read's `title`.
|
|
107
|
+
- **On names/titles read off a request** — `createTab()`, `renameTab()` (both old and new
|
|
108
|
+
name), `deleteTab()`, `reorderTabs()`, `filterKnownTabs()`, `createProject()`, and
|
|
109
|
+
`upload()` titles.
|
|
110
|
+
|
|
111
|
+
Both sides are required: decoding only reads would make `renameTab()` compare a decoded
|
|
112
|
+
`H&H` against a stored `H&H` and fail with *"No such tab."*; decoding only writes would
|
|
113
|
+
leave already-bad stored data permanently double-escaped on screen.
|
|
114
|
+
|
|
115
|
+
**Self-healing:** every tab write rewrites `tabs.json` and each affected `project.json` in
|
|
116
|
+
full, so pre-encoded stored values are re-persisted as plain text on the next
|
|
117
|
+
rename/delete/reorder. No manual repair of the `forward` repo is needed.
|
|
118
|
+
|
|
95
119
|
### Actions & UX
|
|
96
120
|
|
|
97
121
|
- **Publish a version** — uploads a self-contained HTML export as the next `vN/index.html`,
|
|
@@ -147,10 +171,27 @@ None — internal/shared design-team tool.
|
|
|
147
171
|
`curl_close()` calls. Audit ported code for other deprecated-in-8.x calls.
|
|
148
172
|
- **Contents API inlines only ≤1 MB** — read exports with the raw media type (see above).
|
|
149
173
|
- **Publishing pushes to `forward`'s `_main`** and triggers EB auto-deploy; no staging.
|
|
174
|
+
- **Double-escaped names (`H&H` shown literally, rename fails with "No such tab.")** —
|
|
175
|
+
caused by values that arrived **already HTML-encoded** and were then correctly escaped once
|
|
176
|
+
more at display. The render path was never wrong; no current code path produces the
|
|
177
|
+
encoding (`createTab`/`createProject` and the JS transport `window.prompt` →
|
|
178
|
+
`URLSearchParams` → `$postField` trim/`mb_substr` are clean), so encoded values entered from
|
|
179
|
+
outside — a paste from rendered HTML, or a pre-fix build. Fixed defensively with
|
|
180
|
+
`decodeEntities()` on both the read and request sides (see above). Generated project stubs
|
|
181
|
+
were affected identically, because `buildStub()`'s single correct `htmlspecialchars()` was a
|
|
182
|
+
*second* escape on a pre-encoded title; storing titles plain fixes the stubs too.
|
|
150
183
|
- Never hardcode the token in tracked source — a leaked `ghp_`/PAT must be rotated.
|
|
151
184
|
|
|
152
185
|
## Change history
|
|
153
186
|
|
|
187
|
+
- 2026-07-30 — Fixed HTML-entity double-escaping of tab names and project titles (a tab named
|
|
188
|
+
`H&H` rendered as `H&H`, and renaming it failed with "No such tab."). Added
|
|
189
|
+
`App_Design_Github::decodeEntities()` and applied it on every read out of storage
|
|
190
|
+
(`loadTabs`, `normalizeTabs` incl. legacy scalar `tab`, project `title`) and on every
|
|
191
|
+
name/title read off a request (`createTab`, `renameTab`, `deleteTab`, `reorderTabs`,
|
|
192
|
+
`filterKnownTabs`, `createProject`, `upload`), so all comparisons are decoded-vs-decoded and
|
|
193
|
+
storage re-persists plain text. Defensive — no current code path produced the encoding.
|
|
194
|
+
Verified by the developer; not yet committed to `tools`. (jcardinal)
|
|
154
195
|
- 2026-07-24 — Moved the Design Demo Admin from `forward` into the SSO-protected `tools` app
|
|
155
196
|
(`App_Design_Github` + `/design` MVC route + namespaced assets + nav entry); still commits
|
|
156
197
|
to `agilantsolutions/forward` `_main` and demos stay hosted at `demo.togatech.com`. Added
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Front-End Standards
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
project: Library
|
|
5
|
+
client: shared
|
|
6
|
+
type: standard
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-07-30
|
|
9
|
+
owners: [jcardinal]
|
|
10
|
+
files: []
|
|
11
|
+
related:
|
|
12
|
+
- ./backend-php.md
|
|
13
|
+
- ./framework-rules.md
|
|
14
|
+
- ../apps/tools/features/design-demo-admin.md
|
|
15
|
+
- ../apps/tools/features/talos-kb-documents-admin.md
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# Front-End Standards (1.0 Legacy — `Browser_` / server-rendered UI)
|
|
19
|
+
|
|
20
|
+
> **Scope.** Server-rendered UI and browser assets for the legacy **1.0** framework
|
|
21
|
+
> (`Browser_` classes, `mvc/` get/post handlers, `assets/js`, `assets/css`). 1.0 is in
|
|
22
|
+
> maintenance mode and historically inconsistent — match the file you are editing, and
|
|
23
|
+
> prefer the cleaner pattern for new code. Do not retrofit 2.0 front-end conventions.
|
|
24
|
+
|
|
25
|
+
## Output escaping: storage holds plain text, escape exactly once at display
|
|
26
|
+
|
|
27
|
+
**Rule.** Persisted values — database columns, JSON manifests, config — hold **plain,
|
|
28
|
+
unescaped text**. HTML escaping happens **exactly once**, at the moment the value is
|
|
29
|
+
written into an HTML context (`htmlspecialchars($v, ENT_QUOTES, 'UTF-8')` server-side, or
|
|
30
|
+
the view's `esc()` helper client-side).
|
|
31
|
+
|
|
32
|
+
**Never** escape on the way *in* to storage. An escaped value in storage is a latent bug:
|
|
33
|
+
the display layer escapes it a second time, and the user sees `H&H` instead of `H&H`.
|
|
34
|
+
|
|
35
|
+
### Why it recurs
|
|
36
|
+
|
|
37
|
+
The failure mode is a **round trip**, not a bad escape function:
|
|
38
|
+
|
|
39
|
+
1. A value is rendered into an edit form / prompt already escaped for HTML.
|
|
40
|
+
2. The user submits that form unchanged.
|
|
41
|
+
3. The escaped string is stored verbatim — storage now holds `H&H`.
|
|
42
|
+
4. Display escapes correctly once more → `H&H` on screen.
|
|
43
|
+
|
|
44
|
+
Each round trip adds a level. Equality comparisons then fail too, because a freshly typed
|
|
45
|
+
`H&H` no longer matches the stored `H&H` — which surfaces as a confusing
|
|
46
|
+
"record not found" rather than as a display bug.
|
|
47
|
+
|
|
48
|
+
### Required practice
|
|
49
|
+
|
|
50
|
+
- Pre-fill form inputs and JS prompts with the **plain** value; let the templating layer
|
|
51
|
+
do the one escape it owns.
|
|
52
|
+
- When a store may already contain encoded data, **decode on both sides** of the boundary:
|
|
53
|
+
a single `html_entity_decode($v, ENT_QUOTES, 'UTF-8')` on every read out of storage
|
|
54
|
+
**and** on every value read off a request. Decoding only one side leaves either broken
|
|
55
|
+
comparisons or undisplayable legacy rows.
|
|
56
|
+
- Prefer stores that **rewrite records in full** on update, so the decode-on-write is
|
|
57
|
+
self-healing and no manual data repair is needed.
|
|
58
|
+
- Escape for the **right context**: `htmlspecialchars` for HTML text/attributes,
|
|
59
|
+
`json_encode` (with `JSON_HEX_*` for inline `<script>`) for JavaScript. Never
|
|
60
|
+
hand-build JSON or rely on `htmlspecialchars` inside a JS context.
|
|
61
|
+
|
|
62
|
+
### Precedents in the codebase
|
|
63
|
+
|
|
64
|
+
- `tools/mvc/talos/vocabulary/post.php:74-80` — inline comment recording the rule after
|
|
65
|
+
form inputs pre-filled with `htmlspecialchars()` output accumulated encoding.
|
|
66
|
+
- `tools/_/app/design/github.php` — `App_Design_Github::decodeEntities()` applied to tab
|
|
67
|
+
names and project titles on both the storage-read and request-read sides.
|
|
68
|
+
|
|
69
|
+
## Change history
|
|
70
|
+
|
|
71
|
+
- 2026-07-30 — Created. Establishes the escape-once/plain-text-storage invariant after a
|
|
72
|
+
second independent occurrence in `tools` (Design Demo Admin, following the Talos
|
|
73
|
+
vocabulary tool). (jcardinal)
|
|
74
|
+
</content>
|
|
75
|
+
</invoke>
|
|
@@ -24,7 +24,7 @@
|
|
|
24
24
|
| [_Model magic-field access (__get without __isset)](features/model-magic-field-access.md) | `_Model` exposes DB columns as "magic" properties via `__get()`, but it defines **no** `__isset()`. | _underscore/Model/Core/Model.php |
|
|
25
25
|
| [_Model::save() vs raw _Query — no atomic conditional update](features/model-save-vs-query-atomic-update.md) | `_Model::save()` is a plain load-then-write ORM primitive and **cannot express an atomic conditional update** (an optimistic-concurrency / row-claim guard such | _underscore/Model.php, _underscore/Query.php |
|
|
26
26
|
| [NetSuite REST Client (_Component_Api_Netsuite) — record writes & SuiteQL](features/netsuite-rest-client.md) | `_Component_Api_Netsuite` is the **2.0 `_underscore` NetSuite REST client** — the shared primitive every worker2/api2 NetSuite caller uses for record GETs, Suit | _underscore/Component/Api/Netsuite/Netsuite.php |
|
|
27
|
-
| [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, api2/Controller/Index.php |
|
|
27
|
+
| [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/Query.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, api2/Controller/Index.php |
|
|
28
28
|
| [Persona Name Translation (PersonaTranslations sidecar)](features/persona-name-translation.md) | Serves Persona **names** in multiple languages by adding a per-language **sidecar** table `PersonaTranslations`, reusing the platform's existing metadata-driven | _underscore/Model/Client/PersonaTranslation.php, dbchanges2/Client/2026-07-22b - PersonaTranslations.sql, dbchanges2/Core/2026-07-22a - PersonaTranslationsRecord.sql, dbchanges2/Client/2026-07-22c - PersonaTranslationsAcl.sql, dbchanges2/Client_CompassCanada/2026-07-22 - PersonaTranslationsFrench.sql, toga2-commerce/src/pages/Account/view/MySettingsView.tsx |
|
|
29
29
|
| [Recursive Item Fulfillments (upstream mirroring)](features/recursive-item-fulfillments.md) | In a multi-tier supply chain a sales order (SO) spawns a purchase order (PO) that becomes another SO downstream, and so on. | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillmentItem.php, _underscore/Model/Client/ItemFulfillmentItemUnit.php, _underscore/Model/Client/ItemFulfillmentPackage.php, _underscore/Model/Compass/AdvanceShippingNotice.php, dbchanges2/Core/2026-02-13 - 75601 - RecursiveItemFulfillmentCreation.sql, dbchanges2/Core/2026-06-04 - RecursiveItemFulfillmentPut.sql, dbchanges2/Client_Compass/2026-07-02a - FixSA133377TrackingSerialAndDuplicateIF.sql |
|
|
30
30
|
| [Surface Resolver (_Model_Core_Surface::resolve — replaces Page::meta)](features/surface-resolver.md) | The runtime for the platform-wide **Surface** UI presentation layer: 9 `_underscore` models plus a cached resolver, `_Model_Core_Surface::resolve(&$api, string | _underscore/Model/Core/Surface.php, _underscore/Model/Client/AclRecordScript.php, _underscore/Model/Core/RecordScript.php, dbchanges2/Core/2026-06-30a - SurfaceMetaGroupAndSalesOrderSections.sql, dbchanges2/Client/2026-06-30a - SurfaceMetaGroupAcl.sql, dbchanges2/Client_Quad/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Client_CompassCanada/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Core/2026-06-29b - SurfaceMetaPublicReadAcl.sql, dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql, dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql, dbchanges2/Client_Compass/2026-07-15f - SalesOrderRecordActionsRemoveDeadConfigRuleOverrides.sql, dbchanges2/Core/2026-07-17h - Update - ClearApprovalsFilterButtonConfig.sql, dbchanges2/Client_Compass/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Compass/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_Quad/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Quad/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, _underscore/Model/Core/SurfaceElement.php, _underscore/Model/Core/Action.php, _underscore/Model/Core/Vocabulary.php, _underscore/Model/Core/VocabularyTerm.php, _underscore/Model/Core/Message.php, _underscore/Model/Client/SurfaceOverride.php, _underscore/Model/Client/MessageTranslation.php, _underscore/Model/Client/ThemeToken.php, _underscore/Model/Core/Page.php, api2/Component/Api/V2/V2.php |
|
|
@@ -194,6 +194,23 @@ All non-production environments are **all-in-one**: a single cluster per environ
|
|
|
194
194
|
holds `Core`, `Client_<Tenant>`, `Archive_<Tenant>`, and `Logs_<Tenant>` together
|
|
195
195
|
(no family split). Region/reader routing still applies per cluster.
|
|
196
196
|
|
|
197
|
+
> **⚠ Consequence — never write a query that spans two database families.** Because
|
|
198
|
+
> production splits the families across the four dedicated clusters above, a single SQL
|
|
199
|
+
> statement that joins or sub-selects across them (e.g. `Client_Towfoundation` ↔ `Core`,
|
|
200
|
+
> or `Client_<Tenant>` ↔ `Archive_<Tenant>`) **cannot resolve in production** — the foreign
|
|
201
|
+
> schema is not on that server. And because **every non-production environment is all-in-one,
|
|
202
|
+
> such a query runs perfectly in local/dev/QA/stage**, so the failure surfaces only after
|
|
203
|
+
> release. Treat a successful non-prod run as **no evidence** that a query is cluster-safe.
|
|
204
|
+
> Cross-family data must be assembled in **PHP across two connections** (`_Database` named
|
|
205
|
+
> connections, one per family), never in one statement. Note the corollary that platform
|
|
206
|
+
> databases `Core`, `Forecast`, and `Team` **do** share `prod-core`, so those are same-cluster
|
|
207
|
+
> — but a cross-*database* query is still discouraged, and in `dbchanges2` it is forbidden
|
|
208
|
+
> outright.
|
|
209
|
+
>
|
|
210
|
+
> For **`dbchanges2` migrations this is a hard, mechanically-enforced rule**: a `.sql` file may
|
|
211
|
+
> only reference tables in the one database its folder targets. See
|
|
212
|
+
> [dbchanges2 → Database isolation](../dbchanges2/architecture.md).
|
|
213
|
+
|
|
197
214
|
| Environment (aliases) | Host |
|
|
198
215
|
|---|---|
|
|
199
216
|
| `dev-sandbox` / `sandbox-dev` / developer sandbox | `dev.sandbox.database.togahub.com` |
|
|
@@ -6,10 +6,11 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-30
|
|
10
10
|
owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Database.php
|
|
13
|
+
- _underscore/Query.php
|
|
13
14
|
- _underscore/ApiRequest.php
|
|
14
15
|
- _underscore/Model/Client/Logs/Api.php
|
|
15
16
|
- api2/Controller/Index.php
|
|
@@ -17,6 +18,7 @@ related:
|
|
|
17
18
|
- ../architecture.md
|
|
18
19
|
- ../workflows/local-db-refresh-from-beta.md
|
|
19
20
|
- ./database-alias-repointing.md
|
|
21
|
+
- ../../worker2/features/all-client-email-queue-monitor.md
|
|
20
22
|
---
|
|
21
23
|
|
|
22
24
|
## Summary
|
|
@@ -112,6 +114,31 @@ here — they live in `Config/*.ini`.)
|
|
|
112
114
|
`Core.Clients.clientDatabaseId` → `Core.Databases.name`. This mirrors the log-database resolution
|
|
113
115
|
already done in `_underscore`'s `Email.php`. A string-built name fails only for the clients whose
|
|
114
116
|
names happen to diverge, so it passes local testing and breaks in production.
|
|
117
|
+
- **The same rule applies to the LOG database: never build `'Logs_' . $clientIdentifier`.**
|
|
118
|
+
Resolve `Core.Clients.logDatabaseId` → `Core.Databases.name` (`Compass_Usa` → `Logs_Compass`).
|
|
119
|
+
A string-built log-DB name silently targets a schema that does not exist.
|
|
120
|
+
- **`_Database::registerClientDatabases()` is unsafe for a loop over ALL clients.** It INNER
|
|
121
|
+
JOINs the client database *and* archive database alongside the log database, so a client
|
|
122
|
+
missing either row drops out of the result set entirely — and the method then dereferences the
|
|
123
|
+
null row (`$database->clientId`) and throws. It also opens three connections per client when a
|
|
124
|
+
caller may need only one. For an all-client fan-out, run a **log-database-only** variant with
|
|
125
|
+
**LEFT OUTER JOINs** and report clients that resolve to nothing, rather than letting them
|
|
126
|
+
vanish. Worked example: [All-Client Email Queue Monitor](../../worker2/features/all-client-email-queue-monitor.md).
|
|
127
|
+
- **A per-client loop must NOT reuse the shared aliases.** `DB_CLIENT` / `DB_CLIENT_LOGS` /
|
|
128
|
+
`Archive` are single slots — each iteration clobbers the previous one, and all live connection
|
|
129
|
+
state is alias-keyed. Register each client's database **under its real schema name with no
|
|
130
|
+
alias** (guard with `isset(_Database::$_registers[$name])`) when you need many clients open at
|
|
131
|
+
once.
|
|
132
|
+
- **There is no `_Query::setTimeout()`.** Only `_ApiRequest::setTimeout()` exists. A 2.0 request
|
|
133
|
+
or worker therefore has **no per-query timeout**: a hung database cluster blocks until the
|
|
134
|
+
connection itself times out. This matters most in fan-out loops across many client clusters,
|
|
135
|
+
where one unreachable cluster stalls the whole pass.
|
|
136
|
+
- **Classify driver failures by MySQL error NUMBER, not message text.** `_Query` throws a plain
|
|
137
|
+
`Exception` whose message embeds the driver error on an `Error #: <number>` line
|
|
138
|
+
(`_underscore/Query.php:346`). Matching text like `"doesn't exist"` is locale- and
|
|
139
|
+
engine-dependent (MySQL and MariaDB word it differently). Parse `/Error #:\s*(\d+)/` and
|
|
140
|
+
compare to the code — e.g. `1146` (`ER_NO_SUCH_TABLE`) is how a worker tolerates a per-tenant
|
|
141
|
+
table that predates a migration.
|
|
115
142
|
- **Client DBs do not share a charset, so any hardcoded `COLLATE` is invalid somewhere.** Emitting
|
|
116
143
|
e.g. `COLLATE utf8mb4_unicode_ci` against a `latin1` column errors with `COLLATION ... is not
|
|
117
144
|
valid for CHARACTER SET 'latin1'`. For cross-tenant string comparison use `CAST(expr AS BINARY)`
|
|
@@ -152,6 +179,14 @@ here — they live in `Config/*.ini`.)
|
|
|
152
179
|
|
|
153
180
|
## Change history
|
|
154
181
|
|
|
182
|
+
- 2026-07-30 — Added four framework-level gotchas surfaced while building the all-client email
|
|
183
|
+
queue monitor: the log-DB name has the same never-string-build rule as the client DB;
|
|
184
|
+
`registerClientDatabases()` is unsafe for an all-client loop (INNER JOINs on client+archive
|
|
185
|
+
drop clients then throw on the null row, and it opens three connections); a per-client loop
|
|
186
|
+
must register under the **real schema name with no alias** because the aliases are single
|
|
187
|
+
slots; **`_Query::setTimeout()` does not exist**, so 2.0 has no per-query timeout; and driver
|
|
188
|
+
failures must be classified by MySQL **error number** parsed from `Error #: <n>` rather than
|
|
189
|
+
message text. (jcardinal)
|
|
155
190
|
- 2026-07-28 — **Corrected a wrong rule:** a client's schema name is *not* reliably
|
|
156
191
|
`'Client_' . clientIdentifier` (`Compass_Usa` → `Client_Compass`); resolve
|
|
157
192
|
`Clients.clientDatabaseId` → `Databases.name`. Recorded that client DBs do **not** share a
|
|
@@ -39,6 +39,10 @@ in a folder, then `b`, `c`, …). One folder per database; place client changes
|
|
|
39
39
|
`Client_<Name>` folder. Never edit an already-run migration — add a new dated file instead. The
|
|
40
40
|
`Client/` **blank must carry the baseline DATA seed** (roles, full ACL, reference/lookup tables,
|
|
41
41
|
UI config) — **not just schema**; a schema-only blank produces non-functional clients.
|
|
42
|
+
**A file may only reference tables in the ONE database its folder targets** — `Core`, `Client_*`,
|
|
43
|
+
`Archive_*`, `Logs*`, and `Cache` are on **separate production clusters**, so any
|
|
44
|
+
`OtherDatabase.Table` reference is unrunnable in production even though it works locally
|
|
45
|
+
(see *Database isolation* below; enforced by the `dbchanges2-cluster-isolation` hook).
|
|
42
46
|
|
|
43
47
|
## File naming convention (the execution contract)
|
|
44
48
|
|
|
@@ -90,6 +94,101 @@ Spglobal, Trividiahealth, True, Wje, Wmchealth, Ynhh).
|
|
|
90
94
|
`Client/` and `Logs_Client/` also contain `BLANK_CLIENT_DATABASE` / `BLANK_CLIENT_LOGS_DATABASE`
|
|
91
95
|
seed scripts used to provision a brand-new tenant DB from scratch.
|
|
92
96
|
|
|
97
|
+
## Database isolation — never query across databases (HARD RULE)
|
|
98
|
+
|
|
99
|
+
**A `.sql` file in `dbchanges2` may only reference tables in the ONE database its folder
|
|
100
|
+
targets, and it must reference them UNQUALIFIED.** Any `OtherDatabase.Table` reference — in a
|
|
101
|
+
`JOIN`, a subquery, a `SET @var = (SELECT …)`, an `INSERT … SELECT`, a `NOT EXISTS` guard, or a
|
|
102
|
+
`USE` statement — is a **hard violation**.
|
|
103
|
+
|
|
104
|
+
**Why:** in **production** the 2.0 databases are **not one server**. `Core`, each
|
|
105
|
+
`Client_<Tenant>`, each `Archive_<Tenant>`, `Logs`/`Logs_<Tenant>`, and `Cache` live on
|
|
106
|
+
**entirely separate clusters** (see `2.0/apps/_underscore/architecture.md` → *Database
|
|
107
|
+
architecture*). A cross-database query has no way to resolve the foreign schema there.
|
|
108
|
+
|
|
109
|
+
**Why this is a trap rather than an obvious error:** **locally and in every non-prod
|
|
110
|
+
environment all of these databases sit behind a single endpoint**, so a cross-database query
|
|
111
|
+
runs perfectly, the migration looks correct, review passes — and it fails only when it reaches
|
|
112
|
+
production. Never treat a successful local run as evidence that a query is cluster-safe.
|
|
113
|
+
**Assume every database other than the file's own target is unreachable, always.**
|
|
114
|
+
|
|
115
|
+
### Not allowed
|
|
116
|
+
|
|
117
|
+
This is the canonical violation — a `Client_<Tenant>` migration resolving `Core` ids to copy ACL
|
|
118
|
+
rows. `AclFieldPermissions` is in the client database, `Core.RecordFields` / `Core.Records` are
|
|
119
|
+
on the **core cluster**:
|
|
120
|
+
|
|
121
|
+
```sql
|
|
122
|
+
-- WRONG — Core.* is a different cluster; this cannot run in production
|
|
123
|
+
SET @serviceAddressFieldId = (
|
|
124
|
+
SELECT rf.id FROM Core.RecordFields rf
|
|
125
|
+
JOIN Core.Records r ON r.id = rf.recordId
|
|
126
|
+
WHERE r.`route` = 'entitlements' AND rf.`field` = 'serviceAddressId' LIMIT 1
|
|
127
|
+
);
|
|
128
|
+
SET @saleItemFieldId = (
|
|
129
|
+
SELECT rf.id FROM Core.RecordFields rf
|
|
130
|
+
JOIN Core.Records r ON r.id = rf.recordId
|
|
131
|
+
WHERE r.`route` = 'entitlements' AND rf.`field` = 'saleItemId' LIMIT 1
|
|
132
|
+
);
|
|
133
|
+
|
|
134
|
+
INSERT INTO AclFieldPermissions (uuid, recordFieldId, roleId, isWritable)
|
|
135
|
+
SELECT UUID(), @serviceAddressFieldId, sibling.roleId, 0
|
|
136
|
+
FROM AclFieldPermissions sibling
|
|
137
|
+
WHERE sibling.recordFieldId = @saleItemFieldId
|
|
138
|
+
AND @serviceAddressFieldId IS NOT NULL
|
|
139
|
+
AND NOT EXISTS (
|
|
140
|
+
SELECT 1 FROM AclFieldPermissions existing
|
|
141
|
+
WHERE existing.recordFieldId = @serviceAddressFieldId AND existing.roleId = sibling.roleId
|
|
142
|
+
);
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
### Allowed — replace the foreign lookup
|
|
146
|
+
|
|
147
|
+
Use whichever fits; the goal is that **every table named in the file lives in the file's own
|
|
148
|
+
database**:
|
|
149
|
+
|
|
150
|
+
1. **Hardcode the identifier.** The `Core` row's `uuid` is stable across environments, so
|
|
151
|
+
pre-generate/look it up **once at authoring time** and embed the literal. (Same discipline as
|
|
152
|
+
rule #6 — a v4 UUID literal, never `UUID()`.)
|
|
153
|
+
|
|
154
|
+
```sql
|
|
155
|
+
-- CORRECT — no foreign database read; the uuid literal is resolved at authoring time
|
|
156
|
+
SET @serviceAddressFieldId = (
|
|
157
|
+
SELECT id FROM RecordFields
|
|
158
|
+
WHERE uuid = '<pre-resolved v4 uuid of the serviceAddressId field>' LIMIT 1
|
|
159
|
+
);
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
2. **Resolve by slug / natural key inside the target database.** If the client database carries
|
|
163
|
+
its own copy of the identifying column (`field`, `route`, `name`, a code), match on that
|
|
164
|
+
instead of joining out to `Core`.
|
|
165
|
+
|
|
166
|
+
3. **Split the change into one file per database folder.** If work genuinely spans two
|
|
167
|
+
databases, write a `Core/` file **and** a `Client_<Name>/` (or `Client/`) file, each
|
|
168
|
+
self-contained. Order them by date/letter so the `Core` half lands first, and carry any value
|
|
169
|
+
between them as a **hardcoded literal**, not a query.
|
|
170
|
+
|
|
171
|
+
### Fan-out folders: no qualifier at all
|
|
172
|
+
|
|
173
|
+
In `Client/`, `Logs_Client/`, and `_modules/<module>/` the target database **name is not fixed**
|
|
174
|
+
— the executor runs the same file against every tenant DB. So there is no valid database
|
|
175
|
+
qualifier in these folders whatsoever: **reference every table unqualified**, including tables in
|
|
176
|
+
the database being targeted.
|
|
177
|
+
|
|
178
|
+
### Enforcement
|
|
179
|
+
|
|
180
|
+
This is enforced mechanically, not by convention. The `PreToolUse` hook
|
|
181
|
+
`.claude/hooks/toga/dbchanges2-cluster-isolation.js` inspects every `Write`/`Edit`/`MultiEdit`
|
|
182
|
+
of a `.sql` file under `dbchanges2`, resolves the target database from the folder, and **refuses
|
|
183
|
+
the write** if the SQL qualifies any table with a different database name (`Core`, `Client*`,
|
|
184
|
+
`Logs*`, `Archive*`, `Cache`, `Team`, including backticked and `USE` forms). Comments and string
|
|
185
|
+
literals are stripped first, and table **aliases** (`rf.id`, `sibling.roleId`) are never flagged.
|
|
186
|
+
`HISTORIC/` folders are skipped.
|
|
187
|
+
|
|
188
|
+
> **Scope:** this rule and its hook apply to **`dbchanges2` only**. The 1.0 **`dbchanges`** repo
|
|
189
|
+
> is **not** subject to it. If the hook ever misfires on legitimate SQL, `DBCHANGES2_ISOLATION_DISABLED=1`
|
|
190
|
+
> is an escape hatch **for false positives only** — never to land a cross-database query.
|
|
191
|
+
|
|
93
192
|
## `_modules` — reusable, opt-in change-sets
|
|
94
193
|
|
|
95
194
|
Some change-sets aren't applied to every client — only to clients that use a given **module**
|
|
@@ -189,6 +288,13 @@ its own header.)
|
|
|
189
288
|
`prePost` interceptor is registered unguarded in `Client_Rate/2026-07-15 - PreInterceptor.sql`
|
|
190
289
|
(record 191, `minDepth NULL`); a second guarded registration would have double-fired it. See
|
|
191
290
|
`clients/rate/features/whole-home-warranty-purchase-guard.md`.
|
|
291
|
+
8. **Never reference a database other than the folder's own target.** No `Core.Table` in a
|
|
292
|
+
`Client_<Name>/` file, no `Client_X.Table` in a `Core/` file, and **no database qualifier at
|
|
293
|
+
all** in the fan-out folders (`Client/`, `Logs_Client/`, `_modules/`). These databases are on
|
|
294
|
+
**separate production clusters**; the query works locally and dies in production. Resolve
|
|
295
|
+
foreign ids with a hardcoded v4 UUID literal or an in-database slug/natural key, or split the
|
|
296
|
+
work into one file per database folder. See *Database isolation* above — enforced by the
|
|
297
|
+
`dbchanges2-cluster-isolation` hook.
|
|
192
298
|
|
|
193
299
|
## Bulk data loads — batch, and stage large sets in a temp table
|
|
194
300
|
|
|
@@ -341,6 +447,17 @@ defined in `2.0/apps/_underscore/architecture.md`, and its change files create/a
|
|
|
341
447
|
tables that `_Model_*` classes map to.
|
|
342
448
|
|
|
343
449
|
## Change history
|
|
450
|
+
- 2026-07-29 — **Added *Database isolation — never query across databases* (HARD RULE) + rule #8.**
|
|
451
|
+
A `dbchanges2` `.sql` file may only reference tables in the one database its folder targets, and
|
|
452
|
+
must reference them unqualified; fan-out folders (`Client/`, `Logs_Client/`, `_modules/`) permit
|
|
453
|
+
no database qualifier at all. Rationale: in production `Core`, `Client_<Tenant>`,
|
|
454
|
+
`Archive_<Tenant>`, `Logs`/`Logs_<Tenant>`, and `Cache` are on **entirely separate clusters**,
|
|
455
|
+
while local/non-prod collapse them behind one endpoint — so a cross-database query passes review
|
|
456
|
+
and fails only in production. Replace foreign lookups with hardcoded v4 UUID literals, in-database
|
|
457
|
+
slugs/natural keys, or one file per database folder. Enforced mechanically by the new `PreToolUse`
|
|
458
|
+
hook `dbchanges2-cluster-isolation.js`, which refuses the write. Applies to `dbchanges2` **only** —
|
|
459
|
+
the 1.0 `dbchanges` repo is exempt. Cluster topology recorded in
|
|
460
|
+
`2.0/apps/_underscore/architecture.md` → *Database architecture*. (jcardinal)
|
|
344
461
|
- 2026-07-29 — Added *Self-referencing INSERT...SELECT guard — wrap the existing-set subquery in
|
|
345
462
|
a derived table*: an `INSERT...SELECT` guarded per-key against its own target must wrap the
|
|
346
463
|
existing-set subquery in a derived table (DISTINCT/LIMIT) so MySQL materializes it once —
|
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
|-----|---------|-------|
|
|
5
5
|
| [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php, worker2/composer.json |
|
|
6
6
|
| [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh, worker2/.platform/hooks/prebuild/_shared/041-write-region.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
|
|
7
|
+
| [All-Client Email Queue Monitor (Monitor/Operations/EmailQueue)](features/all-client-email-queue-monitor.md) | `_Worker_Monitor_Operations::EmailQueue()` is the cross-client health check for the 2.0 outbound email queue (`Logs_<Client>.Email` — see [2.0 Email Send Pipeli | worker2/Worker/Monitor/Operations.php, _underscore/Database.php, _underscore/Query.php, dbchanges2/Logs_Client/2026-05-21 - Email.sql |
|
|
7
8
|
| [Automated PR Merger — Concurrent Force-Push Clobber Race](features/automated-pr-merger-force-push-race.md) | The automated PR merger `_Worker_Team_GitHub::Merge` (`worker2` `Worker/Team/Github.php`) merges approved PRs to `_production` by **force-pushing from a clone t | Worker/Team/Github.php |
|
|
8
9
|
| [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | A cron watchdog that emails when the ClickUp integration looks disconnected during business hours. | worker2/Worker/Clickup/Health.php, worker2/Database/ClickupHealthWatchdog.sql |
|
|
9
10
|
| [ClickUp Design Sprint Automation (Final Design Outcome)](features/clickup-design-sprint-automation.md) | `_Worker_Clickup_Design` is meant to drive the design-sprint workflow in ClickUp via the API, replacing a set of native ClickUp automations. | worker2/Worker/Clickup/Design.php, worker2/Worker/Clickup.php, worker2/Controller/ClickupDesignTest.php, _underscore/Component/Api/Clickup/Clickup.php |
|
|
@@ -24,7 +25,7 @@
|
|
|
24
25
|
| [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, worker/crons/toga2/forecast2/import_supporting_records.php |
|
|
25
26
|
| [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |
|
|
26
27
|
| [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
|
|
27
|
-
| [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php |
|
|
28
|
+
| [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php, worker2/Worker/Monitor/Operations.php |
|
|
28
29
|
| [Platform Cache Cleanup (_Worker_Platform_Cache — Clean + Truncate)](features/platform-cache-cleanup.md) | `_Worker_Platform_Cache` owns maintenance of the shared **Cache** cluster that backs api2's [multi-client data retrieval](../../api2/features/cross-client-data- | worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
|
|
29
30
|
| [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |
|
|
30
31
|
| [Talos (TOGa IQ) Meeting-Notes Integration & Token Auto-Refresh (consumer)](features/talos-meeting-notes-integration.md) | How a **dev tool / agent consumes Talos (TOGa IQ)** to query the team meeting-notes corpus programmatically. | .claude/skills/plan-ticket/scripts/talos.js |
|
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: All-Client Email Queue Monitor (Monitor/Operations/EmailQueue)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-30
|
|
10
|
+
owners: ["jcardinal"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Monitor/Operations.php
|
|
13
|
+
- _underscore/Database.php
|
|
14
|
+
- _underscore/Query.php
|
|
15
|
+
- dbchanges2/Logs_Client/2026-05-21 - Email.sql
|
|
16
|
+
related:
|
|
17
|
+
- ./oneuptime-worker2-monitoring.md
|
|
18
|
+
- ./monitoring-framework.md
|
|
19
|
+
- ../../_underscore/features/email-send-pipeline.md
|
|
20
|
+
- ../../_underscore/features/per-client-database-connections.md
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
`_Worker_Monitor_Operations::EmailQueue()` is the cross-client health check for the 2.0
|
|
26
|
+
outbound email queue (`Logs_<Client>.Email` — see
|
|
27
|
+
[2.0 Email Send Pipeline](../../_underscore/features/email-send-pipeline.md)). It resolves
|
|
28
|
+
**every** client from `Core.Clients`, connects to each client's own logs database, counts
|
|
29
|
+
`FAILED` / `PENDING` rows in `Email`, and pushes one aggregate payload to a single OneUptime
|
|
30
|
+
Incoming Request monitor with the **offending clients named** in the body.
|
|
31
|
+
|
|
32
|
+
It was previously hardcoded to `Logs_Compass`, so a failing email queue for any other client
|
|
33
|
+
was completely invisible. The action path and method signature are unchanged, so the existing
|
|
34
|
+
`Core.WorkerJobs` cron entry needed no change. (Out-of-code follow-up: the OneUptime monitor
|
|
35
|
+
was still named "Compass Email Queue" and needs renaming in the UI.)
|
|
36
|
+
|
|
37
|
+
This is **shared internal infrastructure, not a Compass feature** — Compass USA is simply the
|
|
38
|
+
client it used to be limited to.
|
|
39
|
+
|
|
40
|
+
## Key files / entry points
|
|
41
|
+
|
|
42
|
+
- `worker2/Worker/Monitor/Operations.php` — `abstract class _Worker_Monitor_Operations`.
|
|
43
|
+
- `public static function EmailQueue(): string` — the action, route
|
|
44
|
+
`Monitor/Operations/EmailQueue`.
|
|
45
|
+
- `private static function resolveClientLogDatabases(): array` — one `DB_CORE` query
|
|
46
|
+
returning every client + its log database name + this environment's hosts.
|
|
47
|
+
- `private static function isMissingTableError(Throwable $e): bool` — MySQL error-number
|
|
48
|
+
classification (see gotchas).
|
|
49
|
+
- `const MYSQL_ERROR_NO_SUCH_TABLE = 1146;`
|
|
50
|
+
- No other file in any repo references this class (verified by tree-wide grep) — the
|
|
51
|
+
autoloader and `_Worker::runTask()` reach it indirectly by action path.
|
|
52
|
+
|
|
53
|
+
## How it works
|
|
54
|
+
|
|
55
|
+
1. **Resolve every client's log database.** One query on `_underscore::DB_CORE` joins
|
|
56
|
+
`Clients` → `Databases` (via `Clients.logDatabaseId`) → `DatabaseHosts` →
|
|
57
|
+
`Environments` (slug) → `Regions` (code). Environment slug follows the
|
|
58
|
+
`dev-*` → `dev` collapsing used by `_Worker_Team_Transcripts::initialize()`; region comes
|
|
59
|
+
from `_Cloud::getRegion()`. Rows are ordered so the closest region for a client comes
|
|
60
|
+
first; later rows for the same client are skipped.
|
|
61
|
+
2. **Register each client's log DB under its REAL database name, with no alias.** A per-client
|
|
62
|
+
loop cannot reuse the shared `_underscore::DB_CLIENT_LOGS` alias — every iteration would
|
|
63
|
+
clobber it (all live connection state is alias-keyed; see
|
|
64
|
+
[Re-pointing a DB alias mid-request](../../_underscore/features/database-alias-repointing.md)).
|
|
65
|
+
Registration is guarded with `if (!isset(_Database::$_registers[$logDatabaseName]))`.
|
|
66
|
+
3. **Count per client**, index-friendly:
|
|
67
|
+
`SELECT status, COUNT(*) FROM Email WHERE status IN ('FAILED','PENDING') GROUP BY status`.
|
|
68
|
+
4. **Bucket each client** into `checked`, `skipped` (no `Email` table), `unreachable` (any
|
|
69
|
+
other query/connection failure) or `unresolved` (no log database or no host for this
|
|
70
|
+
environment).
|
|
71
|
+
5. **Decide the tokens and push once** to the same OneUptime monitor.
|
|
72
|
+
|
|
73
|
+
### Alarm rule (explicit product decision)
|
|
74
|
+
|
|
75
|
+
Alarm on **`FAILED > 0` only**. `PENDING` is reported as context and **never** alarms — a
|
|
76
|
+
queue with work in it is normal, and the sender cron runs every minute.
|
|
77
|
+
|
|
78
|
+
### Payload contract — `alarm` and `probe` are SEPARATE tokens
|
|
79
|
+
|
|
80
|
+
OneUptime can only string-match a request body (no numeric comparison, no JSON key
|
|
81
|
+
targeting), so the worker decides and emits tokens:
|
|
82
|
+
|
|
83
|
+
| Field | Meaning |
|
|
84
|
+
|---|---|
|
|
85
|
+
| `status` | `reporting`, or `error` reserved for the **Core lookup itself** failing (checker fully blind) |
|
|
86
|
+
| `alarm` | `HIGH` \| `OK` — emails are actually failing (`failedCount > 0`) |
|
|
87
|
+
| `probe` | `OK` \| `DEGRADED` — set `DEGRADED` when `unreachableClients` is non-empty |
|
|
88
|
+
| `failedCount`, `pendingCount`, `alarmFailedCountThreshold` | raw counts + the threshold used |
|
|
89
|
+
| `offendingClients` | comma-separated client **names** with per-client failed counts |
|
|
90
|
+
| `worstClient`, `worstFailedCount` | the single worst tenant |
|
|
91
|
+
| `clientsChecked`, `clientsSkipped` | coverage counters |
|
|
92
|
+
| `unreachableClients`, `unresolvedClients` | named coverage gaps |
|
|
93
|
+
| `checkedAtUtc` | `gmdate('c')` |
|
|
94
|
+
|
|
95
|
+
**Decision:** on a monitor that covers all clients, *"emails are failing"* and *"we could not
|
|
96
|
+
look at some clients"* need distinct OneUptime criteria. Collapsing them means a logs-cluster
|
|
97
|
+
outage reads as healthy email delivery. Hence two independent tokens rather than one.
|
|
98
|
+
|
|
99
|
+
## Data model
|
|
100
|
+
|
|
101
|
+
`Email` / `EmailAttachment` ship in the `Logs_Client` blank
|
|
102
|
+
(`dbchanges2/Logs_Client/2026-05-21 - Email.sql`), so every client logs database should have
|
|
103
|
+
them. `Email.status` is `enum('PENDING','SENT','FAILED')`, with `retryCount` and
|
|
104
|
+
`failureReason`. Index `idx_email_status` exists on `status`.
|
|
105
|
+
|
|
106
|
+
## Client variations
|
|
107
|
+
|
|
108
|
+
None — the check is uniform across every client in `Core.Clients`. Per-client differences that
|
|
109
|
+
matter are the **log database name** and whether the `Email` migration has been applied.
|
|
110
|
+
|
|
111
|
+
## Gotchas / known issues
|
|
112
|
+
|
|
113
|
+
- **A client's database name is NOT derivable from `clientIdentifier`.** `Compass_Usa` →
|
|
114
|
+
`Logs_Compass`. Always resolve `Clients.logDatabaseId` → `Databases.name`. Building
|
|
115
|
+
`'Logs_' . $clientIdentifier` silently targets a nonexistent database. See
|
|
116
|
+
[per-client database connections](../../_underscore/features/per-client-database-connections.md).
|
|
117
|
+
- **Do not use `_Database::registerClientDatabases()` for an all-client fan-out.** It INNER
|
|
118
|
+
JOINs the client and archive databases alongside the log database, so a client missing either
|
|
119
|
+
row drops out and the method then dereferences a null row and throws. It also opens three
|
|
120
|
+
connections when only the log one is needed. Run a log-database-only variant with LEFT OUTER
|
|
121
|
+
JOINs instead.
|
|
122
|
+
- **INNER JOINs in an all-client resolution query silently un-monitor clients.** A client with
|
|
123
|
+
a null `logDatabaseId`, or with no `DatabaseHosts` row for the current environment, vanishes
|
|
124
|
+
from the result set with no log line and no alert token — the exact blind spot the monitor
|
|
125
|
+
exists to eliminate. LEFT OUTER JOIN every level and report those clients under an explicit
|
|
126
|
+
`unresolvedClients` bucket.
|
|
127
|
+
- **Follow-on trap once left-joined:** a `DatabaseHosts` row belonging to a *different*
|
|
128
|
+
environment shows up as a populated host and looks resolved. Null the host columns unless the
|
|
129
|
+
environment join matched:
|
|
130
|
+
`IF(LogEnvironments.id IS NULL, NULL, LogDatabaseHosts.hostCluster)`.
|
|
131
|
+
- **Detect "table does not exist" by MySQL error NUMBER, not message text.** `_Query` throws a
|
|
132
|
+
plain `Exception` whose message embeds the driver error on an `Error #: <number>` line
|
|
133
|
+
(`_underscore/Query.php:346`). Message wording is locale- and engine-dependent (MySQL vs
|
|
134
|
+
MariaDB differ); parse `/Error #:\s*(\d+)/` and compare to `1146` (`ER_NO_SUCH_TABLE`). A
|
|
135
|
+
logs DB predating the `Email` migration is a **skip**, not an outage.
|
|
136
|
+
- **An unconditional `SUM(status = 'X')` cannot use an index.**
|
|
137
|
+
`SELECT SUM(status='FAILED') FROM Email` full-scans, reading every historical `SENT` row
|
|
138
|
+
despite `idx_email_status`. Use `WHERE status IN (...) GROUP BY status`. This runs per client
|
|
139
|
+
on a cron, so the difference compounds.
|
|
140
|
+
- **There is no `_Query::setTimeout()` in the 2.0 framework** (only `_ApiRequest::setTimeout()`).
|
|
141
|
+
A 2.0 worker therefore has **no per-query timeout**, so a hung database cluster blocks the
|
|
142
|
+
fan-out until the connection itself times out. Relevant to any loop over all clients.
|
|
143
|
+
- **A passing dev run does not prove cross-cluster correctness.** Non-prod is a single
|
|
144
|
+
all-in-one database endpoint; production splits Core / `Client_*` / `Logs_*` / `Archive_*` /
|
|
145
|
+
`Cache` across **separate clusters**.
|
|
146
|
+
- **Verification status of this capture:** `php -l` passes; php-reviewer and sql-reviewer both
|
|
147
|
+
returned 0 critical and every actionable finding was fixed. It has **not** been executed
|
|
148
|
+
against any environment and **no database was queried** — the `Core` join shape is verified
|
|
149
|
+
against framework source only, not against real rows.
|
|
150
|
+
|
|
151
|
+
## Change history
|
|
152
|
+
|
|
153
|
+
- 2026-07-30 — Refactored `EmailQueue()` from Compass-only (hardcoded `Logs_Compass` via
|
|
154
|
+
`initialize()` + `const DB_LOGS_COMPASS`, both removed) to an all-client fan-out
|
|
155
|
+
(121 → 309 lines): added `resolveClientLogDatabases()`, per-client registration under the real
|
|
156
|
+
database name with no alias, LEFT-OUTER-JOIN resolution with an `unresolvedClients` bucket,
|
|
157
|
+
error-number-based missing-table skip, index-friendly `GROUP BY status` counting, and the
|
|
158
|
+
separate `alarm` / `probe` tokens. Alarm on `FAILED > 0` only; `PENDING` never alarms. Action
|
|
159
|
+
path and signature unchanged, so no cron change. Uncommitted in the worker2 working tree at
|
|
160
|
+
capture time. (jcardinal)
|
|
161
|
+
|
|
162
|
+
## Related docs
|
|
163
|
+
|
|
164
|
+
- [OneUptime push-metric monitors for 2.0 workers](./oneuptime-worker2-monitoring.md) — the
|
|
165
|
+
push/token pattern and OneUptime criteria this monitor follows.
|
|
166
|
+
- [Monitoring Framework](./monitoring-framework.md) — the parallel DB-driven, email-alert pattern.
|
|
167
|
+
- [2.0 Email Send Pipeline](../../_underscore/features/email-send-pipeline.md) — what fills the
|
|
168
|
+
queue this monitor watches.
|
|
169
|
+
- [Per-Client Database Connections](../../_underscore/features/per-client-database-connections.md)
|
|
170
|
+
— log-database name resolution.
|
|
@@ -6,15 +6,17 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-30
|
|
10
10
|
owners: ["jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Monitor/Compass.php
|
|
13
13
|
- worker2/Worker/Client/Compass.php
|
|
14
14
|
- worker2/composer.json
|
|
15
15
|
- _underscore/Cloud.php
|
|
16
|
+
- worker2/Worker/Monitor/Operations.php
|
|
16
17
|
related:
|
|
17
18
|
- ./monitoring-framework.md
|
|
19
|
+
- ./all-client-email-queue-monitor.md
|
|
18
20
|
- ../../_underscore/features/cloud-s3-helpers.md
|
|
19
21
|
- ../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md
|
|
20
22
|
---
|
|
@@ -98,6 +100,17 @@ OneUptime criteria:
|
|
|
98
100
|
prevents flapping back to Operational while the alarm is still HIGH).
|
|
99
101
|
- **not received in 15 min** → Offline.
|
|
100
102
|
|
|
103
|
+
### Separate the ALARM signal from the PROBE-health signal on multi-client monitors
|
|
104
|
+
|
|
105
|
+
When one monitor covers **every** client (rather than one client's integration), "the metric is
|
|
106
|
+
bad" and "we could not look at some clients" need **distinct** OneUptime criteria. Collapsing
|
|
107
|
+
them means a logs-cluster outage reads as healthy delivery. Emit **two independent string
|
|
108
|
+
tokens**: `alarm` (`HIGH`/`OK`) for the measured condition and `probe` (`OK`/`DEGRADED`) for
|
|
109
|
+
coverage gaps, and name the affected clients in the body so the incident is actionable without a
|
|
110
|
+
log dive. `status: "error"` stays reserved for the checker being **fully** blind (its own
|
|
111
|
+
resolution query failed). Worked example:
|
|
112
|
+
[All-Client Email Queue Monitor](./all-client-email-queue-monitor.md).
|
|
113
|
+
|
|
101
114
|
### Heartbeat / cron cadence timing
|
|
102
115
|
|
|
103
116
|
The 10-min-Degraded / 15-min-Offline heartbeat thresholds pair with a **5-minute** cron
|
|
@@ -179,6 +192,11 @@ check for another client.
|
|
|
179
192
|
monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
|
|
180
193
|
|
|
181
194
|
## Change history
|
|
195
|
+
- 2026-07-30 — Added the multi-client refinement of the payload contract: a **separate `probe`
|
|
196
|
+
token** (`OK`/`DEGRADED`) alongside `alarm`, so a partial logs-cluster outage cannot read as a
|
|
197
|
+
healthy metric, plus naming the offending clients in the body. Recorded that the previously
|
|
198
|
+
Compass-only `Monitor/Operations/EmailQueue` is now an all-client monitor — see
|
|
199
|
+
[All-Client Email Queue Monitor](./all-client-email-queue-monitor.md). (jcardinal)
|
|
182
200
|
- 2026-07-14 — Grew `_Worker_Monitor_Compass` to a 9-monitor suite (S3/DB/mailbox);
|
|
183
201
|
finalized the payload contract (`status`/`alarm`/`checkedAtUtc`) and the not-received
|
|
184
202
|
10-min-Degraded / 15-min-Offline + OK-gated recovery criteria; standardized cron on
|
|
@@ -113,6 +113,30 @@ Never create a `.sql` file inside `worker2`, `api2`, `_underscore`, or any other
|
|
|
113
113
|
application repo. The SQL that belongs in those repos is only query strings embedded in
|
|
114
114
|
PHP code — not standalone migration files.
|
|
115
115
|
|
|
116
|
+
### Database isolation in `dbchanges2` — one database per file (HARD RULE)
|
|
117
|
+
|
|
118
|
+
**A `dbchanges2` `.sql` file may only reference tables in the ONE database its folder targets,
|
|
119
|
+
and must reference them UNQUALIFIED.** Never write `Core.Table` in a `Client_<Name>/` file,
|
|
120
|
+
never `Client_X.Table` in a `Core/` file, and use **no database qualifier at all** in the
|
|
121
|
+
fan-out folders (`Client/`, `Logs_Client/`, `_modules/`), where the tenant database name is not
|
|
122
|
+
fixed. This covers `JOIN`s, subqueries, `SET @var = (SELECT …)`, `INSERT … SELECT`, `NOT EXISTS`
|
|
123
|
+
guards, and `USE`.
|
|
124
|
+
|
|
125
|
+
In **production** `Core`, `Client_<Tenant>`, `Archive_<Tenant>`, `Logs`/`Logs_<Tenant>`, and
|
|
126
|
+
`Cache` are on **entirely separate clusters**, so a cross-database statement cannot resolve
|
|
127
|
+
there. **Local and every non-prod environment put all of them behind one endpoint**, so the
|
|
128
|
+
query runs fine, passes review, and fails only in production — never trust a local run as proof
|
|
129
|
+
it is safe.
|
|
130
|
+
|
|
131
|
+
Instead: hardcode a pre-resolved **v4 UUID literal**, resolve rows by a **slug / natural key that
|
|
132
|
+
exists in the target database**, or **split the work into one file per database folder** (passing
|
|
133
|
+
values between them as literals, not queries).
|
|
134
|
+
|
|
135
|
+
Enforced mechanically by the `PreToolUse` hook
|
|
136
|
+
`.claude/hooks/toga/dbchanges2-cluster-isolation.js`, which refuses the write. Full rule,
|
|
137
|
+
examples, and the approved rewrites: `2.0/apps/dbchanges2/architecture.md` → *Database
|
|
138
|
+
isolation*. **Applies to `dbchanges2` only — the 1.0 `dbchanges` repo is exempt.**
|
|
139
|
+
|
|
116
140
|
## Checking dependencies before touching shared code
|
|
117
141
|
|
|
118
142
|
`dependsOn` in `knowledge/registry.json` means a repo extends or depends on another repo's classes. Before modifying a class in a dependency repo (e.g. `_underscore` core):
|
package/knowledge/INDEX.md
CHANGED
|
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
19
19
|
## 2.0 framework
|
|
20
20
|
|
|
21
21
|
- **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
22
|
-
- **worker2** (Worker) —
|
|
22
|
+
- **worker2** (Worker) — 36 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
23
23
|
- **api2** (API) — 20 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
24
24
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
25
25
|
- **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
package/package.json
CHANGED