toga-ai 1.0.477 → 1.0.479

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,7 +6,7 @@ project: Tools
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-24
9
+ updated: 2026-07-30
10
10
  owners: [jcardinal]
11
11
  files:
12
12
  - tools/_/app/design/github.php
@@ -19,6 +19,7 @@ files:
19
19
  - tools/composer.json
20
20
  related:
21
21
  - ../architecture.md
22
+ - ../../../standards/frontend.md
22
23
  - ../features/persona-gated-navigation.md
23
24
  - ../features/saml-sso-auth.md
24
25
  - ../workflows/deploy-to-elastic-beanstalk-al2023.md
@@ -92,6 +93,29 @@ on the next write of that project.json. Tab management via a per-pill "⋯" menu
92
93
  - **Reorder** — a single-file `tabs.json` commit, validated as a **permutation** of the
93
94
  existing list.
94
95
 
96
+ ### Text-storage invariant: JSON holds plain text, escape once at display
97
+
98
+ `tabs.json` tab names and `project.json` `title` values are stored as **plain text**. The
99
+ single correct escape happens at display time in `assets/js/design.js` `esc()` (and in
100
+ `buildStub()`, which `htmlspecialchars()` the title into the generated `index.html`).
101
+
102
+ `App_Design_Github::decodeEntities()` (one pass of `html_entity_decode` with
103
+ `ENT_QUOTES | UTF-8`) enforces that invariant at two chokepoints:
104
+
105
+ - **On read out of storage** — `loadTabs()`, `normalizeTabs()` (including the legacy scalar
106
+ `tab`), and the project read's `title`.
107
+ - **On names/titles read off a request** — `createTab()`, `renameTab()` (both old and new
108
+ name), `deleteTab()`, `reorderTabs()`, `filterKnownTabs()`, `createProject()`, and
109
+ `upload()` titles.
110
+
111
+ Both sides are required: decoding only reads would make `renameTab()` compare a decoded
112
+ `H&H` against a stored `H&H` and fail with *"No such tab."*; decoding only writes would
113
+ leave already-bad stored data permanently double-escaped on screen.
114
+
115
+ **Self-healing:** every tab write rewrites `tabs.json` and each affected `project.json` in
116
+ full, so pre-encoded stored values are re-persisted as plain text on the next
117
+ rename/delete/reorder. No manual repair of the `forward` repo is needed.
118
+
95
119
  ### Actions & UX
96
120
 
97
121
  - **Publish a version** — uploads a self-contained HTML export as the next `vN/index.html`,
@@ -147,10 +171,27 @@ None — internal/shared design-team tool.
147
171
  `curl_close()` calls. Audit ported code for other deprecated-in-8.x calls.
148
172
  - **Contents API inlines only ≤1 MB** — read exports with the raw media type (see above).
149
173
  - **Publishing pushes to `forward`'s `_main`** and triggers EB auto-deploy; no staging.
174
+ - **Double-escaped names (`H&H` shown literally, rename fails with "No such tab.")** —
175
+ caused by values that arrived **already HTML-encoded** and were then correctly escaped once
176
+ more at display. The render path was never wrong; no current code path produces the
177
+ encoding (`createTab`/`createProject` and the JS transport `window.prompt` →
178
+ `URLSearchParams` → `$postField` trim/`mb_substr` are clean), so encoded values entered from
179
+ outside — a paste from rendered HTML, or a pre-fix build. Fixed defensively with
180
+ `decodeEntities()` on both the read and request sides (see above). Generated project stubs
181
+ were affected identically, because `buildStub()`'s single correct `htmlspecialchars()` was a
182
+ *second* escape on a pre-encoded title; storing titles plain fixes the stubs too.
150
183
  - Never hardcode the token in tracked source — a leaked `ghp_`/PAT must be rotated.
151
184
 
152
185
  ## Change history
153
186
 
187
+ - 2026-07-30 — Fixed HTML-entity double-escaping of tab names and project titles (a tab named
188
+ `H&H` rendered as `H&H`, and renaming it failed with "No such tab."). Added
189
+ `App_Design_Github::decodeEntities()` and applied it on every read out of storage
190
+ (`loadTabs`, `normalizeTabs` incl. legacy scalar `tab`, project `title`) and on every
191
+ name/title read off a request (`createTab`, `renameTab`, `deleteTab`, `reorderTabs`,
192
+ `filterKnownTabs`, `createProject`, `upload`), so all comparisons are decoded-vs-decoded and
193
+ storage re-persists plain text. Defensive — no current code path produced the encoding.
194
+ Verified by the developer; not yet committed to `tools`. (jcardinal)
154
195
  - 2026-07-24 — Moved the Design Demo Admin from `forward` into the SSO-protected `tools` app
155
196
  (`App_Design_Github` + `/design` MVC route + namespaced assets + nav entry); still commits
156
197
  to `agilantsolutions/forward` `_main` and demos stay hosted at `demo.togatech.com`. Added
@@ -0,0 +1,75 @@
1
+ ---
2
+ title: Front-End Standards
3
+ framework: "1.0"
4
+ project: Library
5
+ client: shared
6
+ type: standard
7
+ status: active
8
+ updated: 2026-07-30
9
+ owners: [jcardinal]
10
+ files: []
11
+ related:
12
+ - ./backend-php.md
13
+ - ./framework-rules.md
14
+ - ../apps/tools/features/design-demo-admin.md
15
+ - ../apps/tools/features/talos-kb-documents-admin.md
16
+ ---
17
+
18
+ # Front-End Standards (1.0 Legacy — `Browser_` / server-rendered UI)
19
+
20
+ > **Scope.** Server-rendered UI and browser assets for the legacy **1.0** framework
21
+ > (`Browser_` classes, `mvc/` get/post handlers, `assets/js`, `assets/css`). 1.0 is in
22
+ > maintenance mode and historically inconsistent — match the file you are editing, and
23
+ > prefer the cleaner pattern for new code. Do not retrofit 2.0 front-end conventions.
24
+
25
+ ## Output escaping: storage holds plain text, escape exactly once at display
26
+
27
+ **Rule.** Persisted values — database columns, JSON manifests, config — hold **plain,
28
+ unescaped text**. HTML escaping happens **exactly once**, at the moment the value is
29
+ written into an HTML context (`htmlspecialchars($v, ENT_QUOTES, 'UTF-8')` server-side, or
30
+ the view's `esc()` helper client-side).
31
+
32
+ **Never** escape on the way *in* to storage. An escaped value in storage is a latent bug:
33
+ the display layer escapes it a second time, and the user sees `H&H` instead of `H&H`.
34
+
35
+ ### Why it recurs
36
+
37
+ The failure mode is a **round trip**, not a bad escape function:
38
+
39
+ 1. A value is rendered into an edit form / prompt already escaped for HTML.
40
+ 2. The user submits that form unchanged.
41
+ 3. The escaped string is stored verbatim — storage now holds `H&H`.
42
+ 4. Display escapes correctly once more → `H&H` on screen.
43
+
44
+ Each round trip adds a level. Equality comparisons then fail too, because a freshly typed
45
+ `H&H` no longer matches the stored `H&H` — which surfaces as a confusing
46
+ "record not found" rather than as a display bug.
47
+
48
+ ### Required practice
49
+
50
+ - Pre-fill form inputs and JS prompts with the **plain** value; let the templating layer
51
+ do the one escape it owns.
52
+ - When a store may already contain encoded data, **decode on both sides** of the boundary:
53
+ a single `html_entity_decode($v, ENT_QUOTES, 'UTF-8')` on every read out of storage
54
+ **and** on every value read off a request. Decoding only one side leaves either broken
55
+ comparisons or undisplayable legacy rows.
56
+ - Prefer stores that **rewrite records in full** on update, so the decode-on-write is
57
+ self-healing and no manual data repair is needed.
58
+ - Escape for the **right context**: `htmlspecialchars` for HTML text/attributes,
59
+ `json_encode` (with `JSON_HEX_*` for inline `<script>`) for JavaScript. Never
60
+ hand-build JSON or rely on `htmlspecialchars` inside a JS context.
61
+
62
+ ### Precedents in the codebase
63
+
64
+ - `tools/mvc/talos/vocabulary/post.php:74-80` — inline comment recording the rule after
65
+ form inputs pre-filled with `htmlspecialchars()` output accumulated encoding.
66
+ - `tools/_/app/design/github.php` — `App_Design_Github::decodeEntities()` applied to tab
67
+ names and project titles on both the storage-read and request-read sides.
68
+
69
+ ## Change history
70
+
71
+ - 2026-07-30 — Created. Establishes the escape-once/plain-text-storage invariant after a
72
+ second independent occurrence in `tools` (Design Demo Admin, following the Talos
73
+ vocabulary tool). (jcardinal)
74
+ </content>
75
+ </invoke>
@@ -24,7 +24,7 @@
24
24
  | [_Model magic-field access (__get without __isset)](features/model-magic-field-access.md) | `_Model` exposes DB columns as "magic" properties via `__get()`, but it defines **no** `__isset()`. | _underscore/Model/Core/Model.php |
25
25
  | [_Model::save() vs raw _Query — no atomic conditional update](features/model-save-vs-query-atomic-update.md) | `_Model::save()` is a plain load-then-write ORM primitive and **cannot express an atomic conditional update** (an optimistic-concurrency / row-claim guard such | _underscore/Model.php, _underscore/Query.php |
26
26
  | [NetSuite REST Client (_Component_Api_Netsuite) — record writes & SuiteQL](features/netsuite-rest-client.md) | `_Component_Api_Netsuite` is the **2.0 `_underscore` NetSuite REST client** — the shared primitive every worker2/api2 NetSuite caller uses for record GETs, Suit | _underscore/Component/Api/Netsuite/Netsuite.php |
27
- | [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, api2/Controller/Index.php |
27
+ | [Per-Client Database Connections & the Local Logs Trap](features/per-client-database-connections.md) | When `_underscore` serves a request for a client it opens **three distinct per-client database connections**, not one. | _underscore/Database.php, _underscore/Query.php, _underscore/ApiRequest.php, _underscore/Model/Client/Logs/Api.php, api2/Controller/Index.php |
28
28
  | [Persona Name Translation (PersonaTranslations sidecar)](features/persona-name-translation.md) | Serves Persona **names** in multiple languages by adding a per-language **sidecar** table `PersonaTranslations`, reusing the platform's existing metadata-driven | _underscore/Model/Client/PersonaTranslation.php, dbchanges2/Client/2026-07-22b - PersonaTranslations.sql, dbchanges2/Core/2026-07-22a - PersonaTranslationsRecord.sql, dbchanges2/Client/2026-07-22c - PersonaTranslationsAcl.sql, dbchanges2/Client_CompassCanada/2026-07-22 - PersonaTranslationsFrench.sql, toga2-commerce/src/pages/Account/view/MySettingsView.tsx |
29
29
  | [Recursive Item Fulfillments (upstream mirroring)](features/recursive-item-fulfillments.md) | In a multi-tier supply chain a sales order (SO) spawns a purchase order (PO) that becomes another SO downstream, and so on. | _underscore/Model/Client/ItemFulfillment.php, _underscore/Model/Client/ItemFulfillmentItem.php, _underscore/Model/Client/ItemFulfillmentItemUnit.php, _underscore/Model/Client/ItemFulfillmentPackage.php, _underscore/Model/Compass/AdvanceShippingNotice.php, dbchanges2/Core/2026-02-13 - 75601 - RecursiveItemFulfillmentCreation.sql, dbchanges2/Core/2026-06-04 - RecursiveItemFulfillmentPut.sql, dbchanges2/Client_Compass/2026-07-02a - FixSA133377TrackingSerialAndDuplicateIF.sql |
30
30
  | [Surface Resolver (_Model_Core_Surface::resolve — replaces Page::meta)](features/surface-resolver.md) | The runtime for the platform-wide **Surface** UI presentation layer: 9 `_underscore` models plus a cached resolver, `_Model_Core_Surface::resolve(&$api, string | _underscore/Model/Core/Surface.php, _underscore/Model/Client/AclRecordScript.php, _underscore/Model/Core/RecordScript.php, dbchanges2/Core/2026-06-30a - SurfaceMetaGroupAndSalesOrderSections.sql, dbchanges2/Client/2026-06-30a - SurfaceMetaGroupAcl.sql, dbchanges2/Client_Quad/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Client_CompassCanada/2026-07-01a - GrantSurfacesMetaGroupScriptAcl.sql, dbchanges2/Core/2026-06-29b - SurfaceMetaPublicReadAcl.sql, dbchanges2/Client/2026-06-29c - SurfaceRecordScriptAcl.sql, dbchanges2/Core/2026-06-29c - SurfaceDebugPhpMethodFix.sql, dbchanges2/Client_Compass/2026-07-15f - SalesOrderRecordActionsRemoveDeadConfigRuleOverrides.sql, dbchanges2/Core/2026-07-17h - Update - ClearApprovalsFilterButtonConfig.sql, dbchanges2/Client_Compass/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Compass/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_CompassCanada/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, dbchanges2/Client_Quad/2026-07-17a - SalesOrderApprovalActionsOverride.sql, dbchanges2/Client_Quad/2026-07-17b - SalesOrderApprovalsFilterButtonOverride.sql, _underscore/Model/Core/SurfaceElement.php, _underscore/Model/Core/Action.php, _underscore/Model/Core/Vocabulary.php, _underscore/Model/Core/VocabularyTerm.php, _underscore/Model/Core/Message.php, _underscore/Model/Client/SurfaceOverride.php, _underscore/Model/Client/MessageTranslation.php, _underscore/Model/Client/ThemeToken.php, _underscore/Model/Core/Page.php, api2/Component/Api/V2/V2.php |
@@ -194,6 +194,23 @@ All non-production environments are **all-in-one**: a single cluster per environ
194
194
  holds `Core`, `Client_<Tenant>`, `Archive_<Tenant>`, and `Logs_<Tenant>` together
195
195
  (no family split). Region/reader routing still applies per cluster.
196
196
 
197
+ > **⚠ Consequence — never write a query that spans two database families.** Because
198
+ > production splits the families across the four dedicated clusters above, a single SQL
199
+ > statement that joins or sub-selects across them (e.g. `Client_Towfoundation` ↔ `Core`,
200
+ > or `Client_<Tenant>` ↔ `Archive_<Tenant>`) **cannot resolve in production** — the foreign
201
+ > schema is not on that server. And because **every non-production environment is all-in-one,
202
+ > such a query runs perfectly in local/dev/QA/stage**, so the failure surfaces only after
203
+ > release. Treat a successful non-prod run as **no evidence** that a query is cluster-safe.
204
+ > Cross-family data must be assembled in **PHP across two connections** (`_Database` named
205
+ > connections, one per family), never in one statement. Note the corollary that platform
206
+ > databases `Core`, `Forecast`, and `Team` **do** share `prod-core`, so those are same-cluster
207
+ > — but a cross-*database* query is still discouraged, and in `dbchanges2` it is forbidden
208
+ > outright.
209
+ >
210
+ > For **`dbchanges2` migrations this is a hard, mechanically-enforced rule**: a `.sql` file may
211
+ > only reference tables in the one database its folder targets. See
212
+ > [dbchanges2 → Database isolation](../dbchanges2/architecture.md).
213
+
197
214
  | Environment (aliases) | Host |
198
215
  |---|---|
199
216
  | `dev-sandbox` / `sandbox-dev` / developer sandbox | `dev.sandbox.database.togahub.com` |
@@ -13,6 +13,7 @@ files:
13
13
  - worker2/Worker/Infrastructure/Email/Send.php
14
14
  related:
15
15
  - email-template-sending.md
16
+ - ../../worker2/features/all-client-email-queue-monitor.md
16
17
  - ../../ai-bdr/features/web-funnel-app.md
17
18
  ---
18
19
 
@@ -6,10 +6,11 @@ project: _Underscore
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-28
9
+ updated: 2026-07-30
10
10
  owners: ["dfranks", "jcardinal", "mhammontree", "apeterson", "kyalamarthi"]
11
11
  files:
12
12
  - _underscore/Database.php
13
+ - _underscore/Query.php
13
14
  - _underscore/ApiRequest.php
14
15
  - _underscore/Model/Client/Logs/Api.php
15
16
  - api2/Controller/Index.php
@@ -17,6 +18,7 @@ related:
17
18
  - ../architecture.md
18
19
  - ../workflows/local-db-refresh-from-beta.md
19
20
  - ./database-alias-repointing.md
21
+ - ../../worker2/features/all-client-email-queue-monitor.md
20
22
  ---
21
23
 
22
24
  ## Summary
@@ -112,6 +114,31 @@ here — they live in `Config/*.ini`.)
112
114
  `Core.Clients.clientDatabaseId` → `Core.Databases.name`. This mirrors the log-database resolution
113
115
  already done in `_underscore`'s `Email.php`. A string-built name fails only for the clients whose
114
116
  names happen to diverge, so it passes local testing and breaks in production.
117
+ - **The same rule applies to the LOG database: never build `'Logs_' . $clientIdentifier`.**
118
+ Resolve `Core.Clients.logDatabaseId` → `Core.Databases.name` (`Compass_Usa` → `Logs_Compass`).
119
+ A string-built log-DB name silently targets a schema that does not exist.
120
+ - **`_Database::registerClientDatabases()` is unsafe for a loop over ALL clients.** It INNER
121
+ JOINs the client database *and* archive database alongside the log database, so a client
122
+ missing either row drops out of the result set entirely — and the method then dereferences the
123
+ null row (`$database->clientId`) and throws. It also opens three connections per client when a
124
+ caller may need only one. For an all-client fan-out, run a **log-database-only** variant with
125
+ **LEFT OUTER JOINs** and report clients that resolve to nothing, rather than letting them
126
+ vanish. Worked example: [All-Client Email Queue Monitor](../../worker2/features/all-client-email-queue-monitor.md).
127
+ - **A per-client loop must NOT reuse the shared aliases.** `DB_CLIENT` / `DB_CLIENT_LOGS` /
128
+ `Archive` are single slots — each iteration clobbers the previous one, and all live connection
129
+ state is alias-keyed. Register each client's database **under its real schema name with no
130
+ alias** (guard with `isset(_Database::$_registers[$name])`) when you need many clients open at
131
+ once.
132
+ - **There is no `_Query::setTimeout()`.** Only `_ApiRequest::setTimeout()` exists. A 2.0 request
133
+ or worker therefore has **no per-query timeout**: a hung database cluster blocks until the
134
+ connection itself times out. This matters most in fan-out loops across many client clusters,
135
+ where one unreachable cluster stalls the whole pass.
136
+ - **Classify driver failures by MySQL error NUMBER, not message text.** `_Query` throws a plain
137
+ `Exception` whose message embeds the driver error on an `Error #: <number>` line
138
+ (`_underscore/Query.php:346`). Matching text like `"doesn't exist"` is locale- and
139
+ engine-dependent (MySQL and MariaDB word it differently). Parse `/Error #:\s*(\d+)/` and
140
+ compare to the code — e.g. `1146` (`ER_NO_SUCH_TABLE`) is how a worker tolerates a per-tenant
141
+ table that predates a migration.
115
142
  - **Client DBs do not share a charset, so any hardcoded `COLLATE` is invalid somewhere.** Emitting
116
143
  e.g. `COLLATE utf8mb4_unicode_ci` against a `latin1` column errors with `COLLATION ... is not
117
144
  valid for CHARACTER SET 'latin1'`. For cross-tenant string comparison use `CAST(expr AS BINARY)`
@@ -152,6 +179,14 @@ here — they live in `Config/*.ini`.)
152
179
 
153
180
  ## Change history
154
181
 
182
+ - 2026-07-30 — Added four framework-level gotchas surfaced while building the all-client email
183
+ queue monitor: the log-DB name has the same never-string-build rule as the client DB;
184
+ `registerClientDatabases()` is unsafe for an all-client loop (INNER JOINs on client+archive
185
+ drop clients then throw on the null row, and it opens three connections); a per-client loop
186
+ must register under the **real schema name with no alias** because the aliases are single
187
+ slots; **`_Query::setTimeout()` does not exist**, so 2.0 has no per-query timeout; and driver
188
+ failures must be classified by MySQL **error number** parsed from `Error #: <n>` rather than
189
+ message text. (jcardinal)
155
190
  - 2026-07-28 — **Corrected a wrong rule:** a client's schema name is *not* reliably
156
191
  `'Client_' . clientIdentifier` (`Compass_Usa` → `Client_Compass`); resolve
157
192
  `Clients.clientDatabaseId` → `Databases.name`. Recorded that client DBs do **not** share a
@@ -39,6 +39,10 @@ in a folder, then `b`, `c`, …). One folder per database; place client changes
39
39
  `Client_<Name>` folder. Never edit an already-run migration — add a new dated file instead. The
40
40
  `Client/` **blank must carry the baseline DATA seed** (roles, full ACL, reference/lookup tables,
41
41
  UI config) — **not just schema**; a schema-only blank produces non-functional clients.
42
+ **A file may only reference tables in the ONE database its folder targets** — `Core`, `Client_*`,
43
+ `Archive_*`, `Logs*`, and `Cache` are on **separate production clusters**, so any
44
+ `OtherDatabase.Table` reference is unrunnable in production even though it works locally
45
+ (see *Database isolation* below; enforced by the `dbchanges2-cluster-isolation` hook).
42
46
 
43
47
  ## File naming convention (the execution contract)
44
48
 
@@ -90,6 +94,101 @@ Spglobal, Trividiahealth, True, Wje, Wmchealth, Ynhh).
90
94
  `Client/` and `Logs_Client/` also contain `BLANK_CLIENT_DATABASE` / `BLANK_CLIENT_LOGS_DATABASE`
91
95
  seed scripts used to provision a brand-new tenant DB from scratch.
92
96
 
97
+ ## Database isolation — never query across databases (HARD RULE)
98
+
99
+ **A `.sql` file in `dbchanges2` may only reference tables in the ONE database its folder
100
+ targets, and it must reference them UNQUALIFIED.** Any `OtherDatabase.Table` reference — in a
101
+ `JOIN`, a subquery, a `SET @var = (SELECT …)`, an `INSERT … SELECT`, a `NOT EXISTS` guard, or a
102
+ `USE` statement — is a **hard violation**.
103
+
104
+ **Why:** in **production** the 2.0 databases are **not one server**. `Core`, each
105
+ `Client_<Tenant>`, each `Archive_<Tenant>`, `Logs`/`Logs_<Tenant>`, and `Cache` live on
106
+ **entirely separate clusters** (see `2.0/apps/_underscore/architecture.md` → *Database
107
+ architecture*). A cross-database query has no way to resolve the foreign schema there.
108
+
109
+ **Why this is a trap rather than an obvious error:** **locally and in every non-prod
110
+ environment all of these databases sit behind a single endpoint**, so a cross-database query
111
+ runs perfectly, the migration looks correct, review passes — and it fails only when it reaches
112
+ production. Never treat a successful local run as evidence that a query is cluster-safe.
113
+ **Assume every database other than the file's own target is unreachable, always.**
114
+
115
+ ### Not allowed
116
+
117
+ This is the canonical violation — a `Client_<Tenant>` migration resolving `Core` ids to copy ACL
118
+ rows. `AclFieldPermissions` is in the client database, `Core.RecordFields` / `Core.Records` are
119
+ on the **core cluster**:
120
+
121
+ ```sql
122
+ -- WRONG — Core.* is a different cluster; this cannot run in production
123
+ SET @serviceAddressFieldId = (
124
+ SELECT rf.id FROM Core.RecordFields rf
125
+ JOIN Core.Records r ON r.id = rf.recordId
126
+ WHERE r.`route` = 'entitlements' AND rf.`field` = 'serviceAddressId' LIMIT 1
127
+ );
128
+ SET @saleItemFieldId = (
129
+ SELECT rf.id FROM Core.RecordFields rf
130
+ JOIN Core.Records r ON r.id = rf.recordId
131
+ WHERE r.`route` = 'entitlements' AND rf.`field` = 'saleItemId' LIMIT 1
132
+ );
133
+
134
+ INSERT INTO AclFieldPermissions (uuid, recordFieldId, roleId, isWritable)
135
+ SELECT UUID(), @serviceAddressFieldId, sibling.roleId, 0
136
+ FROM AclFieldPermissions sibling
137
+ WHERE sibling.recordFieldId = @saleItemFieldId
138
+ AND @serviceAddressFieldId IS NOT NULL
139
+ AND NOT EXISTS (
140
+ SELECT 1 FROM AclFieldPermissions existing
141
+ WHERE existing.recordFieldId = @serviceAddressFieldId AND existing.roleId = sibling.roleId
142
+ );
143
+ ```
144
+
145
+ ### Allowed — replace the foreign lookup
146
+
147
+ Use whichever fits; the goal is that **every table named in the file lives in the file's own
148
+ database**:
149
+
150
+ 1. **Hardcode the identifier.** The `Core` row's `uuid` is stable across environments, so
151
+ pre-generate/look it up **once at authoring time** and embed the literal. (Same discipline as
152
+ rule #6 — a v4 UUID literal, never `UUID()`.)
153
+
154
+ ```sql
155
+ -- CORRECT — no foreign database read; the uuid literal is resolved at authoring time
156
+ SET @serviceAddressFieldId = (
157
+ SELECT id FROM RecordFields
158
+ WHERE uuid = '<pre-resolved v4 uuid of the serviceAddressId field>' LIMIT 1
159
+ );
160
+ ```
161
+
162
+ 2. **Resolve by slug / natural key inside the target database.** If the client database carries
163
+ its own copy of the identifying column (`field`, `route`, `name`, a code), match on that
164
+ instead of joining out to `Core`.
165
+
166
+ 3. **Split the change into one file per database folder.** If work genuinely spans two
167
+ databases, write a `Core/` file **and** a `Client_<Name>/` (or `Client/`) file, each
168
+ self-contained. Order them by date/letter so the `Core` half lands first, and carry any value
169
+ between them as a **hardcoded literal**, not a query.
170
+
171
+ ### Fan-out folders: no qualifier at all
172
+
173
+ In `Client/`, `Logs_Client/`, and `_modules/<module>/` the target database **name is not fixed**
174
+ — the executor runs the same file against every tenant DB. So there is no valid database
175
+ qualifier in these folders whatsoever: **reference every table unqualified**, including tables in
176
+ the database being targeted.
177
+
178
+ ### Enforcement
179
+
180
+ This is enforced mechanically, not by convention. The `PreToolUse` hook
181
+ `.claude/hooks/toga/dbchanges2-cluster-isolation.js` inspects every `Write`/`Edit`/`MultiEdit`
182
+ of a `.sql` file under `dbchanges2`, resolves the target database from the folder, and **refuses
183
+ the write** if the SQL qualifies any table with a different database name (`Core`, `Client*`,
184
+ `Logs*`, `Archive*`, `Cache`, `Team`, including backticked and `USE` forms). Comments and string
185
+ literals are stripped first, and table **aliases** (`rf.id`, `sibling.roleId`) are never flagged.
186
+ `HISTORIC/` folders are skipped.
187
+
188
+ > **Scope:** this rule and its hook apply to **`dbchanges2` only**. The 1.0 **`dbchanges`** repo
189
+ > is **not** subject to it. If the hook ever misfires on legitimate SQL, `DBCHANGES2_ISOLATION_DISABLED=1`
190
+ > is an escape hatch **for false positives only** — never to land a cross-database query.
191
+
93
192
  ## `_modules` — reusable, opt-in change-sets
94
193
 
95
194
  Some change-sets aren't applied to every client — only to clients that use a given **module**
@@ -189,6 +288,13 @@ its own header.)
189
288
  `prePost` interceptor is registered unguarded in `Client_Rate/2026-07-15 - PreInterceptor.sql`
190
289
  (record 191, `minDepth NULL`); a second guarded registration would have double-fired it. See
191
290
  `clients/rate/features/whole-home-warranty-purchase-guard.md`.
291
+ 8. **Never reference a database other than the folder's own target.** No `Core.Table` in a
292
+ `Client_<Name>/` file, no `Client_X.Table` in a `Core/` file, and **no database qualifier at
293
+ all** in the fan-out folders (`Client/`, `Logs_Client/`, `_modules/`). These databases are on
294
+ **separate production clusters**; the query works locally and dies in production. Resolve
295
+ foreign ids with a hardcoded v4 UUID literal or an in-database slug/natural key, or split the
296
+ work into one file per database folder. See *Database isolation* above — enforced by the
297
+ `dbchanges2-cluster-isolation` hook.
192
298
 
193
299
  ## Bulk data loads — batch, and stage large sets in a temp table
194
300
 
@@ -341,6 +447,17 @@ defined in `2.0/apps/_underscore/architecture.md`, and its change files create/a
341
447
  tables that `_Model_*` classes map to.
342
448
 
343
449
  ## Change history
450
+ - 2026-07-29 — **Added *Database isolation — never query across databases* (HARD RULE) + rule #8.**
451
+ A `dbchanges2` `.sql` file may only reference tables in the one database its folder targets, and
452
+ must reference them unqualified; fan-out folders (`Client/`, `Logs_Client/`, `_modules/`) permit
453
+ no database qualifier at all. Rationale: in production `Core`, `Client_<Tenant>`,
454
+ `Archive_<Tenant>`, `Logs`/`Logs_<Tenant>`, and `Cache` are on **entirely separate clusters**,
455
+ while local/non-prod collapse them behind one endpoint — so a cross-database query passes review
456
+ and fails only in production. Replace foreign lookups with hardcoded v4 UUID literals, in-database
457
+ slugs/natural keys, or one file per database folder. Enforced mechanically by the new `PreToolUse`
458
+ hook `dbchanges2-cluster-isolation.js`, which refuses the write. Applies to `dbchanges2` **only** —
459
+ the 1.0 `dbchanges` repo is exempt. Cluster topology recorded in
460
+ `2.0/apps/_underscore/architecture.md` → *Database architecture*. (jcardinal)
344
461
  - 2026-07-29 — Added *Self-referencing INSERT...SELECT guard — wrap the existing-set subquery in
345
462
  a derived table*: an `INSERT...SELECT` guarded per-key against its own target must wrap the
346
463
  existing-set subquery in a derived table (DISTINCT/LIMIT) so MySQL materializes it once —
@@ -4,6 +4,7 @@
4
4
  |-----|---------|-------|
5
5
  | [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php, worker2/composer.json |
6
6
  | [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh, worker2/.platform/hooks/prebuild/_shared/041-write-region.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
7
+ | [All-Client Email Queue Monitor (Monitor/Operations/EmailQueue)](features/all-client-email-queue-monitor.md) | `_Worker_Monitor_Operations::EmailQueue()` is the cross-client health check for the 2.0 outbound email queue (`Logs_<Client>.Email` — see [2.0 Email Send Pipeli | worker2/Worker/Monitor/Operations.php, _underscore/Database.php, _underscore/Query.php, dbchanges2/Logs_Client/2026-05-21 - Email.sql |
7
8
  | [Automated PR Merger — Concurrent Force-Push Clobber Race](features/automated-pr-merger-force-push-race.md) | The automated PR merger `_Worker_Team_GitHub::Merge` (`worker2` `Worker/Team/Github.php`) merges approved PRs to `_production` by **force-pushing from a clone t | Worker/Team/Github.php |
8
9
  | [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | A cron watchdog that emails when the ClickUp integration looks disconnected during business hours. | worker2/Worker/Clickup/Health.php, worker2/Database/ClickupHealthWatchdog.sql |
9
10
  | [ClickUp Design Sprint Automation (Final Design Outcome)](features/clickup-design-sprint-automation.md) | `_Worker_Clickup_Design` is meant to drive the design-sprint workflow in ClickUp via the API, replacing a set of native ClickUp automations. | worker2/Worker/Clickup/Design.php, worker2/Worker/Clickup.php, worker2/Controller/ClickupDesignTest.php, _underscore/Component/Api/Clickup/Clickup.php |
@@ -24,7 +25,7 @@
24
25
  | [NetSuite Supporting-Record Webhook Importer (the reusable recipe)](features/netsuite-supporting-record-webhook-importer.md) | A single **repeatable recipe** for porting a legacy daily-pull NetSuite *supporting-record* importer (the lookup/dimension tables behind Forecast2 — Employees, | worker2/Worker/Netsuite/Employee.php, worker2/Worker/Netsuite/Account.php, worker2/Worker/Netsuite/Classification.php, worker2/Worker/Netsuite/Customer.php, worker2/Worker/Netsuite/Item.php, worker2/Worker/Netsuite.php, _underscore/Model/Forecast/Employee.php, _underscore/Model/Forecast/Account.php, _underscore/Model/Forecast/Classification.php, _underscore/Component/Forecast/Db/Db.php, test/@dave/test_employee_lifecycle.php, test/@dave/test_account_lifecycle.php, test/@dave/test_classification_lifecycle.php, test/@dave/NetSuite/api-message-queue/ue_api_msg_queue_enqueue.js, worker/crons/toga2/forecast2/import_supporting_records.php |
25
26
  | [Background Email-Template Worker (_Worker_Notification_EmailTemplate)](features/notification-email-template.md) | `_Worker_Notification_EmailTemplate::Send(...)` dispatches a **stored, client-defined `EmailTemplates` row off-thread** as a background WorkerJob. | worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, _underscore/Model/Client/EmailTemplate.php |
26
27
  | [DB-Driven Notification (Internal) Email](features/notification-email.md) | Internal/notification emails (merge-conflict alerts, ops notices — anything system-generated, not client-facing transactional mail) are sent through one worker | worker2/Worker/Notification/Email.php, _underscore/Model/Client/EmailTemplate.php, dbchanges2/Client/2026-06-23a - EmailTemplateWrapper.sql, dbchanges2/Client_True/2026-06-23a - EmailTemplateWrapper.sql |
27
- | [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php |
28
+ | [OneUptime push-metric monitors for 2.0 workers](features/oneuptime-worker2-monitoring.md) | A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, ported from the 1.0 `App_SystemMonitor_Compass` monitors. | worker2/Worker/Monitor/Compass.php, worker2/Worker/Client/Compass.php, worker2/composer.json, _underscore/Cloud.php, worker2/Worker/Monitor/Operations.php |
28
29
  | [Platform Cache Cleanup (_Worker_Platform_Cache — Clean + Truncate)](features/platform-cache-cleanup.md) | `_Worker_Platform_Cache` owns maintenance of the shared **Cache** cluster that backs api2's [multi-client data retrieval](../../api2/features/cross-client-data- | worker2/Worker/Platform/Cache.php, worker2/Controller/Index.php, worker2/_.php, dbchanges2/Core/2026-07-27a - PlatformCacheCleanCron.sql |
29
30
  | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. | worker2/Worker/Startech.php |
30
31
  | [Talos (TOGa IQ) Meeting-Notes Integration & Token Auto-Refresh (consumer)](features/talos-meeting-notes-integration.md) | How a **dev tool / agent consumes Talos (TOGa IQ)** to query the team meeting-notes corpus programmatically. | .claude/skills/plan-ticket/scripts/talos.js |
@@ -0,0 +1,170 @@
1
+ ---
2
+ title: All-Client Email Queue Monitor (Monitor/Operations/EmailQueue)
3
+ framework: "2.0"
4
+ repo: worker2
5
+ project: Worker
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-07-30
10
+ owners: ["jcardinal"]
11
+ files:
12
+ - worker2/Worker/Monitor/Operations.php
13
+ - _underscore/Database.php
14
+ - _underscore/Query.php
15
+ - dbchanges2/Logs_Client/2026-05-21 - Email.sql
16
+ related:
17
+ - ./oneuptime-worker2-monitoring.md
18
+ - ./monitoring-framework.md
19
+ - ../../_underscore/features/email-send-pipeline.md
20
+ - ../../_underscore/features/per-client-database-connections.md
21
+ ---
22
+
23
+ ## Summary
24
+
25
+ `_Worker_Monitor_Operations::EmailQueue()` is the cross-client health check for the 2.0
26
+ outbound email queue (`Logs_<Client>.Email` — see
27
+ [2.0 Email Send Pipeline](../../_underscore/features/email-send-pipeline.md)). It resolves
28
+ **every** client from `Core.Clients`, connects to each client's own logs database, counts
29
+ `FAILED` / `PENDING` rows in `Email`, and pushes one aggregate payload to a single OneUptime
30
+ Incoming Request monitor with the **offending clients named** in the body.
31
+
32
+ It was previously hardcoded to `Logs_Compass`, so a failing email queue for any other client
33
+ was completely invisible. The action path and method signature are unchanged, so the existing
34
+ `Core.WorkerJobs` cron entry needed no change. (Out-of-code follow-up: the OneUptime monitor
35
+ was still named "Compass Email Queue" and needs renaming in the UI.)
36
+
37
+ This is **shared internal infrastructure, not a Compass feature** — Compass USA is simply the
38
+ client it used to be limited to.
39
+
40
+ ## Key files / entry points
41
+
42
+ - `worker2/Worker/Monitor/Operations.php` — `abstract class _Worker_Monitor_Operations`.
43
+ - `public static function EmailQueue(): string` — the action, route
44
+ `Monitor/Operations/EmailQueue`.
45
+ - `private static function resolveClientLogDatabases(): array` — one `DB_CORE` query
46
+ returning every client + its log database name + this environment's hosts.
47
+ - `private static function isMissingTableError(Throwable $e): bool` — MySQL error-number
48
+ classification (see gotchas).
49
+ - `const MYSQL_ERROR_NO_SUCH_TABLE = 1146;`
50
+ - No other file in any repo references this class (verified by tree-wide grep) — the
51
+ autoloader and `_Worker::runTask()` reach it indirectly by action path.
52
+
53
+ ## How it works
54
+
55
+ 1. **Resolve every client's log database.** One query on `_underscore::DB_CORE` joins
56
+ `Clients` → `Databases` (via `Clients.logDatabaseId`) → `DatabaseHosts` →
57
+ `Environments` (slug) → `Regions` (code). Environment slug follows the
58
+ `dev-*` → `dev` collapsing used by `_Worker_Team_Transcripts::initialize()`; region comes
59
+ from `_Cloud::getRegion()`. Rows are ordered so the closest region for a client comes
60
+ first; later rows for the same client are skipped.
61
+ 2. **Register each client's log DB under its REAL database name, with no alias.** A per-client
62
+ loop cannot reuse the shared `_underscore::DB_CLIENT_LOGS` alias — every iteration would
63
+ clobber it (all live connection state is alias-keyed; see
64
+ [Re-pointing a DB alias mid-request](../../_underscore/features/database-alias-repointing.md)).
65
+ Registration is guarded with `if (!isset(_Database::$_registers[$logDatabaseName]))`.
66
+ 3. **Count per client**, index-friendly:
67
+ `SELECT status, COUNT(*) FROM Email WHERE status IN ('FAILED','PENDING') GROUP BY status`.
68
+ 4. **Bucket each client** into `checked`, `skipped` (no `Email` table), `unreachable` (any
69
+ other query/connection failure) or `unresolved` (no log database or no host for this
70
+ environment).
71
+ 5. **Decide the tokens and push once** to the same OneUptime monitor.
72
+
73
+ ### Alarm rule (explicit product decision)
74
+
75
+ Alarm on **`FAILED > 0` only**. `PENDING` is reported as context and **never** alarms — a
76
+ queue with work in it is normal, and the sender cron runs every minute.
77
+
78
+ ### Payload contract — `alarm` and `probe` are SEPARATE tokens
79
+
80
+ OneUptime can only string-match a request body (no numeric comparison, no JSON key
81
+ targeting), so the worker decides and emits tokens:
82
+
83
+ | Field | Meaning |
84
+ |---|---|
85
+ | `status` | `reporting`, or `error` reserved for the **Core lookup itself** failing (checker fully blind) |
86
+ | `alarm` | `HIGH` \| `OK` — emails are actually failing (`failedCount > 0`) |
87
+ | `probe` | `OK` \| `DEGRADED` — set `DEGRADED` when `unreachableClients` is non-empty |
88
+ | `failedCount`, `pendingCount`, `alarmFailedCountThreshold` | raw counts + the threshold used |
89
+ | `offendingClients` | comma-separated client **names** with per-client failed counts |
90
+ | `worstClient`, `worstFailedCount` | the single worst tenant |
91
+ | `clientsChecked`, `clientsSkipped` | coverage counters |
92
+ | `unreachableClients`, `unresolvedClients` | named coverage gaps |
93
+ | `checkedAtUtc` | `gmdate('c')` |
94
+
95
+ **Decision:** on a monitor that covers all clients, *"emails are failing"* and *"we could not
96
+ look at some clients"* need distinct OneUptime criteria. Collapsing them means a logs-cluster
97
+ outage reads as healthy email delivery. Hence two independent tokens rather than one.
98
+
99
+ ## Data model
100
+
101
+ `Email` / `EmailAttachment` ship in the `Logs_Client` blank
102
+ (`dbchanges2/Logs_Client/2026-05-21 - Email.sql`), so every client logs database should have
103
+ them. `Email.status` is `enum('PENDING','SENT','FAILED')`, with `retryCount` and
104
+ `failureReason`. Index `idx_email_status` exists on `status`.
105
+
106
+ ## Client variations
107
+
108
+ None — the check is uniform across every client in `Core.Clients`. Per-client differences that
109
+ matter are the **log database name** and whether the `Email` migration has been applied.
110
+
111
+ ## Gotchas / known issues
112
+
113
+ - **A client's database name is NOT derivable from `clientIdentifier`.** `Compass_Usa` →
114
+ `Logs_Compass`. Always resolve `Clients.logDatabaseId` → `Databases.name`. Building
115
+ `'Logs_' . $clientIdentifier` silently targets a nonexistent database. See
116
+ [per-client database connections](../../_underscore/features/per-client-database-connections.md).
117
+ - **Do not use `_Database::registerClientDatabases()` for an all-client fan-out.** It INNER
118
+ JOINs the client and archive databases alongside the log database, so a client missing either
119
+ row drops out and the method then dereferences a null row and throws. It also opens three
120
+ connections when only the log one is needed. Run a log-database-only variant with LEFT OUTER
121
+ JOINs instead.
122
+ - **INNER JOINs in an all-client resolution query silently un-monitor clients.** A client with
123
+ a null `logDatabaseId`, or with no `DatabaseHosts` row for the current environment, vanishes
124
+ from the result set with no log line and no alert token — the exact blind spot the monitor
125
+ exists to eliminate. LEFT OUTER JOIN every level and report those clients under an explicit
126
+ `unresolvedClients` bucket.
127
+ - **Follow-on trap once left-joined:** a `DatabaseHosts` row belonging to a *different*
128
+ environment shows up as a populated host and looks resolved. Null the host columns unless the
129
+ environment join matched:
130
+ `IF(LogEnvironments.id IS NULL, NULL, LogDatabaseHosts.hostCluster)`.
131
+ - **Detect "table does not exist" by MySQL error NUMBER, not message text.** `_Query` throws a
132
+ plain `Exception` whose message embeds the driver error on an `Error #: <number>` line
133
+ (`_underscore/Query.php:346`). Message wording is locale- and engine-dependent (MySQL vs
134
+ MariaDB differ); parse `/Error #:\s*(\d+)/` and compare to `1146` (`ER_NO_SUCH_TABLE`). A
135
+ logs DB predating the `Email` migration is a **skip**, not an outage.
136
+ - **An unconditional `SUM(status = 'X')` cannot use an index.**
137
+ `SELECT SUM(status='FAILED') FROM Email` full-scans, reading every historical `SENT` row
138
+ despite `idx_email_status`. Use `WHERE status IN (...) GROUP BY status`. This runs per client
139
+ on a cron, so the difference compounds.
140
+ - **There is no `_Query::setTimeout()` in the 2.0 framework** (only `_ApiRequest::setTimeout()`).
141
+ A 2.0 worker therefore has **no per-query timeout**, so a hung database cluster blocks the
142
+ fan-out until the connection itself times out. Relevant to any loop over all clients.
143
+ - **A passing dev run does not prove cross-cluster correctness.** Non-prod is a single
144
+ all-in-one database endpoint; production splits Core / `Client_*` / `Logs_*` / `Archive_*` /
145
+ `Cache` across **separate clusters**.
146
+ - **Verification status of this capture:** `php -l` passes; php-reviewer and sql-reviewer both
147
+ returned 0 critical and every actionable finding was fixed. It has **not** been executed
148
+ against any environment and **no database was queried** — the `Core` join shape is verified
149
+ against framework source only, not against real rows.
150
+
151
+ ## Change history
152
+
153
+ - 2026-07-30 — Refactored `EmailQueue()` from Compass-only (hardcoded `Logs_Compass` via
154
+ `initialize()` + `const DB_LOGS_COMPASS`, both removed) to an all-client fan-out
155
+ (121 → 309 lines): added `resolveClientLogDatabases()`, per-client registration under the real
156
+ database name with no alias, LEFT-OUTER-JOIN resolution with an `unresolvedClients` bucket,
157
+ error-number-based missing-table skip, index-friendly `GROUP BY status` counting, and the
158
+ separate `alarm` / `probe` tokens. Alarm on `FAILED > 0` only; `PENDING` never alarms. Action
159
+ path and signature unchanged, so no cron change. Uncommitted in the worker2 working tree at
160
+ capture time. (jcardinal)
161
+
162
+ ## Related docs
163
+
164
+ - [OneUptime push-metric monitors for 2.0 workers](./oneuptime-worker2-monitoring.md) — the
165
+ push/token pattern and OneUptime criteria this monitor follows.
166
+ - [Monitoring Framework](./monitoring-framework.md) — the parallel DB-driven, email-alert pattern.
167
+ - [2.0 Email Send Pipeline](../../_underscore/features/email-send-pipeline.md) — what fills the
168
+ queue this monitor watches.
169
+ - [Per-Client Database Connections](../../_underscore/features/per-client-database-connections.md)
170
+ — log-database name resolution.
@@ -6,15 +6,17 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-07-14
9
+ updated: 2026-07-30
10
10
  owners: ["jcardinal"]
11
11
  files:
12
12
  - worker2/Worker/Monitor/Compass.php
13
13
  - worker2/Worker/Client/Compass.php
14
14
  - worker2/composer.json
15
15
  - _underscore/Cloud.php
16
+ - worker2/Worker/Monitor/Operations.php
16
17
  related:
17
18
  - ./monitoring-framework.md
19
+ - ./all-client-email-queue-monitor.md
18
20
  - ../../_underscore/features/cloud-s3-helpers.md
19
21
  - ../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md
20
22
  ---
@@ -98,6 +100,17 @@ OneUptime criteria:
98
100
  prevents flapping back to Operational while the alarm is still HIGH).
99
101
  - **not received in 15 min** → Offline.
100
102
 
103
+ ### Separate the ALARM signal from the PROBE-health signal on multi-client monitors
104
+
105
+ When one monitor covers **every** client (rather than one client's integration), "the metric is
106
+ bad" and "we could not look at some clients" need **distinct** OneUptime criteria. Collapsing
107
+ them means a logs-cluster outage reads as healthy delivery. Emit **two independent string
108
+ tokens**: `alarm` (`HIGH`/`OK`) for the measured condition and `probe` (`OK`/`DEGRADED`) for
109
+ coverage gaps, and name the affected clients in the body so the incident is actionable without a
110
+ log dive. `status: "error"` stays reserved for the checker being **fully** blind (its own
111
+ resolution query failed). Worked example:
112
+ [All-Client Email Queue Monitor](./all-client-email-queue-monitor.md).
113
+
101
114
  ### Heartbeat / cron cadence timing
102
115
 
103
116
  The 10-min-Degraded / 15-min-Offline heartbeat thresholds pair with a **5-minute** cron
@@ -179,6 +192,11 @@ check for another client.
179
192
  monitors must pass the region (see [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md)).
180
193
 
181
194
  ## Change history
195
+ - 2026-07-30 — Added the multi-client refinement of the payload contract: a **separate `probe`
196
+ token** (`OK`/`DEGRADED`) alongside `alarm`, so a partial logs-cluster outage cannot read as a
197
+ healthy metric, plus naming the offending clients in the body. Recorded that the previously
198
+ Compass-only `Monitor/Operations/EmailQueue` is now an all-client monitor — see
199
+ [All-Client Email Queue Monitor](./all-client-email-queue-monitor.md). (jcardinal)
182
200
  - 2026-07-14 — Grew `_Worker_Monitor_Compass` to a 9-monitor suite (S3/DB/mailbox);
183
201
  finalized the payload contract (`status`/`alarm`/`checkedAtUtc`) and the not-received
184
202
  10-min-Degraded / 15-min-Offline + OK-gated recovery criteria; standardized cron on
@@ -113,6 +113,30 @@ Never create a `.sql` file inside `worker2`, `api2`, `_underscore`, or any other
113
113
  application repo. The SQL that belongs in those repos is only query strings embedded in
114
114
  PHP code — not standalone migration files.
115
115
 
116
+ ### Database isolation in `dbchanges2` — one database per file (HARD RULE)
117
+
118
+ **A `dbchanges2` `.sql` file may only reference tables in the ONE database its folder targets,
119
+ and must reference them UNQUALIFIED.** Never write `Core.Table` in a `Client_<Name>/` file,
120
+ never `Client_X.Table` in a `Core/` file, and use **no database qualifier at all** in the
121
+ fan-out folders (`Client/`, `Logs_Client/`, `_modules/`), where the tenant database name is not
122
+ fixed. This covers `JOIN`s, subqueries, `SET @var = (SELECT …)`, `INSERT … SELECT`, `NOT EXISTS`
123
+ guards, and `USE`.
124
+
125
+ In **production** `Core`, `Client_<Tenant>`, `Archive_<Tenant>`, `Logs`/`Logs_<Tenant>`, and
126
+ `Cache` are on **entirely separate clusters**, so a cross-database statement cannot resolve
127
+ there. **Local and every non-prod environment put all of them behind one endpoint**, so the
128
+ query runs fine, passes review, and fails only in production — never trust a local run as proof
129
+ it is safe.
130
+
131
+ Instead: hardcode a pre-resolved **v4 UUID literal**, resolve rows by a **slug / natural key that
132
+ exists in the target database**, or **split the work into one file per database folder** (passing
133
+ values between them as literals, not queries).
134
+
135
+ Enforced mechanically by the `PreToolUse` hook
136
+ `.claude/hooks/toga/dbchanges2-cluster-isolation.js`, which refuses the write. Full rule,
137
+ examples, and the approved rewrites: `2.0/apps/dbchanges2/architecture.md` → *Database
138
+ isolation*. **Applies to `dbchanges2` only — the 1.0 `dbchanges` repo is exempt.**
139
+
116
140
  ## Checking dependencies before touching shared code
117
141
 
118
142
  `dependsOn` in `knowledge/registry.json` means a repo extends or depends on another repo's classes. Before modifying a class in a dependency repo (e.g. `_underscore` core):
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
19
19
  ## 2.0 framework
20
20
 
21
21
  - **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
22
- - **worker2** (Worker) — 35 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
22
+ - **worker2** (Worker) — 36 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
23
23
  - **api2** (API) — 20 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
24
24
  - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
25
25
  - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.477",
3
+ "version": "1.0.479",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",