toga-ai 1.0.595 → 1.0.597
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/2.0/apps/toga-blox/features/primary-table-templates.md +16 -0
- package/knowledge/2.0/apps/toga-blox/features/table.md +40 -19
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
- package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md +1 -0
- package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md +182 -0
- package/knowledge/INDEX.md +1 -1
- package/package.json +1 -1
|
@@ -55,6 +55,13 @@ Constants in `PrimaryTable.tsx`: `DEFAULT_MIN_COLUMN_WIDTH = 80`, `ABSOLUTE_MIN_
|
|
|
55
55
|
|
|
56
56
|
1. **Measure** — on first paint, measure header + widest body cell content per column
|
|
57
57
|
(`measureCellContentWidth`), store in `frozenWidths`; per-column floors in `contentFloors`.
|
|
58
|
+
The **full header width is a HARD floor for every column** (`headerFloors` state): it is not
|
|
59
|
+
capped by max-width, not overridden by a consumer `minColumnWidth`, and is applied even to
|
|
60
|
+
explicitly-pinned (`columnWidths`) columns — the earlier `return def.size` bypass was why
|
|
61
|
+
pinned columns stayed clipped. Header width is measured **deterministically** = label text
|
|
62
|
+
`scrollWidth` + a fixed ~34px reserve per sort/filter control, **not** by measuring the
|
|
63
|
+
icon-font/SVG control elements (which can report ~0 width pre-paint on the freeze pass,
|
|
64
|
+
causing both header abbreviation and icon overlap).
|
|
58
65
|
2. **Freeze** — apply `table-layout: fixed` + a `<colgroup>` of frozen pixel widths so columns
|
|
59
66
|
don't reflow while virtualizing. Resets when the leaf column set changes. Empty tables stay
|
|
60
67
|
on auto-layout.
|
|
@@ -78,6 +85,10 @@ Sticky columns use a `--sticky-offset` CSS variable summed from neighboring colu
|
|
|
78
85
|
## Gotchas
|
|
79
86
|
|
|
80
87
|
- Frozen widths measure only **mounted** rows; virtualized tables size from the first paint set.
|
|
88
|
+
- **Header labels must never abbreviate or overlap.** The full header width is a hard floor
|
|
89
|
+
(above `maxColumnWidth`, `minColumnWidth`, and pinned `columnWidths`). Measure header width
|
|
90
|
+
deterministically (label `scrollWidth` + fixed per-control reserve), never by measuring the
|
|
91
|
+
async-loading sort/filter icon elements — they can report ~0 width before they paint.
|
|
81
92
|
- `headerSpan` adds a separate top row (`colSpan`); only the leaf header row maps to colgroup.
|
|
82
93
|
- Resize is non-persistent (local state, resets when columns change).
|
|
83
94
|
- **A visible column can look "missing" under `shrinkColumns`.** `shrinkColumns` (modal tables)
|
|
@@ -87,6 +98,11 @@ Sticky columns use a `--sticky-offset` CSS variable summed from neighboring colu
|
|
|
87
98
|
not the column set.
|
|
88
99
|
|
|
89
100
|
## Change history
|
|
101
|
+
- 2026-08-17 — Fixed clipped/abbreviated headers: full header width is now a hard floor for every
|
|
102
|
+
column (not capped by max-width, not overridden by `minColumnWidth`, applied even to pinned
|
|
103
|
+
`columnWidths` — the old `return def.size` bypass left pinned columns clipped). Added a
|
|
104
|
+
`headerFloors` state; header width is now measured deterministically (label `scrollWidth` +
|
|
105
|
+
fixed ~34px per sort/filter control) instead of measuring async-loading icon elements. (apeterson)
|
|
90
106
|
- 2026-08-17 — Noted that under `shrinkColumns` a genuinely visible column can render at a sliver width and look "missing" — check `isVisible` and the measured/frozen width, not the column set (apeterson).
|
|
91
107
|
- 2026-06-23 — Documented the three Primary Table entry points, measure-and-freeze sizing, virtualization, and sticky columns (apeterson).
|
|
92
108
|
- 2026-06-22 — Added `shrinkColumns` / `minColumnWidth` shrink-to-fit for modal tables; freeze logic uses natural content width (apeterson).
|
|
@@ -88,13 +88,18 @@ isHovered, skin, isDisabled }`. They are inserted as columns via `buildActionCol
|
|
|
88
88
|
|
|
89
89
|
Per-column header controls, each a popover trigger with `data-active` / `data-open`:
|
|
90
90
|
|
|
91
|
-
- **`HeaderFilterSearch`** — text filter,
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
91
|
+
- **`HeaderFilterSearch`** — text filter, **multi-chip with per-chip mode**. Each confirmed
|
|
92
|
+
search becomes its own `TextFilterTag = { value, mode }` (mode ∈ Includes / Starts with /
|
|
93
|
+
Ends with / Exactly / Excludes). The mode dropdown sets the mode for the **next** chip
|
|
94
|
+
added; existing chips keep the mode they were created with. All chips on a column are
|
|
95
|
+
combined with **AND**. `onFilterChange` emits the full tags array (`{ tags }`); the
|
|
96
|
+
component rehydrates every chip from props and renders each as "Mode: value". `isActive =
|
|
97
|
+
tags.length > 0`. The mode dropdown is portalled to `document.body` with fixed positioning
|
|
98
|
+
recomputed on scroll. The search icon (solid when active, regular otherwise) is hidden
|
|
99
|
+
while the input is focused; both the trigger button and the icon span carry `data-active`.
|
|
100
|
+
The clear button must not widen the input row (fixed-width menu).
|
|
101
|
+
(Previously a column allowed only one term / one mode — a second search replaced the first
|
|
102
|
+
and switching mode re-applied it to everything.)
|
|
98
103
|
- **`HeaderFilterRange`** — numeric, modes exactly/moreThan/lessThan + a range toggle
|
|
99
104
|
(min/max); emits `{ key, value }[]` (`eq`/`min`/`max`/`between`). Reads its persisted
|
|
100
105
|
operator/values back from a `filterParams` prop (not `filterValue`, which collapses to the
|
|
@@ -134,6 +139,10 @@ Per-column header controls, each a popover trigger with `data-active` / `data-op
|
|
|
134
139
|
## Gotchas
|
|
135
140
|
|
|
136
141
|
- `MIN_COLUMN_WIDTH = 120` is the shrink floor only for columns with **no** explicit width.
|
|
142
|
+
- **Never truncate header labels.** `truncateCells` (`.truncateCells .headerCellLabel` in
|
|
143
|
+
`toga.module.css`) is for **body** values only; it must apply only `white-space: nowrap` to
|
|
144
|
+
the header (no `overflow:hidden`/`text-overflow:ellipsis`/wrapping) or headers show ellipses.
|
|
145
|
+
Headers must render in full on one line, never abbreviated, never overlapping.
|
|
137
146
|
- The filter `includes` mode treats comma-separated values as OR.
|
|
138
147
|
- `EditableCell` currency formats **on blur**, phone formats **on keystroke**.
|
|
139
148
|
- Cell type config (`columnTypeConfig` prop) overrides the hardcoded `COLUMN_TYPE_CONFIG`.
|
|
@@ -151,24 +160,36 @@ Per-column header controls, each a popover trigger with `data-active` / `data-op
|
|
|
151
160
|
never for data fetching. A visible column can still *look* missing when `shrinkColumns` freezes
|
|
152
161
|
it to a tiny width because its measured header+body content is minimal (see
|
|
153
162
|
[primary-table-templates](primary-table-templates.md)).
|
|
154
|
-
- **Filter presentational components must seed their internal
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
(
|
|
163
|
+
- **Filter presentational components must seed their internal state from props.**
|
|
164
|
+
Filter state survives the URL round-trip and is rehydrated into TanStack `columnFilters` by
|
|
165
|
+
toga25-supply `useServerTableUrlState`, carried through `PrimaryTableHeaderCell`. But
|
|
166
|
+
`HeaderFilterSearch`/`HeaderFilterRange` are remounted fresh each time a popover opens, so
|
|
167
|
+
they must seed local state from props or reopening *shows the wrong state*.
|
|
168
|
+
**Text (`HeaderFilterSearch`)** now persists as a single readable URL param
|
|
169
|
+
`<slug>_<col>_filter` = comma-separated `mode:value` pairs (values URL-encoded so `:`/`,`
|
|
170
|
+
inside a term can't corrupt the delimiters); `getDataTableData` parses it and pushes **one
|
|
171
|
+
AND WHERE condition per chip** (mode→op: includes→contains, startsWith→starts,
|
|
172
|
+
endsWith→ends, exactly→eq, excludes→excludes). The component rehydrates the full `tags`
|
|
173
|
+
array from props. Old base64/legacy single-mode `_starts`/`_ends`/… URLs don't parse under
|
|
174
|
+
the new scheme (graceful — just re-apply the filter). The browser may percent-encode `:`/`,`
|
|
175
|
+
in the address bar but it round-trips.
|
|
176
|
+
**Range (`HeaderFilterRange`)** still persists as key suffixes (`_min`/`_max`/`_eq`/
|
|
177
|
+
`_between`); it reads mode + value(s) back from `filterParams` via a rehydration effect keyed
|
|
178
|
+
on `JSON.stringify(filterParams)` (`eq`→Exactly, `min`→MoreThan, `max`→LessThan,
|
|
179
|
+
`between`→range min/max).
|
|
166
180
|
- **Table/surface meta is cached with `Infinity` staleTime/gcTime** (`useFetchTablePageMeta`), so
|
|
167
181
|
after a DB metadata change (`TableViewFields`, `Core.RecordFields`) a **hard reload** is required
|
|
168
182
|
to see it. And the app consumes toga-blox's built **`dist/`**, not `src/` — editing blox source
|
|
169
183
|
requires `npm run build` before it takes effect at runtime.
|
|
170
184
|
|
|
171
185
|
## Change history
|
|
186
|
+
- 2026-08-17 — Built multi-chip text column filter: each `HeaderFilterSearch` search is now its
|
|
187
|
+
own `TextFilterTag {value, mode}`, the mode dropdown sets the mode for the next chip, existing
|
|
188
|
+
chips keep their mode, and all chips AND together. Persisted as a single readable URL param
|
|
189
|
+
`<slug>_<col>_filter` = `mode:value,...` (URL-encoded values); `getDataTableData` emits one AND
|
|
190
|
+
WHERE per chip. Also fixed: header labels were being ellipsis-truncated (and overlapping the
|
|
191
|
+
next column) — `truncateCells` is body-only, so `.truncateCells .headerCellLabel` is now just
|
|
192
|
+
`white-space: nowrap`. (apeterson)
|
|
172
193
|
- 2026-08-17 — Fixed: column filter **mode** was not restored when a filter popover was reopened
|
|
173
194
|
(text reverted to `includes`, range to `exactly`). Root cause: the URL round-trip and
|
|
174
195
|
`PrimaryTableHeaderCell` preserved the mode, but `HeaderFilterSearch` (destructured
|
|
@@ -18,6 +18,7 @@
|
|
|
18
18
|
| [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
|
|
19
19
|
| [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
|
|
20
20
|
| [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php |
|
|
21
|
+
| [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's **enhanced-health** status | worker2/Worker/Infrastructure/CloudWatch.php |
|
|
21
22
|
| [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
|
|
22
23
|
| [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Notification/Email.php, worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql, dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql |
|
|
23
24
|
| [Error-Issue Auto-Resolution & Reopen (frequency-decay lifecycle)](features/error-issue-auto-resolution.md) | The error system could escalate and de-escalate an Issue's *urgency* but had no concept of an Issue being **resolved**. | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, _underscore/Model/Core/Logs/Issue.php, tools/mvc/errors/get.php, tools/mvc/errors/issue/get.php, dbchanges2/Logs/2026-08-05a - Issue status baseline and auto-resolution.sql |
|
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-08-17
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Infrastructure/CloudWatch.php
|
|
13
|
+
related:
|
|
14
|
+
- ./cross-account-aws-access.md
|
|
15
|
+
- ./oneuptime-worker2-monitoring.md
|
|
16
|
+
- ./monitoring-framework.md
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Summary
|
|
20
|
+
|
|
21
|
+
`_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads
|
|
22
|
+
each Elastic Beanstalk (EB) environment's **enhanced-health** status and reports it to a
|
|
23
|
+
OneUptime **Incoming Request** monitor, which pages the team. It is a **"dumb reporter,
|
|
24
|
+
smart monitor"** heartbeat (the same Pattern-B contract as
|
|
25
|
+
[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)): the worker decides
|
|
26
|
+
pass/fail and POSTs decided **string tokens** (`alarm=HIGH/OK`, `probe=DEGRADED/OK`) that
|
|
27
|
+
OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
|
|
28
|
+
|
|
29
|
+
The load-bearing design decision this doc records is the **paging (alarm) criteria**: page
|
|
30
|
+
on EB `Degraded`/`Severe`, but **suppress the common "an occasional HTTP 500 tripped
|
|
31
|
+
Degraded" false alarm** unless 5xx errors actually dominate the request mix.
|
|
32
|
+
|
|
33
|
+
**Critical behavior:** the method is **non-fatal and never throws** — it always returns a
|
|
34
|
+
string. worker2 has **no DLQ and a 3600s SQS visibility timeout**, so any uncaught 500
|
|
35
|
+
becomes a poison-message storm; an outer `catch(\Throwable)` is the backstop and the push
|
|
36
|
+
itself runs non-fatal.
|
|
37
|
+
|
|
38
|
+
## Key files / entry points
|
|
39
|
+
|
|
40
|
+
- `worker2/Worker/Infrastructure/CloudWatch.php` — `_Worker_Infrastructure_CloudWatch`,
|
|
41
|
+
action `ElasticBeanstalkHealth`, dispatched via
|
|
42
|
+
`_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
|
|
43
|
+
environments, oneuptimeUrl})`.
|
|
44
|
+
- AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
|
|
45
|
+
a read-only role per account, one assume reused across regions) — see
|
|
46
|
+
[Cross-account AWS access](./cross-account-aws-access.md), which also documents the
|
|
47
|
+
`environments` region→env-names map parameter shape.
|
|
48
|
+
- `oneuptimeUrl` arrives as a **cron parameter and is a push credential** — never log it and
|
|
49
|
+
never record its value in a doc (see gotchas).
|
|
50
|
+
|
|
51
|
+
## Alarm / paging criteria (the decision logic)
|
|
52
|
+
|
|
53
|
+
Per environment, `alarm=HIGH` (page) when the EB `HealthStatus` is `Degraded` **or**
|
|
54
|
+
`Severe`. `Suspended` never pages.
|
|
55
|
+
|
|
56
|
+
- **`Severe` always pages.**
|
|
57
|
+
- **`Degraded` for a non-5xx reason** (latency, instances down, …) **always pages.**
|
|
58
|
+
- **`Degraded` attributed to HTTP 5xx errors pages only when 5xx errors dominate** — i.e.
|
|
59
|
+
the 5xx share of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80). An occasional
|
|
60
|
+
500 that trips Degraded is noise and must **not** page; a flood is a real problem and
|
|
61
|
+
**must** page.
|
|
62
|
+
- **Tiny-sample guard:** below `HTTP_5XX_MIN_REQUEST_COUNT` (default 20) requests in the
|
|
63
|
+
window, do **not** trust the ratio — page rather than let a 2-of-3 blip read as "80%
|
|
64
|
+
failing."
|
|
65
|
+
|
|
66
|
+
### Fail-safe: a blind read pages, never reads healthy
|
|
67
|
+
|
|
68
|
+
If the health read (`describeEnvironmentHealth`) throws `AwsException` for an environment,
|
|
69
|
+
the monitor sets **`alarm=HIGH`** for that environment (a blind read must never masquerade
|
|
70
|
+
as healthy) **and** sets **`probe=DEGRADED`** (coverage gap, distinct from a measured-bad
|
|
71
|
+
signal — same alarm-vs-probe split as the multi-client monitors in
|
|
72
|
+
[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)).
|
|
73
|
+
|
|
74
|
+
### Configurable constants (top of the class)
|
|
75
|
+
|
|
76
|
+
| Constant | Default | Meaning |
|
|
77
|
+
|---|---|---|
|
|
78
|
+
| `HEALTH_ALARM_STATUSES` | `['Degraded','Severe']` | HealthStatus values that page |
|
|
79
|
+
| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a 5xx-driven `Degraded` pages |
|
|
80
|
+
| `HTTP_5XX_MIN_REQUEST_COUNT` | `20` | Below this many requests in the window, page rather than trust the ratio |
|
|
81
|
+
| `HTTP_5XX_CAUSE_MARKERS` | `['5xx','http 5']` | Case-insensitive substrings identifying a 5xx-attributed EB `Cause` |
|
|
82
|
+
|
|
83
|
+
### Implementation shape
|
|
84
|
+
|
|
85
|
+
- `shouldPageForHealth(healthStatus, causes, applicationMetrics): [bool, ?float]` — returns
|
|
86
|
+
the page decision and the observed 5xx ratio.
|
|
87
|
+
- `causesAttributeTo5xx(causes): bool` — substring-matches `HTTP_5XX_CAUSE_MARKERS` against
|
|
88
|
+
the EB `Causes` strings (case-insensitive).
|
|
89
|
+
- The ratio is computed as `StatusCodes.Status5xx / RequestCount` from `ApplicationMetrics`
|
|
90
|
+
(raw counts — see below), **not** by parsing the percentage out of the `Causes` text.
|
|
91
|
+
- `describeEnvironmentHealth` is called with
|
|
92
|
+
`AttributeNames = ['HealthStatus','Status','Color','Causes','ApplicationMetrics']`.
|
|
93
|
+
- Per-environment report + OneUptime payload now include the per-env `alarm` token and the
|
|
94
|
+
observed `fivexxRatio`.
|
|
95
|
+
|
|
96
|
+
## How EB enhanced-health is read (durable AWS reference)
|
|
97
|
+
|
|
98
|
+
Facts verified against current AWS docs; they inform why the logic above is shaped as it is.
|
|
99
|
+
|
|
100
|
+
- **HealthStatus severity ladder:** `Ok → Info → Warning → Degraded → Severe` (plus the grey
|
|
101
|
+
states `Pending`/`Unknown`/`Suspended`/`NoData`). `Degraded` is the "high failure" tier
|
|
102
|
+
and is **routinely tripped by benign transients** (Auto Scaling scale-up, a mid-deploy
|
|
103
|
+
dip), which is exactly why `Degraded` alone is noisy. `Severe` = "very high failure /
|
|
104
|
+
environment effectively not serving."
|
|
105
|
+
- **`ApplicationMetrics` returns RAW COUNTS, not percentages** — despite the API-reference
|
|
106
|
+
prose saying "percentage"/"per second". Fields: `Duration` (Integer seconds, usually 10),
|
|
107
|
+
`RequestCount` (Integer, **total** requests over the window),
|
|
108
|
+
`StatusCodes.{Status2xx,Status3xx,Status4xx,Status5xx}` (Integer counts). AWS's own
|
|
109
|
+
example: 2xx 3391 + 5xx 843 = RequestCount 4234. With no traffic, `RequestCount=0` and
|
|
110
|
+
`StatusCodes` may be **absent** → treat as "no data," not "zero failures." The 5xx-ratio
|
|
111
|
+
denominator is `RequestCount` (the authoritative total).
|
|
112
|
+
- **Why an env is Degraded comes from `Causes`:** EB `Causes` strings literally contain e.g.
|
|
113
|
+
`"19.9 % of the requests are failing with HTTP 5xx."` A substring check reliably
|
|
114
|
+
**classifies** 5xx-driven vs. not (fail-safe: a wording change → treated as a normal
|
|
115
|
+
`Degraded` → pages). The **ratio itself must come from `ApplicationMetrics`**, never by
|
|
116
|
+
parsing the number out of `Causes`.
|
|
117
|
+
- **A failed application DEPLOYMENT is NOT reliably reflected in HealthStatus** — it can read
|
|
118
|
+
`Warning`/`Degraded`, or fail before any red request-failure signal. The authoritative
|
|
119
|
+
structured deploy signal is per-instance `describeInstancesHealth` →
|
|
120
|
+
`Deployment.Status ∈ {'In Progress','Deployed','Failed'}`. (This detector was evaluated and
|
|
121
|
+
removed this iteration — see Design history.)
|
|
122
|
+
- **IAM:** `describeEnvironmentHealth` / `describeInstancesHealth` / `describeEvents` are all
|
|
123
|
+
covered by the managed `AWSElasticBeanstalkReadOnly` policy and require enhanced health
|
|
124
|
+
enabled. No per-call charge.
|
|
125
|
+
|
|
126
|
+
## OneUptime monitor config (external, not code)
|
|
127
|
+
|
|
128
|
+
Paging depends on the OneUptime Incoming-Request monitor's **string-match** criteria on the
|
|
129
|
+
POSTed body — `Contains "alarm":"HIGH"` (and, if desired, `Contains "probe":"DEGRADED"`).
|
|
130
|
+
This is external OneUptime configuration, not worker2 code.
|
|
131
|
+
|
|
132
|
+
## Design history — rejected direction (this session)
|
|
133
|
+
|
|
134
|
+
An earlier iteration set `HEALTH_ALARM_STATUSES = ['Severe']` (dropping `Degraded` entirely)
|
|
135
|
+
and **added a separate deployment-failure detector** (`describeInstancesHealth` →
|
|
136
|
+
`Deployment.Status = 'Failed'`) to still catch deploy failures. That was **reverted** after a
|
|
137
|
+
scope change: the team decided they **do** want to page on `Degraded` (to catch a flood of
|
|
138
|
+
500s and non-500 degradations), with only the occasional-500 case suppressed via the 80%
|
|
139
|
+
ratio. The deployment-failure detector was **removed** — deploy failures that surface as
|
|
140
|
+
`Degraded` are now covered by the `Degraded` alarm. **Residual gap accepted for this
|
|
141
|
+
iteration:** a deploy that fails fast *without ever degrading health* is not caught.
|
|
142
|
+
|
|
143
|
+
## Known gaps / follow-ups (not done)
|
|
144
|
+
|
|
145
|
+
- **Per-region client construction is not individually guarded.** Each
|
|
146
|
+
`new ElasticBeanstalkClient` is not in its own try/catch, so a bad region/creds aborts the
|
|
147
|
+
**whole run** via the outer backstop instead of paging just that region's envs as blind.
|
|
148
|
+
Candidate follow-up now that blind reads page.
|
|
149
|
+
- **No first-party test harness exists in worker2** (all tests are vendor/).
|
|
150
|
+
`shouldPageForHealth()` and `causesAttributeTo5xx()` are pure and ideal to unit-test
|
|
151
|
+
(ratio at exactly 0.80; count 19 vs 20; `Severe` over a 5xx cause; empty causes; null
|
|
152
|
+
metrics) — deferred pending a harness.
|
|
153
|
+
|
|
154
|
+
## Gotchas / known issues
|
|
155
|
+
|
|
156
|
+
- **Keep the method non-fatal — never let it throw.** worker2 has no DLQ and a 3600s SQS
|
|
157
|
+
visibility timeout, so any uncaught 500 becomes a poison-message storm. The push runs
|
|
158
|
+
non-fatal and an outer `catch(\Throwable)` is the backstop.
|
|
159
|
+
- **A blind health read must page, not read healthy** — an `AwsException` on
|
|
160
|
+
`describeEnvironmentHealth` sets `alarm=HIGH` + `probe=DEGRADED` for that env.
|
|
161
|
+
- **`ApplicationMetrics` is raw counts, not percentages** — divide `Status5xx` by
|
|
162
|
+
`RequestCount`; `RequestCount=0`/absent `StatusCodes` means "no data," not "zero failures."
|
|
163
|
+
- **Classify 5xx-attribution from `Causes` text, but take the ratio from `ApplicationMetrics`
|
|
164
|
+
— never parse the percentage out of `Causes`.**
|
|
165
|
+
- **`oneuptimeUrl` is a push credential** — it arrives as a cron parameter; never log it or
|
|
166
|
+
record its value in a doc.
|
|
167
|
+
|
|
168
|
+
## Change history
|
|
169
|
+
- 2026-08-17 — Created. Documented the `ElasticBeanstalkHealth` alarm/paging criteria (page
|
|
170
|
+
on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless the 5xx share ≥
|
|
171
|
+
`HTTP_5XX_ALARM_RATIO` 0.80 with an `HTTP_5XX_MIN_REQUEST_COUNT` 20 tiny-sample guard;
|
|
172
|
+
`Severe` and non-5xx `Degraded` always page), the blind-read fail-safe (`alarm=HIGH` +
|
|
173
|
+
`probe=DEGRADED`), the four configurable constants, and the `shouldPageForHealth` /
|
|
174
|
+
`causesAttributeTo5xx` helper shape. Captured durable EB enhanced-health facts (severity
|
|
175
|
+
ladder; `ApplicationMetrics` returns raw counts not percentages; `Causes` classifies but
|
|
176
|
+
`ApplicationMetrics` sets the ratio; failed deploys aren't reliably in HealthStatus —
|
|
177
|
+
`describeInstancesHealth.Deployment.Status` is authoritative; `AWSElasticBeanstalkReadOnly`
|
|
178
|
+
covers the reads). Recorded the reverted `['Severe']`-only + deployment-failure-detector
|
|
179
|
+
direction and the accepted residual gap (a deploy that fails without degrading health).
|
|
180
|
+
(jcardinal)
|
|
181
|
+
</content>
|
|
182
|
+
</invoke>
|
package/knowledge/INDEX.md
CHANGED
|
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
19
19
|
## 2.0 framework
|
|
20
20
|
|
|
21
21
|
- **_underscore** (_Underscore) _(framework core)_ — 58 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
22
|
-
- **worker2** (Worker) —
|
|
22
|
+
- **worker2** (Worker) — 50 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
23
23
|
- **api2** (API) — 24 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
24
24
|
- **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
25
25
|
- **toga2-supply** (TOGa Supply) — 6 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
package/package.json
CHANGED