toga-ai 1.0.619 → 1.0.620

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-17
+updated: 2026-08-19
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -28,7 +28,10 @@ OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
 
 The load-bearing design decision this doc records is the **paging (alarm) criteria**: page
 on EB `Degraded`/`Severe`, but **suppress the common "an occasional HTTP 500 tripped
-Degraded" false alarm** unless 5xx errors actually dominate the request mix.
+Degraded/Severe" false alarm** unless 5xx errors actually dominate the request mix (by
+share) or reach a real absolute volume. The 5xx gate applies to **both** `Degraded` and
+`Severe`, and only when EB attributes the problem **solely** to 5xx — a mixed cause always
+pages.
 
 **Critical behavior:** the method is **non-fatal and never throws** — it always returns a
 string. worker2 has **no DLQ and a 3600s SQS visibility timeout**, so any uncaught 500
@@ -53,15 +56,33 @@ itself runs non-fatal.
 Per environment, `alarm=HIGH` (page) when the EB `HealthStatus` is `Degraded` **or**
 `Severe`. `Suspended` never pages.
 
-- **`Severe` always pages.**
-- **`Degraded` for a non-5xx reason** (latency, instances down, …) **always pages.**
-- **`Degraded` attributed to HTTP 5xx errors pages only when 5xx errors dominate** — i.e.
-  the 5xx share of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80). An occasional
-  500 that trips Degraded is noise and must **not** page; a flood is a real problem and
-  **must** page.
-- **Tiny-sample guard:** below `HTTP_5XX_MIN_REQUEST_COUNT` (default 20) requests in the
-  window, do **not** trust the ratio — page rather than let a 2-of-3 blip read as "80%
-  failing."
+The **5xx gate applies to both `Degraded` and `Severe`.** A non-5xx or mixed-cause
+`Degraded`/`Severe` — and any env with no stated cause — **always pages**; the gate only
+ever suppresses an env EB attributes **solely** to 5xx.
+
+- **Non-5xx or mixed-cause `Degraded`/`Severe`** (latency, instances down, a failed deploy,
+  or a 5xx trickle *alongside* a real non-5xx failure) **always pages.**
+- **Solely-5xx `Degraded`/`Severe` pages only on a genuine flood** — either:
+  - the **absolute** 5xx count in the window is at/above `HTTP_5XX_ALARM_ABSOLUTE_COUNT`
+    (default 20), **or**
+  - the **5xx share** of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80).
+- **Zero requests in the window → no page** (nothing observed; also avoids divide-by-zero).
+
+**Why the gate now covers `Severe` too:** EB escalates an environment to `Severe` precisely
+when the 5xx share is high, so real 5xx floods reach `Severe` *first* and never touch the
+gated `Degraded` branch. When the gate applied only to `Degraded`, the 80% threshold was
+effectively dead — a real 50%-5xx `Severe` event paged despite the setting. Gating both
+tiers makes the threshold actually govern.
+
+**Why "solely" 5xx, not "any" 5xx cause:** a mixed-cause env (a 5xx trickle plus a real
+non-5xx failure such as a failed deploy) must not be silenced by a sub-threshold ratio. The
+ratio gate only applies when **every** stated cause references 5xx (and there is at least
+one); anything else pages.
+
+**Why an absolute floor in addition to the ratio:** the ratio alone is magnitude-blind —
+60k failing out of 100k reads as 60% and would be suppressed, though it is plainly a flood.
+`HTTP_5XX_ALARM_ABSOLUTE_COUNT` pages such an env regardless of share; tune it to the busiest
+monitored environment's request volume.
 
 ### Fail-safe: a blind read pages, never reads healthy
 
@@ -76,18 +97,23 @@ signal — same alarm-vs-probe split as the multi-client monitors in
 | Constant | Default | Meaning |
 |---|---|---|
 | `HEALTH_ALARM_STATUSES` | `['Degraded','Severe']` | HealthStatus values that page |
-| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a 5xx-driven `Degraded` pages |
-| `HTTP_5XX_MIN_REQUEST_COUNT` | `20` | Below this many requests in the window, page rather than trust the ratio |
+| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a solely-5xx `Degraded`/`Severe` pages |
+| `HTTP_5XX_ALARM_ABSOLUTE_COUNT` | `20` | Absolute 5xx count in the window at/above which a solely-5xx `Degraded`/`Severe` pages regardless of share |
 | `HTTP_5XX_CAUSE_MARKERS` | `['5xx','http 5']` | Case-insensitive substrings identifying a 5xx-attributed EB `Cause` |
 
 ### Implementation shape
 
 - `shouldPageForHealth(healthStatus, causes, applicationMetrics): [bool, ?float]` — returns
   the page decision and the observed 5xx ratio.
-- `causesAttributeTo5xx(causes): bool` — substring-matches `HTTP_5XX_CAUSE_MARKERS` against
-  the EB `Causes` strings (case-insensitive).
+- `causesAttributeSolelyTo5xx(causes): bool` — true only when **every** EB `Cause` references
+  5xx **and there is at least one** cause. Replaces the former `causesAttributeTo5xx()`, which
+  returned true if *any* cause mentioned 5xx and so let a mixed-cause env be gated.
+- `causeIsFivexx(cause): bool` — helper that substring-matches `HTTP_5XX_CAUSE_MARKERS`
+  against a single EB `Cause` string (case-insensitive); `causesAttributeSolelyTo5xx` requires
+  it to hold for all causes.
 - The ratio is computed as `StatusCodes.Status5xx / RequestCount` from `ApplicationMetrics`
-  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text.
+  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text. The
+  absolute-count floor reads `StatusCodes.Status5xx` directly.
 - `describeEnvironmentHealth` is called with
   `AttributeNames = ['HealthStatus','Status','Color','Causes','ApplicationMetrics']`.
 - Per-environment report + OneUptime payload now include the per-env `alarm` token and the
@@ -147,9 +173,10 @@ iteration:** a deploy that fails fast *without ever degrading health* is not cau
   **whole run** via the outer backstop instead of paging just that region's envs as blind.
   Candidate follow-up now that blind reads page.
 - **No first-party test harness exists in worker2** (all tests are vendor/).
-  `shouldPageForHealth()` and `causesAttributeTo5xx()` are pure and ideal to unit-test
-  (ratio at exactly 0.80; count 19 vs 20; `Severe` over a 5xx cause; empty causes; null
-  metrics) — deferred pending a harness.
+  `shouldPageForHealth()` and `causesAttributeSolelyTo5xx()` are pure and ideal to unit-test
+  (ratio at exactly 0.80; absolute count 19 vs 20; `Severe` over a solely-5xx cause; a
+  mixed 5xx + non-5xx cause; empty causes; null metrics; zero requests) — deferred pending a
+  harness.
 
 ## Gotchas / known issues
 
@@ -164,8 +191,28 @@ iteration:** a deploy that fails fast *without ever degrading health* is not cau
   — never parse the percentage out of `Causes`.**
 - **`oneuptimeUrl` is a push credential** — it arrives as a cron parameter; never log it or
   record its value in a doc.
+- **Near-idle ratio edge (residual, accepted).** With the tiny-sample guard removed, a
+  near-idle env whose window holds a tiny all-error sample (e.g. its only request being a
+  500 = 100%) still trips the ratio and pages. Optional future guard: require a minimum 5xx
+  count before the ratio applies (the absolute floor gates high volume, not this low-volume
+  edge).
 
 ## Change history
+- 2026-08-19 — Overhauled the 5xx alarm gating in `shouldPageForHealth()`. The 80% share
+  gate now applies to **both** `Degraded` and `Severe` (was `Degraded`-only, which left the
+  threshold dead because EB escalates real 5xx floods straight to `Severe` — a 50%-5xx
+  `Severe` had been paging despite the 0.80 setting). This Severe-gating trade-off was made
+  deliberately with an independent architecture second opinion on record (cto returned
+  DISAGREE-WITH-ALTERNATIVE; developer accepted). Narrowed 5xx classification: replaced
+  `causesAttributeTo5xx()` (any cause mentions 5xx) with `causesAttributeSolelyTo5xx()` (every
+  cause references 5xx, ≥1) + a `causeIsFivexx()` helper, so a mixed cause (5xx trickle beside
+  a real non-5xx failure) always pages — a security-review gap the Severe change would have
+  widened. Added `HTTP_5XX_ALARM_ABSOLUTE_COUNT` (20): a solely-5xx env pages regardless of
+  share once the absolute 5xx count reaches the floor (closes the ratio's magnitude-blindness,
+  e.g. 60k of 100k). Removed `HTTP_5XX_MIN_REQUEST_COUNT` (was 20) and its tiny-sample
+  page-anyway guard — a small number of 5xx is now judged purely on the ratio; below the
+  absolute floor, page only if share ≥ 0.80, and zero requests → no page. Accepted residual:
+  a near-idle env with a tiny all-error sample still trips the ratio. (jcardinal)
 - 2026-08-17 — Created. Documented the `ElasticBeanstalkHealth` alarm/paging criteria (page
   on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless the 5xx share ≥
   `HTTP_5XX_ALARM_RATIO` 0.80 with an `HTTP_5XX_MIN_REQUEST_COUNT` 20 tiny-sample guard;
@@ -178,5 +225,3 @@ iteration:** a deploy that fails fast *without ever degrading health* is not cau
   covers the reads). Recorded the reverted `['Severe']`-only + deployment-failure-detector
   direction and the accepted residual gap (a deploy that fails without degrading health).
   (jcardinal)
-</content>
-</invoke>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.619",
+  "version": "1.0.620",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",