toga-ai 1.0.596 → 1.0.597

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -18,6 +18,7 @@
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php |
+| [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's **enhanced-health** status | worker2/Worker/Infrastructure/CloudWatch.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Notification/Email.php, worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql, dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql |
 | [Error-Issue Auto-Resolution & Reopen (frequency-decay lifecycle)](features/error-issue-auto-resolution.md) | The error system could escalate and de-escalate an Issue's *urgency* but had no concept of an Issue being **resolved**. | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, _underscore/Model/Core/Logs/Issue.php, tools/mvc/errors/get.php, tools/mvc/errors/issue/get.php, dbchanges2/Logs/2026-08-05a - Issue status baseline and auto-resolution.sql |

package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md

@@ -17,6 +17,7 @@ related:
   - ../../_underscore/features/config-group-access.md
   - ../../_underscore/features/cloud-s3-helpers.md
   - ./oneuptime-worker2-monitoring.md
+  - ./elastic-beanstalk-health-monitor.md
 ---
 
 ## Summary

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -0,0 +1,182 @@
+---
+title: Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-08-17
+owners: [jcardinal]
+files:
+  - worker2/Worker/Infrastructure/CloudWatch.php
+related:
+  - ./cross-account-aws-access.md
+  - ./oneuptime-worker2-monitoring.md
+  - ./monitoring-framework.md
+---
+
+## Summary
+
+`_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads
+each Elastic Beanstalk (EB) environment's **enhanced-health** status and reports it to a
+OneUptime **Incoming Request** monitor, which pages the team. It is a **"dumb reporter,
+smart monitor"** heartbeat (the same Pattern-B contract as
+[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)): the worker decides
+pass/fail and POSTs decided **string tokens** (`alarm=HIGH/OK`, `probe=DEGRADED/OK`) that
+OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
+
+The load-bearing design decision this doc records is the **paging (alarm) criteria**: page
+on EB `Degraded`/`Severe`, but **suppress the common "an occasional HTTP 500 tripped
+Degraded" false alarm** unless 5xx errors actually dominate the request mix.
+
+**Critical behavior:** the method is **non-fatal and never throws** — it always returns a
+string. worker2 has **no DLQ and a 3600s SQS visibility timeout**, so any uncaught 500
+becomes a poison-message storm; an outer `catch(\Throwable)` is the backstop and the push
+itself runs non-fatal.
+
+## Key files / entry points
+
+- `worker2/Worker/Infrastructure/CloudWatch.php` — `_Worker_Infrastructure_CloudWatch`,
+  action `ElasticBeanstalkHealth`, dispatched via
+  `_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
+  environments, oneuptimeUrl})`.
+- AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
+  a read-only role per account, one assume reused across regions) — see
+  [Cross-account AWS access](./cross-account-aws-access.md), which also documents the
+  `environments` region→env-names map parameter shape.
+- `oneuptimeUrl` arrives as a **cron parameter and is a push credential** — never log it and
+  never record its value in a doc (see gotchas).
+
+## Alarm / paging criteria (the decision logic)
+
+Per environment, `alarm=HIGH` (page) when the EB `HealthStatus` is `Degraded` **or**
+`Severe`. `Suspended` never pages.
+
+- **`Severe` always pages.**
+- **`Degraded` for a non-5xx reason** (latency, instances down, …) **always pages.**
+- **`Degraded` attributed to HTTP 5xx errors pages only when 5xx errors dominate** — i.e.
+  the 5xx share of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80). An occasional
+  500 that trips Degraded is noise and must **not** page; a flood is a real problem and
+  **must** page.
+- **Tiny-sample guard:** below `HTTP_5XX_MIN_REQUEST_COUNT` (default 20) requests in the
+  window, do **not** trust the ratio — page rather than let a 2-of-3 blip read as "80%
+  failing."
+
+### Fail-safe: a blind read pages, never reads healthy
+
+If the health read (`describeEnvironmentHealth`) throws `AwsException` for an environment,
+the monitor sets **`alarm=HIGH`** for that environment (a blind read must never masquerade
+as healthy) **and** sets **`probe=DEGRADED`** (coverage gap, distinct from a measured-bad
+signal — same alarm-vs-probe split as the multi-client monitors in
+[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)).
+
+### Configurable constants (top of the class)
+
+| Constant | Default | Meaning |
+|---|---|---|
+| `HEALTH_ALARM_STATUSES` | `['Degraded','Severe']` | HealthStatus values that page |
+| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a 5xx-driven `Degraded` pages |
+| `HTTP_5XX_MIN_REQUEST_COUNT` | `20` | Below this many requests in the window, page rather than trust the ratio |
+| `HTTP_5XX_CAUSE_MARKERS` | `['5xx','http 5']` | Case-insensitive substrings identifying a 5xx-attributed EB `Cause` |
+
+### Implementation shape
+
+- `shouldPageForHealth(healthStatus, causes, applicationMetrics): [bool, ?float]` — returns
+  the page decision and the observed 5xx ratio.
+- `causesAttributeTo5xx(causes): bool` — substring-matches `HTTP_5XX_CAUSE_MARKERS` against
+  the EB `Causes` strings (case-insensitive).
+- The ratio is computed as `StatusCodes.Status5xx / RequestCount` from `ApplicationMetrics`
+  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text.
+- `describeEnvironmentHealth` is called with
+  `AttributeNames = ['HealthStatus','Status','Color','Causes','ApplicationMetrics']`.
+- Per-environment report + OneUptime payload now include the per-env `alarm` token and the
+  observed `fivexxRatio`.
+
+## How EB enhanced-health is read (durable AWS reference)
+
+Facts verified against current AWS docs; they inform why the logic above is shaped as it is.
+
+- **HealthStatus severity ladder:** `Ok → Info → Warning → Degraded → Severe` (plus the grey
+  states `Pending`/`Unknown`/`Suspended`/`NoData`). `Degraded` is the "high failure" tier
+  and is **routinely tripped by benign transients** (Auto Scaling scale-up, a mid-deploy
+  dip), which is exactly why `Degraded` alone is noisy. `Severe` = "very high failure /
+  environment effectively not serving."
+- **`ApplicationMetrics` returns RAW COUNTS, not percentages** — despite the API-reference
+  prose saying "percentage"/"per second". Fields: `Duration` (Integer seconds, usually 10),
+  `RequestCount` (Integer, **total** requests over the window),
+  `StatusCodes.{Status2xx,Status3xx,Status4xx,Status5xx}` (Integer counts). AWS's own
+  example: 2xx 3391 + 5xx 843 = RequestCount 4234. With no traffic, `RequestCount=0` and
+  `StatusCodes` may be **absent** → treat as "no data," not "zero failures." The 5xx-ratio
+  denominator is `RequestCount` (the authoritative total).
+- **Why an env is Degraded comes from `Causes`:** EB `Causes` strings literally contain e.g.
+  `"19.9 % of the requests are failing with HTTP 5xx."` A substring check reliably
+  **classifies** 5xx-driven vs. not (fail-safe: a wording change → treated as a normal
+  `Degraded` → pages). The **ratio itself must come from `ApplicationMetrics`**, never by
+  parsing the number out of `Causes`.
+- **A failed application DEPLOYMENT is NOT reliably reflected in HealthStatus** — it can read
+  `Warning`/`Degraded`, or fail before any red request-failure signal. The authoritative
+  structured deploy signal is per-instance `describeInstancesHealth` →
+  `Deployment.Status ∈ {'In Progress','Deployed','Failed'}`. (This detector was evaluated and
+  removed this iteration — see Design history.)
+- **IAM:** `describeEnvironmentHealth` / `describeInstancesHealth` / `describeEvents` are all
+  covered by the managed `AWSElasticBeanstalkReadOnly` policy and require enhanced health
+  enabled. No per-call charge.
+
+## OneUptime monitor config (external, not code)
+
+Paging depends on the OneUptime Incoming-Request monitor's **string-match** criteria on the
+POSTed body — `Contains "alarm":"HIGH"` (and, if desired, `Contains "probe":"DEGRADED"`).
+This is external OneUptime configuration, not worker2 code.
+
+## Design history — rejected direction (this session)
+
+An earlier iteration set `HEALTH_ALARM_STATUSES = ['Severe']` (dropping `Degraded` entirely)
+and **added a separate deployment-failure detector** (`describeInstancesHealth` →
+`Deployment.Status = 'Failed'`) to still catch deploy failures. That was **reverted** after a
+scope change: the team decided they **do** want to page on `Degraded` (to catch a flood of
+500s and non-500 degradations), with only the occasional-500 case suppressed via the 80%
+ratio. The deployment-failure detector was **removed** — deploy failures that surface as
+`Degraded` are now covered by the `Degraded` alarm. **Residual gap accepted for this
+iteration:** a deploy that fails fast *without ever degrading health* is not caught.
+
+## Known gaps / follow-ups (not done)
+
+- **Per-region client construction is not individually guarded.** Each
+  `new ElasticBeanstalkClient` is not in its own try/catch, so a bad region/creds aborts the
+  **whole run** via the outer backstop instead of paging just that region's envs as blind.
+  Candidate follow-up now that blind reads page.
+- **No first-party test harness exists in worker2** (all tests are vendor/).
+  `shouldPageForHealth()` and `causesAttributeTo5xx()` are pure and ideal to unit-test
+  (ratio at exactly 0.80; count 19 vs 20; `Severe` over a 5xx cause; empty causes; null
+  metrics) — deferred pending a harness.
+
+## Gotchas / known issues
+
+- **Keep the method non-fatal — never let it throw.** worker2 has no DLQ and a 3600s SQS
+  visibility timeout, so any uncaught 500 becomes a poison-message storm. The push runs
+  non-fatal and an outer `catch(\Throwable)` is the backstop.
+- **A blind health read must page, not read healthy** — an `AwsException` on
+  `describeEnvironmentHealth` sets `alarm=HIGH` + `probe=DEGRADED` for that env.
+- **`ApplicationMetrics` is raw counts, not percentages** — divide `Status5xx` by
+  `RequestCount`; `RequestCount=0`/absent `StatusCodes` means "no data," not "zero failures."
+- **Classify 5xx-attribution from `Causes` text, but take the ratio from `ApplicationMetrics`
+  — never parse the percentage out of `Causes`.**
+- **`oneuptimeUrl` is a push credential** — it arrives as a cron parameter; never log it or
+  record its value in a doc.
+
+## Change history
+- 2026-08-17 — Created. Documented the `ElasticBeanstalkHealth` alarm/paging criteria (page
+  on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless the 5xx share ≥
+  `HTTP_5XX_ALARM_RATIO` 0.80 with an `HTTP_5XX_MIN_REQUEST_COUNT` 20 tiny-sample guard;
+  `Severe` and non-5xx `Degraded` always page), the blind-read fail-safe (`alarm=HIGH` +
+  `probe=DEGRADED`), the four configurable constants, and the `shouldPageForHealth` /
+  `causesAttributeTo5xx` helper shape. Captured durable EB enhanced-health facts (severity
+  ladder; `ApplicationMetrics` returns raw counts not percentages; `Causes` classifies but
+  `ApplicationMetrics` sets the ratio; failed deploys aren't reliably in HealthStatus —
+  `describeInstancesHealth.Deployment.Status` is authoritative; `AWSElasticBeanstalkReadOnly`
+  covers the reads). Recorded the reverted `['Severe']`-only + deployment-failure-detector
+  direction and the accepted residual gap (a deploy that fails without degrading health).
+  (jcardinal)
+</content>
+</invoke>

package/knowledge/INDEX.md

@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 58 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 49 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 50 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 24 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 8 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 6 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.596",
+  "version": "1.0.597",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",