toga-ai 1.0.650 → 1.0.651

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -18,7 +18,7 @@
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | `_Component_Aws_Workloads` is the **standard, and only sanctioned, way any new worker2 cron obtains AWS access** — for any account, any region, any AWS SDK clie | worker2/Component/Aws/Workloads/Workloads.php, worker2/Config/production.ini, worker2/Worker/Infrastructure/CloudWatch.php, worker2/Worker/Monitor/Fleet.php, library/app/worker.php, library/app/cloud.php |
-| [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's **enhanced-health** status | worker2/Worker/Infrastructure/CloudWatch.php |
+| [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads each Elastic Beanstalk (EB) environment's health and reports it to a | worker2/Worker/Infrastructure/CloudWatch.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | `_Worker_Infrastructure_Errors::Escalate` (renamed from `SyncWithClickup`) is the sole owner of **escalation, de-escalation, ClickUp ticketing, reminders, busin | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Notification/Email.php, worker2/Worker/Notification/EmailTemplate.php, worker2/Worker/Client/True.php, worker2/Worker/Clickup/ErrorTask.php, worker2/Worker/Clickup.php, worker2/Controller/Index.php, worker2/Config/production.ini, _underscore/Model/Core/Logs/Issue.php, dbchanges2/Core/2026-07-30a - Error escalation cron job.sql, dbchanges2/Logs/2026-08-03a - Issue clickupPriority.sql, dbchanges2/Core/2026-08-04a - Error neglect digest cron job.sql |
 | [Error-Issue Auto-Resolution & Reopen (frequency-decay lifecycle)](features/error-issue-auto-resolution.md) | The error system could escalate and de-escalate an Issue's *urgency* but had no concept of an Issue being **resolved**. | worker2/Worker/Infrastructure/Errors.php, worker2/Worker/Clickup/ErrorTask.php, _underscore/Model/Core/Logs/Issue.php, tools/mvc/errors/get.php, tools/mvc/errors/issue/get.php, dbchanges2/Logs/2026-08-05a - Issue status baseline and auto-resolution.sql |

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-19
+updated: 2026-08-25
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -19,209 +19,253 @@ related:
 ## Summary
 
 `_Worker_Infrastructure_CloudWatch::ElasticBeanstalkHealth()` is a worker2 cron that reads
-each Elastic Beanstalk (EB) environment's **enhanced-health** status and reports it to a
-OneUptime **Incoming Request** monitor, which pages the team. It is a **"dumb reporter,
-smart monitor"** heartbeat (the same Pattern-B contract as
+each Elastic Beanstalk (EB) environment's health and reports it to a OneUptime **Incoming
+Request** monitor, which pages the team. It is a **"dumb reporter, smart monitor"**
+heartbeat (the same Pattern-B contract as
 [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)): the worker decides
 pass/fail and POSTs decided **string tokens** (`alarm=HIGH/OK`, `probe=DEGRADED/OK`) that
 OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
 
-The load-bearing design decision this doc records is the **paging (alarm) criteria**: page
-on EB `Degraded`/`Severe`, but **suppress the common "an occasional HTTP 500 tripped
-Degraded/Severe" false alarm** unless 5xx errors actually dominate the request mix (by
-share) or reach a real absolute volume. The 5xx gate applies to **both** `Degraded` and
-`Severe`, and only when EB attributes the problem **solely** to 5xx — a mixed cause always
-pages.
+The load-bearing design decision this doc records is the **paging (alarm) criteria**, and
+the key **2026-08-25** change: the 5xx page is now driven by a **CloudWatch windowed
+request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. A non-5xx
+`Degraded`/`Severe` still pages immediately off EB health; but whether a *5xx* problem pages
+is decided by the true per-environment 5xx **rate over a trailing window** (default 5 min),
+not by EB's ~10-second `ApplicationMetrics` snapshot.
+
+**Why the change:** the old gate read EB `describeEnvironmentHealth`'s `ApplicationMetrics`,
+a ~10-**second** window. At low traffic a single 500 in a near-empty bucket reads as 100%
+5xx and false-pages. This actually fired: worker-production paged "100% requests failing"
+while the healthd log showed **16 500s / 516 req = 3.1%** over 17 minutes — the 100% was one
+500 alone in a 1-request 10s bucket. The windowed CloudWatch gate averages over minutes and
+across instances, so a lone 500 can no longer dominate.
 
 **Critical behavior:** the method is **non-fatal and never throws** — it always returns a
 string. worker2 has **no DLQ and a 3600s SQS visibility timeout**, so any uncaught 500
 becomes a poison-message storm; an outer `catch(\Throwable)` is the backstop and the push
-itself runs non-fatal.
+itself runs non-fatal. CloudWatch read failures are caught **internally** and return `null`.
 
 ## Key files / entry points
 
 - `worker2/Worker/Infrastructure/CloudWatch.php` — `_Worker_Infrastructure_CloudWatch`,
   action `ElasticBeanstalkHealth`, dispatched via
   `_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
-  environments, oneuptimeUrl})`.
+  environments, oneuptimeUrl, …thresholds})`.
+- New private `fetchWindowedRequestCounts()` reads the CloudWatch metrics (below).
 - AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
   a read-only role per account, one assume reused across regions) — see
   [Cross-account AWS access](./cross-account-aws-access.md), which also documents the
-  `environments` region→env-names map parameter shape.
+  `environments` region→env-names map parameter shape. The **same assumed credentials** build
+  both the `ElasticBeanstalkClient` and the new `CloudWatchClient`.
 - `oneuptimeUrl` arrives as a **cron parameter and is a push credential** — never log it and
   never record its value in a doc (see gotchas).
 
 ## Alarm / paging criteria (the decision logic)
 
-Per environment, `alarm=HIGH` (page) when the EB `HealthStatus` is `Degraded` **or**
-`Severe`. `Suspended` never pages.
+Two independent paths now feed `alarm=HIGH`:
 
-The **5xx gate applies to both `Degraded` and `Severe`.** A non-5xx or mixed-cause
-`Degraded`/`Severe` — and any env with no stated cause — **always pages**; the gate only
-ever suppresses an env EB attributes **solely** to 5xx.
+1. **EB-health path (non-5xx).** EB `HealthStatus` `Degraded` or `Severe` that is **not**
+   attributable solely to 5xx (latency, instances down, a failed deploy, a mixed cause, or
+   no stated cause) **pages immediately** — unchanged, still driven off EB health.
+   `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx event
+   apart from everything else.
+2. **CloudWatch windowed 5xx-rate path.** For a solely-5xx `Degraded`/`Severe`, the page is
+   decided **not** by EB but by the true windowed rate from CloudWatch. Given the window's
+   `requests` (ApplicationRequestsTotal) and `fivexx` (ApplicationRequests5xx), page when:
+   - `fivexx >= min5xxAbsolute`  **(absolute-volume floor)** — pages regardless of share, or
+   - `requests >= minRequests` **AND** `sharePercent >= min5xxPercent` **(share gate)** — the
+     `minRequests` floor keeps a near-idle window off the share gate so a lone 500 can't page.
 
-- **Non-5xx or mixed-cause `Degraded`/`Severe`** (latency, instances down, a failed deploy,
-  or a 5xx trickle *alongside* a real non-5xx failure) **always pages.**
-- **Solely-5xx `Degraded`/`Severe` pages only on a genuine flood** — either:
-  - the **absolute** 5xx count in the window is at/above `HTTP_5XX_ALARM_ABSOLUTE_COUNT`
-    (default 20), **or**
-  - the **5xx share** of requests is at/above `HTTP_5XX_ALARM_RATIO` (default 0.80).
-- **Zero requests in the window → no page** (nothing observed; also avoids divide-by-zero).
+Because the 5xx decision reads CloudWatch directly, a real 5xx flood pages **even if EB
+hasn't escalated yet**, and a benign low-traffic 5xx blip does not page even though EB flags
+`Degraded`.
 
-**Why the gate now covers `Severe` too:** EB escalates an environment to `Severe` precisely
-when the 5xx share is high, so real 5xx floods reach `Severe` *first* and never touch the
-gated `Degraded` branch. When the gate applied only to `Degraded`, the 80% threshold was
-effectively dead — a real 50%-5xx `Severe` event paged despite the setting. Gating both
-tiers makes the threshold actually govern.
+### Cron parameters (replaces the former class constants)
 
-**Why "solely" 5xx, not "any" 5xx cause:** a mixed-cause env (a 5xx trickle plus a real
-non-5xx failure such as a failed deploy) must not be silenced by a sub-threshold ratio. The
-ratio gate only applies when **every** stated cause references 5xx (and there is at least
-one); anything else pages.
+The four thresholds are **optional cron parameters** spread as **named arguments** from
+`CronJobs.parameters`, with method-signature defaults:
 
-**Why an absolute floor in addition to the ratio:** the ratio alone is magnitude-blind —
-60k failing out of 100k reads as 60% and would be suppressed, though it is plainly a flood.
-`HTTP_5XX_ALARM_ABSOLUTE_COUNT` pages such an env regardless of share; tune it to the busiest
-monitored environment's request volume.
+| Parameter | Default | Meaning |
+|---|---|---|
+| `windowMinutes` | `5` | Trailing window (minutes) over which the CloudWatch rate is summed |
+| `minRequests` | `100` | Minimum windowed request count before the **share** gate applies |
+| `min5xxPercent` | `20.0` | 5xx share (percent) at/above which a solely-5xx env pages, once `minRequests` is met |
+| `min5xxAbsolute` | `20` | Absolute windowed 5xx count at/above which a solely-5xx env pages regardless of share |
 
-### Fail-safe: a blind read pages, never reads healthy
+Removed this iteration: `shouldPageForHealth()` and the old
+`HTTP_5XX_ALARM_RATIO` / `HTTP_5XX_ALARM_ABSOLUTE_COUNT` constants (superseded by the
+windowed gate and the four parameters).
 
-If the health read (`describeEnvironmentHealth`) throws `AwsException` for an environment,
-the monitor sets **`alarm=HIGH`** for that environment (a blind read must never masquerade
-as healthy) **and** sets **`probe=DEGRADED`** (coverage gap, distinct from a measured-bad
-signal — same alarm-vs-probe split as the multi-client monitors in
-[OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md)).
+### Per-monitor threshold tuning (one cron watches many environments)
 
-### Configurable constants (top of the class)
+A single `ElasticBeanstalkHealth` cron watches environments of **very different scale**, so
+thresholds are tuned per monitor. Measured 3h baselines: api-production-1 ~922–1737 req/5min,
+apiproxy ~400–1650, but eu-west-1 envs only ~40–90 req/5min; 5xx was 0 across all.
 
-| Constant | Default | Meaning |
-|---|---|---|
-| `HEALTH_ALARM_STATUSES` | `['Degraded','Severe']` | HealthStatus values that page |
-| `HTTP_5XX_ALARM_RATIO` | `0.80` | 5xx share (0.0–1.0) at/above which a solely-5xx `Degraded`/`Severe` pages |
-| `HTTP_5XX_ALARM_ABSOLUTE_COUNT` | `20` | Absolute 5xx count in the window at/above which a solely-5xx `Degraded`/`Severe` pages regardless of share |
-| `HTTP_5XX_CAUSE_MARKERS` | `['5xx','http 5']` | Case-insensitive substrings identifying a 5xx-attributed EB `Cause` |
-
-### Implementation shape
-
-- `shouldPageForHealth(healthStatus, causes, applicationMetrics): [bool, ?float]` — returns
-  the page decision and the observed 5xx ratio.
-- `causesAttributeSolelyTo5xx(causes): bool` — true only when **every** EB `Cause` references
-  5xx **and there is at least one** cause. Replaces the former `causesAttributeTo5xx()`, which
-  returned true if *any* cause mentioned 5xx and so let a mixed-cause env be gated.
-- `causeIsFivexx(cause): bool` — helper that substring-matches `HTTP_5XX_CAUSE_MARKERS`
-  against a single EB `Cause` string (case-insensitive); `causesAttributeSolelyTo5xx` requires
-  it to hold for all causes.
-- The ratio is computed as `StatusCodes.Status5xx / RequestCount` from `ApplicationMetrics`
-  (raw counts — see below), **not** by parsing the percentage out of the `Causes` text. The
-  absolute-count floor reads `StatusCodes.Status5xx` directly.
-- `describeEnvironmentHealth` is called with
-  `AttributeNames = ['HealthStatus','Status','Color','Causes','ApplicationMetrics']`.
-- Per-environment report + OneUptime payload now include the per-env `alarm` token and the
-  observed `fivexxRatio`.
-
-## How EB enhanced-health is read (durable AWS reference)
-
-Facts verified against current AWS docs; they inform why the logic above is shaped as it is.
+- **Worker + worker 1.0 monitors** keep the **method defaults** (`100 / 20% / 20`) — the low
+  absolute floor suits ~150 req/5min traffic.
+- **API and API-Proxy monitors** (which each also include the low-traffic eu-west-1 env) are
+  overridden to **`windowMinutes=5, minRequests=30, min5xxPercent=20, min5xxAbsolute=100`** —
+  the lower request floor keeps the small env on the share gate, while the higher absolute
+  floor stops the busy envs paging on minor blips (20 scattered 500s on ~1300 req = <2%). The
+  override is applied by the dbchanges2 migration below.
+
+### Fail-safe: an unmeasurable solely-5xx event still pages
+
+The windowed metrics are **per-instance only** (see below), and some envs never publish them
+(the 1.0 worker env `agilant-worker` publishes only `EnvironmentHealth`). For such an env the
+gate has **no data** and returns **zeros** (not `null`). To avoid silently missing a real
+flood, when EB reports a **solely-5xx `Degraded`/`Severe`** but the window is **empty
+(unmeasurable)**, the monitor **pages** with gate token **`health-5xx-unmeasured`** rather
+than trusting the empty read.
+
+A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`** (coverage
+gap), distinct from a measured-bad `alarm=HIGH` — same alarm-vs-probe split as the
+multi-client monitors in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md).
+
+## Reading the windowed 5xx rate from CloudWatch (durable AWS reference)
+
+- **EB application-request metrics are PER-INSTANCE ONLY.** `ApplicationRequestsTotal`,
+  `ApplicationRequests5xx` (and `…2xx/3xx/4xx`) are published under dimensions
+  `{EnvironmentName, InstanceId}` in namespace `AWS/ElasticBeanstalk`. **There is no
+  environment-only rollup.** You must sum across instances yourself with metric-math over a
+  search expression:
+
+  ```
+  SUM(SEARCH('{AWS/ElasticBeanstalk,EnvironmentName,InstanceId} MetricName="ApplicationRequestsTotal" EnvironmentName="<env>"','Sum',<periodSeconds>))
+  ```
+
+  The `SEARCH` auto-discovers every instance's series and `SUM` folds them into one, so the
+  window total is correct as instances come and go. Query total and 5xx as two expressions
+  over `windowMinutes`.
+- **Not every environment publishes these metrics.** worker-production, api-production-*, and
+  apiproxy-production-* all publish them; the **1.0 worker env `agilant-worker` publishes
+  ONLY `EnvironmentHealth`** (no app-request metrics) → the gate reads zeros there → the
+  `health-5xx-unmeasured` fail-safe above covers it.
+- **IAM already sufficient.** The `WorkloadsRuntime` role grants `cloudwatch:*` in **both**
+  account `654654170868` (worker/api) and `502614707982` (legacy worker), so no IAM change
+  was needed for the CloudWatch reads.
+
+### EB enhanced-health facts (still relevant — the EB-health path)
 
 - **HealthStatus severity ladder:** `Ok → Info → Warning → Degraded → Severe` (plus the grey
-  states `Pending`/`Unknown`/`Suspended`/`NoData`). `Degraded` is the "high failure" tier
-  and is **routinely tripped by benign transients** (Auto Scaling scale-up, a mid-deploy
-  dip), which is exactly why `Degraded` alone is noisy. `Severe` = "very high failure /
-  environment effectively not serving."
-- **`ApplicationMetrics` returns RAW COUNTS, not percentages** — despite the API-reference
-  prose saying "percentage"/"per second". Fields: `Duration` (Integer seconds, usually 10),
-  `RequestCount` (Integer, **total** requests over the window),
-  `StatusCodes.{Status2xx,Status3xx,Status4xx,Status5xx}` (Integer counts). AWS's own
-  example: 2xx 3391 + 5xx 843 = RequestCount 4234. With no traffic, `RequestCount=0` and
-  `StatusCodes` may be **absent** → treat as "no data," not "zero failures." The 5xx-ratio
-  denominator is `RequestCount` (the authoritative total).
+  states `Pending`/`Unknown`/`Suspended`/`NoData`). `Degraded` is routinely tripped by benign
+  transients (Auto Scaling scale-up, a mid-deploy dip), which is why a solely-5xx `Degraded`
+  is now gated on the CloudWatch rate rather than paged outright.
 - **Why an env is Degraded comes from `Causes`:** EB `Causes` strings literally contain e.g.
   `"19.9 % of the requests are failing with HTTP 5xx."` A substring check reliably
-  **classifies** 5xx-driven vs. not (fail-safe: a wording change → treated as a normal
-  `Degraded` → pages). The **ratio itself must come from `ApplicationMetrics`**, never by
-  parsing the number out of `Causes`.
-- **A failed application DEPLOYMENT is NOT reliably reflected in HealthStatus** — it can read
-  `Warning`/`Degraded`, or fail before any red request-failure signal. The authoritative
-  structured deploy signal is per-instance `describeInstancesHealth` →
-  `Deployment.Status ∈ {'In Progress','Deployed','Failed'}`. (This detector was evaluated and
-  removed this iteration — see Design history.)
-- **IAM:** `describeEnvironmentHealth` / `describeInstancesHealth` / `describeEvents` are all
-  covered by the managed `AWSElasticBeanstalkReadOnly` policy and require enhanced health
-  enabled. No per-call charge.
+  **classifies** solely-5xx vs. not; a wording change fails safe (treated as non-5xx →
+  pages). `describeEnvironmentHealth` is called with `AttributeNames` including
+  `HealthStatus`, `Status`, `Color`, `Causes`.
+- **A failed application DEPLOYMENT is NOT reliably reflected in HealthStatus** — the
+  authoritative structured deploy signal is per-instance `describeInstancesHealth` →
+  `Deployment.Status`. (A deploy that fails without ever degrading health remains an accepted
+  residual gap.)
+- **IAM:** `describeEnvironmentHealth` is covered by the managed `AWSElasticBeanstalkReadOnly`
+  policy and requires enhanced health enabled.
+
+## Deploy order (mandatory) — code BEFORE the migration
+
+worker2 code **must** deploy **before** the dbchanges2 threshold migration runs.
+
+- new-code + old-params → **fine** (the params are optional with defaults).
+- **old-code + new-params → FATAL.** The dispatcher spreads `CronJobs.parameters` as **named
+  arguments**; an old method signature that lacks `windowMinutes`/`minRequests`/… throws
+  **"Unknown named parameter"** and **every run fails** → the monitor goes dark → OneUptime
+  raises an offline incident.
+
+Correct order: **1) deploy worker2 → 2) apply the SQL → 3) add the OneUptime criterion.**
+
+## dbchanges2 migration (the API/API-Proxy override)
+
+`dbchanges2/Core/2026-08-25a - ApiApiproxy Beanstalk Health 5xx Thresholds.sql` applies the
+API/API-Proxy override with a JSON_SET that **adds only the four keys**, preserving
+`awsAccountId`/`environments`/`oneuptimeUrl`:
+
+```sql
+UPDATE CronJobs
+SET parameters = JSON_SET(parameters,
+    '$.windowMinutes', 5, '$.minRequests', 30,
+    '$.min5xxPercent', 20, '$.min5xxAbsolute', 100)
+WHERE action = 'Infrastructure/CloudWatch/ElasticBeanstalkHealth'
+  AND name IN (<the two API monitor names>);
+```
+
+`Core.CronJobs.parameters` is confirmed a native `JSON` column, so `JSON_SET` merges rather
+than overwrites.
 
 ## OneUptime monitor config (external, not code)
 
 Paging depends on the OneUptime Incoming-Request monitor's **string-match** criteria on the
-POSTed body — `Contains "alarm":"HIGH"` (and, if desired, `Contains "probe":"DEGRADED"`).
-This is external OneUptime configuration, not worker2 code.
-
-## Design history — rejected direction (this session)
+POSTed body — `Contains "alarm":"HIGH"` pages.
 
-An earlier iteration set `HEALTH_ALARM_STATUSES = ['Severe']` (dropping `Degraded` entirely)
-and **added a separate deployment-failure detector** (`describeInstancesHealth` →
-`Deployment.Status = 'Failed'`) to still catch deploy failures. That was **reverted** after a
-scope change: the team decided they **do** want to page on `Degraded` (to catch a flood of
-500s and non-500 degradations), with only the occasional-500 case suppressed via the 80%
-ratio. The deployment-failure detector was **removed** — deploy failures that surface as
-`Degraded` are now covered by the `Degraded` alarm. **Residual gap accepted for this
-iteration:** a deploy that fails fast *without ever degrading health* is not caught.
+**Known alerting gap (recommended fix):** the current monitors match `"status":"error"`,
+`"alarm":"HIGH"`, and heartbeat-gap rules, but **nothing matches `"probe":"DEGRADED"`**. So a
+CloudWatch-metrics **read failure while EB reads Ok** (body = `alarm:OK, probe:DEGRADED,
+status:reporting`) is **invisible to alerting**. Add a criterion matching
+`Contains "probe":"DEGRADED"` → a **Degraded-severity, auto-resolving** incident, placed
+**before** the "online" recovery rule (which also matches `alarm:OK`). The code already emits
+the `probe` token; this is a OneUptime config change the developer applies.
 
 ## Known gaps / follow-ups (not done)
 
-- **Per-region client construction is not individually guarded.** Each
-  `new ElasticBeanstalkClient` is not in its own try/catch, so a bad region/creds aborts the
-  **whole run** via the outer backstop instead of paging just that region's envs as blind.
-  Candidate follow-up now that blind reads page.
-- **No first-party test harness exists in worker2** (all tests are vendor/).
-  `shouldPageForHealth()` and `causesAttributeSolelyTo5xx()` are pure and ideal to unit-test
-  (ratio at exactly 0.80; absolute count 19 vs 20; `Severe` over a solely-5xx cause; a
-  mixed 5xx + non-5xx cause; empty causes; null metrics; zero requests) — deferred pending a
-  harness.
+- **No first-party test harness in worker2** (all tests are vendor/). The gate decision and
+  `causesAttributeSolelyTo5xx()` are pure and ideal to unit-test (share at exactly
+  `min5xxPercent`; absolute at `min5xxAbsolute-1` vs `min5xxAbsolute`; `requests` just below
+  `minRequests`; empty/zero window; mixed 5xx + non-5xx cause) — deferred pending a harness.
+- **Deploy fails without degrading health** — still not caught (accepted residual).
 
 ## Gotchas / known issues
 
+- **Deploy worker2 code BEFORE the threshold migration** — old-code + new-params is fatal
+  ("Unknown named parameter" on every run → monitor dark). See Deploy order.
 - **Keep the method non-fatal — never let it throw.** worker2 has no DLQ and a 3600s SQS
-  visibility timeout, so any uncaught 500 becomes a poison-message storm. The push runs
-  non-fatal and an outer `catch(\Throwable)` is the backstop.
-- **A blind health read must page, not read healthy** — an `AwsException` on
-  `describeEnvironmentHealth` sets `alarm=HIGH` + `probe=DEGRADED` for that env.
-- **`ApplicationMetrics` is raw counts, not percentages** — divide `Status5xx` by
-  `RequestCount`; `RequestCount=0`/absent `StatusCodes` means "no data," not "zero failures."
-- **Classify 5xx-attribution from `Causes` text, but take the ratio from `ApplicationMetrics`
-  — never parse the percentage out of `Causes`.**
+  visibility timeout; an outer `catch(\Throwable)` is the backstop and CloudWatch failures are
+  caught internally (return `null`).
+- **EB app-request metrics are per-instance only** — there is no env rollup; you must
+  `SUM(SEARCH(...))` across instances or you undercount.
+- **An empty window ≠ zero failures.** An env with no published app-request metrics (e.g. 1.0
+  `agilant-worker`) reads zeros; a solely-5xx EB event over an empty window pages via the
+  `health-5xx-unmeasured` fail-safe rather than reading healthy.
+- **`minRequests` floor guards the near-idle edge** the old code couldn't — the share gate is
+  suppressed until the window has real volume, so a lone 500 in a quiet window no longer pages.
+- **OneUptime has no `probe:DEGRADED` criterion** — a metrics-read failure while EB is Ok is
+  currently unalerted until that criterion is added.
 - **`oneuptimeUrl` is a push credential** — it arrives as a cron parameter; never log it or
   record its value in a doc.
-- **Near-idle ratio edge (residual, accepted).** With the tiny-sample guard removed, a
-  near-idle env whose window holds a tiny all-error sample (e.g. its only request being a
-  500 = 100%) still trips the ratio and pages. Optional future guard: require a minimum 5xx
-  count before the ratio applies (the absolute floor gates high volume, not this low-volume
-  edge).
 
 ## Change history
+- 2026-08-25 — Replaced the EB `ApplicationMetrics` 5xx gate (a ~10s window that false-paged
+  "100% failing" on a lone 500 in a near-empty bucket — real worker-production incident: 16
+  500s / 516 req = 3.1% over 17 min) with a **CloudWatch windowed request-rate gate**. Reads
+  true per-env `ApplicationRequestsTotal`/`ApplicationRequests5xx` over a trailing window
+  (default 5 min) via `SUM(SEARCH('{AWS/ElasticBeanstalk,EnvironmentName,InstanceId}…','Sum',
+  period))` (these metrics are per-instance only — no env rollup). The 5xx page is now
+  **decoupled from EB `HealthStatus`**: pages when `5xx >= min5xxAbsolute` OR `requests >=
+  minRequests AND share >= min5xxPercent`; a non-5xx `Degraded`/`Severe` still pages
+  immediately. Added four optional cron params (`windowMinutes=5, minRequests=100,
+  min5xxPercent=20.0, min5xxAbsolute=20`) spread as named arguments; removed
+  `shouldPageForHealth()` and the `HTTP_5XX_ALARM_RATIO`/`HTTP_5XX_ALARM_ABSOLUTE_COUNT`
+  constants. Added a `health-5xx-unmeasured` fail-safe (solely-5xx over an empty/unpublished
+  window pages rather than reading healthy). Per-monitor tuning: worker + worker-1.0 keep
+  defaults; API/API-Proxy overridden to `30 / 20% / 100` (dbchanges2
+  `2026-08-25a - ApiApiproxy Beanstalk Health 5xx Thresholds.sql`, JSON_SET on the native
+  `CronJobs.parameters` JSON column). Recorded the mandatory deploy order (code before the
+  migration — old-code+new-params throws "Unknown named parameter") and the OneUptime gap
+  (no `probe:DEGRADED` criterion, so a metrics-read failure while EB reads Ok is unalerted).
+  WorkloadsRuntime already grants `cloudwatch:*` in accounts 654654170868 and 502614707982 —
+  no IAM change. (jcardinal)
 - 2026-08-19 — Overhauled the 5xx alarm gating in `shouldPageForHealth()`. The 80% share
-  gate now applies to **both** `Degraded` and `Severe` (was `Degraded`-only, which left the
-  threshold dead because EB escalates real 5xx floods straight to `Severe` — a 50%-5xx
-  `Severe` had been paging despite the 0.80 setting). This Severe-gating trade-off was made
-  deliberately with an independent architecture second opinion on record (cto returned
-  DISAGREE-WITH-ALTERNATIVE; developer accepted). Narrowed 5xx classification: replaced
-  `causesAttributeTo5xx()` (any cause mentions 5xx) with `causesAttributeSolelyTo5xx()` (every
-  cause references 5xx, ≥1) + a `causeIsFivexx()` helper, so a mixed cause (5xx trickle beside
-  a real non-5xx failure) always pages — a security-review gap the Severe change would have
-  widened. Added `HTTP_5XX_ALARM_ABSOLUTE_COUNT` (20): a solely-5xx env pages regardless of
-  share once the absolute 5xx count reaches the floor (closes the ratio's magnitude-blindness,
-  e.g. 60k of 100k). Removed `HTTP_5XX_MIN_REQUEST_COUNT` (was 20) and its tiny-sample
-  page-anyway guard — a small number of 5xx is now judged purely on the ratio; below the
-  absolute floor, page only if share ≥ 0.80, and zero requests → no page. Accepted residual:
-  a near-idle env with a tiny all-error sample still trips the ratio. (jcardinal)
+  gate now applied to **both** `Degraded` and `Severe` (was `Degraded`-only, which left the
+  threshold dead because EB escalates real 5xx floods straight to `Severe`). Narrowed 5xx
+  classification: replaced `causesAttributeTo5xx()` (any cause mentions 5xx) with
+  `causesAttributeSolelyTo5xx()` (every cause references 5xx, ≥1). Added
+  `HTTP_5XX_ALARM_ABSOLUTE_COUNT` (20). Removed `HTTP_5XX_MIN_REQUEST_COUNT` tiny-sample
+  guard. (jcardinal)
 - 2026-08-17 — Created. Documented the `ElasticBeanstalkHealth` alarm/paging criteria (page
-  on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless the 5xx share ≥
-  `HTTP_5XX_ALARM_RATIO` 0.80 with an `HTTP_5XX_MIN_REQUEST_COUNT` 20 tiny-sample guard;
-  `Severe` and non-5xx `Degraded` always page), the blind-read fail-safe (`alarm=HIGH` +
-  `probe=DEGRADED`), the four configurable constants, and the `shouldPageForHealth` /
-  `causesAttributeTo5xx` helper shape. Captured durable EB enhanced-health facts (severity
-  ladder; `ApplicationMetrics` returns raw counts not percentages; `Causes` classifies but
-  `ApplicationMetrics` sets the ratio; failed deploys aren't reliably in HealthStatus —
-  `describeInstancesHealth.Deployment.Status` is authoritative; `AWSElasticBeanstalkReadOnly`
-  covers the reads). Recorded the reverted `['Severe']`-only + deployment-failure-detector
-  direction and the accepted residual gap (a deploy that fails without degrading health).
-  (jcardinal)
+  on `Degraded`/`Severe`; suppress a 5xx-driven `Degraded` unless share ≥ ratio), the
+  blind-read fail-safe (`alarm=HIGH` + `probe=DEGRADED`), and durable EB enhanced-health
+  facts (severity ladder; `ApplicationMetrics` raw counts; `Causes` classifies; failed
+  deploys aren't reliably in HealthStatus). (jcardinal)
+</content>
+</invoke>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.650",
+  "version": "1.0.651",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",