toga-ai 1.0.717 → 1.0.718

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-08-28
+updated: 2026-09-01
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -315,6 +315,52 @@ worker2 EB health monitor now applies a **persistence-delay gate** (a non-5xx `D
 must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages), which suppresses these flaps.
 See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
 
+**A sixth cause — a 4xx BURST flips the env to `Degraded`/`Severe` while the ALB stays fully
+healthy, because EB Enhanced Health counts application/ELB 4xx toward environment health.**
+Distinct from every cause above: it is not a deploy (cause #4), not worker starvation (cause #3),
+and not idle flappiness (cause #5) — it is EB reacting to *client* 4xx responses that are working
+as designed. EB Enhanced Health's `ConfigDocument` (namespace
+`aws:elasticbeanstalk:healthreporting:system`) ships with two rules **Enabled by default**:
+`Rules.Environment.Application.ApplicationRequests4xx` and `Rules.Environment.ELB.ELBRequests4xx`.
+So a short burst of client 4xx (e.g. a client polling with an expired bearer token) flips the env
+to `Degraded`/`Severe` even though the ALB never marked a target unhealthy. Verified live (AWS CLI
+`describe-events`/`describe-configuration-settings` + CloudWatch + prod `Logs.Api`, acct
+`654654170868`):
+- `api-production-1` (us-east-1) flip 2026-09-01 01:39–01:44 UTC, no cause cited ("TargetGroups
+  reduced health state"), **not** during a deploy. Throughout: `HealthyHostCount=2`,
+  `UnHealthyHostCount=0`, **0** target 5xx, **0** ELB 5xx, **0** connection errors — the ALB never
+  deregistered a target.
+- Trigger burst ~5 min earlier at 01:34 UTC: `RequestCount` 738/min, ~560 of them 4xx (~77%),
+  `TargetResponseTime` max 5.2 s. Prod `Logs.Api` for the window: **398 × HTTP 401 on
+  `GET /v2/users/me`** from `sourceIp 34.232.23.158` (the Compass/Veyer feed) on `api1.togahub.com`,
+  split across both instances (`i-0fa76fc1365cdf583`, `i-068e89c284eb26133`) — a client polling with
+  an expired/invalid bearer token.
+- `api-production-3` (eu-west-1) 2026-09-01 01:06 UTC stated it plainly: "Environment health
+  transitioned Ok→Severe: 80.7% of requests erroring with HTTP 4xx" — same root cause.
+- Other causes ruled out: `/health` confirmed served as a **static Apache file** (response carries
+  `Last-Modified`/`ETag`/`Accept-Ranges`, body `OK`, `Content-Length 3`, no PHP) → cause #3 not in
+  play; `DeploymentPolicy=Rolling`, `BatchSize=1` Fixed confirmed → cause #4 not in play. This is
+  why the prior four fixes (culminating in Rolling/batch-1, which only addressed cause #4) never
+  stopped these flips — they are unrelated to deploys.
+
+**Rule:** an EB "% 4xx" flag, or a no-cause "TargetGroup reduced health" flip, where
+`HealthyHostCount` never drops and target/ELB **5xx = 0**, is EB Enhanced Health reacting to a 4xx
+burst — **not** a backend failure. Check the ALB 4xx metric and `Logs.Api` for a client 4xx flood
+before chasing app bugs.
+
+**Fix shipped 2026-09-01 (config-only, all three prod envs).** Both 4xx rules were **disabled** on
+`api-production-1/2/3` by updating the `ConfigDocument` option (namespace
+`aws:elasticbeanstalk:healthreporting:system`) via `aws elasticbeanstalk update-environment
+--option-settings`, setting `ApplicationRequests4xx.Enabled=false` and
+`ELBRequests4xx.Enabled=false` while preserving every other rule and the `CloudWatchMetrics` block
+(**5xx and latency rules stay ON, so real backend failures still page**). It is a config-only update
+(~1–2 min "Updating", no instance replacement) and is **reversible** by flipping the flags back to
+`true`. Verified after apply: all three Ready/Green/Ok with both 4xx rules `Enabled=false`. This is
+the **server-side complement** to the worker2 monitor's non-5xx persistence-delay gate
+(`nonFivexxPersistMinutes`, cause #5) — one de-sensitizes the alert, this stops EB from generating
+the flip at all. (The dev ran the three `update-environment` commands; the harness auto-mode
+classifier correctly blocks the assistant from mutating prod infra or self-granting that permission.)
+
 **Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504 to find
 the cause:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout
 (worker saturation); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing
@@ -400,6 +446,20 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-09-01 — Recorded a **sixth EB-`Degraded` cause — a client 4xx BURST**: EB Enhanced Health
+  counts application/ELB 4xx toward env health via two default-Enabled `ConfigDocument` rules
+  (`ApplicationRequests4xx`, `ELBRequests4xx`, namespace
+  `aws:elasticbeanstalk:healthreporting:system`), so a short 4xx spike flips the env to
+  `Degraded`/`Severe` while the ALB stays fully healthy (`HealthyHostCount`=2, 0 target/ELB 5xx).
+  Verified on `api-production-1` (2026-09-01 01:39 UTC flip, not a deploy) from a 01:34 UTC burst of
+  398×HTTP 401 on `GET /v2/users/me` from the Compass/Veyer feed (`34.232.23.158`) polling with an
+  expired token; `api-production-3` stated it as "80.7% of requests erroring with HTTP 4xx". Ruled
+  out causes #3 (static-file `/health` confirmed) and #4 (Rolling/batch-1 confirmed) — which is why
+  the prior deploy-focused fixes never stopped it. Triage rule: no-cause/%-4xx flip + steady
+  `HealthyHostCount` + 0 5xx = a 4xx burst, check the ALB 4xx metric and `Logs.Api` first. Fixed
+  config-only on all three prod envs — disabled both 4xx rules via `update-environment
+  --option-settings` (5xx/latency rules kept ON), reversible; the server-side complement to the
+  worker2 monitor's non-5xx persistence-delay gate. (jcardinal)
 - 2026-08-29 — Recorded a **fifth EB-`Degraded` cause**: a brief self-healing "no-cause reduced
   health" flip on a low-traffic *idle* 2-instance env (e.g. `api-production-3`, eu-west-1) is EB
   Enhanced Health flappiness (a single missed ALB `/health` check), distinct from the busy-env

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.717",
+  "version": "1.0.718",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",