toga-ai 1.0.869 → 1.0.870

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-09-16
+updated: 2026-09-22
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -134,7 +134,9 @@ api2's EB target group health-checks **`/health`** (interval 15 s, timeout 5 s,
 
 **A fourth cause (deploy-triggered) — a burst of ELB `502`s when both instances restart at once.** Confirmed on `api-production-1` (2026-08-28 23:51 UTC deploy: **43 × 502 in one minute**, the only ELB 5xx in a 7-hour window). Under the old `AllAtOnce`/100% policy, every deploy to a 2-instance env restarts Apache/PHP-FPM on **both** instances simultaneously, so in-flight requests have their connections dropped mid-response → a burst of **ELB-generated HTTP 502**. EB flags "X% of requests to the ELB are failing with HTTP 5xx" `Degraded` ~3–4 min later and recovers in ~5 min. Fingerprint vs. a real app failure: it is **502** (dropped backend connection), **not 503** (no healthy backend) or **504** (backend timeout); **`HTTPCode_Target_5XX_Count` (app 5xx) = 0**; **`HealthyHostCount` never drops below 2**; and there is **NOTHING in `Logs.Api`** — api2 logs a request only *after* the response, and a 502 never completes one. Diagnosis path: CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`, plus EB `describe-events` (a version change) and CodePipeline `list-pipeline-executions` — because a push to `_production` auto-deploys all three prod tiers, "we didn't deploy" can be false (see [CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md)). **Rule:** an EB "% 5xx" `Degraded` with an **empty `Logs.Api` and 0 target-5xx** right after a deploy is a deploy-restart 502 artifact, not an application bug. **Fixed 2026-08-28:** all three prod envs moved to **Rolling / batch-size 1** — config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
 
-**A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.** Distinct from the third cause (the *busy* env under Compass ASN saturation). This is a single missed ALB `/health` check — a network/agent transient. Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2, **0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no server-side app fix applies** — do not chase it. Mitigation is alert de-sensitization: the worker2 EB health monitor applies a **persistence-delay gate** (a non-5xx `Degraded`/`Severe` must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages). See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
+**A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.** Distinct from the third cause (the *busy* env under Compass ASN saturation). This is a single missed ALB `/health` check — a network/agent transient. Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed; **recurred 2026-09-22** — three ~5-min Ok→Warning→Ok cycles, no cause cited): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2, **0** target/ELB 5xx, sub-second latency (or NO DATA on a fully idle env), idle CPU, near-zero traffic, empty ALB `RequestCount`; ALB target health already reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no server-side app fix applies** — do not chase it.
+
+Two paging facts confirmed 2026-09-22 on `api-production-3`: (1) **EB native notifications are OFF** (`aws:elasticbeanstalk:sns:topics` endpoint/topic all null) — EB itself never emails on a transition; the **only** paging path is the worker2 → OneUptime cron `Infrastructure/CloudWatch/ElasticBeanstalkHealth`. (2) That monitor pages only on `Degraded`/`Severe`, so a **`Warning`-only flip is below its alarm threshold and never pages** — the 2026-09-22 Warning flaps did not page. Mitigation is alert de-sensitization: the monitor's non-5xx persistence gate (`nonFivexxPersistMinutes`) was **raised from the default 5 to 8** on the API + API-Proxy monitors (dbchanges2 `2026-09-22d`), clearing the ~5-min idle flaps with margin; the 2026-09-01 4xx-rule disable (#6) and Rolling/batch-1 (#4) remain in place. See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
 
 **A sixth cause — a 4xx BURST flips the env to `Degraded`/`Severe` while the ALB stays fully healthy, because EB Enhanced Health counts application/ELB 4xx toward environment health.** Not a deploy (#4), not worker starvation (#3), not idle flappiness (#5) — EB reacting to *client* 4xx responses working as designed. EB Enhanced Health's `ConfigDocument` (namespace `aws:elasticbeanstalk:healthreporting:system`) ships with two rules **Enabled by default**: `Rules.Environment.Application.ApplicationRequests4xx` and `Rules.Environment.ELB.ELBRequests4xx`. So a short burst of client 4xx (e.g. a client polling with an expired bearer token) flips the env to `Degraded`/`Severe` even though the ALB never marked a target unhealthy. Verified live (AWS CLI + CloudWatch + prod `Logs.Api`, acct `654654170868`):
 - `api-production-1` (us-east-1) flip 2026-09-01 01:39–01:44 UTC, no cause cited, **not** during a deploy. Throughout: `HealthyHostCount=2`, `UnHealthyHostCount=0`, **0** target 5xx, **0** ELB 5xx, **0** connection errors.

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-22
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -65,8 +65,8 @@ Removed this iteration: `shouldPageForHealth()` and the old `HTTP_5XX_ALARM_RATI
 
 A single `ElasticBeanstalkHealth` cron watches environments of **very different scale**, so thresholds are tuned per monitor. Measured 3h baselines: api-production-1 ~922–1737 req/5min, apiproxy ~400–1650, but eu-west-1 envs only ~40–90 req/5min; 5xx was 0 across all.
 
-- **Worker + worker 1.0 monitors** keep the **method defaults** (`100 / 20% / 20`) — the low absolute floor suits ~150 req/5min traffic.
-- **API and API-Proxy monitors** (each also including the low-traffic eu-west-1 env) are overridden to **`windowMinutes=5, minRequests=30, min5xxPercent=20, min5xxAbsolute=100`** — the lower request floor keeps the small env on the share gate, while the higher absolute floor stops the busy envs paging on minor blips (20 scattered 500s on ~1300 req = <2%). The override is applied by the dbchanges2 migration below.
+- **Worker + worker 1.0 monitors** keep the **method defaults** (`100 / 20% / 20`, and `nonFivexxPersistMinutes` default 5) — the low absolute floor suits ~150 req/5min traffic.
+- **API and API-Proxy monitors** (each also including the low-traffic eu-west-1 env) are overridden to **`windowMinutes=5, minRequests=30, min5xxPercent=20, min5xxAbsolute=100, nonFivexxPersistMinutes=8`** — the lower request floor keeps the small env on the share gate, the higher absolute floor stops the busy envs paging on minor blips (20 scattered 500s on ~1300 req = <2%), and the **8-minute** non-5xx persistence raises the gate above the observed ~5-min idle-flap length (the default 5 sat at the flap boundary). The overrides are applied by the two dbchanges2 migrations below (`2026-08-25a` = the four rate keys; `2026-09-22d` = `nonFivexxPersistMinutes=8`).
 
 #### Fail-safe: an unmeasurable solely-5xx event still pages
 
@@ -78,7 +78,7 @@ A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`**
 
 A non-5xx `Degraded`/`Severe` pages only after the **current non-Ok episode has persisted ≥ `nonFivexxPersistMinutes`** (default 5). This exists because the brief "no-cause TargetGroup reduced health" flips on low-traffic 2-instance envs are **EB Enhanced Health flappiness** — a single missed ALB `/health` check (a network/agent transient), not a real app problem — and they self-heal in ~2–5 min. A persistence delay suppresses those while a genuinely sustained non-5xx degradation still pages, delayed only by the threshold. (The windowed 5xx-flood gate is **unchanged** — it has its own, independent decision.)
 
-**Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already reads "healthy" by the time you look. On an idle env there is **no worker starvation**, so nothing to fix server-side — the mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under ASN worker saturation, fixed by the static-`/health` file) — see [api2 architecture — EB "Degraded" health](../../api2/architecture.md).
+**Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health` build deployed; **recurred 2026-09-22** as three ~5-min Ok→Warning→Ok cycles on the same env, no cause cited): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, latency/4xx/5xx = NO DATA (idle, zero traffic), ALB `RequestCount` empty; ALB target health already reads "healthy" by the time you look. The 2026-09-22 recurrence sat at the **~5-min boundary of the default `nonFivexxPersistMinutes=5`**, which is why the API + API-Proxy monitors were raised to **8** (migration `2026-09-22d`). Note the 2026-09-22 flips peaked at `Warning`, which is **below the monitor's alarm threshold** (the EB-health path pages only on `Degraded`/`Severe`), so they never paged; the gate raise is **preventive** for the `Severe`-flavored idle flap (e.g. 2026-08-29). On an idle env there is **no worker starvation**, so nothing to fix server-side — the mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under ASN worker saturation, fixed by the static-`/health` file) — see [api2 architecture — EB "Degraded" health](../../api2/architecture.md).
 
 **Measuring episode age — `fetchNonOkDepartureAgeSeconds()`** reads EB `describeEvents`:
 - The most recent **"transitioned from Ok to …"** event marks the current episode's start. Once non-Ok, later transitions stay between non-Ok states until a to-Ok event; because `describeEvents` returns **newest-first**, the first "from Ok to …" match wins.
@@ -129,6 +129,8 @@ WHERE action = 'Infrastructure/CloudWatch/ElasticBeanstalkHealth'
 
 `Core.CronJobs.parameters` is confirmed a native `JSON` column, so `JSON_SET` merges rather than overwrites.
 
+**`dbchanges2/Core/2026-09-22d - Beanstalk Health NonFivexx Persist Gate.sql`** (same pattern) adds only `$.nonFivexxPersistMinutes = 8` on the same two rows (matched by `action` + `name IN (API-Production, API Proxy-Production)`), raising the non-5xx persistence gate above the ~5-min idle-flap length seen in the 2026-09-22 `api-production-3` recurrence. Existing threshold keys are `JSON_TYPE=INTEGER`, so a bare `8` lands as an integer. Same **code-before-params deploy order** applies: worker2's `nonFivexxPersistMinutes` support (added 2026-08-29) must be live on `_production` first, or the monitor throws "Unknown named parameter" and goes dark. Ran on prod Core 2026-09-22.
+
 ### OneUptime monitor config (external, not code)
 
 Paging depends on the OneUptime Incoming-Request monitor's **string-match** criteria on the POSTed body — `Contains "alarm":"HIGH"` pages.
@@ -146,6 +148,12 @@ Paging depends on the OneUptime Incoming-Request monitor's **string-match** crit
 - **Deploy fails without degrading health** — still not caught (accepted residual).
 - **`ElasticBeanstalkHealth()` is long** and grew with the persistence gate — php-reviewer (0 critical) flagged it as a decomposition candidate; deferred pending the missing harness.
 
+## Change history
+
+- 2026-09-22 — API + API-Proxy monitors set `nonFivexxPersistMinutes=8` (was default 5) via dbchanges2 `2026-09-22d`, after the `api-production-3` idle-flap recurred at the ~5-min default boundary; default 5 kept on Worker/Worker-1.0. (jcardinal)
+- 2026-08-29 — added the non-5xx persistence gate (`nonFivexxPersistMinutes`, default 5) to suppress brief self-healing idle-env EB flaps. (jcardinal)
+- 2026-08-25 — 5xx page moved to a CloudWatch windowed request-rate gate; API/API-Proxy override applied via dbchanges2 `2026-08-25a`. (jcardinal)
+
 ## Related
 
 - [Cross-account AWS access](./cross-account-aws-access.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.869",
+  "version": "1.0.870",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",