toga-ai 1.0.696 → 1.0.697

package/knowledge/2.0/apps/api2/architecture.md

@@ -302,6 +302,27 @@ the deploy/version timeline before hunting app bugs. **Fixed 2026-08-28:** all t
 to **Rolling / batch-size 1** (≥1 healthy backend throughout a deploy), which eliminates the blip —
 config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
 
+**A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a
+LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.**
+Distinct from the third cause (that one is the *busy* env, `api-production-1`, under Compass ASN
+worker saturation). This is a single missed ALB `/health` check — a network/agent transient.
+Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health`
+build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum),
+**0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already
+reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no
+server-side app fix applies** — do not chase it. The mitigation is alert de-sensitization: the
+worker2 EB health monitor now applies a **persistence-delay gate** (a non-5xx `Degraded`/`Severe`
+must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages), which suppresses these flaps.
+See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
+
+**Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504 to find
+the cause:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout
+(worker saturation); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing
+in-flight requests, or a PHP worker crash). Because **ELB-generated 5xx (any of these) never reach
+PHP**, they leave **no row in `Logs.Api`** (api2 logs only *after* a completed response) — so an EB
+"% 5xx" flag with an **empty `Logs.Api`** means an **ELB-level** 5xx, not an app 5xx. Discriminate
+with CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`.
+
 ## Known issues / accepted risks
 
 Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -379,6 +400,14 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-08-29 — Recorded a **fifth EB-`Degraded` cause**: a brief self-healing "no-cause reduced
+  health" flip on a low-traffic *idle* 2-instance env (e.g. `api-production-3`, eu-west-1) is EB
+  Enhanced Health flappiness (a single missed ALB `/health` check), distinct from the busy-env
+  PHP-`/health`-starvation cause — no server-side fix applies; mitigated by the worker2 monitor's
+  new non-5xx persistence-delay gate. Added an ELB **502/503/504 discrimination** triage aid
+  (503=no healthy backend, 504=target timeout/worker saturation, 502=dropped connection from an
+  AllAtOnce deploy or PHP crash); ELB-generated 5xx never reach PHP so they leave no `Logs.Api`
+  row — an EB "% 5xx" flag with an empty `Logs.Api` is ELB-level, not app 5xx. (jcardinal)
 - 2026-08-28 — Recorded a **fourth EB-`Degraded` cause — the deploy-restart ELB 502**: under the old
   `AllAtOnce`/100% policy a deploy restarts both instances of a 2-instance env at once, dropping
   in-flight connections → an ELB-502 burst (43 in one minute on `api-production-1`, 2026-08-28) that

package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md

@@ -152,6 +152,10 @@ one 15 s interval (~45 s → 30 s) of per-batch health settle.
   `BatchSize`); `aws:elasticbeanstalk:environment:process:default` (`HealthyThresholdCount`).
 - Trade-off: Rolling is slower end-to-end than the old ~31 s `AllAtOnce`, and capacity briefly
   drops to 1/2 instances per batch (fine at current traffic).
+- **Confirmed working in production (2026-08-29).** The 2026-08-29 00:49 UTC deploy ran as
+  Rolling batches ("Batch 1 / Batch 2, one instance at a time, register and wait for healthy")
+  across all three API prod envs with **no post-deploy 5xx flip** — the `AllAtOnce`
+  deploy-restart 502 blip is eliminated in practice, not just in config.
 
 ### Deploy-policy guidance (don't reach for immutable expecting speed)
 - **Immutable does NOT reduce deploy time.** It is the **slowest** policy — it boots a fresh
@@ -295,6 +299,10 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 ```
 
 ## Change history
+- 2026-08-29 — Confirmed the Rolling/batch-1 policy works in production: the 2026-08-29
+  00:49 UTC deploy ran as Rolling batches (one instance at a time, register + wait for healthy)
+  on all three API prod envs with **no post-deploy 5xx flip** — the AllAtOnce deploy-restart
+  502 blip is eliminated. (jcardinal)
 - 2026-08-28 — Recorded the **production deploy trigger + policy change**: the `api-production-1/2/3`
   CodePipelines **auto-deploy on every push to `_production`** (WebhookV2 on the CodeConnections
   connection, repos `_underscore` + `api2`), so a merge to `_production` **is** a three-region prod

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-25
+updated: 2026-08-29
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -28,10 +28,13 @@ OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
 
 The load-bearing design decision this doc records is the **paging (alarm) criteria**, and
 the key **2026-08-25** change: the 5xx page is now driven by a **CloudWatch windowed
-request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. A non-5xx
-`Degraded`/`Severe` still pages immediately off EB health; but whether a *5xx* problem pages
-is decided by the true per-environment 5xx **rate over a trailing window** (default 5 min),
-not by EB's ~10-second `ApplicationMetrics` snapshot.
+request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. Whether a *5xx*
+problem pages is decided by the true per-environment 5xx **rate over a trailing window**
+(default 5 min), not by EB's ~10-second `ApplicationMetrics` snapshot. As of **2026-08-29**,
+a non-5xx `Degraded`/`Severe` no longer pages immediately either — it must **persist** past a
+threshold (`nonFivexxPersistMinutes`, default 5) before paging, which suppresses the brief
+self-healing "reduced health" flips that EB Enhanced Health throws on low-traffic 2-instance
+envs (see the persistence gate below).
 
 **Why the change:** the old gate read EB `describeEnvironmentHealth`'s `ApplicationMetrics`,
 a ~10-**second** window. At low traffic a single 500 in a near-empty bucket reads as 100%
@@ -52,6 +55,10 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
   `_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
   environments, oneuptimeUrl, …thresholds})`.
 - New private `fetchWindowedRequestCounts()` reads the CloudWatch metrics (below).
+- New private `fetchNonOkDepartureAgeSeconds(ElasticBeanstalkClient $eb, string
+  $environmentName, int $lookbackSeconds): ?int` returns how long the current non-Ok episode
+  has persisted, from EB `describeEvents` (below). **Never throws** — catches `AwsException`
+  and returns `null`.
 - AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
   a read-only role per account, one assume reused across regions) — see
   [Cross-account AWS access](./cross-account-aws-access.md), which also documents the
@@ -64,11 +71,14 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
 
 Two independent paths now feed `alarm=HIGH`:
 
-1. **EB-health path (non-5xx).** EB `HealthStatus` `Degraded` or `Severe` that is **not**
-   attributable solely to 5xx (latency, instances down, a failed deploy, a mixed cause, or
-   no stated cause) **pages immediately** — unchanged, still driven off EB health.
-   `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx event
-   apart from everything else.
+1. **EB-health path (non-5xx) — now gated on PERSISTENCE (2026-08-29).** EB `HealthStatus`
+   `Degraded` or `Severe` that is **not** attributable solely to 5xx (latency, instances
+   down, a failed deploy, a mixed cause, or no stated cause) pages **only once the current
+   non-Ok episode has persisted ≥ `nonFivexxPersistMinutes`** (new optional param, default 5)
+   — it no longer pages on the first snapshot. A sub-threshold blip is **reported but does not
+   page**. `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx
+   event apart from everything else. See the persistence-gate section below for how episode
+   age is measured and the fail-safe.
 2. **CloudWatch windowed 5xx-rate path.** For a solely-5xx `Degraded`/`Severe`, the page is
    decided **not** by EB but by the true windowed rate from CloudWatch. Given the window's
    `requests` (ApplicationRequestsTotal) and `fivexx` (ApplicationRequests5xx), page when:
@@ -91,6 +101,7 @@ The four thresholds are **optional cron parameters** spread as **named arguments
 | `minRequests` | `100` | Minimum windowed request count before the **share** gate applies |
 | `min5xxPercent` | `20.0` | 5xx share (percent) at/above which a solely-5xx env pages, once `minRequests` is met |
 | `min5xxAbsolute` | `20` | Absolute windowed 5xx count at/above which a solely-5xx env pages regardless of share |
+| `nonFivexxPersistMinutes` | `5` | Minutes a **non-5xx** `Degraded`/`Severe` episode must persist before it pages (added 2026-08-29; a trailing optional param — existing `CronJobs.parameters` without the key still run) |
 
 Removed this iteration: `shouldPageForHealth()` and the old
 `HTTP_5XX_ALARM_RATIO` / `HTTP_5XX_ALARM_ABSOLUTE_COUNT` constants (superseded by the
@@ -123,6 +134,43 @@ A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`**
 gap), distinct from a measured-bad `alarm=HIGH` — same alarm-vs-probe split as the
 multi-client monitors in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md).
 
+## Persistence gate on the non-5xx page (2026-08-29)
+
+A non-5xx `Degraded`/`Severe` now pages only after the **current non-Ok episode has persisted
+≥ `nonFivexxPersistMinutes`** (default 5). This exists because the brief "no-cause TargetGroup
+reduced health" flips on low-traffic 2-instance envs are **EB Enhanced Health flappiness** —
+a single missed ALB `/health` check (a network/agent transient), not a real app problem — and
+they self-heal in ~2–5 min. A persistence delay suppresses those while a genuinely sustained
+non-5xx degradation still pages, delayed only by the threshold. (The windowed 5xx-flood gate
+is **unchanged** — it has its own, independent decision.)
+
+**Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29
+13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout,
+`HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, sub-second latency, idle
+CPU, near-zero traffic; ALB target health already reads "healthy" by the time you look. On an
+idle env there is **no worker starvation**, so there is nothing to fix server-side — the
+mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from
+the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under
+ASN worker saturation, fixed by the static-`/health` file) — see
+[api2 architecture — EB "Degraded" health](../../api2/architecture.md).
+
+**Measuring episode age — `fetchNonOkDepartureAgeSeconds()`** reads EB `describeEvents`:
+
+- The most recent **"transitioned from Ok to …"** event marks the current episode's start.
+  Once non-Ok, later transitions stay between non-Ok states until a to-Ok event; because
+  `describeEvents` returns **newest-first**, the first "from Ok to …" match wins.
+- The query is **TIME-bounded**: `StartTime = now − (persistWindow +
+  HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS = 600)`, with a `MaxRecords = 100` cap — so a chatty
+  env's deploy/config events cannot push the departure event out of range.
+- **Fail-safe:** an episode **older than the window** returns `null` → the caller **pages**.
+  That is correct: if the departure is beyond the lookback, the episode has already persisted
+  past the threshold, so paging is the safe outcome. A read failure also returns `null`
+  (the method catches `AwsException` and never throws) → same fail-safe.
+
+**Auditable suppression:** a sub-threshold blip is reported, not paged, and the suppression is
+made visible — `pendingPersistSeconds` is added to the OneUptime payload and a `pending=Ns`
+annotation to the WorkerJobs summary — so a suppressed episode is never silent.
+
 ## Reading the windowed 5xx rate from CloudWatch (durable AWS reference)
 
 - **EB application-request metrics are PER-INSTANCE ONLY.** `ApplicationRequestsTotal`,
@@ -214,6 +262,8 @@ the `probe` token; this is a OneUptime config change the developer applies.
   `min5xxPercent`; absolute at `min5xxAbsolute-1` vs `min5xxAbsolute`; `requests` just below
   `minRequests`; empty/zero window; mixed 5xx + non-5xx cause) — deferred pending a harness.
 - **Deploy fails without degrading health** — still not caught (accepted residual).
+- **`ElasticBeanstalkHealth()` is long** and grew with the persistence gate — php-reviewer
+  (0 critical) flagged it as a decomposition candidate; deferred pending the missing harness.
 
 ## Gotchas / known issues
 
@@ -235,6 +285,21 @@ the `probe` token; this is a OneUptime config change the developer applies.
   record its value in a doc.
 
 ## Change history
+- 2026-08-29 — Added a **persistence gate on the non-5xx page**: a non-5xx `Degraded`/`Severe`
+  no longer pages on the first snapshot — it must persist ≥ `nonFivexxPersistMinutes` (new
+  optional trailing param, default 5; existing `CronJobs.parameters` without the key still
+  run) before paging. Episode age comes from EB `describeEvents` via new never-throwing helper
+  `fetchNonOkDepartureAgeSeconds()` (most recent "from Ok to …" marks episode start; events
+  are newest-first so the first match wins; TIME-bounded `StartTime = now − (persistWindow +
+  HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS=600)`, `MaxRecords=100`, so an episode older than the
+  window returns `null` → the caller pages — correct fail-safe). Sub-threshold blips are
+  reported not paged, with `pendingPersistSeconds` (OneUptime payload) + `pending=Ns`
+  (WorkerJobs summary) so suppressions stay auditable. The windowed 5xx gate is unchanged.
+  Rationale: the brief self-healing "no-cause reduced health" flips on low-traffic 2-instance
+  envs are EB Enhanced Health flappiness (a single missed ALB `/health` check), not app bugs
+  — distinct from the busy-env PHP-`/health`-starvation cause. worker2-only; the param is
+  optional-with-default so no `_underscore`/dbchanges2 change. Not yet committed/deployed.
+  (jcardinal)
 - 2026-08-25 — Replaced the EB `ApplicationMetrics` 5xx gate (a ~10s window that false-paged
   "100% failing" on a lone 500 in a near-empty bucket — real worker-production incident: 16
   500s / 516 req = 3.1% over 17 min) with a **CloudWatch windowed request-rate gate**. Reads

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.696",
+  "version": "1.0.697",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",