toga-ai 1.0.695 → 1.0.697

package/knowledge/2.0/apps/api2/INDEX.md

@@ -25,5 +25,5 @@
 | [V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay the whole request in-process**, so a transient collision self-re | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, _underscore/Database.php, _underscore/Query.php, _underscore/Model/Compass/AdvanceShippingNotice.php |
 | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
 | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
-| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
+| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
 | [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/composer.json, api2/composer.lock, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |

package/knowledge/2.0/apps/api2/architecture.md

@@ -197,6 +197,12 @@ always exits 0). See
 [the worker2 reference implementation](../worker2/features/alb-target-group-auto-registration.md) —
 api2's copy is the **unhardened original** and must be brought up to it.
 
+**Deploy policy (2026-08-28):** the three production envs (`api-production-1/2/3`, 2 instances each)
+run **Rolling / `BatchSize` 1** (was `AllAtOnce`/100%), target-group `HealthyThresholdCount` 2 — so
+a deploy keeps ≥1 healthy backend and no longer emits the ELB-502 blip. Each CodePipeline
+auto-deploys on a push to `_production`. See
+[the CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
+
 ## CI — commit message policy
 
 `.github/workflows/true-devteam-requirements.yml` enforces **`TRUE-{ticket}: {Subject}`** —
@@ -275,6 +281,48 @@ probe was starved of a PHP-FPM worker — do **not** chase client-traffic 5xx. *
 liveness from PHP-FPM worker availability so a slow-request burst can no longer flap it — see
 [health-check endpoint](features/health-check-endpoint.md).
 
+**A fourth cause (deploy-triggered) — a burst of ELB `502`s when both instances restart at once.**
+Confirmed on `api-production-1` (2026-08-28 23:51 UTC deploy: **43 × 502 in one minute**, the only
+ELB 5xx in a 7-hour window). Under the old `AllAtOnce`/100% deploy policy, every deploy to a
+2-instance env restarts Apache/PHP-FPM on **both** instances simultaneously, so in-flight requests
+have their connections dropped mid-response → a burst of **ELB-generated HTTP 502**. EB flags "X%
+of requests to the ELB are failing with HTTP 5xx" `Degraded` ~3–4 min later and recovers in ~5 min.
+Distinguishing fingerprint vs. a real app failure: it is **502** (a dropped backend connection),
+**not 503** (no healthy backend) or **504** (backend timeout); **`HTTPCode_Target_5XX_Count` (app
+5xx) = 0**; **`HealthyHostCount` never drops below 2**; and there is **NOTHING in `Logs.Api`** —
+api2 logs a request only *after* the response, and a 502 never completes one, so the request leaves
+no log row. Diagnosis path: CloudWatch `HTTPCode_ELB_502/503/504_Count` vs
+`HTTPCode_Target_5XX_Count`, plus EB `describe-events` (a version change) and CodePipeline
+`list-pipeline-executions` — because a push to `_production` auto-deploys all three prod tiers, "we
+didn't deploy" can be false (see
+[the CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md)).
+**Rule:** an EB "% 5xx" `Degraded` with an **empty `Logs.Api` and 0 target-5xx** right after a
+deploy is a deploy-restart 502 artifact, not an application bug — check the ELB-5xx breakdown and
+the deploy/version timeline before hunting app bugs. **Fixed 2026-08-28:** all three prod envs moved
+to **Rolling / batch-size 1** (≥1 healthy backend throughout a deploy), which eliminates the blip —
+config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
+
+**A fifth cause — a brief, self-healing "no-cause TargetGroup reduced health" flip on a
+LOW-TRAFFIC, IDLE 2-instance env is EB Enhanced Health FLAPPINESS, not worker starvation.**
+Distinct from the third cause (that one is the *busy* env, `api-production-1`, under Compass ASN
+worker saturation). This is a single missed ALB `/health` check — a network/agent transient.
+Fingerprint (verified `api-production-3`, eu-west-1, 2026-08-29 13:31 UTC `Severe`, static-`/health`
+build deployed): self-heals in ~2–5 min; throughout, `HealthyHostCount` stays 2 (1-min minimum),
+**0** target/ELB 5xx, sub-second latency, idle CPU, near-zero traffic; ALB target health already
+reads "healthy" by the time you check. On an idle env there is no worker starvation, so **no
+server-side app fix applies** — do not chase it. The mitigation is alert de-sensitization: the
+worker2 EB health monitor now applies a **persistence-delay gate** (a non-5xx `Degraded`/`Severe`
+must persist ≥ `nonFivexxPersistMinutes`, default 5, before it pages), which suppresses these flaps.
+See [worker2 EB health monitor](../worker2/features/elastic-beanstalk-health-monitor.md).
+
+**Triage aid — break an EB "% requests failing with HTTP 5xx" flag into ELB 502/503/504 to find
+the cause:** **503** = no healthy backend (a deploy with no capacity); **504** = target timeout
+(worker saturation); **502** = connection dropped mid-response (an `AllAtOnce` deploy killing
+in-flight requests, or a PHP worker crash). Because **ELB-generated 5xx (any of these) never reach
+PHP**, they leave **no row in `Logs.Api`** (api2 logs only *after* a completed response) — so an EB
+"% 5xx" flag with an **empty `Logs.Api`** means an **ELB-level** 5xx, not an app 5xx. Discriminate
+with CloudWatch `HTTPCode_ELB_502/503/504_Count` vs `HTTPCode_Target_5XX_Count`.
+
 ## Known issues / accepted risks
 
 Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -352,6 +400,23 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-08-29 — Recorded a **fifth EB-`Degraded` cause**: a brief self-healing "no-cause reduced
+  health" flip on a low-traffic *idle* 2-instance env (e.g. `api-production-3`, eu-west-1) is EB
+  Enhanced Health flappiness (a single missed ALB `/health` check), distinct from the busy-env
+  PHP-`/health`-starvation cause — no server-side fix applies; mitigated by the worker2 monitor's
+  new non-5xx persistence-delay gate. Added an ELB **502/503/504 discrimination** triage aid
+  (503=no healthy backend, 504=target timeout/worker saturation, 502=dropped connection from an
+  AllAtOnce deploy or PHP crash); ELB-generated 5xx never reach PHP so they leave no `Logs.Api`
+  row — an EB "% 5xx" flag with an empty `Logs.Api` is ELB-level, not app 5xx. (jcardinal)
+- 2026-08-28 — Recorded a **fourth EB-`Degraded` cause — the deploy-restart ELB 502**: under the old
+  `AllAtOnce`/100% policy a deploy restarts both instances of a 2-instance env at once, dropping
+  in-flight connections → an ELB-502 burst (43 in one minute on `api-production-1`, 2026-08-28) that
+  EB flags as "X% 5xx" Degraded ~3–4 min later. Fingerprint vs. an app bug: **502** (not 503/504),
+  **`HTTPCode_Target_5XX_Count`=0**, `HealthyHostCount` never < 2, and **nothing in `Logs.Api`** (a
+  502 never completes a PHP response, which api2 logs only after). Diagnose via the ELB-5xx breakdown
+  + `describe-events`/`list-pipeline-executions` — a push to `_production` auto-deploys all three prod
+  tiers, so "we didn't deploy" can be false. Fixed by moving to Rolling/batch-1 (deploy config +
+  policy tradeoffs captured in the CodePipeline deploy workflow). (jcardinal)
 - 2026-08-28 — Recorded the third EB-`Degraded` cause: a no-stated-cause "reduced health" flip on `api-production-1` is the **PHP-served `/health` probe starved of a PHP-FPM worker** by off-peak bursts of slow failing Compass ASN POSTs (17–63 s, ~100% 500) — invisible in ALB `RequestCount`/`5xx` and never dropping `HealthyHostCount` (flap < 75 s deregistration). Correlated a 06:00–06:08 CST ASN burst to the 11:05 UTC Severe flip. Noted the fix (static-file `/health` decouples liveness from PHP-FPM) and updated the "Don't break `/health`" rule to the static-file form. (jcardinal)
 - 2026-08-26 — Documented and fixed a `/v2/auth/public` OOM: a present-but-null client `transactionId` (`{"transactionId": null}`) flowed into the request-logger's `_Model::load()` dup-check; because `Logs.Api.transactionId` is `FIELD_CHAR`, a null value collapses the WHERE and full-table-scans prod `Logs.Api` → memory exhaustion (the `_underscore` collapsed-WHERE mechanism). V2 only auto-uuids an *absent* transactionId (~L543), so a present null fell through (~L553 read → null). Fix (commit `36ef31f`): coerce null/empty transactionId → uuid in the auth block, and skip `load()` in the final logger dup-check (~L2277) when null/empty. Added the "don't blame the latest deploy — check first-occurrence" triage note. (jcardinal)
 - 2026-08-25 — **Resolved the auto-generated `Api.transactionId` 1062 collision** (was the 2026-07-28

package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md

@@ -6,12 +6,13 @@ project: API
 client: shared
 type: workflow
 status: active
-updated: 2026-08-25
+updated: 2026-08-28
 owners: ["jcardinal", "mhammontree", "bala", "apeterson"]
 files:
   - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
   - api2/ebs/register_instance_to_shared_application_load_balancer.php
   - api2/.platform/hooks/prebuild/git.sh
+  - api2/.platform/hooks/postdeploy/015_install_composer.sh
   - api2/.ebextensions/git.php
   - api2/.ebextensions/git.sandbox-dev.json
 related:
@@ -62,8 +63,11 @@ are actually changing before you merge anything:
   `_<environment>`). Consequence: **a framework-only change requires REDEPLOYING THE CONSUMING
   TIER even when `api2` itself has zero changes** — an instance **restart will not pick it up**,
   because the clone happens in prebuild.
-- **`api2` has no deploy workflow.** Its only GitHub Action is the `TRUE-#####:` commit-message
-  linter, so **EB deploys are manual**. Do not wait for CI to ship a framework change.
+- **`api2` has no GitHub *Actions* deploy workflow** — its only GitHub Action is the
+  `TRUE-#####:` commit-message linter. **The production tiers, however, auto-deploy via AWS
+  CodePipeline on every push to `_production`** (a different system from GitHub Actions — see
+  *Production deploy: auto-trigger + rolling policy* below). Non-pipeline-triggered tiers are
+  still deployed by hand; don't wait for a GitHub Action to ship a framework change.
 
 ### ⚠ The `_underscore` SOURCE ACTION has its own branch trigger — and a green pipeline hides a wrong one
 
@@ -105,6 +109,68 @@ beta env actually received and returned when Sentry isn't wired up. (The Rate AI
 scans this same `Logs_Rate.Api` for failed outbound calls — see
 `clients/rate/features/aig-contract-creation.md`.)
 
+## Production deploy: auto-trigger + rolling policy (2026-08-28)
+
+### Production auto-deploys on every push to `_production` — a merge IS a deployment
+The `api-production-1` (us-east-1), `api-production-2` (us-west-2), and `api-production-3`
+(eu-west-1) CodePipelines each **auto-deploy on every push to branch `_production`** — a
+**WebhookV2** trigger on the CodeConnections GitHub connection, watching source repos
+`agilantsolutions/_underscore` and `agilantsolutions/api2`. There is **no separate manual deploy
+step for production**: pushing or merging to `_production` **is** a production deployment to all
+three regions. (This is why deploying a fix can itself page a moment later — see the
+deploy-restart 502 below.)
+
+**Triage consequence:** "we didn't deploy" can be false — any push to `_production` is a deploy.
+Correlate an incident against the actual pipeline runs, not just remembered manual actions:
+```bash
+aws codepipeline list-pipeline-executions --pipeline-name api-production-1 --region us-east-1
+aws elasticbeanstalk describe-events --environment-name api-production-1 --region us-east-1  # version-change events
+```
+
+### ⚠ An EB `Degraded` right after a deploy = the deploy-restart 502
+Under the old `AllAtOnce`/100% policy every deploy restarted Apache/PHP-FPM on **both**
+instances of a 2-instance env simultaneously, dropping in-flight connections → a burst of
+ELB-generated **HTTP 502** that EB flags as an "X% 5xx" `Degraded` ~3–4 min later (recovers in
+~5 min). This is a **deploy artifact, not an application failure** — the full diagnostic
+fingerprint (502 not 503/504, app `HTTPCode_Target_5XX_Count`=0, `HealthyHostCount` never < 2,
+**nothing in `Logs.Api`** because a 502 never completes a PHP response) lives in
+[api2 architecture — EB "Degraded" health](../architecture.md). The rolling policy below
+eliminates it.
+
+### Deploy policy: Rolling, batch size 1 (changed 2026-08-28)
+All three production envs (app **"API"**, 2 instances each) were switched from
+`AllAtOnce`/`Percentage`/`100` to **`DeploymentPolicy=Rolling`, `BatchSizeType=Fixed`,
+`BatchSize=1`**, and the target group **`HealthyThresholdCount` dropped 3 → 2** (health path
+`/health`, `HealthCheckInterval` 15 s, `UnhealthyThresholdCount` 5 unchanged). Rolling batch-1 on
+a 2-instance env deploys in **2 batches with ≥1 healthy backend throughout**, so a deploy no
+longer drops in-flight requests → **no more deploy-restart 502 blip**; the 3→2 threshold shaves
+one 15 s interval (~45 s → 30 s) of per-batch health settle.
+
+- Applied via a **config-only `update-environment`** (AWS CLI, per env) — no instance
+  replacement, no downtime; **takes effect on the NEXT deploy**.
+- Namespaces: `aws:elasticbeanstalk:command` (`DeploymentPolicy` / `BatchSizeType` /
+  `BatchSize`); `aws:elasticbeanstalk:environment:process:default` (`HealthyThresholdCount`).
+- Trade-off: Rolling is slower end-to-end than the old ~31 s `AllAtOnce`, and capacity briefly
+  drops to 1/2 instances per batch (fine at current traffic).
+- **Confirmed working in production (2026-08-29).** The 2026-08-29 00:49 UTC deploy ran as
+  Rolling batches ("Batch 1 / Batch 2, one instance at a time, register and wait for healthy")
+  across all three API prod envs with **no post-deploy 5xx flip** — the `AllAtOnce`
+  deploy-restart 502 blip is eliminated in practice, not just in config.
+
+### Deploy-policy guidance (don't reach for immutable expecting speed)
+- **Immutable does NOT reduce deploy time.** It is the **slowest** policy — it boots a fresh
+  instance batch, health-checks it, then swaps, and temporarily **doubles** the instance count.
+  It does fully eliminate deploy 5xx and gives instant rollback, but it is not "faster."
+- **`AllAtOnce` is fast (~31 s) precisely BECAUSE both instances restart at once** — which is
+  what causes the 502s. You cannot have "as fast as AllAtOnce" **and** "zero 502s."
+- **Rolling batch-1 is the sweet spot** for a 2-instance env.
+- **`RollingWithAdditionalBatch`** is the option only if zero capacity loss during a deploy is
+  ever required (it adds a temporary instance; slower and costlier).
+- The real deploy-**time** levers are independent of the policy: the health-check settle
+  (`HealthCheckInterval` × `HealthyThresholdCount`) and per-instance postdeploy work — api2 runs
+  `composer install` on **every** deploy via
+  `.platform/hooks/postdeploy/015_install_composer.sh` (a caching / pre-bake candidate).
+
 ## Steps
 1. A push to the watched branch (e.g. `_stage`) triggers the pipeline.
 2. The Source stage uses `CodeStarSourceConnection` actions, each referencing:
@@ -233,6 +299,24 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 ```
 
 ## Change history
+- 2026-08-29 — Confirmed the Rolling/batch-1 policy works in production: the 2026-08-29
+  00:49 UTC deploy ran as Rolling batches (one instance at a time, register + wait for healthy)
+  on all three API prod envs with **no post-deploy 5xx flip** — the AllAtOnce deploy-restart
+  502 blip is eliminated. (jcardinal)
+- 2026-08-28 — Recorded the **production deploy trigger + policy change**: the `api-production-1/2/3`
+  CodePipelines **auto-deploy on every push to `_production`** (WebhookV2 on the CodeConnections
+  connection, repos `_underscore` + `api2`), so a merge to `_production` **is** a three-region prod
+  deploy — correlate incidents against `list-pipeline-executions`, not remembered manual actions.
+  Switched all three 2-instance envs from `AllAtOnce`/100% to **Rolling / Fixed / BatchSize 1** and
+  dropped the target-group **`HealthyThresholdCount` 3 → 2** (config-only `update-environment`,
+  effective next deploy) to eliminate the **deploy-restart 502 blip** (both instances restarting at
+  once dropped in-flight requests → ELB 502 → EB "X% 5xx" Degraded). Added deploy-policy guidance:
+  immutable is the **slowest** policy (not faster) and doubles instances; `AllAtOnce` is fast only
+  because it restarts both at once (the 502 cause); Rolling batch-1 is the sweet spot;
+  `RollingWithAdditionalBatch` only if zero capacity loss is needed; the real deploy-time levers are
+  the health-check settle and the per-deploy `015_install_composer.sh` `composer install`. Corrected
+  the "EB deploys are manual" note — that is only true of the absent GitHub *Actions* workflow; prod
+  deploys via CodePipeline are automatic. (jcardinal)
 - 2026-08-25 — Documented that the Source stage's **`_underscore` action has its own AWS-side branch
   trigger**, independent of `git.<env>.json` and invisible in the repo — a second way to silently run
   the wrong framework branch. `ApiSandboxClient` ran `_production` from a mis-set source action while

package/knowledge/2.0/apps/worker2/features/elastic-beanstalk-health-monitor.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-08-25
+updated: 2026-08-29
 owners: [jcardinal]
 files:
   - worker2/Worker/Infrastructure/CloudWatch.php
@@ -28,10 +28,13 @@ OneUptime string-matches — OneUptime cannot compare numbers on a pushed body.
 
 The load-bearing design decision this doc records is the **paging (alarm) criteria**, and
 the key **2026-08-25** change: the 5xx page is now driven by a **CloudWatch windowed
-request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. A non-5xx
-`Degraded`/`Severe` still pages immediately off EB health; but whether a *5xx* problem pages
-is decided by the true per-environment 5xx **rate over a trailing window** (default 5 min),
-not by EB's ~10-second `ApplicationMetrics` snapshot.
+request-rate gate**, fully **decoupled from EB's own `HealthStatus`**. Whether a *5xx*
+problem pages is decided by the true per-environment 5xx **rate over a trailing window**
+(default 5 min), not by EB's ~10-second `ApplicationMetrics` snapshot. As of **2026-08-29**,
+a non-5xx `Degraded`/`Severe` no longer pages immediately either — it must **persist** past a
+threshold (`nonFivexxPersistMinutes`, default 5) before paging, which suppresses the brief
+self-healing "reduced health" flips that EB Enhanced Health throws on low-traffic 2-instance
+envs (see the persistence gate below).
 
 **Why the change:** the old gate read EB `describeEnvironmentHealth`'s `ApplicationMetrics`,
 a ~10-**second** window. At low traffic a single 500 in a near-empty bucket reads as 100%
@@ -52,6 +55,10 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
   `_Worker::runTask('Infrastructure/CloudWatch/ElasticBeanstalkHealth', {awsAccountId,
   environments, oneuptimeUrl, …thresholds})`.
 - New private `fetchWindowedRequestCounts()` reads the CloudWatch metrics (below).
+- New private `fetchNonOkDepartureAgeSeconds(ElasticBeanstalkClient $eb, string
+  $environmentName, int $lookbackSeconds): ?int` returns how long the current non-Ok episode
+  has persisted, from EB `describeEvents` (below). **Never throws** — catches `AwsException`
+  and returns `null`.
 - AWS access is obtained through `_Component_Aws_Workloads::assumeCredentials()` (STS-assume
   a read-only role per account, one assume reused across regions) — see
   [Cross-account AWS access](./cross-account-aws-access.md), which also documents the
@@ -64,11 +71,14 @@ itself runs non-fatal. CloudWatch read failures are caught **internally** and re
 
 Two independent paths now feed `alarm=HIGH`:
 
-1. **EB-health path (non-5xx).** EB `HealthStatus` `Degraded` or `Severe` that is **not**
-   attributable solely to 5xx (latency, instances down, a failed deploy, a mixed cause, or
-   no stated cause) **pages immediately** — unchanged, still driven off EB health.
-   `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx event
-   apart from everything else.
+1. **EB-health path (non-5xx) — now gated on PERSISTENCE (2026-08-29).** EB `HealthStatus`
+   `Degraded` or `Severe` that is **not** attributable solely to 5xx (latency, instances
+   down, a failed deploy, a mixed cause, or no stated cause) pages **only once the current
+   non-Ok episode has persisted ≥ `nonFivexxPersistMinutes`** (new optional param, default 5)
+   — it no longer pages on the first snapshot. A sub-threshold blip is **reported but does not
+   page**. `Suspended` never pages. `causesAttributeSolelyTo5xx()` is what tells a solely-5xx
+   event apart from everything else. See the persistence-gate section below for how episode
+   age is measured and the fail-safe.
 2. **CloudWatch windowed 5xx-rate path.** For a solely-5xx `Degraded`/`Severe`, the page is
    decided **not** by EB but by the true windowed rate from CloudWatch. Given the window's
    `requests` (ApplicationRequestsTotal) and `fivexx` (ApplicationRequests5xx), page when:
@@ -91,6 +101,7 @@ The four thresholds are **optional cron parameters** spread as **named arguments
 | `minRequests` | `100` | Minimum windowed request count before the **share** gate applies |
 | `min5xxPercent` | `20.0` | 5xx share (percent) at/above which a solely-5xx env pages, once `minRequests` is met |
 | `min5xxAbsolute` | `20` | Absolute windowed 5xx count at/above which a solely-5xx env pages regardless of share |
+| `nonFivexxPersistMinutes` | `5` | Minutes a **non-5xx** `Degraded`/`Severe` episode must persist before it pages (added 2026-08-29; a trailing optional param — existing `CronJobs.parameters` without the key still run) |
 
 Removed this iteration: `shouldPageForHealth()` and the old
 `HTTP_5XX_ALARM_RATIO` / `HTTP_5XX_ALARM_ABSOLUTE_COUNT` constants (superseded by the
@@ -123,6 +134,43 @@ A CloudWatch read *failure* (as opposed to empty data) sets **`probe=DEGRADED`**
 gap), distinct from a measured-bad `alarm=HIGH` — same alarm-vs-probe split as the
 multi-client monitors in [OneUptime push-metric monitors](./oneuptime-worker2-monitoring.md).
 
+## Persistence gate on the non-5xx page (2026-08-29)
+
+A non-5xx `Degraded`/`Severe` now pages only after the **current non-Ok episode has persisted
+≥ `nonFivexxPersistMinutes`** (default 5). This exists because the brief "no-cause TargetGroup
+reduced health" flips on low-traffic 2-instance envs are **EB Enhanced Health flappiness** —
+a single missed ALB `/health` check (a network/agent transient), not a real app problem — and
+they self-heal in ~2–5 min. A persistence delay suppresses those while a genuinely sustained
+non-5xx degradation still pages, delayed only by the threshold. (The windowed 5xx-flood gate
+is **unchanged** — it has its own, independent decision.)
+
+**Fingerprint of the idle-env flap** (verified `api-production-3`, eu-west-1, 2026-08-29
+13:31 UTC `Severe`, static-`/health` build deployed): self-heals in ~2–5 min; throughout,
+`HealthyHostCount` stays 2 (1-min minimum), **0** target/ELB 5xx, sub-second latency, idle
+CPU, near-zero traffic; ALB target health already reads "healthy" by the time you look. On an
+idle env there is **no worker starvation**, so there is nothing to fix server-side — the
+mitigation is this monitor persistence-delay gate, not more app code. **This is DISTINCT from
+the PHP-`/health`-starvation cause** (that one is the *busy* env, `api-production-1`, under
+ASN worker saturation, fixed by the static-`/health` file) — see
+[api2 architecture — EB "Degraded" health](../../api2/architecture.md).
+
+**Measuring episode age — `fetchNonOkDepartureAgeSeconds()`** reads EB `describeEvents`:
+
+- The most recent **"transitioned from Ok to …"** event marks the current episode's start.
+  Once non-Ok, later transitions stay between non-Ok states until a to-Ok event; because
+  `describeEvents` returns **newest-first**, the first "from Ok to …" match wins.
+- The query is **TIME-bounded**: `StartTime = now − (persistWindow +
+  HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS = 600)`, with a `MaxRecords = 100` cap — so a chatty
+  env's deploy/config events cannot push the departure event out of range.
+- **Fail-safe:** an episode **older than the window** returns `null` → the caller **pages**.
+  That is correct: if the departure is beyond the lookback, the episode has already persisted
+  past the threshold, so paging is the safe outcome. A read failure also returns `null`
+  (the method catches `AwsException` and never throws) → same fail-safe.
+
+**Auditable suppression:** a sub-threshold blip is reported, not paged, and the suppression is
+made visible — `pendingPersistSeconds` is added to the OneUptime payload and a `pending=Ns`
+annotation to the WorkerJobs summary — so a suppressed episode is never silent.
+
 ## Reading the windowed 5xx rate from CloudWatch (durable AWS reference)
 
 - **EB application-request metrics are PER-INSTANCE ONLY.** `ApplicationRequestsTotal`,
@@ -214,6 +262,8 @@ the `probe` token; this is a OneUptime config change the developer applies.
   `min5xxPercent`; absolute at `min5xxAbsolute-1` vs `min5xxAbsolute`; `requests` just below
   `minRequests`; empty/zero window; mixed 5xx + non-5xx cause) — deferred pending a harness.
 - **Deploy fails without degrading health** — still not caught (accepted residual).
+- **`ElasticBeanstalkHealth()` is long** and grew with the persistence gate — php-reviewer
+  (0 critical) flagged it as a decomposition candidate; deferred pending the missing harness.
 
 ## Gotchas / known issues
 
@@ -235,6 +285,21 @@ the `probe` token; this is a OneUptime config change the developer applies.
   record its value in a doc.
 
 ## Change history
+- 2026-08-29 — Added a **persistence gate on the non-5xx page**: a non-5xx `Degraded`/`Severe`
+  no longer pages on the first snapshot — it must persist ≥ `nonFivexxPersistMinutes` (new
+  optional trailing param, default 5; existing `CronJobs.parameters` without the key still
+  run) before paging. Episode age comes from EB `describeEvents` via new never-throwing helper
+  `fetchNonOkDepartureAgeSeconds()` (most recent "from Ok to …" marks episode start; events
+  are newest-first so the first match wins; TIME-bounded `StartTime = now − (persistWindow +
+  HEALTH_EVENTS_LOOKBACK_BUFFER_SECONDS=600)`, `MaxRecords=100`, so an episode older than the
+  window returns `null` → the caller pages — correct fail-safe). Sub-threshold blips are
+  reported not paged, with `pendingPersistSeconds` (OneUptime payload) + `pending=Ns`
+  (WorkerJobs summary) so suppressions stay auditable. The windowed 5xx gate is unchanged.
+  Rationale: the brief self-healing "no-cause reduced health" flips on low-traffic 2-instance
+  envs are EB Enhanced Health flappiness (a single missed ALB `/health` check), not app bugs
+  — distinct from the busy-env PHP-`/health`-starvation cause. worker2-only; the param is
+  optional-with-default so no `_underscore`/dbchanges2 change. Not yet committed/deployed.
+  (jcardinal)
 - 2026-08-25 — Replaced the EB `ApplicationMetrics` 5xx gate (a ~10s window that false-paged
   "100% failing" on a lone 500 in a near-empty bucket — real worker-production incident: 16
   500s / 516 req = 3.1% over 17 min) with a **CloudWatch windowed request-rate gate**. Reads

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.695",
+  "version": "1.0.697",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",