toga-ai 1.0.695 → 1.0.696

package/knowledge/2.0/apps/api2/INDEX.md

@@ -25,5 +25,5 @@
 | [V2 request retry (Prong A) — deadlock AND retriable unique-key race, route-scoped in-process replay](features/v2-deadlock-retry.md) | api2's front controller ("**Prong A**") can **detect a retriable concurrency failure and replay the whole request in-process**, so a transient collision self-re | api2/Controller/Index.php, api2/Component/Api/V2/V2.php, _underscore/Database.php, _underscore/Query.php, _underscore/Model/Compass/AdvanceShippingNotice.php |
 | [V2 REST query contract (params, where grammar, encoding, ACL behavior)](features/v2-rest-query-contract.md) | What an **HTTP client** has to get right to query the Toga v2 REST API: which query params are recognized, the exact `where` grammar, how the query string is (n | api2/Component/Api/V2/V2.php |
 | [V2 reverse hasMany collections must be named in the fetch fields whitelist](features/v2-reverse-hasmany-fields-whitelist.md) | In the V2 JSON engine, a **reverse hasMany** relationship — the collection of child records that foreign-key back to a parent (e.g. | api2/Component/Api/V2/V2.php |
-| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
+| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php, api2/.platform/hooks/prebuild/git.sh, api2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.ebextensions/git.php, api2/.ebextensions/git.sandbox-dev.json |
 | [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/composer.json, api2/composer.lock, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |

package/knowledge/2.0/apps/api2/architecture.md

@@ -197,6 +197,12 @@ always exits 0). See
 [the worker2 reference implementation](../worker2/features/alb-target-group-auto-registration.md) —
 api2's copy is the **unhardened original** and must be brought up to it.
 
+**Deploy policy (2026-08-28):** the three production envs (`api-production-1/2/3`, 2 instances each)
+run **Rolling / `BatchSize` 1** (was `AllAtOnce`/100%), target-group `HealthyThresholdCount` 2 — so
+a deploy keeps ≥1 healthy backend and no longer emits the ELB-502 blip. Each CodePipeline
+auto-deploys on a push to `_production`. See
+[the CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
+
 ## CI — commit message policy
 
 `.github/workflows/true-devteam-requirements.yml` enforces **`TRUE-{ticket}: {Subject}`** —
@@ -275,6 +281,27 @@ probe was starved of a PHP-FPM worker — do **not** chase client-traffic 5xx. *
 liveness from PHP-FPM worker availability so a slow-request burst can no longer flap it — see
 [health-check endpoint](features/health-check-endpoint.md).
 
+**A fourth cause (deploy-triggered) — a burst of ELB `502`s when both instances restart at once.**
+Confirmed on `api-production-1` (2026-08-28 23:51 UTC deploy: **43 × 502 in one minute**, the only
+ELB 5xx in a 7-hour window). Under the old `AllAtOnce`/100% deploy policy, every deploy to a
+2-instance env restarts Apache/PHP-FPM on **both** instances simultaneously, so in-flight requests
+have their connections dropped mid-response → a burst of **ELB-generated HTTP 502**. EB flags "X%
+of requests to the ELB are failing with HTTP 5xx" `Degraded` ~3–4 min later and recovers in ~5 min.
+Distinguishing fingerprint vs. a real app failure: it is **502** (a dropped backend connection),
+**not 503** (no healthy backend) or **504** (backend timeout); **`HTTPCode_Target_5XX_Count` (app
+5xx) = 0**; **`HealthyHostCount` never drops below 2**; and there is **NOTHING in `Logs.Api`** —
+api2 logs a request only *after* the response, and a 502 never completes one, so the request leaves
+no log row. Diagnosis path: CloudWatch `HTTPCode_ELB_502/503/504_Count` vs
+`HTTPCode_Target_5XX_Count`, plus EB `describe-events` (a version change) and CodePipeline
+`list-pipeline-executions` — because a push to `_production` auto-deploys all three prod tiers, "we
+didn't deploy" can be false (see
+[the CodePipeline deploy workflow](workflows/codepipeline-codeconnections-deploy.md)).
+**Rule:** an EB "% 5xx" `Degraded` with an **empty `Logs.Api` and 0 target-5xx** right after a
+deploy is a deploy-restart 502 artifact, not an application bug — check the ELB-5xx breakdown and
+the deploy/version timeline before hunting app bugs. **Fixed 2026-08-28:** all three prod envs moved
+to **Rolling / batch-size 1** (≥1 healthy backend throughout a deploy), which eliminates the blip —
+config + tradeoffs in the [deploy workflow](workflows/codepipeline-codeconnections-deploy.md).
+
 ## Known issues / accepted risks
 
 Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -352,6 +379,15 @@ they are the known sharp edges. Do not re-discover these from scratch.
   single-caller branches in `V2.php` as unverified until exercised directly.
 
 ## Change history
+- 2026-08-28 — Recorded a **fourth EB-`Degraded` cause — the deploy-restart ELB 502**: under the old
+  `AllAtOnce`/100% policy a deploy restarts both instances of a 2-instance env at once, dropping
+  in-flight connections → an ELB-502 burst (43 in one minute on `api-production-1`, 2026-08-28) that
+  EB flags as "X% 5xx" Degraded ~3–4 min later. Fingerprint vs. an app bug: **502** (not 503/504),
+  **`HTTPCode_Target_5XX_Count`=0**, `HealthyHostCount` never < 2, and **nothing in `Logs.Api`** (a
+  502 never completes a PHP response, which api2 logs only after). Diagnose via the ELB-5xx breakdown
+  + `describe-events`/`list-pipeline-executions` — a push to `_production` auto-deploys all three prod
+  tiers, so "we didn't deploy" can be false. Fixed by moving to Rolling/batch-1 (deploy config +
+  policy tradeoffs captured in the CodePipeline deploy workflow). (jcardinal)
 - 2026-08-28 — Recorded the third EB-`Degraded` cause: a no-stated-cause "reduced health" flip on `api-production-1` is the **PHP-served `/health` probe starved of a PHP-FPM worker** by off-peak bursts of slow failing Compass ASN POSTs (17–63 s, ~100% 500) — invisible in ALB `RequestCount`/`5xx` and never dropping `HealthyHostCount` (flap < 75 s deregistration). Correlated a 06:00–06:08 CST ASN burst to the 11:05 UTC Severe flip. Noted the fix (static-file `/health` decouples liveness from PHP-FPM) and updated the "Don't break `/health`" rule to the static-file form. (jcardinal)
 - 2026-08-26 — Documented and fixed a `/v2/auth/public` OOM: a present-but-null client `transactionId` (`{"transactionId": null}`) flowed into the request-logger's `_Model::load()` dup-check; because `Logs.Api.transactionId` is `FIELD_CHAR`, a null value collapses the WHERE and full-table-scans prod `Logs.Api` → memory exhaustion (the `_underscore` collapsed-WHERE mechanism). V2 only auto-uuids an *absent* transactionId (~L543), so a present null fell through (~L553 read → null). Fix (commit `36ef31f`): coerce null/empty transactionId → uuid in the auth block, and skip `load()` in the final logger dup-check (~L2277) when null/empty. Added the "don't blame the latest deploy — check first-occurrence" triage note. (jcardinal)
 - 2026-08-25 — **Resolved the auto-generated `Api.transactionId` 1062 collision** (was the 2026-07-28

package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md

@@ -6,12 +6,13 @@ project: API
 client: shared
 type: workflow
 status: active
-updated: 2026-08-25
+updated: 2026-08-28
 owners: ["jcardinal", "mhammontree", "bala", "apeterson"]
 files:
   - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
   - api2/ebs/register_instance_to_shared_application_load_balancer.php
   - api2/.platform/hooks/prebuild/git.sh
+  - api2/.platform/hooks/postdeploy/015_install_composer.sh
   - api2/.ebextensions/git.php
   - api2/.ebextensions/git.sandbox-dev.json
 related:
@@ -62,8 +63,11 @@ are actually changing before you merge anything:
   `_<environment>`). Consequence: **a framework-only change requires REDEPLOYING THE CONSUMING
   TIER even when `api2` itself has zero changes** — an instance **restart will not pick it up**,
   because the clone happens in prebuild.
-- **`api2` has no deploy workflow.** Its only GitHub Action is the `TRUE-#####:` commit-message
-  linter, so **EB deploys are manual**. Do not wait for CI to ship a framework change.
+- **`api2` has no GitHub *Actions* deploy workflow** — its only GitHub Action is the
+  `TRUE-#####:` commit-message linter. **The production tiers, however, auto-deploy via AWS
+  CodePipeline on every push to `_production`** (a different system from GitHub Actions — see
+  *Production deploy: auto-trigger + rolling policy* below). Non-pipeline-triggered tiers are
+  still deployed by hand; don't wait for a GitHub Action to ship a framework change.
 
 ### ⚠ The `_underscore` SOURCE ACTION has its own branch trigger — and a green pipeline hides a wrong one
 
@@ -105,6 +109,64 @@ beta env actually received and returned when Sentry isn't wired up. (The Rate AI
 scans this same `Logs_Rate.Api` for failed outbound calls — see
 `clients/rate/features/aig-contract-creation.md`.)
 
+## Production deploy: auto-trigger + rolling policy (2026-08-28)
+
+### Production auto-deploys on every push to `_production` — a merge IS a deployment
+The `api-production-1` (us-east-1), `api-production-2` (us-west-2), and `api-production-3`
+(eu-west-1) CodePipelines each **auto-deploy on every push to branch `_production`** — a
+**WebhookV2** trigger on the CodeConnections GitHub connection, watching source repos
+`agilantsolutions/_underscore` and `agilantsolutions/api2`. There is **no separate manual deploy
+step for production**: pushing or merging to `_production` **is** a production deployment to all
+three regions. (This is why deploying a fix can itself page a moment later — see the
+deploy-restart 502 below.)
+
+**Triage consequence:** "we didn't deploy" can be false — any push to `_production` is a deploy.
+Correlate an incident against the actual pipeline runs, not just remembered manual actions:
+```bash
+aws codepipeline list-pipeline-executions --pipeline-name api-production-1 --region us-east-1
+aws elasticbeanstalk describe-events --environment-name api-production-1 --region us-east-1  # version-change events
+```
+
+### ⚠ An EB `Degraded` right after a deploy = the deploy-restart 502
+Under the old `AllAtOnce`/100% policy every deploy restarted Apache/PHP-FPM on **both**
+instances of a 2-instance env simultaneously, dropping in-flight connections → a burst of
+ELB-generated **HTTP 502** that EB flags as an "X% 5xx" `Degraded` ~3–4 min later (recovers in
+~5 min). This is a **deploy artifact, not an application failure** — the full diagnostic
+fingerprint (502 not 503/504, app `HTTPCode_Target_5XX_Count`=0, `HealthyHostCount` never < 2,
+**nothing in `Logs.Api`** because a 502 never completes a PHP response) lives in
+[api2 architecture — EB "Degraded" health](../architecture.md). The rolling policy below
+eliminates it.
+
+### Deploy policy: Rolling, batch size 1 (changed 2026-08-28)
+All three production envs (app **"API"**, 2 instances each) were switched from
+`AllAtOnce`/`Percentage`/`100` to **`DeploymentPolicy=Rolling`, `BatchSizeType=Fixed`,
+`BatchSize=1`**, and the target group **`HealthyThresholdCount` dropped 3 → 2** (health path
+`/health`, `HealthCheckInterval` 15 s, `UnhealthyThresholdCount` 5 unchanged). Rolling batch-1 on
+a 2-instance env deploys in **2 batches with ≥1 healthy backend throughout**, so a deploy no
+longer drops in-flight requests → **no more deploy-restart 502 blip**; the 3→2 threshold shaves
+one 15 s interval (~45 s → 30 s) of per-batch health settle.
+
+- Applied via a **config-only `update-environment`** (AWS CLI, per env) — no instance
+  replacement, no downtime; **takes effect on the NEXT deploy**.
+- Namespaces: `aws:elasticbeanstalk:command` (`DeploymentPolicy` / `BatchSizeType` /
+  `BatchSize`); `aws:elasticbeanstalk:environment:process:default` (`HealthyThresholdCount`).
+- Trade-off: Rolling is slower end-to-end than the old ~31 s `AllAtOnce`, and capacity briefly
+  drops to 1/2 instances per batch (fine at current traffic).
+
+### Deploy-policy guidance (don't reach for immutable expecting speed)
+- **Immutable does NOT reduce deploy time.** It is the **slowest** policy — it boots a fresh
+  instance batch, health-checks it, then swaps, and temporarily **doubles** the instance count.
+  It does fully eliminate deploy 5xx and gives instant rollback, but it is not "faster."
+- **`AllAtOnce` is fast (~31 s) precisely BECAUSE both instances restart at once** — which is
+  what causes the 502s. You cannot have "as fast as AllAtOnce" **and** "zero 502s."
+- **Rolling batch-1 is the sweet spot** for a 2-instance env.
+- **`RollingWithAdditionalBatch`** is the option only if zero capacity loss during a deploy is
+  ever required (it adds a temporary instance; slower and costlier).
+- The real deploy-**time** levers are independent of the policy: the health-check settle
+  (`HealthCheckInterval` × `HealthyThresholdCount`) and per-instance postdeploy work — api2 runs
+  `composer install` on **every** deploy via
+  `.platform/hooks/postdeploy/015_install_composer.sh` (a caching / pre-bake candidate).
+
 ## Steps
 1. A push to the watched branch (e.g. `_stage`) triggers the pipeline.
 2. The Source stage uses `CodeStarSourceConnection` actions, each referencing:
@@ -233,6 +295,20 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 ```
 
 ## Change history
+- 2026-08-28 — Recorded the **production deploy trigger + policy change**: the `api-production-1/2/3`
+  CodePipelines **auto-deploy on every push to `_production`** (WebhookV2 on the CodeConnections
+  connection, repos `_underscore` + `api2`), so a merge to `_production` **is** a three-region prod
+  deploy — correlate incidents against `list-pipeline-executions`, not remembered manual actions.
+  Switched all three 2-instance envs from `AllAtOnce`/100% to **Rolling / Fixed / BatchSize 1** and
+  dropped the target-group **`HealthyThresholdCount` 3 → 2** (config-only `update-environment`,
+  effective next deploy) to eliminate the **deploy-restart 502 blip** (both instances restarting at
+  once dropped in-flight requests → ELB 502 → EB "X% 5xx" Degraded). Added deploy-policy guidance:
+  immutable is the **slowest** policy (not faster) and doubles instances; `AllAtOnce` is fast only
+  because it restarts both at once (the 502 cause); Rolling batch-1 is the sweet spot;
+  `RollingWithAdditionalBatch` only if zero capacity loss is needed; the real deploy-time levers are
+  the health-check settle and the per-deploy `015_install_composer.sh` `composer install`. Corrected
+  the "EB deploys are manual" note — that is only true of the absent GitHub *Actions* workflow; prod
+  deploys via CodePipeline are automatic. (jcardinal)
 - 2026-08-25 — Documented that the Source stage's **`_underscore` action has its own AWS-side branch
   trigger**, independent of `git.<env>.json` and invisible in the repo — a second way to silently run
   the wrong framework branch. `ApiSandboxClient` ran `_production` from a mis-set source action while

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.695",
+  "version": "1.0.696",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",