toga-ai 1.0.908 → 1.0.909

package/knowledge/2.0/apps/worker2/architecture.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-09-25
+updated: 2026-09-28
 owners: [jcardinal, ajean, rgirish]
 files:
   - worker2/Controller/Index.php
@@ -151,6 +151,8 @@ For **programmatic** enqueue via `_Worker::runTask()`, the MySQL-first rule was
 
 ## Gotchas
 
+- **Shrinking `WorkerJobs.output` MEDIUMTEXT → TEXT (planned, not yet done).** TEXT holds 65,535 **bytes** (utf8mb4 = up to 4 bytes/char), not chars. Trim byte-safe before the ALTER, repeating until 0 rows: `UPDATE WorkerJobs SET output = LEFT(output, CHAR_LENGTH(output) - (LENGTH(output) - 65535)) WHERE LENGTH(output) > 65535 LIMIT 1000`. After the change, every producer must truncate first or the result UPDATE fails with 1406 Data too long (strict mode) and the job result is lost: PHP check-out (both paths in `Controller/Index.php`) and the JobScheduler watchdog Lambda UPDATE (repackage + redeploy it). The `MODIFY` must restate the column COMMENT. On MySQL 8 the type change rebuilds the table.
+
 - **⚠ OBSERVED `Core.WorkerJobs` retention is ~30 days, not 90 — unexplained.** Measured 2026-09-15 in prod: `MIN(dtCreated)` across the **whole** table is **2026-08-16 02:00:46** (790,597 rows). Either the cleanup cron does not behave as described, or something else prunes the table. **Not investigated** — no cause is asserted.
 - **A missing `WorkerJobs` row is NOT proof the job never ran.** Any "there is no job for this record, so the webhook never fired" conclusion is invalid past the ~30-day floor. That exact wrong call was made and retracted on 2026-09-15. Same failure family as *an empty log table is not proof a code path never ran*.
 - **The watchdog does not stop the PHP process** — an over-budget action keeps running past a false failure row and typically loses its result UPDATE to MySQL 2006 (prod job 1075223). See JobScheduler Part B above.

package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-09-28
 owners: [jcardinal, bala]
 files:
   - worker2/Component/Aws/Workloads/Workloads.php
@@ -82,6 +82,21 @@ Each `WorkloadsRuntime` role has broad service permissions (`elasticbeanstalk:*`
 
 The helper adoption was captured earlier: the prior inline STS/credential code and the `HEALTH_ASSUME_ROLE_NAME` / `HEALTH_ASSUME_ROLE_SESSION` constants were removed in favor of `_Component_Aws_Workloads` + the `[cloud]` config keys. The prior `CheckBeanstalks()` action was renamed `ElasticBeanstalkInstances()` (dbchanges2 `Core/2026-08-10c` updates the CronJobs action name; `Core/2026-08-10d` seeds the parameterized `ElasticBeanstalkHealth` cron row, `isActive=0`, and carries the `environments`-map sample parameters).
 
+### Second consumer — ElasticBeanstalkInstances (auto-terminate stuck instances)
+
+`_Worker_Infrastructure_CloudWatch::ElasticBeanstalkInstances(?regions)` — prod `Core.CronJobs` id 39, every 10 min. Rewritten 2026-09-28 (not yet verified by a real prod run).
+- **Why:** the old version used the aws CLI via `_Cloud::$preCommands` + the legacy static `[cloud]` key, which cannot see 868's EB environments → failed **100% of runs** since 2026-08-29 (`CloudWatch: No environments returned for region us-east-1`). Auto-terminate never worked.
+- **Now:** `assumeCredentials()` once, SDK `ElasticBeanstalkClient` + `Ec2Client` per region (same shape as ElasticBeanstalkHealth).
+- Account is const `INSTANCES_AWS_ACCOUNT_ID = 654654170868`, **not overridable** (it terminates). Optional `regions` must be in `INSTANCES_ALLOWED_REGIONS` (us-east-1, us-west-2, eu-west-1) — region goes into the SDK endpoint host, so allowlist it.
+- Returns JSON `{isSuccess, awsAccountId, environmentCount, terminatedCount, alarmCount, errorCount, terminated[], alarms[], errors[], environments[]}`. Throws only on: region not allowed, assume-role fails, or no region readable. Unhealthy instances no longer throw (developer choice).
+- Guards: 240s budget (`INSTANCES_TIME_BUDGET_SECONDS`; unreached envs go to `errors`); max 3 terminations per run; max floor(50%) of an env's instances but always at least 1 (a single-instance env's only instance may be terminated — EB replaces it; developer choice). Capped / over-budget items go to `errors`, never silently skipped.
+- Termination email kept (private `notifyInstanceTerminated` → Notification/EmailTemplate/Send).
+- `_Cloud::terminateInstance()` in `_underscore` now has no known callers in worker2/_underscore (left in place).
+
+### Terminate guard on 868 WorkloadsRuntime
+
+Applied 2026-09-28 (AWS CLI) to inline policy `WorkloadsRuntimePermissions` in 654654170868: statement `DenyTerminateNonBeanstalk` — Deny `ec2:TerminateInstances` on `*` when `aws:ResourceTag/elasticbeanstalk:environment-name` is `Null`. Any caller of WorkloadsRuntime can terminate **only EB-managed instances**. The other two accounts' roles do not have this statement.
+
 ### Legacy static key — being retired
 
 The legacy `[cloud]` static access key is a **personal IAM user in account 502614707982**. Being retired for new code but **still used by older code paths** — ZoomInfo, VipSupport, and the `_Cloud` CLI path. **Retrofit of that existing code is deliberately deferred** (developer decision, this session): document the new approach; do not migrate old code yet. New code must not adopt the static key.

package/knowledge/2.0/apps/worker2/features/tracking-status-refresh.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: ["bala"]
+updated: 2026-09-28
+owners: ["bala", "jcardinal"]
 files:
   - worker2/Worker/Sync/TrackingNumbers.php
   - _underscore/Component/Library/Carriers/Fedex/Fedex.php
@@ -156,6 +156,8 @@ bookkeeping columns.
 
 ## Gotchas
 
+- **Every `c_` refresh column must exist in every `Client_*` DB.** One missing column fails that client's `RefreshClient` every run: Adyen failed 144 runs (2026-09-21 14:00 → 2026-09-22 13:50) on `Unknown column 'TrackingNumbers.c_trackingCheckFailures'` until the column was added. Check the fan-out when a new client is provisioned.
+
 ### ⚠ GO-LIVE GATE — worker2 → api2 hard-crashes on PHP 8.5
 Writes go through api2 PUT, and a worker2 → api2 call **hard-crashes the request on PHP 8.5 (OpenSSL
 3.5.5): a process-level 500 with no catchable trace, and the shutdown handler never fires.** The prod

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.908",
+  "version": "1.0.909",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",