toga-ai 1.0.1002 → 1.0.1003

package/knowledge/1.0/apps/library/INDEX.md

@@ -6,6 +6,7 @@
 | [Address Validation Gateway (App_Api_OfficeDepot::validateAddress + USPS fallback)](features/address-validation-gateway.md) | The shared 1.0 address-validation gateway (`App_Api_OfficeDepot::validateAddress` → USPS fallback); open when standardizing/verifying a typed address before sav |
 | [API Request Logging & Retention (App_ApiTransaction → Logs.API / Logs_<Tenant>)](features/api-request-logging-and-retention.md) | Where every outbound API call is logged (1.0 `Logs.API` vs the 2.0 `Logs_<Tenant>` cluster), how the routing is chosen, and the 30-day retention trim; open befo |
 | [Where a new App_ class goes — the app/ folder IS a behavioral contract](features/app-class-placement-base-contracts.md) | Where a new `App_` class goes — each `app/` folder's base class carries real inherited behavior; open before creating any class under `app/model|client|api`. |
+| [1.0 AWS credentials — App_Cloud assumes WorkloadsRuntime (no IAM keys)](features/app-cloud-aws-credentials.md) | How every 1.0 AWS call (App_Cloud CLI + SDK, deploy scripts, `setup_git_libraries.php`) gets credentials by assuming `WorkloadsRuntime` from the EC2 instance ro |
 | [App_Sso — Reusable 1.0 SSO Initiation (SP-initiated SAML via saml.togahub.com)](features/app-sso-initiation.md) | Reusable 1.0 SP-initiated SAML SSO via `saml.togahub.com` (`App_Sso::initiate`); the canonical doc to follow for any "add SSO to `<1.0 app>`" request. |
 | [Carrier detection from a tracking number (App_Misc) — patterns, check digits, shipping-method fallback](features/carrier-detection-from-tracking-number.md) | How 1.0 decides which carrier a tracking number belongs to (`App_Misc`, `library/app/misc.php`): certain vs shared-shape patterns, the FedEx check digit, and th |
 | [Cron Execution Monitoring (App_Framework check-in/out → CronJobExecutions)](features/cron-execution-monitoring.md) | 1.0 cron check-in/out lifecycle via `App_Framework::cronInitialization()`/`cronFinished()` → `Common.CronJobExecutions`; open to monitor a 1.0 cron or read its |

package/knowledge/1.0/apps/library/features/app-cloud-aws-credentials.md

@@ -0,0 +1,63 @@
+---
+title: "1.0 AWS credentials — App_Cloud assumes WorkloadsRuntime (no IAM keys)"
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-10-12
+owners: [jcardinal]
+files:
+  - library/app/cloud.php
+  - library/app/systemmonitor/rdsproductioncluster.php
+  - worker/ebs/cron.worker.php
+  - toga/ebs/register_instance_to_shared_application_load_balancer.php
+  - tools/ebs/check_s3fs.php
+related:
+  - ../../../../2.0/apps/worker2/features/cross-account-aws-access.md
+  - ../../../../2.0/apps/_underscore/features/cloud-s3-helpers.md
+  - ../../tools/workflows/deploy-to-elastic-beanstalk-al2023.md
+  - ../architecture.md
+---
+
+How every 1.0 AWS call (App_Cloud CLI + SDK, deploy scripts, `setup_git_libraries.php`) gets credentials by assuming `WorkloadsRuntime` from the EC2 instance role; open before touching 1.0 AWS access or EB deploy hooks.
+
+## Summary
+
+No long-lived IAM keys anywhere (TRUE-82705, 2026-10-12 — the old 502 IAM-user key leaked and was deleted). 1.0 code assumes role `WorkloadsRuntime` via STS using the EB EC2 instance role. Default account = legacy **502614707982** (same access the deleted key had; `toga-private`, `toga-public`, `agilant-as2`, `asifiles` live there). Role model, accounts, trusts, permissions: [cross-account AWS access](../../../../2.0/apps/worker2/features/cross-account-aws-access.md#the-workloadsruntime-role-model). 2.0 twin: [`_Cloud` credentials](../../../../2.0/apps/_underscore/features/cloud-s3-helpers.md#credentials).
+
+## How it works
+
+### App_Cloud (`library/app/cloud.php`)
+- `App_Cloud::getCredentialProvider($awsAccountId = AWS_ACCOUNT_ID_LEGACY)` → memoized SDK provider per account. Session name `library-app-cloud`.
+- STS client credentials = `CredentialProvider::memoize(CredentialProvider::instanceProfile())` — **never the default chain** (see Gotchas).
+- Local dev: if `AWS_PROFILE` is set, that profile (SSO) is used directly, no assume.
+- `initialize()` rebuilds `App_Cloud::$preCommands` on **every** call: exports `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` for the CLI. Refreshes when < 900 s left (`AWS_CLI_CREDENTIALS_MIN_SECONDS_LEFT`). Long crons must call `initialize()` (any App_Cloud CLI method does) — never cache `$preCommands`.
+- `addSqsMessage()` assumes into the account named in the queue URL (1.0 worker queue + 2.0 `WorkerProductionQueue` are in 868).
+- `library/app/systemmonitor/rdsproductioncluster.php`, `worker/crons/infrastructure/cloudwatch.php`, `worker/crons/maintenance/migrate_s3_files.php` use App_Cloud — no own credentials.
+
+### Standalone deploy scripts (no library loaded)
+`worker/ebs/cron.worker.php`; `ebs/register_instance_to_shared_application_load_balancer.php` in `toga`, `store`, `vision`, `api`, `toga2-commerce`; `tools/ebs/check_s3fs.php`. Pattern:
+1. `unset` every `AWS_*` env var, then `aws sts assume-role --role-arn arn:aws:iam::502614707982:role/WorkloadsRuntime ...`.
+2. Export the 3 values with `escapeshellarg()` as a command prefix.
+3. On failure: `error_log` and continue — never abort a crontab install or deploy.
+`cron.worker.php` also validates the instance id (`/^i-[0-9a-f]{8,17}$/`) before terminate.
+
+### `s3://asifiles/system/setup_git_libraries.php` (not in any git repo)
+Downloaded by **every** legacy EB deploy (`.ebextensions/020_setup_git_libraries.config`, `wget`); clones `library`, `_underscore`, etc. after a CodePipeline lookup. Uses the same assume-role pattern (session `setup-git-libraries`). Edit = download, change, re-upload **with `--acl public-read`** (wget is anonymous).
+
+### 1.0 worker deploy
+CodePipeline `Worker` (502, us-west-2) sources `_production` of **both** `agilantsolutions/worker` and `agilantsolutions/library` → EB `agilant-worker`. A library-only merge to `_production` redeploys the worker.
+
+## Gotchas
+
+- **Never use the default credential chain for the STS client.** Old EB env properties (`agilant-worker` has `AWS_ACCESS_KEY_ID`/`AWS_SECRET_KEY`) still carry the dead key, and the default chain picks env vars first → `InvalidClientTokenId`. Always `instanceProfile()`.
+- **Never put `$cmd` in an exception or log message.** It contains `$preCommands` = live temporary credentials.
+- **CLI temp creds are visible in argv (`ps`).** Open follow-up: CLI profile with `credential_source=Ec2InstanceMetadata`.
+- **`setup_git_libraries.php` failing is silent.** When the key died its pipeline lookup failed and **library was not updated on deploy** — the old library kept running. If a library change "didn't deploy", check this script first. It was public-read with the key inside — a likely leak path; never put a credential in it.
+- Verify assume-role in CloudTrail (`AssumeRole` on `WorkloadsRuntime`, by session name: `library-app-cloud`, `setup-git-libraries`, `register-instance`, `worker-deploy`).
+- Open, not fixed (developer owns): remove `AWS_ACCESS_KEY_ID`/`AWS_SECRET_KEY` from `agilant-worker` EB env properties.
+
+## Change history
+- 2026-10-12 — Created: removed the static IAM key from all 1.0 code/config; App_Cloud, deploy scripts and `setup_git_libraries.php` assume `WorkloadsRuntime` (TRUE-82705) (jcardinal)

package/knowledge/1.0/apps/library/features/error-capture-1-0.md

@@ -6,7 +6,7 @@ project: Library
 client: shared
 type: feature
 status: active
-updated: 2026-09-18
+updated: 2026-10-12
 owners: ["jcardinal", "mhammontree"]
 files:
   - library/app/error/capture.php
@@ -121,10 +121,12 @@ Two things are specifically 1.0:
 - **⚠ A grouped `Logs.Issue` row can show a STALE error id — read the live api2 transaction log for the current error.** Two facts combine to mislead: (1) `Event.errorMessage` is **`varchar(255)`**, so a long API response is **truncated** — often before the trailing error id; and (2) `Logs.Issue.subject` / `Issue.errorMessage` hold the **first-occurrence** text, so a long-lived issue keeps showing the error id from whenever it was first seen even after the per-call error has changed. Observed 2026-08-17: issue `3H` still displayed `EO-1` / `3G-6` (the original undefined-method fatal) while the current per-call failure was actually `EV-11` (an FK 1451 on a DELETE). **To see what is failing *now*, read the live api2 transaction log (`Logs_<client>.Api.responsePayload`) for the specific call — not the grouped `Issue` row.**
 - **Any PHP warning inside capture code would kill the request.** 1.0's `App_Error::handleError()` routes **every** PHP warning into `handleException()`, which renders the red box and calls `exit()` — so a single warning raised while capturing (an unreachable IMDS endpoint is the everyday case: `file_get_contents` warns before returning `false`) terminates the request or cron mid-run while reporting an unrelated error. 2.0's equivalent handler *throws*, which a `catch` absorbs. `captureException()` therefore sets `App_Error::setThrowExceptionsEnabled(false)` for its own duration and restores it in `finally`. This disables the **handler**, not real throws — `App_Query` raises a plain `throw new Exception` on DB errors, so genuine failures still reach the catch. **This is a general 1.0 trap for any code that must not escalate**, not just error capture.
 - **No typed properties anywhere in `library/`.** 1.0 runs **PHP 7.2**; a deploy failed with `syntax error, unexpected '?'` on `private static ?string $memoryReserve`. Nine typed properties across `error.php`, `error/capture.php`, and `exception/business.php` were converted to untyped with `@var` docblocks. Parameter and return types are fine (7.0/7.1).
-- **An early `return` can make a fallback unreachable.** Two bugs in the first EB-environment attempt: a `return` when `get-config` was not executable skipped the `describe-tags` fallback entirely — i.e. the fallback was dead on the only platform that needed it — and the CLI call omitted `App_Cloud::$preCommands`, so it did not authenticate the way every other 1.0 AWS call does. **Unverified:** whether that IAM identity actually holds `ec2:DescribeTags`.
+- **An early `return` can make a fallback unreachable.** Two bugs in the first EB-environment attempt: a `return` when `get-config` was not executable skipped the `describe-tags` fallback entirely — i.e. the fallback was dead on the only platform that needed it — and the CLI call omitted `App_Cloud::$preCommands`, so it did not authenticate the way every other 1.0 AWS call does. (`$preCommands` now assumes `WorkloadsRuntime`, which has `ec2:*` — see [1.0 AWS credentials](./app-cloud-aws-credentials.md).)
 - **Fingerprint scheme changes re-group existing issues once.** Moving to file+function frames and the class+entryPoint fallback changes hashes, so open issues re-fingerprint on first occurrence after deploy. Deliberate; merge old hashes onto the curated Issue with the Tools merge-fingerprint box.
 - **⚠ The legacy pre-Sentry handler dumps the entire config on any error it renders.** `library/app/DELETE_DOA_PRE-SENTRY_error.php` is a dead-named but still-reachable handler whose `htmlBacktrace()` does `print_r($entry['args'])` for **every** frame. On 2026-08-05 a real error page on **https://towfoundation.togaview.com** rendered ~26 pages containing the whole config array — production DB host/user/password, SES SMTP credentials, NetSuite credentials, Payeezy API key/secret/merchant token, SSL Store token — plus ticket PII. The current `library/app/error.php` is safe (it dumps only under `App_Registry::inDevMode()`), but **both** handlers gate on `inDevMode()`, so the affected host was **evaluating as dev mode in production**. Two actions: (1) verify `inDevMode()` per host in production before trusting any error page not to leak, and (2) delete the legacy handler. Rotation of every exposed credential **was escalated to Jeff on 2026-08-05; confirmation that it was completed is still owed** — treat those credentials as compromised until confirmed. (No values are recorded here or anywhere in the KB.)
-- **Pre-existing committed secrets (location only — not fixed, out of scope).** `library/app/cloud.php` holds a hardcoded **AWS access key id and secret access key** as class constants/statics (~lines 23–24), used by `App_Cloud::$preCommands` for every 1.0 AWS CLI call. Unrelated to this work. Treat as compromised: move to SSM like the other parameters and rotate.
+
+## Change history
+- 2026-10-12 — Removed the committed-AWS-key gotcha: key deleted, `App_Cloud` assumes `WorkloadsRuntime` (TRUE-82705) (jcardinal)
 
 ## Related
 

package/knowledge/1.0/apps/tools/features/cloudfront-client-setup.md

@@ -6,7 +6,7 @@ project: Tools
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-12
 owners: [jcardinal]
 files:
   - tools/_/app/devops/cloudfront.php
@@ -84,7 +84,10 @@ Behind Tools SSO; the nav action and page are gated to the **`Development Team`*
 - **Alternate domain names require a covering ACM certificate.** A wildcard cert covers only a single label — `*.togasupply.com` does not cover `a.b.togasupply.com`. Ensure the distribution has a cert covering the FQDN before adding it.
 - **Route 53 CloudFront alias records use the fixed `HostedZoneId` `Z2FDTNDATAQYW2`** (an AWS global constant for all CloudFront alias targets — not the zone's own ID).
 - **1.0 promotes undefined-index warnings to a thrown `ErrorException`** that would abort the whole "never fatally abort" run — all AWS-response array access is guarded. The SDK client-builder `catch` is widened to **`\Throwable`** because the constructor can throw `InvalidArgumentException`, not just `AwsException`.
-- **Credential exposure caveat.** This tool wields the prod IAM key, which amplified the impact of the separate config-file web-root exposure fixed this session — see the 1.0 back-end security standard and the Tools architecture Gotchas. Every secret in `config.production.ini` still owes rotation because the file was publicly readable.
+- **Credential exposure caveat.** The static IAM key this tool used is deleted (2026-10-12); AWS access now comes from the instance role — see [1.0 AWS credentials](../../library/features/app-cloud-aws-credentials.md). Every other secret in `config.production.ini` still owes rotation because the file was publicly readable (Tools architecture Gotchas).
+
+## Change history
+- 2026-10-12 — Static IAM key deleted; caveat now points at the instance-role credentials doc (TRUE-82705) (jcardinal)
 
 ## Related
 - [Tools Architecture](../architecture.md)

package/knowledge/1.0/apps/worker/architecture.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-09-16
+updated: 2026-10-12
 owners: [jcardinal, sking, bala]
 files:
   - worker/index.php
@@ -171,4 +171,4 @@ Container commands set up the box per deploy: apache tuning, force HTTP→HTTPS,
 - **Timezone is `America/Chicago`** everywhere; cron expressions are in that zone, but several `togadesk/` UMA jobs are commented with their UTC equivalents — read the `name` carefully.
 - **Two stale-worker cutoffs exist** (270 s in the heartbeat, 7 min in the deploy script) and the `$desiredWorkers` list is duplicated in `ebs/cron.worker.php` and `App_Worker` — keep them aligned when changing roles.
 - **Per-role Elastic IPs** mean a role's outbound IP is stable across instance replacement — relevant for partner allow-lists (EDI/SFTP endpoints).
-- **⚠ Committed secrets.** Several files contain **live credentials checked into the repo**: AWS access keys (`ebs/cron.worker.php`, and in a comment in `crons/worker/worker_heartbeat.php`), GitHub PATs (`ebs/git.json`), and a Sentry DSN (`_/app/framework.php`). Rotate and move to environment variables / SSM Parameter Store. (Values deliberately not reproduced here.)
+- **⚠ Committed secrets.** Live credentials checked into the repo: GitHub PATs (`ebs/git.json`) and a Sentry DSN (`_/app/framework.php`). Rotate and move to env/SSM. The AWS keys (`ebs/cron.worker.php`, and the `crons/worker/worker_heartbeat.php` comment) are gone (2026-10-12): the deploy script now assumes `WorkloadsRuntime` — see [1.0 AWS credentials](../library/features/app-cloud-aws-credentials.md). The 1.0 worker deploys from CodePipeline `Worker` (502, us-west-2), which sources `_production` of both worker and library. (Values not reproduced.)

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -16,7 +16,7 @@
 | [Carrier Shipping Labels (UPS/FedEx) & NetSuite Item Fulfillment](features/carrier-shipping-labels.md) | Backend mechanics behind Fulfill & Ship: buying/persisting a carrier label (UPS/FedEx) and creating the NetSuite Item Fulfillment; open for label, reprint, retu |
 | [Running 2.0 code from a bare CLI script (bootstrap + transactions)](features/cli-script-bootstrap.md) | Minimum recipe + traps for running real 2.0 framework code (`_Model`/`_Query`/`_Database`) from a bare `php script.php`; open before writing a CLI harness/backf |
 | [Client Users require a contactId — the postPost backfill interceptor & every creation path](features/client-user-contactid-required.md) | Every `Client_<Tenant>.Users` row must have a non-null `contactId`; the shared `_Model_Client_User::postPost` interceptor guarantees it for api2-created users, |
-| [_Cloud S3 helpers (copy / get / delete / list)](features/cloud-s3-helpers.md) | `_Cloud` S3 helpers (copy/get/delete/list) with per-bucket region override and a 5 GB single-PUT limit; open before doing S3 access from 2.0 code. |
+| [_Cloud credentials (WorkloadsRuntime) + S3 helpers (copy / get / delete / list)](features/cloud-s3-helpers.md) | `_Cloud` credentials (assumes `WorkloadsRuntime`, no IAM keys) and S3 helpers (copy/get/delete/list, per-bucket region, 5 GB single-PUT limit); open before any |
 | [_Cloud Secrets Manager helper (_Cloud::getSecret)](features/cloud-secrets-manager.md) | `_Cloud::getSecret` reads AWS Secrets Manager (own default-provider auth, never throws, per-process cache); open before pulling secret material at request time. |
 | [_Component_*/_Model_* project-namespace registration (autoloader) & backslash-qualify traps](features/component-model-namespace-registration.md) | Project-local `_Component_*`/`_Model_*` must declare the app namespace AND live in a `<Name>/<Name>.php` directory, or the autoloader fatals at runtime; open wh |
 | [_Config group access — the two-argument "optional" form does NOT rescue a missing GROUP](features/config-group-access.md) | `_Config::<group>('prop', false)` does NOT rescue a MISSING GROUP (only a missing prop) — guard with `_Config::isGroupSet()`; open before reading per-environmen |

package/knowledge/2.0/apps/_underscore/features/cloud-s3-helpers.md

@@ -1,22 +1,24 @@
 ---
-title: _Cloud S3 helpers (copy / get / delete / list)
+title: _Cloud credentials (WorkloadsRuntime) + S3 helpers (copy / get / delete / list)
 framework: "2.0"
 repo: _underscore
 project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-12
 owners: ["jcardinal", "bala", "rgirish"]
 files:
   - _underscore/Cloud.php
 related:
+  - ../../worker2/features/cross-account-aws-access.md
+  - ../../../../1.0/apps/library/features/app-cloud-aws-credentials.md
   - ../../worker2/features/oneuptime-worker2-monitoring.md
   - ../../worker2/features/compass-vip-support-importer.md
   - ../../worker2/features/s3-folder-retention-cleanup.md
 ---
 
-`_Cloud` S3 helpers (copy/get/delete/list) with per-bucket region override and a 5 GB single-PUT limit; open before doing S3 access from 2.0 code.
+`_Cloud` credentials (assumes `WorkloadsRuntime`, no IAM keys) and S3 helpers (copy/get/delete/list, per-bucket region, 5 GB single-PUT limit); open before any AWS access from 2.0 code.
 
 ## Summary
 
@@ -24,6 +26,17 @@ related:
 
 ## How it works
 
+### Credentials
+
+No static key (TRUE-82705, 2026-10-12). Role model + accounts: [cross-account AWS access](../../worker2/features/cross-account-aws-access.md#the-workloadsruntime-role-model). 1.0 twin: [App_Cloud credentials](../../../../1.0/apps/library/features/app-cloud-aws-credentials.md).
+- `_Cloud::getCredentialProvider(?string $awsAccountId = null)` → memoized, self-refreshing SDK provider for `WorkloadsRuntime` in that account (default `AWS_ACCOUNT_ID_DEFAULT` = 502614707982; session `_underscore-cloud`; `[cloud] workloads_role_name` / `workloads_role_session` override). Pass it as `'credentials'` to any SDK client — worker2 `VipSupport` and `ZoomInfo` do.
+- The STS client uses `CredentialProvider::instanceProfile()` (memoized), never the default chain. If `AWS_PROFILE` is set (local dev), that profile is used directly.
+- `initialize()` rebuilds `_Cloud::$preCommands` on every call (exports `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` for the CLI); refreshes when < 900 s left (`AWS_CLI_CREDENTIALS_MIN_SECONDS_LEFT`).
+- `addSqsMessage()` assumes into the account in the queue URL (`getAccountIdFromSqsQueueUrl()`).
+- `getParameter()` returns `''` when credentials fail, so `_Database` falls back to its backup host instead of fataling.
+
+### S3 helpers
+
 - **Entry point:** `_underscore/Cloud.php` — `_Cloud::getS3Objects($s3BucketName, $s3BucketPath = '', ?string $awsRegion = null)`.
 - `getS3Objects()` uses the AWS SDK `S3Client->listObjectsV2` and transparently paginates (`listObjectsV2` caps at 1000 keys per page), returning entries shaped:
   - `Key` — object key
@@ -37,11 +50,18 @@ The config-default region is **us-east-1**. Buckets in another region must pass
 
 ## Gotchas
 
+- **Never put a CLI `$cmd` in an exception or log** — it starts with `$preCommands` (live temp creds).
+- **Never use the default credential chain for STS.** Old EB env properties can still hold the dead key; env vars win in the default chain → `InvalidClientTokenId`.
+- **Deploy order:** `_underscore` is pulled at deploy, and api2/worker2/saml ini no longer carry a key — merge `_underscore` to `_production` **before** them.
+
 - Callers get every matching key regardless of count; pagination is handled internally, so do not add your own `ContinuationToken` loop on top.
 - A bucket outside the config-default us-east-1 (e.g. `agilant-as2` in us-west-2) throws `AuthorizationHeaderMalformed` unless you pass `$awsRegion`.
 - **⚠ `copyFileToS3` is a single `putObject` — it CANNOT upload a file larger than 5 GB.** Verified 2026-09-10 at `_underscore/Cloud.php:49`: the method issues one `putObject` call, and S3 caps a single PUT at **5 GB**. There is **no `MultipartUploader` anywhere in `_underscore` or `worker2`**. Anything that could produce a large artifact — a database dump above all — will fail on this helper once it crosses the limit. If you need large-object upload, add `MultipartUploader` explicitly; do not assume this helper scales.
 - **`getFileFromS3` (the GET method) is single-region.** It builds the `S3Client` with region `_Config::cloud('aws_region')` (the worker's own region) and only swallows HTTP 404. Reading an object from a bucket in a different region raises a `301 PermanentRedirect`, which it re-throws. To read a cross-region (or public) S3 object, use a direct HTTPS download instead, or construct an `S3Client` with the bucket's own region. (Unlike `getS3Objects()`, `getFileFromS3` has no region-override parameter.)
 
+## Change history
+- 2026-10-12 — Added Credentials: `_Cloud` assumes `WorkloadsRuntime` via the instance role; static `[cloud]` key removed (TRUE-82705) (jcardinal)
+
 ## Related
 
 - [OneUptime worker2 monitoring](../../worker2/features/oneuptime-worker2-monitoring.md)

package/knowledge/2.0/apps/_underscore/features/cloud-secrets-manager.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
+updated: 2026-10-12
 owners: ["jcardinal"]
 files:
   - _underscore/Cloud.php
@@ -27,7 +27,7 @@ related:
 Entry point: `_underscore/Cloud.php` — `_Cloud::getSecret(string $secretName, ?string $awsRegion = null): ?array`.
 
 1. Builds a `SecretsManagerClient` and calls `GetSecretValue` for `$secretName`.
-2. **Auth is the AWS default provider chain** — the EC2 instance role in prod, a local profile/env in dev. It deliberately does **NOT** use the shared static `_Config::cloud` access key that the S3 helpers use. Reason: keep the blast radius small — leaking that one shared key must not also expose the SAML SP private keys. Grant the reading role its own Secrets Manager permission instead.
+2. **Auth is the AWS default provider chain** — the EC2 instance role in prod, a local profile/env in dev. It does **not** assume `WorkloadsRuntime` like the other `_Cloud` methods (see [_Cloud credentials](./cloud-s3-helpers.md#credentials)) — the reading instance role gets its own Secrets Manager grant, so the broad workloads role never needs access to the SAML SP private keys.
 3. `$awsRegion` defaults to `null`; pass the secret's region explicitly (SAML passes `'us-east-1'`).
 4. Returns the secret's JSON decoded to an **assoc array**. For the SSL secrets the keys are `certificate`, `privateKey`, `certificateChain` (optional), `passphrase` (optional).
 5. **Caches per PHP process** (function-static) — repeated reads of the same secret in one request hit the cache, not the API.
@@ -36,8 +36,11 @@ Entry point: `_underscore/Cloud.php` — `_Cloud::getSecret(string $secretName,
 ## Gotchas
 
 - **Returns `null` on failure, does not throw.** A caller MUST null-check the result; a silent `null` means "secret unavailable", not "empty secret".
-- **The role needs its own Secrets Manager grant.** Because it does not use the `_Config::cloud` key, an instance/role with no `secretsmanager:GetSecretValue` gets `null` for every secret. The `saml-production` EB instance role needed an inline policy added (see the SAML architecture Deployment notes).
-- **A committed AWS access key + secret were found in a code comment in `Cloud.php` `initialize()` and removed (2026-09-15).** Treat that key as leaked — it must be rotated separately from this change. Never store credential VALUES in source or comments; reference secret names only.
+- **The role needs its own Secrets Manager grant.** Because it uses the instance role directly, an instance/role with no `secretsmanager:GetSecretValue` gets `null` for every secret. The `saml-production` EB instance role needed an inline policy added (see the SAML architecture Deployment notes).
+- Never store credential VALUES in source or comments; reference secret names only. (A key found in a `Cloud.php` comment 2026-09-15 was the leaked `[cloud]` key — deleted 2026-10-12.)
+
+## Change history
+- 2026-10-12 — Static `[cloud]` key is gone; wording now contrasts the instance role with `WorkloadsRuntime` (TRUE-82705) (jcardinal)
 
 ## Related
 

package/knowledge/2.0/apps/_underscore/features/event-publish-sqs.md

@@ -6,8 +6,8 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: [rgirish]
+updated: 2026-10-12
+owners: [rgirish, jcardinal]
 files:
   - _underscore/Event.php
 related:
@@ -29,8 +29,8 @@ response.
 
 `publish(object &$api, string $httpMethod, array $routeVars)`:
 
-1. Reads `aws_event_queue_url` / `aws_event_queue_region` from `_Config::cloud()`; returns
-   early if no queue URL is configured.
+1. Reads `aws_event_queue_url` / `aws_event_queue_region` from `_Config::cloud()` (no
+   default — see Gotchas).
 2. Determines the affected record's `recordRoute`, `recordId`, `uuid`.
 3. Derives `parentRoute` / `parentUuid` from `$routeVars` when the URL is a nested resource
    (`count($routeVars) >= 3` → `parentRoute = $routeVars[0]`, `parentUuid = $routeVars[1]`).
@@ -38,10 +38,17 @@ response.
    parentRoute, parentUuid, dtEvent }` and enqueues via `_Cloud::addSqsMessage()`.
 
 SQS credentials come from `_Config::cloud('aws_event_queue_access_key_id' /
-'..._secret_access_key')` — values live in config/cloud, never in code or this doc.
+'..._secret_access_key')` — a static key, only in `api2/Config/beta.ini` (open: replace with
+the `WorkloadsRuntime` provider, see [_Cloud credentials](./cloud-s3-helpers.md#credentials)).
+Never put the values in a doc.
 
 ## Gotchas
 
+- **⚠ Event publishing to SQS silently fails in every env except beta.** `aws_event_queue_*`
+  is read with no default, and only `beta.ini` sets it, so `_Config::cloud()` throws; the
+  `try/catch (\Throwable)` logs and moves on. Known (2026-10-12, TRUE-82705); developer chose to
+  leave it. Before relying on `record:changed` outside beta, add the keys or a default.
+
 - **`recordRoute` must come from `$routeVars[0]`, not `$api->record->route`.** Post-`POST`
   interceptors can swap `$api->record` to a *different* model before `publish()` runs (e.g.
   `service-requests`' `postPost` switches `$api->record` to the `service-request-bundles`
@@ -54,6 +61,9 @@ SQS credentials come from `_Config::cloud('aws_event_queue_access_key_id' /
   published events. Cast first: `$data = (array) $api->response->data;` then
   `$topRecord = (object) $data[array_key_first($data)];` before reading `->id` / `->uuid`.
 
+## Change history
+- 2026-10-12 — Recorded that publishing throws (caught) outside beta: `aws_event_queue_*` has no default; static queue key still open (TRUE-82705) (jcardinal)
+
 ## Related
 
 - [websocket SQS fan-out](../../../../standalone/apps/websocket/features/sqs-fanout.md)

package/knowledge/2.0/apps/api2/architecture.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: architecture
 status: active
-updated: 2026-09-22
+updated: 2026-10-12
 owners: [jcardinal, bala, mhammontree, dfranks]
 files:
   - api2/Controller/Index.php
@@ -115,7 +115,7 @@ Same shape as other 2.0 tiers. `.ebextensions/php_include_underscore.config` add
 
 ### Security note (committed secrets)
 
-- `Config/production.ini` (and dev configs) contain **plaintext production credentials** — Core/Client/Logs DB passwords, **AWS access key id + secret**, and third-party API keys (FedEx, UPS, PayPal, NetSuite, Cal.com, AIG, OptimumDesk, VAPI). The **`.ebextensions/git.*.json`** files (per-env — e.g. `git.json`, `git.sandbox-dev.json`) each carry a **plaintext GitHub Personal Access Token**, used by the `.platform/hooks/prebuild/git.sh` clone hook to pull `_underscore` at build. These must be **rotated** (treat the committed tokens as compromised) and moved to SSM Parameter Store / EB env properties. **Flag this if you touch config or deploy.** (Location + remediation only — do not record the token value anywhere.)
+- `Config/production.ini` (and dev configs) contain **plaintext production credentials** — Core/Client/Logs DB passwords and third-party API keys (the `[cloud]` AWS key was removed and deleted 2026-10-12; `beta.ini` still holds the event-queue AWS key — open) (FedEx, UPS, PayPal, NetSuite, Cal.com, AIG, OptimumDesk, VAPI). The **`.ebextensions/git.*.json`** files (per-env — e.g. `git.json`, `git.sandbox-dev.json`) each carry a **plaintext GitHub Personal Access Token**, used by the `.platform/hooks/prebuild/git.sh` clone hook to pull `_underscore` at build. These must be **rotated** (treat the committed tokens as compromised) and moved to SSM Parameter Store / EB env properties. **Flag this if you touch config or deploy.** (Location + remediation only — do not record the token value anywhere.)
 - **Committed IAM access key (found 2026-07-28).** An IAM access key id + secret access key are hardcoded as PHP constants in `ebs/register_instance_to_shared_application_load_balancer.php` (**lines 29–30**) and are in **committed git history** (`git log -S`; commit subject *"Auto-registering to target groups"*) — exposed to every reader, clone, and EB bundle. **Remediation:** deactivate + delete the key in IAM, audit CloudTrail for its use, replace the block with the instance-profile pattern worker2 now uses. **Rotation, not history rewrite, closes the exposure**; a history rewrite is a separate sign-off-required decision because the team git rule forbids force-pushing `_main` in application repos. The same file also passes the IMDSv2 token on the `curl` command line (visible in `ps`/`/proc/<pid>/cmdline`, 6h TTL) and falls back silently to IMDSv1 — fix both alongside the key. *(Location + remediation only; no key material recorded.)*
 
 ### EB "Degraded" health alerts vs. the ALB's own view (read before triaging)

package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md

@@ -6,7 +6,7 @@ project: API
 client: shared
 type: workflow
 status: active
-updated: 2026-09-16
+updated: 2026-10-12
 owners: ["jcardinal", "mhammontree", "bala", "apeterson"]
 files:
   - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
@@ -87,6 +87,8 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 
 ## Gotchas
 
+- **Prod approval gates differ by pipeline (868).** `api-production-1` and `saml-production` have a manual **Approve** stage; `Worker-Production` does **not** — a worker2 `_production` merge deploys at once. `_underscore` is pulled at deploy, so merge an `_underscore` change before the api2/worker2/saml change that needs it.
+
 - **⚠ The `_underscore` SOURCE ACTION has its own branch trigger — and a green pipeline hides a wrong one.** The Source stage's **`_underscore` action carries its own `BranchName`/branch trigger, configured in AWS and independent of the api2 environment it deploys to.** Nothing in the repo reveals it. A **second**, separate way for an environment to run the wrong framework branch — the first being `.ebextensions/git.<env>.json`. Both must be right. **Incident 2026-08-25 (`ApiSandboxClient`, multi-hour investigation):** the `_underscore` source action was pointed at **`_production`**, so the environment ran `_production` framework code, while: `git.sandbox-client.json` was **correct** (no `branch` key), so the one config you can `grep` exonerated itself; the pipeline UI displayed the **`_sandbox-client` merge commit `2b523788`**; the pipeline reported **"All actions succeeded"**; the prebuild clone deletes `.git`, so the instance carries **no commit id**. Two apparently unrelated live bugs (a Quad approval-email 500 and a Quad tooltip falling back to Core defaults) were the **same** stale box; `git merge-base --is-ancestor <fix> _production` proved neither fix was on the branch actually deployed. **Triage rule:** a green pipeline showing the right commit proves the *source artifact* was fetched — it does **not** prove which **branch** each source action watches. Read the source action config:
 
   ```bash
@@ -116,6 +118,9 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 - **Credentials live in the committed `Config/*.ini` files (finding, 2026-08-10).** Both `worker2` and `api2` carry **live secrets in `Config/*.ini` files committed to those repos** — NetSuite OAuth `clientId`/`certificateId`, AWS keys, database passwords, a Talos agent key. Document only **where** they live, never the values. **Recommendation (not actioned):** move them to **EB environment properties or SSM Parameter Store** and treat anything currently committed as rotate-on-exposure. Related: api2's ALB auto-registration copy also carries a committed IAM access key (see the auto-registration note above).
 - **On-instance composer install (when a dep is missing post-deploy).** If a Composer dep is missing on the running instance (e.g. `Class "FPDF" not found` because `composer.lock` wasn't committed), you can install it on the box over SSH/PuTTY — but `composer require setasign/fpdf:^1.8` must have **no space** after the colon (`fpdf: ^1.8` parses `^1.8` as a separate package and errors), and run it as the web app user / `chown` the result so the webserver can read `vendor/`. The real fix is to **commit `composer.lock`** so EB's `composer install` picks it up — the on-instance install is a stopgap that a redeploy wipes.
 
+## Change history
+- 2026-10-12 — Added prod Approve-stage gotcha and `_underscore`-first merge order (TRUE-82705) (jcardinal)
+
 ## Related
 
 - [ENVIRONMENT drives the `_underscore` branch](../features/environment-variable-drives-underscore-branch.md)

package/knowledge/2.0/apps/saml/architecture.md

@@ -6,7 +6,7 @@ project: SAML SSO Gateway
 client: shared
 type: architecture
 status: active
-updated: 2026-09-28
+updated: 2026-10-12
 owners: ["rgirish", "jcardinal"]
 files:
   - saml/index.php
@@ -93,13 +93,13 @@ Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentica
 
 - **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026).** Any code that registers DBs then writes must call `_Database::transactionCommit()` before exiting — otherwise MySQL silently rolls back all writes on connection close. `/acs` calls `transactionCommit()` after `getAuthenticatedSsoUser()` succeeds and before the redirect. See `_underscore/Database.php:48`.
 - **SAML signature verification is commented out — repo's top security priority.** The IdP X509 cert verification block in `/acs` is disabled, so the gateway accepts any well-formed status-success `SAMLResponse` without proving it was signed by the expected IdP. The only remaining gate is the encrypted RelayState, which authenticates the *originating TOGa request* — not the *asserting IdP*. Threat: an attacker who can craft/replay a `SAMLResponse` (or a malicious/compromised IdP) can forge an identity assertion for any user. Re-enabling per-IdP X509 signature verification is the single highest-impact security fix. The per-client IdP verification certs already exist in source as base64 blobs (Compass, Office Depot, Prudential, RUMCSI, Agilant) — they are what a re-enabled verifier would load; verification is per-client and each IdP's signing cert must match.
-- **Plaintext secrets in source — move to AWS SSM Parameter Store.** Checked-in credentials, all live and to be rotated: `saml/Config/production.ini` holds a **DB password, an AWS access key id + secret, and NetSuite credentials** in cleartext; `saml/Controller/Index.php` hardcodes the **Sentry DSN**; `.ebextensions/git.json` carries a **GitHub PAT** (same leak class as `apiproxy`). Migrate to AWS SSM Parameter Store (SecureString), load at runtime; source tree should reference parameter names only, never literal credential values.
+- **Plaintext secrets in source — move to AWS SSM Parameter Store.** Checked-in credentials, all live and to be rotated: `saml/Config/production.ini` holds a **DB password and NetSuite credentials** (its AWS key was removed 2026-10-12; AWS access = instance role → `WorkloadsRuntime`) in cleartext; `saml/Controller/Index.php` hardcodes the **Sentry DSN**; `.ebextensions/git.json` carries a **GitHub PAT** (same leak class as `apiproxy`). Migrate to AWS SSM Parameter Store (SecureString), load at runtime; source tree should reference parameter names only, never literal credential values.
 - **Stale test scaffolding in `/acs` — remove it.** A large commented-out block in `saml/Controller/Index.php` hardcodes captured real `SAMLResponse` payloads and a synthetic `RelayState` for Office Depot / Compass testing. It is inert (commented out) but captured assertions living in source are a confidentiality/hygiene risk — delete it.
 - **RelayState is still not replay-proof, even after the TTL fix.** The check was written backwards (`$relayState->time <= time() + MAX_TIME_TO_LIVE`, true for every past timestamp) and is now `abs(time() - $relayState->time) <= MAX_TIME_TO_LIVE` on `TRUE-81993` / `_sandbox-client`. **That is hygiene, not a control:** `_String::encryptWithKey()` is AES-256-CBC with **no MAC and an attacker-supplied IV**, so the `time` field can be rewritten without the key, and there is no nonce or single-use marking. Fix direction is AES-256-GCM or an HMAC checked with `hash_equals()` before decrypt. See [/acs assertion trust gaps](features/acs-assertion-trust-gaps.md).
 - **`/acs` must redirect only to a URL rebuilt by `safeRedirectUrl()`.** `_Page::headerLocationRedirect()` is a bare `header('Location: ')`, and the `?saml=` payload exchanges for real tokens at `POST /auth/encrypted-user-uuid` — so the destination is a credential delivery target. Redirect to the **return value**, never to the stored `Core.Domains.domain`: appending `'?saml='` to the stored string lets a stored path or query steal the handoff. A `Core.Domains` row (owned by the right client + environment) **is** the authorization — there is deliberately no host allowlist; do not add one. On `TRUE-81993` / `_sandbox-client`, not yet on `_production`.
 - **`/meta` cert lines must be `trim()`-ed (CRLF).** The `.crt` files in `_underscore/Assets/ssl` are CRLF-encoded; cert-format code does `explode("\n", ...)` and indents each line, so without a per-line `trim()` a stray `\r` ends up on every base64 line inside `<ds:X509Certificate>`. Trim each line in both signing and encryption blocks.
 - **SP encryption certs load from Secrets Manager at request time.** `/acs` holds no encryption key material on disk (`_underscore/Assets/ssl/` no longer carries the SP encryption key). Rotation = edit the `ENCRYPTION_SECRET_NAMES` list + the secrets themselves, not a redeploy of on-disk `.crt`/`.key` files.
 - **Non-prod SAML depends on prod KMS/Secrets Manager; one SP key serves every env** — a non-prod leak is a prod key leak. `ENCRYPTION_SECRET_NAMES` must hold full ARNs (short names resolve only in the caller's account).
-- **Committed AWS key must be rotated.** A hardcoded AWS key was removed from `_underscore/Cloud.php`; because it was committed to git history it is compromised — rotate it. Still open: plaintext creds in `production.ini` and the hardcoded Sentry DSN.
+- **The leaked AWS key is deleted (2026-10-12, TRUE-82705).** `_Cloud` now assumes `WorkloadsRuntime` — see [_Cloud credentials](../_underscore/features/cloud-s3-helpers.md#credentials). `saml-production` has a manual Approve stage, so merge `_underscore` to `_production` first. Still open: plaintext DB/NetSuite creds in `production.ini`, and the hardcoded Sentry DSN.
 - **SLS not implemented.** `/sls` falls through to the default banner.
 - **`set_exception_handler(null)` at top of `saml()`.** Unhandled exceptions output raw PHP errors. All exceptions from `getAuthenticatedSsoUser()` are caught and sent to Sentry.

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -24,7 +24,7 @@
 | [worker2 — Committed Secrets & Config Exposure (debt register)](features/committed-secrets-and-config-debt.md) | Standing list of credentials committed to the worker2 repo (locations only); open before adding a credential to worker2 or assuming a config value is private. |
 | [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | worker2 action that ingests Compass's quarterly VIP spreadsheet and sets each VIP's support tech (`Users.c_supportedByUserId` in `Client_Compass`); open when wo |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). |
-| [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | The standard, and only sanctioned, way any new worker2 cron obtains AWS access (any account/region/SDK client) via role-assume; open before writing worker2 code |
+| [Cross-account AWS access for worker2 crons (_Component_Aws_Workloads)](features/cross-account-aws-access.md) | The `WorkloadsRuntime` role model (accounts, trusts, permissions — no IAM keys anywhere, 1.0 or 2.0) and worker2's `_Component_Aws_Workloads`; open before writi |
 | [Elastic Beanstalk health monitor → OneUptime (ElasticBeanstalkHealth)](features/elastic-beanstalk-health-monitor.md) | worker2 cron that reads each EB environment's health and pages the team via a OneUptime Incoming-Request monitor; open when tuning EB paging, its 5xx gate, or i |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` syncs Freshservice webhook events (tickets + conversations) into TOGA 2; open when working on Elite ticket sync or its Freshservice/TOGA-2 API q |
 | [Error Escalation Cron (Errors::Escalate → ClickUp / email)](features/error-escalation-cron.md) | Sole owner of error-pipeline notification (escalation, de-escalation, ClickUp tickets, reminders, business email, owner learning, GC); open when touching alerti |

package/knowledge/2.0/apps/worker2/features/committed-secrets-and-config-debt.md

@@ -6,8 +6,8 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-06
-owners: ["kyalamarthi", "ajean", "mhammontree"]
+updated: 2026-10-12
+owners: ["kyalamarthi", "ajean", "mhammontree", "jcardinal"]
 files:
   - worker2/.gitignore
   - worker2/Worker/Team/Github.php
@@ -46,6 +46,7 @@ the publish on a credential literal.
 | NetSuite EC private key | `worker2/Config/netsuite_ec_key.pem` | Tracked in git as a file. |
 | TOGa Talos (`api.togaiq.com`) API key — **old** release-notes key | git history only: `worker2/Worker/Team/Sprint.php` before PR #179 (still visible at commit `8613ace`) | Code no longer uses it, but it was committed, so it **must be revoked** in TOGa Talos. Open until revoked. |
 | TOGa Talos API key — **current** (`[talos] api_key`) | `worker2/Config/production.ini` `[talos]` group (~L230) | Shared by release notes and the transcript pipeline. Committed like every `Config/*.ini` (see Summary). |
+| AWS access key (`[talos]` group) | `worker2/Config/*.ini` `[talos]` | Open — not part of TRUE-82705; rotate + replace with the instance role. |
 | PayPal webhook signature verification **off** in production | `worker2/Config/production.ini:182` | Impact already written up — a caller who knows the URL can create a paid SalesOrder. See [Rate PayPal subscription webhook](../../../../clients/rate/features/paypal-subscription-purchase-webhook.md). |
 
 ### Secrets stored at runtime in `Core.WorkerJobs` (not committed, same exposure)
@@ -57,6 +58,11 @@ the publish on a credential literal.
 
 ### Fixed / no longer true
 
+- **AWS `[cloud]` static access key** (2026-10-12, TRUE-82705) — key deleted in IAM and removed from
+  every `Config/*.ini` (plus the dead `[teams]` `s3_*` keys). AWS access is the instance role assuming
+  `WorkloadsRuntime` — see [cross-account AWS access](./cross-account-aws-access.md). Still in git
+  history, but dead.
+
 - **ClickUp API token in `Sprint.php`** — the hardcoded `CLICKUP_API_ACCESS_TOKEN` constant is
   **gone**. All 6 call sites now use `_Component_Api_Clickup::accessToken()`, which reads
   `[clickup] token` from config. The component's own hardcoded fallback token, and the plaintext
@@ -89,6 +95,7 @@ the publish on a credential literal.
 - [Rate PayPal Subscription Purchase & Webhook Pipeline](../../../../clients/rate/features/paypal-subscription-purchase-webhook.md)
 
 ## Change history
+- 2026-10-12 — Moved the `[cloud]` AWS key to Fixed (deleted, TRUE-82705); listed the `[talos]` AWS key as still open (jcardinal)
 - 2026-10-06 — WorkerJobs runtime exposures: trace arguments (`zend.exception_ignore_args` devops option), NYCDOE FTP passwords in old job params (mhammontree)
 - 2026-10-01 — TOGa Talos release-notes key moved to `[talos] api_key`; old key listed for revocation (ajean)
 - 2026-09-25 — renamed product reference from "TOGa IQ" to "TOGa Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)

package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md

@@ -6,31 +6,35 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-09-28
+updated: 2026-10-12
 owners: [jcardinal, bala]
 files:
   - worker2/Component/Aws/Workloads/Workloads.php
   - worker2/Config/production.ini
   - worker2/Worker/Infrastructure/CloudWatch.php
   - worker2/Worker/Monitor/Fleet.php
-  - library/app/worker.php
+  - _underscore/Cloud.php
   - library/app/cloud.php
 related:
   - ../../_underscore/features/component-model-namespace-registration.md
   - ../../_underscore/features/config-group-access.md
   - ../../_underscore/features/cloud-s3-helpers.md
+  - ../../../../1.0/apps/library/features/app-cloud-aws-credentials.md
   - ./oneuptime-worker2-monitoring.md
   - ./elastic-beanstalk-health-monitor.md
   - ./worker-fleet-role-assignment-monitor.md
 ---
 
-The standard, and only sanctioned, way any new worker2 cron obtains AWS access (any account/region/SDK client) via role-assume; open before writing worker2 code that touches AWS.
+The `WorkloadsRuntime` role model (accounts, trusts, permissions — no IAM keys anywhere, 1.0 or 2.0) and worker2's `_Component_Aws_Workloads`; open before writing any code that touches AWS.
 
 ## Summary
 
-`_Component_Aws_Workloads` replaces hand-rolled `StsClient`/credential code and the **legacy static IAM-user access key** with a role-based model: the Elastic Beanstalk EC2 **instance role is the worker's identity**, and it **assumes a per-account `WorkloadsRuntime` role** via STS to reach each target account.
+**No long-lived IAM keys anywhere, 1.0 or 2.0** (TRUE-82705, 2026-10-12 — the static `[cloud]` key leaked and was deleted). Every AWS caller uses the EB EC2 **instance role as its identity** and **assumes `WorkloadsRuntime`** via STS in the target account. This doc owns the role model; the per-stack helpers are:
+- worker2 SDK clients: `_Component_Aws_Workloads` (below).
+- 2.0 `_Cloud` (S3/SQS/SSM/CLI): [`_Cloud` credentials](../../_underscore/features/cloud-s3-helpers.md#credentials) — `VipSupport` and `ZoomInfo` use `_Cloud::getCredentialProvider()`.
+- 1.0 `App_Cloud` + deploy scripts: [1.0 AWS credentials](../../../../1.0/apps/library/features/app-cloud-aws-credentials.md).
 
-**Standard for all new AWS-touching worker2 code:** obtain credentials/clients through `_Component_Aws_Workloads`. Do **not** instantiate `StsClient` by hand, and do **not** use the legacy `[cloud]` static access key in new code paths (see [Legacy static key](#legacy-static-key--being-retired)).
+Do **not** instantiate `StsClient` by hand and never add an access key to config.
 
 ## How it works
 
@@ -50,7 +54,7 @@ Role name and session are read from the `[cloud]` config group via `_Config::clo
 
 Another concrete instance of the app-repo namespace pattern: `namespace worker` + `extends \_Component`, referenced unqualified from `_Worker_*` classes via the autoloader alias (see [Component/Model namespace registration](../../_underscore/features/component-model-namespace-registration.md)).
 
-### The WorkloadsRuntime role model (three accounts)
+### The WorkloadsRuntime role model
 
 IAM is **global**, so **one `WorkloadsRuntime` role per account** covers all regions — region is only a runtime argument to `client()`, never a reason for a second role.
 
@@ -58,7 +62,11 @@ IAM is **global**, so **one `WorkloadsRuntime` role per account** covers all reg
 - **502614707982** — legacy account, `us-west-2`. **This is where the 1.0 `worker` fleet actually runs** — see the account-confusion gotcha below.
 - **975050298201** — non-production account, `us-east-1`.
 
-Each `WorkloadsRuntime` role has broad service permissions (`elasticbeanstalk:*`, `ec2:*`, `s3:*`, `sqs:*`, `cloudwatch:*`, `logs:*`, `ssm:*`, `autoscaling:*`, `elasticloadbalancing:*`) and a **trust policy allowing the 868 instance role to assume it**.
+**One broad role per account, on purpose** (developer decision 2026-10-12): no per-env roles, no resource scoping.
+- Permissions: `elasticbeanstalk:*`, `ec2:*`, `s3:*`, `sqs:*`, `cloudwatch:*`, `logs:*`, `ssm:*`, `autoscaling:*`, `elasticloadbalancing:*`. The 502 role also has `transfer:*`, `rds:*`, `ses:*`, `guardduty:*`, `codepipeline:*`, `route53:*`, `iam:PassRole` (what the deleted key could do).
+- Trust (502 and 868 roles): **both** instance roles — 868 `ElasticBeanstalk-EC2-Instance-Profile` (all 2.0 prod envs) and 502 `aws-elasticbeanstalk-ec2-role` (shared by ~65 legacy EB envs). Each instance role has inline policy `AssumeWorkloadsRuntime` (`sts:AssumeRole` on `arn:aws:iam::*:role/WorkloadsRuntime`).
+- Default account when the caller names none = **502** (`_Cloud`, `App_Cloud`, deploy scripts). SQS: assume in the account in the queue URL (2.0 `WorkerProductionQueue` is in 868).
+- Verify in CloudTrail: `AssumeRole` on `WorkloadsRuntime`, filter by session name (`worker2-workloads`, `_underscore-cloud`, `library-app-cloud`, …).
 
 ### First consumer — ElasticBeanstalkHealth (worked example)
 
@@ -97,11 +105,7 @@ The helper adoption was captured earlier: the prior inline STS/credential code a
 
 Applied 2026-09-28 (AWS CLI) to inline policy `WorkloadsRuntimePermissions` in 654654170868: statement `DenyTerminateNonBeanstalk` — Deny `ec2:TerminateInstances` on `*` when `aws:ResourceTag/elasticbeanstalk:environment-name` is `Null`. Any caller of WorkloadsRuntime can terminate **only EB-managed instances**. The other two accounts' roles do not have this statement.
 
-### Legacy static key — being retired
-
-The legacy `[cloud]` static access key is a **personal IAM user in account 502614707982**. Being retired for new code but **still used by older code paths** — ZoomInfo, VipSupport, and the `_Cloud` CLI path. **Retrofit of that existing code is deliberately deferred** (developer decision, this session): document the new approach; do not migrate old code yet. New code must not adopt the static key.
-
-**Never record the access-key values in knowledge.** They exist in `production.ini`; that is their only home.
+### Which account holds what
 
 #### Which account the 1.0 worker fleet lives in (easy and expensive to get wrong)
 
@@ -109,7 +113,7 @@ The 1.0 `worker` EC2 instances are in **502614707982**, `us-west-2` (EB environm
 
 **`App_Worker::AWS_WORKER_QUEUE_URL` in `library/app/worker.php` points at 654654170868 and misleads** — that account holds only the **SQS queue the 1.0 workers read from**, not the instances. Anything that needs to *see* the worker boxes (EB, EC2, health) must target **502614707982**.
 
-worker 1.0 authenticates there with a **long-lived hardcoded IAM access key in `library/app/cloud.php`** — a committed credential that should be rotated and moved to a role (value deliberately not recorded here; see [Legacy static key](#legacy-static-key--being-retired)). worker2 does **not** use it: it assumes `WorkloadsRuntime` cross-account from its EB instance role, which is exactly the migration this component exists to enable. First worker2 consumer in that account is the [Worker Fleet Role-Assignment Monitor](./worker-fleet-role-assignment-monitor.md).
+worker2 reaches it by assuming `WorkloadsRuntime` in 502 from its 868 instance role. First worker2 consumer in that account is the [Worker Fleet Role-Assignment Monitor](./worker-fleet-role-assignment-monitor.md).
 
 **Scope EB reads by `EnvironmentName`, never account-wide.** 502614707982 also runs `agilant-worker-alpha` and `agilant-worker-beta` as **separate EB environments that have no role registry at all**. An account-level EC2 scan would sweep them in and misreport them.
 
@@ -130,10 +134,15 @@ First S3 consumer of this pattern: the [S3 folder retention cleanup action](./s3
 - **`agilant-as2` is cross-account (owner 502614707982), not in the 868 hub.** Assume into the owner account; the `AccessDenied ... no resource-based policy` wording is the cross-account tell, and `ListBucket`/`GetBucketLocation` need the bare bucket ARN while `DeleteObject` needs the `/*` ARN. See [the section above](#agilant-as2-is-in-the-legacy-account-502614707982--s3-work-on-it-is-cross-account).
 - **Do not assume the 1.0 worker fleet is in the 868 production hub.** `AWS_WORKER_QUEUE_URL` (654654170868) is the SQS queue only; the instances are in the legacy account 502614707982, `us-west-2`. See [the section above](#which-account-the-10-worker-fleet-lives-in-easy-and-expensive-to-get-wrong).
 - Do not construct `StsClient` or pass explicit credentials by hand in new crons — the whole point is that the instance role is the caller and the component owns the assume-role step.
-- The instance-role-as-caller only works **on the EB tier**. Off-tier (e.g. a local CLI run) the SDK default chain resolves differently; that is why the legacy `_Cloud` CLI path still uses the static key for now.
+- The instance-role-as-caller only works **on the EB tier**. Locally, set `AWS_PROFILE` (SSO); `_Cloud`/`App_Cloud` then use the profile directly.
+- **`_Component_Aws_Workloads` builds its `StsClient` from the default chain; `_Cloud`/`App_Cloud` use `instanceProfile()`.** The default chain reads env vars first, and old EB env properties can still carry the dead key (`agilant-worker` does) → `InvalidClientTokenId`. If an env shows that, remove the `AWS_*` env props; new helpers should pin `instanceProfile()` like `_Cloud`.
+- `assumeCredentials()` returns plain creds with **no refresh** (~1 h). For a long-running process use `_Cloud::getCredentialProvider()` (memoized, refreshes).
 - Config keys are optional by design (in-code defaults), so a missing `[cloud]` `workloads_role_*` key silently uses `WorkloadsRuntime` / `worker2-workloads` — set them explicitly in prod ini to override.
 - **`$environments` must be typed `object|array`, not `array`.** The cron dispatcher casts only the top level of `CronJobs.parameters` to an array, so this nested JSON **object** arrives as a PHP `stdClass` and a strict `array` type throws a `TypeError` at dispatch. Declare it `object|array` and `(array)`-cast at the top. General rule for any object-valued cron param — see [creating-worker-actions.md](./creating-worker-actions.md).
 
+## Change history
+- 2026-10-12 — Static `[cloud]` key deleted everywhere (TRUE-82705); this doc now owns the role model for 1.0 + 2.0: broad role per account, trusts both instance roles, default account 502, SQS account from queue URL (jcardinal)
+
 ## Related
 
 - [Component/Model namespace registration](../../_underscore/features/component-model-namespace-registration.md)

package/knowledge/2.0/standards/backend-php.md

@@ -5,7 +5,7 @@ project: _Underscore
 client: shared
 type: standard
 status: active
-updated: 2026-10-07
+updated: 2026-10-12
 owners: [jcardinal, mhammontree, dfranks, bala, ajean]
 files: []
 related:
@@ -1009,24 +1009,22 @@ tsc will not catch a wrong shape — it fails silently.
 
 * Ensure code is well-documented with comments and usage examples.
 
-## AWS access (worker2)
+## AWS access — never IAM keys (1.0 and 2.0)
 
-### Obtain AWS credentials via _Component_Aws_Workloads — never static keys in new code
+No long-lived IAM access key in code, config, or deploy scripts — ever. AWS access = the EB
+EC2 instance role assuming `WorkloadsRuntime` via STS (one broad role per account; default
+account 502614707982; for SQS, the account in the queue URL).
+- 2.0 SDK clients: `_Cloud::getCredentialProvider($accountId)` (refreshing) or
+  `_Component_Aws_Workloads::client()` / `assumeCredentials()`.
+- 2.0/1.0 CLI: `_Cloud::$preCommands` / `App_Cloud::$preCommands` (rebuilt by `initialize()`).
+- STS must use `CredentialProvider::instanceProfile()`, never the default chain (old EB env
+  props still hold the dead key).
+- Never put a CLI `$cmd` (contains temp creds) in an exception or log.
 
-New worker2 code that touches AWS (any account, any region, any SDK client) must obtain
-credentials/clients through `_Component_Aws_Workloads::assumeCredentials()` /
-`::client()`. Do **not** construct `StsClient` or pass explicit credentials by hand, and do
-**not** use the legacy `[cloud]` static IAM-user access key in new code paths.
-
-The model: the Elastic Beanstalk EC2 instance role (`ElasticBeanstalk-EC2-Instance-Profile`,
-account 654654170868) is the worker's identity and assumes a per-account `WorkloadsRuntime`
-role via STS. IAM is global, so one `WorkloadsRuntime` role per account covers all regions.
-
-Existing static-key paths (ZoomInfo, VipSupport, `_Cloud` CLI) are grandfathered — retrofit
-is deferred, not sanctioned for new work. Never record access-key values anywhere but the ini.
-
-See: 2.0/apps/worker2/features/cross-account-aws-access.md
+See: 2.0/apps/worker2/features/cross-account-aws-access.md (role model),
+1.0/apps/library/features/app-cloud-aws-credentials.md (1.0 + deploy scripts).
 
 ## Change history
 
+- 2026-10-12 — AWS access rule now covers 1.0 + 2.0: no IAM keys, assume WorkloadsRuntime; static-key grandfathering removed (TRUE-82705). (jcardinal)
 - 2026-10-07 — Added the short-file-docblock rule (design history belongs in the knowledge base). (mhammontree)

package/knowledge/INDEX.md

@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 36 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **library** (Library) _(framework core)_ — 37 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 42 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
@@ -16,6 +16,8 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 - **toga** (TOGa) — 4 doc(s) → [1.0/apps/toga/INDEX.md](1.0/apps/toga/INDEX.md)
 - **store** (Store) — 3 doc(s) → [1.0/apps/store/INDEX.md](1.0/apps/store/INDEX.md)
 - **tools** (Tools) — 25 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
+- **vision** (Vision) — 0 doc(s) → [1.0/apps/vision/INDEX.md](1.0/apps/vision/INDEX.md)
+- **api** (API 1.0) — 0 doc(s) → [1.0/apps/api/INDEX.md](1.0/apps/api/INDEX.md)
 
 ## 2.0 framework
 

package/knowledge/clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md

@@ -6,13 +6,14 @@ project: Worker
 client: compass-usa
 type: workflow
 status: active
-updated: 2026-10-08
+updated: 2026-10-12
 owners: ["rgirish", "bala", "dfranks", "jcardinal", "mhammontree"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
   - worker/crons/toga2/compass/workflow/2_transmit_mits_purchase_orders_to_vendors.php
   - worker/crons/toga2/compass/edi/1_download_edi_s3_create_po_toga.php
   - worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php
+  - worker/crons/toga2/compass/workflow/move_office_depot_as2_files_to_import_folder.php
   - worker/crons/toga2/compass/workflow/4_transmit_office_depot_po_acknowledgements.php
   - worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php
   - worker/schedules/cron.worker.sync.json
@@ -147,6 +148,8 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
 - **To confirm a PO actually reached ODP:** check `PurchaseOrders.dtSubmitted` is set **AND** that downstream ODP SO(s) exist via the `PurchaseOrders_SalesOrders` join — their existence proves ODP received the PO and 850'd back.
 
 ## Gotchas
+
+- **⚠ AS2 files reach `OfficeDepot/` only via Lambda `rename_edi_files`.** AS2 drops land in `agilant-as2/processed/`; the Lambda moves the EDI file to `OfficeDepot/` for 3a. AWS suspended the Lambda after the 2026-10 IAM key leak (not firing since 2026-10-10) → ODP POs stopped importing. **Stop-gap:** cron `worker/crons/toga2/compass/workflow/move_office_depot_as2_files_to_import_folder.php` (`*/5`) does the same move (only `.net` files, newer than 2026-10-10 12:30 UTC, older than 120 s). **Disable it once the Lambda is restored.** Symptom of a dead mover: files pile up in `processed/`, `OfficeDepot/` stays empty, no new ODP SalesOrders.
 - **⚠ `edi/1_download_edi_s3_create_po_toga.php` is DEAD CODE (correction 2026-08-04).** It appears in **no** schedule file. The scheduled entry *named* "Download EDI From S3 Files & Create PO - EDI Office Depot" (`*/5 * * * *`, `worker/schedules/cron.worker.sync.json` `:254-259`) actually points at **`toga2/compass/workflow/3a_import_office_depot_purchase_orders.php`**. The two scripts have **different idempotency and different API call shapes** — `edi/1`'s create branch POSTs the SO with *nested* items and a `customer.name`, while **3a POSTs the header alone (uuids) and then POSTs the items in a separate request**. Reasoning from `edi/1` gives wrong answers. **General rule: never infer which script runs from its filename or folder** — verify against `worker/schedules/cron.*.json`, and note the schedule entry's **name can be actively misleading**. The folders `compass/edi/`, `compass/DOA_edi_odp_flow/`, and `compass/DOA_Scripts/` are **legacy / inactive**; `compass/edi/3_send_edi_855(reject).php` is an abandoned sketch using **non-standard** reject codes (`BAK01=A1` / `BAK02=RE`) — **do not** follow it.
 - **⚠ The `OfficeDepot/` listing makes 3a runs overlap (NOT fixed).** An instrumented prod run 2026-08-25 showed the S3 listing returning **138,472 objects** and taking **32 seconds** before any PO was processed, each PO then ~40 seconds. A file-bearing run blows past the `*/5` schedule and the following runs are **silently skipped** by `App_Framework::exitIfProcessRunning()` — leaving **no `CronJobExecutions` row at all**, which reads as "the cron never fired". The object count is driven by `OfficeDepot/SENT/` and `OfficeDepot/OUTBOX/` accumulating: 3a skips those prefixes when *processing* but still *lists* them. Fix candidates (separate work): prune `SENT/`, or list with a tighter prefix. See [Tracing a worker cron run in production](../../1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md).
 - **⚠ `REF~QC` — not `BEG` — is the value to search on for the ODP SalesOrder.** Searching `PurchaseOrders` / `SalesOrders` for the `BEG` number finds nothing for the SO and wastes time.
@@ -178,6 +181,7 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
 - **Editing cron recipient lists — a grep hit is not a live hit.** Email recipients are hardcoded in the cron scripts, and **several dormant copies of the same script exist** under `crons/toga2/compass/edi/` and `crons/toga2/compass/DOA_edi_odp_flow/`. Before changing a recipient list, cross-check the file against `worker/schedules/*.json` for a referencing entry whose `active` is not `0`. Only the scheduled copies matter.
 
 ## Change history
+- 2026-10-12 — Lambda `rename_edi_files` suspended after the IAM key leak; documented the stop-gap mover cron (TRUE-82705) (jcardinal)
 - 2026-10-08 — Cron 5 duplicate-proof send (`fd426d9b`): externalId `toga2-compass-odso-<ODP SO id>` + lookup before/after `add()`, PO fallback for older orders; removed the shutdown alert (false alarm on SA138624) and the try/catch around `add()`; chose add+lookup over upsert (CTO). Added no-beta-testing and ops-email gotchas. (bala)
 - 2026-09-25 — Corrected the fix description (forked stamp via `queryForked`, no `keepAlive`; stamp unconfirmed); widened duplicate scope to 22 POs Sep 15–22; flagged 9 all-Billed POs as open; recorded monitor-only decision and the `NetsuiteDuplicateSalesOrders` backstop. (rgirish)
 - 2026-09-22 — Root-caused the duplicate NetSuite SOs (15 across 11 POs, Sep 21-22) as the dead-MySQL-connection stamp race, not a Toga duplicate; fixed cron 5 with a forked stamp + a `register_shutdown_function` alert (`worker` `afce32ca`). Recorded that the Sep 21 try/catch is dead code and that `fe4d110d` re-enabling the cron is what surfaced the latent bug. (rgirish)

package/knowledge/registry.json

@@ -299,5 +299,23 @@
     "framework": "standalone",
     "role": "app",
     "dependsOn": []
+  },
+  {
+    "repo": "vision",
+    "project": "Vision",
+    "framework": "1.0",
+    "role": "app",
+    "dependsOn": [
+      "library"
+    ]
+  },
+  {
+    "repo": "api",
+    "project": "API 1.0",
+    "framework": "1.0",
+    "role": "app",
+    "dependsOn": [
+      "library"
+    ]
   }
 ]

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1002",
+  "version": "1.0.1003",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",