toga-ai 1.0.461 → 1.0.463

package/knowledge/2.0/apps/_underscore/INDEX.md

@@ -13,6 +13,7 @@
 | [_Cloud S3 helpers (copy / get / delete / list)](features/cloud-s3-helpers.md) | `_Cloud` centralizes AWS SDK S3 usage for the 2.0 stack so the `S3Client` never leaks into workers or app code. | _underscore/Cloud.php |
 | [_Component_*/_Model_* project-namespace registration (autoloader) & backslash-qualify traps](features/component-model-namespace-registration.md) | Every **project-local** `_Component_*` and `_Model_*` class in a 2.0 app **must declare the project namespace** at the top of the file: ```php namespace <NAMESP | _underscore/Loader.php, worker2/_.php, api2/_.php, worker2/Component/Forecast/Db/Db.php, worker2/Component/Forecast/SaleImport/SaleImport.php, api2/Component/Api/Netsuite/Netsuite.php |
 | [Re-pointing a DB alias mid-request (_Database::register park/restore)](features/database-alias-repointing.md) | `_Database` keys **all live per-database runtime state by the connection ALIAS** (`Client` / `_underscore::DB_CLIENT`, `ClientLogs`, `Archive`), **not** by the | _underscore/Database.php, _underscore/Query.php, api2/Component/Api/V2/V2.php, api2/Component/Api/CrossClient/CrossClient.php |
+| [2.0 Email Send Pipeline (queue + Send worker)](features/email-send-pipeline.md) | In 2.0, `_Email::send()` **does not transmit** — it queues the message. | _underscore/Email.php, worker2/Worker/Infrastructure/Email/Send.php |
 | [Client Email Template Sending](features/email-template-sending.md) | `_Model_Client_EmailTemplate` sends a stored, client-defined email template by UUID. | _underscore/Model/Client/EmailTemplate.php, _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php, _underscore/Email.php |
 | [Error Reporting — Issue/Event Aggregation (agreed POST-to-receiver design)](features/error-reporting-issue-event.md) | Platform-wide error-reporting infrastructure for TOGA 2.0, built around a two-table **Issue / Event** aggregation model in the shared **Core Logs DB**. | _underscore/Error.php, _underscore/Model/Core/Logs/Issue.php, _underscore/Model/Core/Logs/Event.php, dbchanges2/Logs/2026-07-06 - Issue and Event tables.sql |
 | [Record-Changed Event Publishing (_Event::publish to SQS)](features/event-publish-sqs.md) | `_Event::publish()` (in `_underscore/Event.php`) is the PHP side of the real-time event pipeline. | _underscore/Event.php |

package/knowledge/2.0/apps/_underscore/features/email-send-pipeline.md

@@ -0,0 +1,69 @@
+---
+title: 2.0 Email Send Pipeline (queue + Send worker)
+framework: "2.0"
+repo: _underscore
+project: _Underscore
+client: shared
+type: feature
+status: active
+updated: 2026-07-28
+owners: ["bala"]
+files:
+  - _underscore/Email.php
+  - worker2/Worker/Infrastructure/Email/Send.php
+related:
+  - email-template-sending.md
+---
+
+## Summary
+
+In 2.0, `_Email::send()` **does not transmit** — it queues the message. It writes a `PENDING`
+row into `Logs_[Client].Email` (the `body` column is `mediumtext`) and stores any attachments as
+BLOBs in `Logs_[Client].EmailAttachment`, then returns. A separate cron worker action,
+`Infrastructure/Email/Send` (`_Worker_Infrastructure_Email_Send::Run`), scheduled in
+`Core.CronJobs` at `* * * * *` (every minute), actually sends the `PENDING` rows via **AWS SES
+SMTP** using PHPMailer, then flips each row to `SENT` or `FAILED`. So a 2.0 email lands within
+~1 minute of the `send()` call — **not instantly**.
+
+## Key files / entry points
+
+- `_underscore/Email.php` — `_Email::send()` builds/validates the PHPMailer message, writes the
+  `PENDING` `Logs_[Client].Email` row (+ attachment BLOBs), and enqueues the send. It **throws**
+  unless both `setClientIdentifier(...)` and a From address (`fromEmailAddress`) are set. The
+  `Logs_[Client]` databases live on a separate cluster (`production-logs-*`), so `send()` points
+  `DB_CLIENT_LOGS` at the correct `Logs_[Client]` schema before inserting.
+- `worker2/Worker/Infrastructure/Email/Send.php` — `Run()` fetches up to 100 `PENDING` rows for
+  the client, and `sendEmail()` transmits each via SES SMTP. Constants: `STATUS_PENDING`,
+  `STATUS_SENT`, `STATUS_FAILED`, `MAX_RETRIES`.
+
+## How it works
+
+1. A caller (model, interceptor, worker) builds `_Email`, sets client identifier + From, adds
+   recipients/subject/body, and calls `send()`.
+2. `send()` inserts a `PENDING` `Logs_[Client].Email` row (attachments → `EmailAttachment` BLOBs)
+   and returns — nothing is transmitted yet.
+3. Every minute, `Infrastructure/Email/Send::Run` selects up to 100 `PENDING` rows and sends each
+   through PHPMailer over SES SMTP.
+4. On success the row flips to `SENT`; on failure `retryCount` is incremented and the row is
+   retried on subsequent runs until `MAX_RETRIES` (3), after which it is marked `FAILED`.
+
+## Gotchas / known issues
+
+- **~1-minute latency, not instant.** Anything that assumes an email is sent synchronously at the
+  `send()` call is wrong — transmission happens on the next Send-worker tick.
+- **The Send worker forces HTML mode.** `sendEmail()` calls `$mailer->IsHTML(true)`
+  **unconditionally**, ignoring the sender's `setIsHtml(false)`. A caller that queued a
+  "plain-text" message is still transmitted HTML-mode. Author bodies accordingly.
+- **⚠ SECURITY — hardcoded AWS SES SMTP credentials in `_underscore/Email.php`.** The
+  `SMTP_USERNAME` / `SMTP_PASSWORD` class constants hold **real** SES SMTP credentials committed
+  in source, violating the no-secrets-in-code rule. Documenting the **location only** — do not
+  copy the values. Remediation: treat as compromised, rotate in AWS SES, and move them into
+  `Config/[environment].ini` (read via `_Config`). Also tracked on
+  [`email-template-sending.md`](email-template-sending.md).
+
+## Change history
+- 2026-07-28 — Created: documented that `_Email::send()` **queues** (writes a `PENDING`
+  `Logs_[Client].Email` row + attachment BLOBs) rather than transmitting, and that the worker2
+  `Infrastructure/Email/Send` cron (`Core.CronJobs` `* * * * *`) does the actual SES SMTP send
+  with `MAX_RETRIES` (3) → `SENT`/`FAILED`, giving ~1-minute delivery latency. Noted the Send
+  worker's unconditional `IsHTML(true)` override and the hardcoded SES SMTP creds location. (bala)

package/knowledge/2.0/apps/_underscore/features/email-template-sending.md

@@ -6,13 +6,14 @@ project: _Underscore
 client: shared
 type: feature
 status: active
-updated: 2026-07-16
+updated: 2026-07-28
 owners: ["jcardinal", "bala", "mhammontree"]
 files:
   - _underscore/Model/Client/EmailTemplate.php
   - _underscore/Model/Client/EmailTemplateOutgoingEmailAddress.php
   - _underscore/Email.php
 related:
+  - email-send-pipeline.md
   - ../../worker2/features/notification-email.md
   - ../../worker2/features/notification-email-template.md
 ---
@@ -86,6 +87,13 @@ can keep using `sendEmail($api, ...)`.
 
 ## Gotchas / known issues
 
+- **`_Email::send()` QUEUES, it does not directly transmit.** `send()` writes a `PENDING` row into
+  `Logs_[Client].Email` (+ attachment BLOBs) and returns; the actual PHPMailer/SES SMTP send,
+  retries, and the `SENT`/`FAILED` outcome happen ~1 minute later in the worker2
+  `Infrastructure/Email/Send` cron. So an email is **not** sent synchronously at the `send()` call,
+  and the "throws when `PHPMailer::Send()` returns false" behavior described under *Failure
+  surfacing* below reflects the pre-queue path — final delivery success/failure is now decided by
+  the Send worker. See [`email-send-pipeline.md`](email-send-pipeline.md) for the full pipeline.
 - **Use `send()` from any non-API context (workers, cron, internal code).** Before
   2026-06-15 the only entry point was `sendEmail(&$api, ...)`, so callers with no API
   context faked one: `$api = (object)['client' => (object)['clientIdentifier' => …]]`.
@@ -138,6 +146,10 @@ worker method) in-process instead.
 
 ## Change history
 
+- 2026-07-28 — Clarified that `_Email::send()` **queues** a `PENDING` `Logs_[Client].Email` row
+  rather than transmitting; the actual SES SMTP send/retry/`SENT`/`FAILED` happens in the worker2
+  `Infrastructure/Email/Send` cron (~1-min latency). Added the new
+  [`email-send-pipeline.md`](email-send-pipeline.md) feature doc and cross-linked it. (bala)
 - 2026-07-16 — Recorded the **hardcoded AWS SES SMTP credentials** security gotcha in
   `_Email` (`SMTP_USERNAME`/`SMTP_PASSWORD` constants; pre-existing — rotate + move to
   `Config`). Surfaced while fixing a production 500 (EO-1) whose root cause was a single

package/knowledge/2.0/apps/api2/INDEX.md

@@ -18,5 +18,5 @@
 | [TableView field/column metadata (TableViewFields, hidden projected columns)](features/tableview-field-metadata.md) | The columns of a 2.0 table view are defined by DB metadata, not code. | _underscore/Model/Client/TableView.php, api2/Component/Api/V2/V2.php, dbchanges2/Client/2026-07-20 - ItemsUuidForPurchaseOrderItemsTableView.sql |
 | [Tickets API (/v2/tickets)](features/tickets-api.md) | The generic ticket endpoint of the 2.0 REST API. | Component/Api/V2/V2.php |
 | [V2 API error/message codes (EV/EZ troubleshooting map)](features/v2-api-error-codes.md) | The V2 JSON engine (`Component/Api/V2/V2.php`) returns short **message codes** in the response `error` field, grouped by family: `EN-*` authentication, `EZ-*` a | api2/Component/Api/V2/V2.php, _underscore/Model/Client/TrackingNumber.php |
-| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. |  |
+| [AWS CodePipeline Deployment via CodeConnections (GitHub → Elastic Beanstalk)](workflows/codepipeline-codeconnections-deploy.md) | 2.0 apps (`api2`, `_underscore`) are deployed through **AWS CodePipeline**. | api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
 | [New Environment Configuration & Provisioning (api2)](workflows/environment-configuration-and-provisioning.md) | What it takes for a 2.0 API environment (e.g. | api2/Config/<environment>.ini, api2/Controller/Index.php, dbchanges2/Core/2026-06-16a - DatabaseHosts for new QA QC stage demo environments.sql, dbchanges2/Logs/, _underscore/Route.php |

package/knowledge/2.0/apps/api2/architecture.md

@@ -107,6 +107,15 @@ One ~2,000-line `execute()` then `processRoutePairs()`:
 5. **Transaction logging** — every request logged (to client/core Logs DB, or as a JSONL
    line shipped by CloudWatch when `[api] log_filepath` is set).
 
+> **⚠ Auto-generated `Api.transactionId` collides under concurrency → 1062 → HTTP 500.**
+> Separate from the *client-supplied* `transactionId` uniqueness check (EV-5, above): the inbound
+> request-logger inserts its `Api` log row with a UNIQUE `transactionId` set to a
+> millisecond-precision timestamp (`Y-m-d H:i:s.v`). Concurrent nested writes generated within the
+> same millisecond collide on that UNIQUE key → MySQL **1062** → **HTTP 500**. This breaks ingestion
+> for high-volume senders (seen on the Compass/Veyer ASN feed, `sourceIp 34.232.23.158`) and is
+> platform-wide. Fix direction: make the logged `transactionId` unique-enough (uuid) or
+> retry-on-1062. Until fixed, a burst of concurrent posts can intermittently 500 with no app-level cause.
+
 ## CRUD engine — `processRoutePairs()`
 
 **Metadata-driven** — routes/models/fields/permissions come from Core/Client DB tables, not
@@ -154,6 +163,13 @@ region-aware DB host selection; CloudWatch agent ships the JSONL log file;
 `long_gateway_timeout.conf` sets `ProxyTimeout 1800`/`Timeout 1800` — **the fix for the
 "exactly 60 second" 504** (EB Apache→PHP-FPM defaults to 60s); `enforce_https.conf`.
 
+`.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh` +
+`ebs/register_instance_to_shared_application_load_balancer.php` register a **non-production**
+instance into the ALB target group whose name equals the EB environment name (production skipped;
+always exits 0). See
+[the worker2 reference implementation](../worker2/features/alb-target-group-auto-registration.md) —
+api2's copy is the **unhardened original** and must be brought up to it.
+
 ## CI — commit message policy
 
 `.github/workflows/true-devteam-requirements.yml` enforces **`TRUE-{ticket}: {Subject}`** —
@@ -171,6 +187,18 @@ build. These must be **rotated** (treat the committed tokens as compromised) and
 Parameter Store / EB env properties. **Flag this if you touch config or deploy.** (Location +
 remediation only — do not record the token value anywhere.)
 
+**Committed IAM access key (found 2026-07-28).** An IAM access key id + secret access key are
+hardcoded as PHP constants in `ebs/register_instance_to_shared_application_load_balancer.php`
+(**lines 29–30**) and are in **committed git history** (`git log -S`; commit subject
+*"Auto-registering to target groups"*) — exposed to every reader of the repo, every clone, and every
+EB bundle. **Remediation:** deactivate + delete the key in IAM, audit CloudTrail for its use, and
+replace the block with the instance-profile pattern worker2 now uses. **Rotation, not history
+rewrite, closes the exposure**; a history rewrite is a separate sign-off-required decision because
+the team git rule forbids force-pushing `_main` in application repos. The same file also passes the
+IMDSv2 token on the `curl` command line (visible in `ps`/`/proc/<pid>/cmdline`, 6h TTL) and falls
+back silently to IMDSv1 — fix both alongside the key. *(Location + remediation only; no key material
+is recorded anywhere.)*
+
 ## Known issues / accepted risks
 
 Open items a maintainer should know before changing this tier. None are "bugs to fix right now" —
@@ -204,6 +232,13 @@ they are the known sharp edges. Do not re-discover these from scratch.
 8. **JWT signing-secret rotation accepts current + previous.** During the overlap window a token
    signed with the retired secret still validates. Revocation is therefore not immediate —
    don't rely on rotation alone to lock out a compromised token.
+9. **Committed IAM access key in the ALB-registration script, not yet rotated (found 2026-07-28).**
+   `ebs/register_instance_to_shared_application_load_balancer.php` lines 29–30, present in committed
+   git history. Treat as compromised: deactivate + delete the key, audit CloudTrail, and move to the
+   EC2 instance profile as
+   [worker2 does](../worker2/features/alb-target-group-auto-registration.md). The same script's
+   IMDSv2 handling (token on the `curl` command line, silent IMDSv1 fallback) must be fixed with it.
+   See the Security note above.
 
 ## When making changes here
 
@@ -221,6 +256,8 @@ they are the known sharp edges. Do not re-discover these from scratch.
   or `execute()`.
 
 ## Change history
+- 2026-07-28 — Documented the previously unrecorded `060_register_instance_to_shared_application_load_balancer` postdeploy hook pair in the Deployment section (non-prod self-registration into the same-named ALB target group; production skipped), and recorded a **committed IAM access key** in `ebs/register_instance_to_shared_application_load_balancer.php` as a security note + Known issue #9 — location, line range, commit subject, and remediation only (rotate, audit CloudTrail, move to the instance profile; history rewrite is a separate sign-off). Flagged api2's copy as the unhardened original vs. the new worker2 reference implementation. (jcardinal)
 - 2026-07-28 — Added a consolidated **Known issues / accepted risks** section (8 items), absorbing the previously free-floating deferred raw-exception-disclosure follow-up as item 1, so the tier's sharp edges (unrotated committed secrets, pre-execute phase still outside the main guard, local Logs DB name mismatch, permissive CORS, unpinned `_underscore` build clone, untested `V2.php` monolith, JWT rotation overlap window) are in one place instead of scattered. Recorded that `DB_CACHE` is resolved by name (`Databases.name = 'Cache'`), never by a hardcoded id, which differs per Core instance. (jcardinal)
+- 2026-07-28 — Added gotcha: the request-logger's auto-generated `Api.transactionId` (millisecond timestamp `Y-m-d H:i:s.v`, UNIQUE) collides under concurrent same-millisecond nested writes → MySQL 1062 → HTTP 500; platform-wide, observed on the Compass/Veyer ASN feed (`sourceIp 34.232.23.158`). Distinct from the client-supplied `transactionId`/EV-5 uniqueness contract. Fix direction: uuid the logged id or retry-on-1062. (bala)
 - 2026-07-27 — Sharpened the committed-secret note: the plaintext GitHub PAT lives in the per-env **`.ebextensions/git.*.json`** files (used by the `prebuild/git.sh` clone hook to pull `_underscore`), must be rotated and moved to SSM / EB env properties (location + remediation only, no value). (mhammontree)
 - 2026-07-23 — Documented the now-guarded Core/Logs DB bootstrap in the front controller: the pre-execute block runs before the `execute()` try/catch, the Core Logs schema name is resolved from a `Core.Database` row (`id = CORE_LOGS_DATABASE_ID`) so a name-mismatched local Logs DB reads as missing, and the failure is now wrapped in `try/catch (\Throwable)` returning `INVALID_CONFIGURATION` + Sentry instead of a fatal (guarded no-op rollback, `Database.php:219–226`). Added the deferred raw-getMessage/getTrace client-disclosure follow-up to the Security note. (jcardinal)

package/knowledge/2.0/apps/api2/workflows/codepipeline-codeconnections-deploy.md

@@ -6,9 +6,11 @@ project: API
 client: shared
 type: workflow
 status: active
-updated: 2026-07-27
+updated: 2026-07-28
 owners: ["jcardinal", "mhammontree"]
-files: []
+files:
+  - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
+  - api2/ebs/register_instance_to_shared_application_load_balancer.php
 related: []
 ---
 
@@ -106,6 +108,15 @@ debugging the code. (Seen 2026-06-18 chasing a "missing FPDF / old code still ru
 beta that was really the LB still pointing at a terminated instance's replacement that was never
 registered.)
 
+**Automated for non-production tiers that carry the `060` hook.** `api2` and (as of 2026-07-28)
+`worker2` ship a postdeploy hook pair that makes a **non-production** instance register *itself*
+into the target group whose name equals the EB environment name — so on those tiers the manual
+step above is only needed if the hook is absent, misnamed, or silently failed (it always exits 0).
+Production is deliberately skipped and still registers by the existing process. See
+[deploy-time auto-registration to the shared ALB target group](../../worker2/features/alb-target-group-auto-registration.md),
+which is also the reference implementation — **api2's copy has known security defects**, including
+a committed IAM access key.
+
 ### On-instance composer install (when a dep is missing post-deploy)
 If a Composer dep is missing on the running instance (e.g. `Class "FPDF" not found` because
 `composer.lock` wasn't committed), you can install it on the box over SSH/PuTTY — but
@@ -128,6 +139,7 @@ aws codeconnections get-connection --connection-arn "<CONN_ARN>" --region "$REGI
 ```
 
 ## Change history
+- 2026-07-28 — Manual LB re-registration is now automated on **non-production** tiers carrying the `.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh` hook pair (api2 already had it; ported to worker2 this session). Linked the new worker2 feature doc as the reference implementation and flagged that api2's copy has known security defects. (jcardinal)
 - 2026-07-27 — Noted that not every EB env pulls `_underscore` from its `_<env>` branch: **`API-Sandbox-Dev`** re-clones `_underscore` from the branch in `.ebextensions/git.sandbox-dev.json` (**`_sandbox-dev`**), overwriting the pipeline copy — so a `_beta` merge doesn't reach it. Added the `Logs_<Client>.Api` server-side request-log note as the reliable way to debug a beta env without Sentry (TRUE-79533). (mhammontree)
 - 2026-07-14 — Documented the branch model: app repos (`_underscore`/`api2`/`toga2-supply`) deploy from long-lived `_beta`/`_production` branches (not `_main`); api2 pulls `_underscore`'s `_<env>` branch at EB build; `dbchanges2` has only `_main` and its migrations are applied per-env by the team process (not by a code deploy). (mhammontree)
 - 2026-06-18 — Added two EB-instance gotchas surfaced during the TOGa Supply beta/prod label deploys: (1) terminated instances must be **manually re-registered** as LB targets (we don't pay for auto-registration) — until then the new code never serves traffic and looks like deploy-lag; (2) on-instance `composer require` stopgap syntax (no space after the colon, fix root cause by committing `composer.lock`). (mhammontree)

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -3,6 +3,7 @@
 | Doc | Summary | Files |
 |-----|---------|-------|
 | [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php |
+| [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh, worker2/.platform/hooks/prebuild/_shared/041-write-region.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
 | [Automated PR Merger — Concurrent Force-Push Clobber Race](features/automated-pr-merger-force-push-race.md) | The automated PR merger `_Worker_Team_GitHub::Merge` (`worker2` `Worker/Team/Github.php`) merges approved PRs to `_production` by **force-pushing from a clone t | Worker/Team/Github.php |
 | [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | A cron watchdog that emails when the ClickUp integration looks disconnected during business hours. | worker2/Worker/Clickup/Health.php, worker2/Database/ClickupHealthWatchdog.sql |
 | [ClickUp Design Sprint Automation (Final Design Outcome)](features/clickup-design-sprint-automation.md) | `_Worker_Clickup_Design` is meant to drive the design-sprint workflow in ClickUp via the API, replacing a set of native ClickUp automations. | worker2/Worker/Clickup/Design.php, worker2/Worker/Clickup.php, worker2/Controller/ClickupDesignTest.php, _underscore/Component/Api/Clickup/Clickup.php |

package/knowledge/2.0/apps/worker2/architecture.md

@@ -15,6 +15,7 @@ files:
   - _underscore/Worker.php
 related:
   - ./features/creating-worker-actions.md
+  - ./features/alb-target-group-auto-registration.md
   - ../_underscore/features/async-query-execution.md
 ---
 
@@ -31,6 +32,18 @@ processes background jobs. It's a `_underscore` 2.0 app (`index.php` is just
 **MySQL is the source of truth; SQS is delivery only.** All job state lives in
 `Core.WorkerJobs`. Every worker invocation reads from that table and writes its result back.
 
+**Production vs. non-production inbound differ.** The SQS/Lambda job pipeline above describes
+**production**. **Non-production worker2 environments do not incorporate SQS at all** — they exist
+for **manual invocation over HTTP** (hence `index.php` dispatch and `.platform/httpd/conf.d/`, and
+no `cron.yaml`). That is precisely why non-prod instances must be reachable through the shared ALB;
+see [deploy-time auto-registration to the shared ALB target group](./features/alb-target-group-auto-registration.md).
+SQS may be added to non-prod later, but is not planned. **Do not assume a non-prod worker2 job ever
+passes through SQS** — a queue-based reproduction of a prod issue will not work there.
+
+**Associated open risk:** because non-prod is HTTP-invoked and now ALB-reachable, the shared-ALB
+listener rules and security groups must restrict non-prod to internal/VPN sources, and the
+HTTP-triggered job endpoints must enforce auth.
+
 ## AWS infrastructure
 
 | Component | Notes |
@@ -260,6 +273,11 @@ the MySQL-first design was departed from **on purpose** here.
 
 ## Change history
 
+- 2026-07-28 — Corrected the inbound model: **non-production** worker2 environments are not
+  SQS-driven at all; they are manually invoked over HTTP (no `cron.yaml`; `index.php` +
+  `.platform/httpd/conf.d/`), which is why they must be reachable through the shared ALB.
+  Production remains SQS-driven. Recorded the resulting non-prod exposure risk (listener/SG
+  restriction + endpoint auth). (jcardinal)
 - 2026-07-28 — Added **Cron actions must return a result on BOTH paths**: a new cron action is
   expected to return a structured result for a success as well as a failure, because both are
   recorded against the job and the successful results are what auditing reads (a completed run that

package/knowledge/2.0/apps/worker2/features/alb-target-group-auto-registration.md

@@ -0,0 +1,162 @@
+---
+title: Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: shared
+type: feature
+status: active
+updated: 2026-07-28
+owners: [jcardinal]
+files:
+  - worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
+  - worker2/ebs/register_instance_to_shared_application_load_balancer.php
+  - worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh
+  - worker2/.platform/hooks/prebuild/_shared/041-write-region.sh
+  - worker2/.platform/hooks/postdeploy/015_install_composer.sh
+  - api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh
+  - api2/ebs/register_instance_to_shared_application_load_balancer.php
+related:
+  - ../architecture.md
+  - ../../api2/workflows/codepipeline-codeconnections-deploy.md
+  - ../../api2/architecture.md
+---
+
+## Summary
+
+TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally
+**not** added to its environment's ALB target group — a fresh or replacement instance serves no
+traffic until someone registers it by hand (see the manual-registration gotcha in the
+[api2 deploy workflow](../../api2/workflows/codepipeline-codeconnections-deploy.md)). This hook
+pair automates that step for **non-production** environments: on every deploy the instance
+registers **itself** into the target group whose **name exactly equals the EB environment name**.
+
+Originally built in `api2`; ported to `worker2` on 2026-07-28 with three security hardenings
+(instance-profile credentials, native-PHP IMDSv2, fail-closed token handling). **`worker2` is now
+the reference implementation — `api2`'s copy has known defects, see below.**
+
+Non-prod `worker2` environments need this because they are reached **over HTTP for manual job
+invocation**, not through SQS.
+
+## How it works
+
+1. **`.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh`**
+   - Reads `get-config environment -k ENVIRONMENT`; if it is `production`, **skips** entirely.
+     Production instances are registered by the existing production process, not by this hook.
+   - Otherwise invokes `ebs/register_instance_to_shared_application_load_balancer.php`.
+   - **Always `exit 0`.** A registration failure must never brick a deploy.
+2. **`ebs/register_instance_to_shared_application_load_balancer.php`**
+   - Reads the instance id and region from
+     `/var/app/current/storage/instance-id.txt` and `storage/region.txt` (written earlier by the
+     `_shared/040-write-instance-id.sh` / `041-write-region.sh` prebuild hooks), falling back to a
+     direct **IMDSv2** lookup.
+   - Reads the EB environment name via `get-config container -k environment_name`.
+   - Uses the AWS SDK (`aws/aws-sdk-php`) ELBv2 client to `DescribeTargetGroups` and selects the
+     target group whose **`TargetGroupName` is an exact string match** for the environment name —
+     this naming equality *is* the wiring convention; there is no tag or config lookup.
+   - Calls `RegisterTargets` with this instance id.
+
+### Hook ordering is load-bearing — why `060`
+
+`060` sorts **after** everything it depends on:
+
+| Hook | Provides |
+|---|---|
+| `prebuild/_shared/040-write-instance-id.sh` | `storage/instance-id.txt` |
+| `prebuild/_shared/041-write-region.sh` | `storage/region.txt` |
+| `postdeploy/015_install_composer.sh` | `vendor/` (the AWS SDK) |
+
+Renumber it below `015` and the SDK autoloader does not exist yet.
+
+### Porting prerequisites (all already satisfied in worker2)
+
+The port needed **no adaptation** because `worker2` already had: `aws/aws-sdk-php` in
+`composer.json`; **byte-identical** `_shared/040-write-instance-id.sh` and `041-write-region.sh`;
+an identical `015_install_composer.sh`; and the same `get-config environment -k ENVIRONMENT`
+convention already used by `.ebextensions/git.php`. Check these four before porting the pair to
+any other 2.0 tier.
+
+## Credentials — instance profile only (decision, 2026-07-28)
+
+**Deploy-time AWS credentials come from the EC2 instance profile. Never from source, never from
+EB environment properties.** The `worker2` version passes **no `credentials`** to the SDK client
+so the default provider chain resolves the instance profile.
+
+**Why EB environment properties were rejected as a fallback:** they are not a secret store. They
+are returned by `elasticbeanstalk:DescribeConfigurationSettings`, displayed in plaintext in the EB
+console and `eb config`, and exported into the process environment — so any `var_dump($_ENV)`,
+`phpinfo()`, or debug handler leaks them. If a static key is ever genuinely unavoidable, fetch it
+at runtime from **SSM Parameter Store SecureString** or Secrets Manager.
+
+### Required IAM on the EB EC2 instance profile — exactly two actions
+
+- **`elasticloadbalancing:DescribeTargetGroups`** — `Resource: "*"` is unavoidable (ELBv2
+  `Describe*` has no resource-level permissions). Constrain it with a condition on
+  `aws:RequestedRegion`.
+- **`elasticloadbalancing:RegisterTargets`** — scope to the **non-production target-group ARN
+  pattern**.
+
+> **Do not use `targetgroup/*/*`.** On a *shared* ALB that lets any instance carrying this profile
+> insert any instance into any target group — a traffic-hijack primitive.
+
+Do **not** grant `DeregisterTargets`, any `Modify*`, or `elasticloadbalancing:*`.
+
+## IMDSv2 must not be fetched by shelling out to `curl`
+
+Passing the IMDSv2 session token as a **`curl` command-line argument** exposes it in
+`/proc/<pid>/cmdline` and in `ps` output to every other process on the instance. The `worker2`
+implementation instead:
+
+- uses a **native PHP stream context** (`stream_context_create` + `file_get_contents`) so the
+  token never enters `argv`;
+- sets the token TTL to **60s instead of 21600s (6h)** — this is a one-shot, sub-second fetch;
+- **fails closed**: if no token can be obtained it aborts, rather than falling back to an
+  unauthenticated IMDSv1 request (a silent downgrade on any instance with `HttpTokens=optional`).
+
+## Gotchas
+
+- **Production is skipped by design.** Adding a prod environment to this hook is a separate
+  decision, not a config tweak.
+- **Naming equality is the contract.** If the target group is not named *exactly* the EB
+  environment name, the hook silently registers nothing (and still exits 0). Check the target
+  group name first when a non-prod env is unreachable after a deploy.
+- **Exit 0 hides failures.** Registration problems will not show in deploy status — read
+  `/var/log/eb-hooks.log` on the instance.
+- **The hook turns on reachability; it does not secure it.** The target group and listener already
+  exist by convention, but because this hook is what actually puts non-prod instances behind the
+  shared ALB, the **listener rules and security groups must restrict non-prod to internal/VPN
+  sources**, and the HTTP-triggered job endpoints must enforce authentication. Open risk, tracked
+  here deliberately.
+
+## api2 divergence — three defects to fix (do not copy the api2 original)
+
+`api2` carries the same hook pair under the same two filenames. Its version predates the
+hardening above and has:
+
+1. **Committed IAM access key — security incident (2026-07-28).** An IAM access key id and secret
+   access key are hardcoded as PHP constants in
+   `api2/ebs/register_instance_to_shared_application_load_balancer.php` (**lines 29–30**) and are
+   present in **committed git history** (found with `git log -S`; commit subject *"Auto-registering
+   to target groups"*). They are exposed to anyone with read access to `api2`, to every clone, and
+   to every EB application bundle built from the repo.
+   **Remediation:** (a) deactivate then delete the key in IAM — rotation, not history rewrite, is
+   what closes the exposure; (b) audit CloudTrail for its use; (c) replace the credential block
+   with the instance-profile pattern above. **Rewriting git history is a separate, sign-off-required
+   decision** — the team git rule forbids force-pushing `_main` in application repos.
+   *(Location + remediation only. Never record the key id, the secret, or any fragment.)*
+2. **IMDSv2 token passed on the `curl` command line** — visible in `ps` / `/proc/<pid>/cmdline`,
+   with a 6-hour TTL.
+3. **Silent IMDSv1 fallback** when no token is obtained — downgrades below the IMDSv2 posture.
+
+This hook pair is also **not yet described in the api2 architecture doc's Deployment section**,
+which lists the other hooks but not `060` — pending an elevated-doc update.
+
+## Change history
+
+- 2026-07-28 — Ported the hook pair from `api2` into `worker2` (new files `060_…​.sh` +
+  `ebs/register_instance_to_shared_application_load_balancer.php`) so non-production worker2
+  environments are reachable through the shared ALB for manual HTTP invocation. Hardened vs. the
+  api2 original: instance-profile credentials only (EB env properties explicitly rejected), native
+  PHP stream-context IMDSv2 with a 60s TTL and fail-closed behaviour, and a two-action least-privilege
+  IAM policy. Recorded the api2 committed-IAM-key incident (location + remediation only) and the
+  non-prod ALB exposure risk. (jcardinal)

package/knowledge/INDEX.md

@@ -17,8 +17,8 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 2.0 framework
 
-- **_underscore** (_Underscore) _(framework core)_ — 39 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 32 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
+- **worker2** (Worker) — 34 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 18 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/prudential/INDEX.md

@@ -3,9 +3,10 @@
 | Doc | Framework | Summary | Files |
 |-----|-----------|---------|-------|
 | [Prudential: Dell ASN units PRE/POST interceptor (legacy key + flat tracking)](features/dell-asn-units-interceptor.md) | 2.0 | After the tracking-number bridge migration, the ASN unit route was renamed (`advance-shipping-notice-units` → `advance-shipping-notice-item-units`), so the inhe | _underscore/Model/Prudential/AdvanceShippingNotice.php, dbchanges2/Client_Prudential/2026-06-10 - AsnUnitsInterceptor.sql |
-| [Prudential: Dell LCH IOP transmissions (LCHRequestV2)](features/dell-lch-iop-transmissions.md) | 1.0 | Outbound order transmissions from the 1.0 worker tier to Dell's Lifecycle Hub (LCH) ITSM integration. | library/app/api/delllch.php, library/app/apitransaction.php, worker/crons/toga2/prudential/transmissions_to_dell_usa.php, worker/crons/toga2/prudential/transmissions_to_dell_ireland.php, worker/crons/toga2/prudential/transmissions_to_dell_india.php |
+| [Prudential: Dell LCH IOP transmissions (LCHRequestV2)](features/dell-lch-iop-transmissions.md) | 1.0 | Outbound order transmissions from the 1.0 worker tier to Dell's Lifecycle Hub (LCH) ITSM integration. | library/app/api/delllch.php, library/app/apitransaction.php, worker/crons/toga2/prudential/transmissions_to_dell_usa.php, worker/crons/toga2/prudential/transmissions_to_dell_ireland.php, worker/crons/toga2/prudential/transmissions_to_dell_india.php, worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php |
 | [Prudential: Device information import + unit→contact linking (import_device_information.php)](features/device-information-import-and-contact-linking.md) | 1.0 | Prudential's **device-sync** cron (`worker/crons/toga2/prudential/import_device_information.php`) pulls device/asset records (from ServiceNow / the Dell CMDB fe | worker/crons/toga2/prudential/import_device_information.php, worker/crons/toga2/prudential/backfill_unit_contacts.php, dbchanges2/Client_Prudential/2026-07-07 - Contact Dedup Merge.sql |
 | [Prudential: Service Request Regional Address Validation](features/service-request-address-validation.md) | 2.0 | The `prePost` interceptor on `_Model_Prudential_ServiceRequest` validates `deliverToAddress` fields differently depending on which Prudential regional customer | _underscore/Model/Prudential/ServiceRequest.php, _underscore/Test/Prudential/ServiceRequestTest.php |
+| [Prudential: Service Request rejection alert email](features/service-request-rejection-alert-email.md) | 2.0 | When a Prudential ServiceNow→TOGa service-request submission (`POST /v2/service-requests`) is **rejected by validation**, TOGa now sends a real-time internal al | worker2/Worker/Client/Prudential/reports/ReqRejectionEmail.php, _underscore/Model/Prudential/ServiceRequest.php |
 | [Prudential Order Shipped Email — transmit_ordershipped_updates_prudential.php](features/transmit-ordershipped-email.md) | 1.0 | Cron script that transmits "Order Shipped" updates to ServiceNow (RITM) and sends a shipped notification email to the end user. | worker/crons/toga2/prudential/transmit_ordershipped_updates_prudential.php, worker/crons/toga2/prudential/transmit_closecomplete_updates_prudential.php, worker/crons/toga2/prudential/transmit_rejected_cancelled_updates_prudential.php, worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php, worker/crons/toga2/prudential_beta/generate_sales_and_purchase_orders_from_service_requests.php, worker/crons/notifications/reports/prudential_exception_report.php |
 | [Prudential Financial](profile.md) | 2.0 | Prudential is a TOGA client whose device-fulfillment flow is driven by **Dell** via the Dell API (`Client_Prudential.Apis.id = 2`). |  |
 | [Prudential: Dell ASN failed POST backfill replay](workflows/dell-asn-backfill-replay.md) | 2.0 | When Dell ASN POSTs fail in bulk (e.g. |  |

package/knowledge/clients/prudential/features/dell-asn-units-interceptor.md

@@ -5,13 +5,14 @@ project: _Underscore
 client: prudential
 type: client-feature
 status: active
-updated: 2026-06-12
-owners: ["jcardinal", "rgirish"]
+updated: 2026-07-28
+owners: ["jcardinal", "rgirish", "bala"]
 files:
   - _underscore/Model/Prudential/AdvanceShippingNotice.php
   - dbchanges2/Client_Prudential/2026-06-10 - AsnUnitsInterceptor.sql
 related:
   - ../../../2.0/apps/_underscore/features/tracking-number-bridges.md
+  - transmit-ordershipped-email.md
 ---
 
 ## Summary
@@ -60,6 +61,24 @@ UPDATE Core.RecordFields SET childPolicy = 'MATCH_CREATE' WHERE id = 2198;
 found. `MATCH` (the default) errors when no row exists — that was the root cause of the Jun 2026
 `EV-12` outage.
 
+## Inbound endpoint & logging (where a Dell ASN lands)
+
+- **Endpoint:** Dell ASNs arrive via inbound `POST /v2/advance-shipping-notices`; **201** =
+  accepted/created.
+- **Success vs. failure log split.** Successful ASN POSTs are logged in `Logs_Prudential.Api`, but
+  **failures often land in the shared base `Logs` schema, not the client log** — so a "no trace in
+  `Logs_Prudential.Api`" ASN may have failed and been recorded in `Logs.Api` instead. Check both.
+- **Filter by `sourceIp` to avoid conflating clients.** Prudential's ASN traffic comes from
+  **`13.86.101.210`** (apiId 2). A *different* client's ASN feed comes from **`34.232.23.158`** —
+  when auditing the shared `Logs` schema, filter on `sourceIp` or you will mix the two clients'
+  ASNs together.
+- **`AdvanceShippingNotices.purchaseOrderId` links an ASN to its PO.** A NULL/unmatched
+  `purchaseOrderId` corresponds to a **400 `EV-12`** on the inbound POST (the ASN could not be
+  matched/created). A successful 201 with a matched PO is what then triggers the shipped-email
+  chain (`transmit_ordershipped_updates_prudential.php` stamps
+  `c_dtTransmittedOrderShippedUpdateToPrudential` + `c_dtEmailSentOrderShipped`) — see
+  `transmit-ordershipped-email.md`.
+
 ## Gotchas
 - Scoped to `apiId = 2` so other (internal) Prudential callers — which already send the new shape —
   are not double-transformed.
@@ -72,6 +91,11 @@ found. `MATCH` (the default) errors when no row exists — that was the root cau
   replay workflow doc.
 
 ## Change history
+- 2026-07-28 — Documented the inbound ASN endpoint (`POST /v2/advance-shipping-notices`, 201 =
+  success), the success-vs-failure log split (successes in `Logs_Prudential.Api`, failures often
+  in shared base `Logs`), the `sourceIp` filter to separate Prudential (`13.86.101.210`) from
+  another client's ASN feed (`34.232.23.158`), and that `AdvanceShippingNotices.purchaseOrderId`
+  links the ASN to its PO (unmatched = 400 `EV-12`) and gates the shipped-email chain. (bala)
 - 2026-06-12 — Fixed interceptor bug: bridge row keys were `trackingNumber`/`returnTrackingNumber`
   instead of `trackingNumberId`/`returnTrackingNumberId`, causing EV-12 on all Dell ASN POSTs. Also
   set `Core.RecordFields childPolicy = MATCH_CREATE` on ids 1437 and 2198. Replayed 28 failed payloads

package/knowledge/clients/prudential/features/dell-lch-iop-transmissions.md

@@ -6,7 +6,7 @@ project: Worker
 client: prudential
 type: client-feature
 status: active
-updated: 2026-07-24
+updated: 2026-07-28
 owners: ["rgirish", "bala"]
 files:
   - library/app/api/delllch.php
@@ -14,6 +14,7 @@ files:
   - worker/crons/toga2/prudential/transmissions_to_dell_usa.php
   - worker/crons/toga2/prudential/transmissions_to_dell_ireland.php
   - worker/crons/toga2/prudential/transmissions_to_dell_india.php
+  - worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php
 related:
   - ../profile.md
   - dell-asn-units-interceptor.md
@@ -57,9 +58,12 @@ WHERE PurchaseOrders.vendorId = 1        # Dell = Vendors.id 1
 ```
 
 - **Region lives on `ServiceRequests.customerId`**, resolved through the join to
-  `Client_Prudential.Customers`. Region rows: **USA = id 3, uuid
-  `b3f7a2c1-4e89-4d6a-9c3b-8f1e5d2a7b04`**; India and Ireland are separate `Customers` rows /
-  uuids. This is NOT the SalesOrder customer — the SO customer is always
+  `Client_Prudential.Customers`. Region rows:
+  - **USA — id 3, uuid `b3f7a2c1-4e89-4d6a-9c3b-8f1e5d2a7b04`**
+  - **Ireland — id 5, uuid `9a4e7d2b-5f13-48c6-b8e1-3d6a9c2f7e85`**
+  - **India — id 4, uuid `e6d1c8f4-2a75-4b3e-a9d7-1c4f6e8b3a52`**
+
+  This is NOT the SalesOrder customer — the SO customer is always
   **"Agilant - Tech Hub" (id 1, uuid `36da53f8-38d2-404f-becf-f58f33c215d3`)**.
 - A REQ is only ever transmitted, validated, or marked once it is selected here. Unit/bundle/SO/PO
   creation happens elsewhere and does **not** send anything to Dell.
@@ -172,6 +176,17 @@ This is the established pattern for all Prudential script changes.
   stuck this way, all with `customerId` NULL. The 2.0-side guard added in
   `service-request-address-validation.md` stops NEW customer-less REQs; already-stuck ones need a
   `customerId` backfill from ship-to country.
+  - **Root cause is a TOGa-side mapping gap, not missing SNOW data.** The inbound ServiceNow
+    payload carries **no customer field** at all — `ServiceRequests.customerId` is assigned
+    *inside TOGa* by `generate_sales_and_purchase_orders_from_service_requests.php`, so a NULL
+    means the generation cron failed to derive/assign a region, not that SNOW omitted anything.
+  - **The country-split cutover (~2026-07-17) is when this started biting.** Before the split,
+    `transmissions_to_dell_v1.php` had **no customer gate**, so every REQ transmitted regardless.
+    The regional split files added `INNER JOIN Customers` + `WHERE Customers.uuid = '<region>'`,
+    which is what drops NULL-`customerId` REQs. ~38 REQs stranded this way since the cutover.
+  - **Remediation:** backfill `ServiceRequests.customerId = 3` (USA) for the stuck US-ship-to
+    REQs; **durable fix** = have the generation cron derive country/`customerId` from
+    `deliverToAddress` so `customerId` is never NULL.
 - **`dtSubmitted` is stamped without inspecting the response (latent bug, not yet fixed).** Every
   regional cron sets `PurchaseOrders.dtSubmitted = NOW()` immediately after the send without
   checking Dell's HTTP status, and `App_ApiTransaction::execute()` (`library/app/apitransaction.php`)
@@ -190,6 +205,14 @@ This is the established pattern for all Prudential script changes.
   do not reintroduce them.
 
 ## Change history
+- 2026-07-28 — Recorded all three region rows (USA id 3, Ireland id 5, India id 4, with uuids) and
+  the root cause of NULL `ServiceRequests.customerId`: the inbound SNOW payload carries no customer
+  field, so `customerId` is assigned TOGa-side by
+  `generate_sales_and_purchase_orders_from_service_requests.php` — a NULL is a TOGa mapping gap.
+  The ~2026-07-17 regional-split cutover added the `INNER JOIN Customers` gate (the old
+  `transmissions_to_dell_v1.php` had none), stranding ~38 REQs. Durable fix = derive
+  country/`customerId` from `deliverToAddress` in the generation cron; interim = backfill
+  `customerId = 3` for stuck US-ship-to REQs. (bala)
 - 2026-07-24 — Documented the live-cron reality (production is the three regional split files under
   `crons/toga2/prudential/`; monolith `transmissions_to_dell.php` disabled, `_v1` dead), the
   selection query, region model on `ServiceRequests.customerId` (USA id 3), the `App_Api_Delllch`

package/knowledge/clients/prudential/features/service-request-address-validation.md

@@ -6,7 +6,7 @@ project: _Underscore
 client: prudential
 type: client-feature
 status: active
-updated: 2026-07-24
+updated: 2026-07-28
 owners: ["rgirish", "bala"]
 files:
   - _underscore/Model/Prudential/ServiceRequest.php
@@ -15,6 +15,7 @@ related:
   - clients/prudential/profile.md
   - clients/prudential/features/dell-asn-units-interceptor.md
   - clients/prudential/features/dell-lch-iop-transmissions.md
+  - clients/prudential/features/service-request-rejection-alert-email.md
   - ../../../2.0/apps/_underscore/features/model-interceptor-unit-testing.md
 ---
 
@@ -47,8 +48,11 @@ back to USA rules.
    `Client_Prudential.Customers` for the matching `name`. Returns `'India'`, `'Ireland'`, or
    `'USA'` (default for anything unrecognised or when uuid is null).
 3. A `match` expression dispatches to the correct validator.
-4. On any validation failure, an `Exception` is thrown with a semicolon-delimited list of all
-   errors — the API engine catches it and returns a 4xx to Dell.
+4. **Errors are collected, not fail-fast.** `prePost` runs all validators and accumulates every
+   error. On any failure it enqueues the internal rejection-alert worker task (see
+   `service-request-rejection-alert-email.md`) and then throws a single `_Exception_Validation`
+   carrying all errors — the API engine maps it to a 400 (behavior unchanged from the caller's
+   perspective; the change is that all errors are now reported at once and staff are alerted).
 
 ## Data model
 
@@ -109,8 +113,22 @@ with a space (e.g. `F92 FP83`). No numeric restriction.
   when Prudential starts including them.
 - **`line3` is India-only.** USA and Ireland validators do not check `line3` (it is not in
   their Dell spec). If Dell starts sending it for USA/Ireland it is silently ignored.
+- **⚠ OPEN (pre-existing HIGH) — unguarded unit access after the validation block throws a 500.**
+  Code that runs *after* the validation block reads
+  `serviceRequestUnits[0]->unit->serialNumber` / `->assetTag` without guarding for a missing
+  `serviceRequestUnits`. A payload with no units throws an **uncaught non-validation
+  `_Exception` → HTTP 500** (which dumps globals in debug mode) and, because it is not an
+  `_Exception_Validation`, it **skips the rejection-alert email** entirely. Fix: guard the unit
+  access (or validate `serviceRequestUnits` presence inside the collected-error block) so it
+  fails as a 400 with an alert instead of a silent 500. Tracked as a follow-up.
 
 ## Change history
+- 2026-07-28 — `prePost` now **collects all validation errors** (was fail-fast) and, on any
+  failure, enqueues the internal rejection-alert worker task before throwing one
+  `_Exception_Validation` with all errors (400 unchanged). See
+  `service-request-rejection-alert-email.md`. Also recorded the open HIGH: unguarded
+  `serviceRequestUnits[0]->unit` access after the validation block throws a 500 and skips the
+  alert. (bala)
 - 2026-07-24 — Added `validateCustomer()` as the first check in `prePost`: rejects a create when
   `customer.uuid` is missing/null/blank (`trim(... ?? '') === ''`) via `_Exception_Validation`
   (→ HTTP 400), stopping customer-less/unroutable REQs at the boundary. Previously a blank uuid

package/knowledge/clients/prudential/features/service-request-rejection-alert-email.md

@@ -0,0 +1,94 @@
+---
+title: "Prudential: Service Request rejection alert email"
+framework: "2.0"
+repo: worker2
+project: Worker
+client: prudential
+type: client-feature
+status: active
+updated: 2026-07-28
+owners: ["bala"]
+files:
+  - worker2/Worker/Client/Prudential/reports/ReqRejectionEmail.php
+  - _underscore/Model/Prudential/ServiceRequest.php
+related:
+  - service-request-address-validation.md
+  - ../../../2.0/apps/_underscore/features/email-send-pipeline.md
+  - ../profile.md
+---
+
+## Summary
+
+When a Prudential ServiceNow→TOGa service-request submission (`POST /v2/service-requests`)
+is **rejected by validation**, TOGa now sends a real-time internal alert email so staff know
+the instant a REQ fails and can tell Prudential exactly what to resubmit. The
+`_Model_Prudential_ServiceRequest::prePost()` interceptor collects **every** validation error
+(not fail-fast), enqueues a worker task carrying the REQ number + reason + full payload, then
+throws a single `_Exception_Validation` — the 400 returned to ServiceNow is unchanged. The
+worker action `_Worker_Client_Prudential_reports_ReqRejectionEmail::sendRejectionEmail` sends
+the alert.
+
+## Scope
+
+- **Validation rejections only.** The MySQL 1062 duplicate-key race on `ServiceRequests.number`
+  (a duplicate SR-number re-POST) is deliberately **out of scope** — it is an internal defect
+  routed to engineering, not a "resubmit this" message for Prudential.
+
+## Key files / entry points
+
+- `_underscore/Model/Prudential/ServiceRequest.php` — `prePost()` collects all validator errors
+  and, on any failure, enqueues the alert via
+  `_Worker::runTask('Client/Prudential/reports/ReqRejectionEmail/sendRejectionEmail', {reqNumber, reason, payload})`
+  before throwing one `_Exception_Validation` carrying all errors.
+- `worker2/Worker/Client/Prudential/reports/ReqRejectionEmail.php` —
+  `sendRejectionEmail(string $reqNumber, string $reason, object|array $payload): ...` builds and
+  sends a **plain-text** email (REQ# + reason + full JSON payload) to a class-constant recipient
+  list (currently `vburks@togatech.com`, `adiamond@togatech.com`).
+
+## How it works
+
+1. `prePost()` runs every validator, **accumulating** errors instead of throwing on the first.
+2. If the error list is non-empty, it enqueues one `ReqRejectionEmail/sendRejectionEmail` worker
+   task with the REQ number, the joined reason string, and the full inbound payload.
+3. It then throws a single `_Exception_Validation` holding all errors → `Controller/Index.php`
+   maps it to **HTTP 400** to ServiceNow (unchanged behavior).
+4. The worker action assembles the plain-text body and sends it via `_Email`. Because sends run
+   through the 2.0 email-send pipeline, the message lands within ~1 minute (see
+   `email-send-pipeline.md`).
+
+## Send guards (why a queue failure can't turn the 400 into a 500)
+
+- `setIsHtml(false)` — plain-text intent (see caveat below).
+- `setClientIdentifier('Prudential')` and a fixed From of `noreply@togatech.com`.
+- `json_encode(..., JSON_INVALID_UTF8_SUBSTITUTE)` with a false-fallback so a bad byte can't
+  fatal the encode; control characters stripped from `reqNumber`.
+- The **whole dispatch is wrapped in a `\Throwable` catch** that `error_log`s an
+  identifier-only message (never PII) — so if the queue/email dispatch fails, the original
+  validation 400 is still what ServiceNow receives; the alert failure never escalates to a 500.
+
+## Accepted residual risk (data-owner decision)
+
+The **full inbound payload (including PII) is emailed** and persisted to `Logs_Prudential.Email`.
+This was an explicit data-owner decision to maximize the staff alert's usefulness, accepted as a
+known residual risk.
+
+## Gotchas / known issues
+
+- **A non-validation exception skips the alert.** The alert only fires for
+  `_Exception_Validation`. A payload that throws a *different* uncaught `_Exception` before the
+  validation block completes (e.g. the unguarded `serviceRequestUnits[0]->unit` access — see
+  `service-request-address-validation.md`) produces a **500 and no alert**. Fix that guard to
+  keep the alert reliable.
+- **"Plain text" is transmitted HTML anyway.** `setIsHtml(false)` is honored by `_Email` when it
+  queues the row, but the worker2 `Infrastructure/Email/Send` action calls `IsHTML(true)`
+  unconditionally, so the message is sent HTML-mode regardless (see `email-send-pipeline.md`).
+
+## Change history
+- 2026-07-28 — Created: real-time internal alert on `POST /v2/service-requests` validation
+  rejection. `prePost()` now collects all validation errors and enqueues the
+  `ReqRejectionEmail/sendRejectionEmail` worker task (REQ# + reason + full payload) before
+  throwing one `_Exception_Validation` (400 unchanged). Recipients are a class constant
+  (`vburks@`, `adiamond@`). Whole dispatch wrapped in `\Throwable` catch so a queue failure can't
+  turn the 400 into a 500. Scope: validation rejections only (the 1062 SR-number duplicate race
+  is routed to engineering). Accepted residual risk: full payload PII is emailed + persisted to
+  `Logs_Prudential.Email`. (bala)

package/knowledge/clients/prudential/profile.md

@@ -7,17 +7,19 @@ apps:
   - dbchanges2
   - websocket
   - worker
+  - worker2
   - library
 project: _Underscore
 client: prudential
 type: profile
 status: active
-updated: 2026-07-07
+updated: 2026-07-28
 owners: ["jcardinal", "rgirish", "bala"]
 files: []
 related:
   - features/dell-asn-units-interceptor.md
   - features/service-request-address-validation.md
+  - features/service-request-rejection-alert-email.md
   - features/dell-lch-iop-transmissions.md
   - features/device-information-import-and-contact-linking.md
 ---
@@ -41,6 +43,23 @@ order-status transmissions.
   POST to Dell's LCH `LCHRequestV2` endpoint via the 1.0 `App_Api_Delllch` client. See the
   Dell LCH IOP transmissions feature doc.
 
+## Reference data (enums)
+
+`Client_Prudential.PurchaseOrders.purchaseOrderStageId` — PO lifecycle stage:
+
+| id | Stage         | id | Stage         |
+|----|---------------|----|---------------|
+| 3  | Open          | 4  | Configuration |
+| 9  | Accepted      | 5  | Ready To Ship |
+| 7  | In Progress   | 2  | Received      |
+| 6  | POD           | 1  | Rejected      |
+| 8  | Cancelled     |    |               |
+
+- PO lifecycle is tracked by `purchaseOrderStageId`. The `dtAcknowledged` column is **unused
+  (always NULL)** — do not key logic on it.
+
+`ServiceRequests.serviceRequestTypeId`: **1 = New Hire, 2 = Breakfix, 3 = Refresh, 5 = Reclaim**.
+
 ## Gotchas
 - Dell will not change their payload shape — see the Dell ASN units interceptor feature doc for the
   PRE/POST translation that keeps their feed working after the tracking-number bridge migration.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.461",
+  "version": "1.0.463",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",