toga-ai 1.0.621 → 1.0.623

package/knowledge/1.0/apps/library/INDEX.md

@@ -9,6 +9,7 @@
 | [Cron Execution Monitoring (App_Framework check-in/out → CronJobExecutions)](features/cron-execution-monitoring.md) | `App_Framework::cronInitialization()` / `App_Framework::cronFinished()` (in `library/app/framework.php`) give every 1.0 (`App_`) cron job a check-in/check-out l | library/app/framework.php |
 | [Diagnostic Dialog — View Recommended Services Routing](features/diagnostic-dialog-view-recommended-services.md) | Two "View Recommended Services" buttons exist in the TOGa Refresh 2026 SR view: 1. | library/app/model/toga/diagnostic.php, library/app/model/servicerequest.php |
 | [Elite Freshservice Sync (library)](features/elite-freshservice-sync.md) | `App_Api_Toga2` in `library/app/api/toga2.php` orchestrates bidirectional sync between TOGA 2 and TOGaDesk. | library/app/api/toga2.php |
+| [App_Email Queued Sending & Attachments (Common.EmailsQueued)](features/email-queue-attachments.md) | `App_Email::send()` can either send **inline** (PHPMailer talks to SES right there) or **queue** the message: `base64(serialize($this))` is inserted into `Commo | library/app/email.php, library/phpmailer/class.phpmailer.php, worker/crons/notifications/infrastructure/send_emails.php, worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php, togadesk/desk/includes/functions.php |
 | [Branded HTML Email Templates (App_Email_Template)](features/email-templates.md) | `App_Email_Template` (`app/email/template.php`) is the base class for branded HTML emails in the 1.0 (`App_`) framework. | library/app/email/template.php, library/app/email/agilant.php |
 | [Error Capture in 1.0 (App_Error_Capture → shared 2.0 Logs DB)](features/error-capture-1-0.md) | The 1.0 side of the platform error-reporting pipeline (TRUE-78188). | library/app/error/capture.php, library/app/error.php, library/app/exception/business.php, library/app/api/toga2.php, library/app/cloud.php, worker/config.worker.ini, worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php |
 | [HTTP 500 Error Monitor (App_SystemMonitor_500Error) — and why its \"Error Type\" is not a diagnosis](features/http-500-error-monitor.md) | `App_SystemMonitor_500Error` (`library/app/systemmonitor/500error.php`, title **"HTTP 500 Error Alert"**) is the 1.0 system monitor that watches **`Logs.Api` fo | library/app/systemmonitor/500error.php, worker/crons/infrastructure/system_monitors.php, api2/Controller/Index.php, _underscore/Error.php |

package/knowledge/1.0/apps/library/features/email-queue-attachments.md

@@ -0,0 +1,112 @@
+---
+title: App_Email Queued Sending & Attachments (Common.EmailsQueued)
+framework: "1.0"
+repo: library
+project: Library
+client: shared
+type: feature
+status: active
+updated: 2026-08-20
+owners: ["mhammontree"]
+files:
+  - library/app/email.php
+  - library/phpmailer/class.phpmailer.php
+  - worker/crons/notifications/infrastructure/send_emails.php
+  - worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php
+  - togadesk/desk/includes/functions.php
+related:
+  - ./email-templates.md
+  - ../architecture.md
+  - ../../togadesk/features/notifications.md
+---
+
+## Summary
+
+`App_Email::send()` can either send **inline** (PHPMailer talks to SES right there) or **queue**
+the message: `base64(serialize($this))` is inserted into `Common.EmailsQueued` and drained later
+by a **different app on a different host** — `worker/crons/notifications/infrastructure/send_emails.php`,
+scheduled `* * * * *`.
+
+Two facts dominate everything else here:
+
+1. **Attachments used to be queued as filesystem *paths*.** The drain host has no such path, and
+   PHPMailer 5 silently swallows an unreadable attachment, so the mail arrived with the body
+   intact and the attachment gone — **no error, no log**. Fixed 2026-08-20 (TRUE-80952) by
+   embedding the file **bytes** at queue time.
+2. **Queued email does not queue outside production.** `setQueueForSending()` is gated on
+   `!App_Registry::inTestMode()`, and every non-prod config has `internal.test_mode = 1`. So on
+   alpha/beta/local the message is sent **inline** off local disk and attachments always work —
+   any test there is a **false pass**.
+
+## How it works
+
+- `App_Email::setQueueForSending($bool)` (`library/app/email.php` ~L161):
+  `$this->queueForSending = (($bool && !App_Registry::inTestMode()) ? … : false);`
+- `send()` with queueing on serializes `$this` and inserts into `Common.EmailsQueued`
+  (`serializedEmailObject`, LONGTEXT).
+- Before serializing, `App_Email::embedAttachmentsForQueue()` (private, added TRUE-80952) reads
+  each attachment into **base64 bytes**. On drain, embedded entries go out via PHPMailer's
+  `AddStringAttachment()`; legacy bare-string path entries still use `AddAttachment()`.
+- **Size budget:** `MAX_QUEUED_ATTACHMENT_BYTES = 7340032` (7 MB raw ≈ 9.8 MB base64, under the
+  SES 10 MB message cap). Over budget, embedding returns false and `send()` **falls back to an
+  inline send** instead of queueing — except when outbound email is disabled, where there is no
+  inline fallback and the case is logged.
+- The file read is wrapped in a **local `set_error_handler` → `ErrorException` with a `finally`
+  restore**. This is mandatory, not stylistic: in 1.0 any PHP warning routes through
+  `App_Error::handleError` → `handleException` → `exit()`, so an unguarded `file_get_contents`
+  warning would kill the `send_emails` cron **mid-loop** and strand every queued email behind it.
+  `is_file()`/`is_readable()` are not sufficient — the file can vanish between check and read (TOCTOU).
+
+## Blast radius (verified 2026-08-20)
+
+Only **two** 1.0 call sites both queue *and* attach: TOGa Desk's `sendEmail()`
+(`togadesk/desk/includes/functions.php`) and
+`worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php` (~L44 + L82, bare
+string path — it had the identical bug and is fixed by the same change). worker's other 16
+`setQueueForSending` sites pass `false`; togaview's two queueing sites (`mvc/signup/post.php`,
+`mvc/reset_password_success/post.php`) attach nothing; `tools` and `walmarttechservices` never
+attach. Inline sending and bare-string path attachments are unchanged.
+
+## Deploy ordering (asymmetric — and so is rollback)
+
+The queue payload format changed, and the queueing app and the draining app deploy separately.
+
+- worker **new** + togadesk **old** → togadesk queues paths, new worker still handles paths. Safe.
+- worker **old** + togadesk **new** → togadesk queues `['data' => …]`, the old worker looks for
+  `['path']`, finds none, and **drops the attachment** — same symptom, new cause.
+
+Therefore: **deploy `worker` before `togadesk`.** Roll back in the inverse order — roll back
+togadesk first, let the queue drain empty, *then* worker.
+
+## Gotchas / known issues
+
+- **You cannot reproduce or validate a queued-email bug outside production.** `test_mode = 1` on
+  alpha, beta and every dev config collapses `setQueueForSending(true)` to `false`. This is the
+  most likely reason TRUE-75813 and TRUE-79924 were "verified" and closed while prod stayed
+  broken — both corrected the upload path *inside togadesk*, which was already working; the file
+  was lost one hop later, at the queue. Locally the queueing app and the drain also share one
+  filesystem, so even with queueing forced on the path resolves — you must **delete the source
+  file between queue and drain** to simulate the drain host.
+- **PHPMailer 5 hides a missing attachment.** `AddAttachment()`
+  (`library/phpmailer/class.phpmailer.php` ~L1358) treats an unreadable path as `STOP_CONTINUE`
+  with `$exceptions` off, and sends the message anyway. Absence of an error proves nothing.
+- **The drain cron's exclusion filter is dead code.** `send_emails.php` ~L34 carries
+  `AND EmailsQueued.serializedEmailObject NOT LIKE '%App_Email%'`. The column holds base64, in
+  which the literal `App_Email` never appears (verified with a PHP round-trip) — the filter
+  matches and excludes nothing. **Not changed**: "fixing" it would stop TOGa Desk mail entirely.
+  Needs a decision from whoever wrote it before anyone touches it.
+- **OPEN RISK — `max_allowed_packet` on the prod Common cluster is unverified.** Rows can now
+  carry ~9.8 MB of base64. The column is LONGTEXT so it is fine, but MySQL 5.7 defaults
+  `max_allowed_packet` to 4 MB (8.0: 64 MB). If prod is below the budget the INSERT fails as an
+  `App_Query` exception on a live ticket reply — worse than a missing attachment. Check it, and
+  lower `MAX_QUEUED_ATTACHMENT_BYTES` beneath it if needed (the oversized path already degrades
+  to an inline send, which is the correct behavior).
+- Regression coverage: `test/@Mark/TRUE-80952/test_queued_email_attachments.php` — 26 assertions,
+  no DB, no SMTP, confirmed failing against pre-fix HEAD.
+
+## Change history
+- 2026-08-20 — TRUE-80952: queued attachments now embed base64 bytes (`embedAttachmentsForQueue()`
+  + `AddStringAttachment`) instead of paths that do not exist on the drain host; added
+  `MAX_QUEUED_ATTACHMENT_BYTES` with inline-send fallback and a scoped warning guard around the
+  file read; recorded the `test_mode` no-queue trap, the dead drain filter, and the
+  worker-before-togadesk deploy order (mhammontree)

package/knowledge/1.0/apps/test/INDEX.md

@@ -10,7 +10,7 @@
 | [Developer Generators (password, UUID)](features/dev-generators.md) | Two tiny **1.0 `App_` framework** convenience scripts for everyday developer needs. | test/team/generate_password.php, test/team/uuid.php |
 | [Forecast vs NetSuite Discrepancy Analysis](features/forecast-netsuite-discrepancy-analysis.md) | `team/forecast-netsuite/discrepancy_analysis.php` detects discrepancies between our **Forecast database** and **NetSuite** (the source of truth for all sales da | test/team/forecast-netsuite/discrepancy_analysis.php |
 | [@goagilant.com → @togatech.com Email-Domain Migration (1.0 + 2.0)](features/goagilant-to-togatech-email-migration.md) | Reference + technique for migrating the company email domain `@goagilant.com` → `@togatech.com` across **both** platforms. | migrate_goagilant_to_togatech_2026-06-26.sql, migrate_goagilant_to_togatech_LEGACY_2026-06-26.sql |
-| [Static (no-DB) Regression Harness for 1.0 Logic + Source Drift Guard](features/static-no-db-regression-harness.md) | 1.0 has **no PHPUnit**, and most of its business logic sits inside methods that also write SQL, so "just call it" means standing up a client database. | test/@Mark/AIG/test_multi_email.php, library/app/database.php, library/app/api/toga2.php |
+| [Static (no-DB) Regression Harness for 1.0 Logic + Source Drift Guard](features/static-no-db-regression-harness.md) | 1.0 has **no PHPUnit**, and most of its business logic sits inside methods that also write SQL, so "just call it" means standing up a client database. | test/@Mark/AIG/test_multi_email.php, test/@Mark/TRUE-80952/test_queued_email_attachments.php, library/app/database.php, library/app/api/toga2.php |
 | [TableView Builder (2.0 TableViews SQL generator)](features/tableview-builder.md) | `team/tableViewBuilder/` generates SQL `INSERT` statements for the **2.0 `TableViews`**, `TableViewFields`, and `TableViewJoins` tables from a plain SQL `SELECT | test/team/tableViewBuilder/TableViewGenerator.php, test/team/tableViewBuilder/index.php, test/team/tableViewBuilder/Instructions.md |
 | [Talos Knowledge Base Pipeline (Uploader + Processor)](features/talos-kb-pipeline.md) | `team/talos/` holds the two-script web tooling that feeds the **TOGa Talos** (TOGa IQ) AI knowledge bases. | test/team/talos/kb_uploader.php, test/team/talos/kb_processor.php, test/team/talos/kb_processor.ini |
 | [TOGa 2.0 Client Onboarding SQL Generator](features/toga2-client-onboarding-sql.md) | > **Superseded by the browser wizard.** The generation logic here was extracted into the reusable > `OnboardingSqlGenerator` class and wrapped in a local browse | test/team/generate_toga2_onboarding_sql.php |

package/knowledge/1.0/apps/test/features/static-no-db-regression-harness.md

@@ -6,10 +6,11 @@ project: Test
 client: shared
 type: feature
 status: active
-updated: 2026-08-03
+updated: 2026-08-20
 owners: ["mhammontree"]
 files:
   - test/@Mark/AIG/test_multi_email.php
+  - test/@Mark/TRUE-80952/test_queued_email_attachments.php
   - library/app/database.php
   - library/app/api/toga2.php
 related:
@@ -75,6 +76,17 @@ method**, which is a large share of `library`.
 entirely. That was out of scope because it changes production code; prefer it when you are
 already editing the method.
 
+## Second worked example — private methods via `ReflectionMethod`
+
+`test/@Mark/TRUE-80952/test_queued_email_attachments.php` (26 assertions, PHP 7.2, no DB, no SMTP)
+extends the same pattern to a **private** method: it `require_once`s `library/app/email.php`
+directly and invokes `App_Email::embedAttachmentsForQueue()` through `ReflectionMethod` with
+`setAccessible(true)`. It covers the embed + serialize round-trip **with the source file deleted**
+(the real production failure mode), a missing file, a directory path, an empty file, the legacy
+bare-string path form, idempotent re-embed, exact-limit and aggregate-limit budget checks, and
+that the scoped error handler is restored. Confirmed failing against pre-fix HEAD — a 1.0
+regression test is only credible if you have run it against the broken code.
+
 ## Gotchas / known issues
 
 - **A mirrored test proves the mirror, not production.** The drift guard is what makes it
@@ -102,6 +114,8 @@ already editing the method.
   **`worker2` / `_underscore` enforce 8.1** — a helper shared between the two must satisfy 7.2.
 
 ## Change history
+- 2026-08-20 — added the TRUE-80952 `ReflectionMethod` variant (private-method coverage, source
+  file deleted to simulate the queue-drain host) as a second worked example (mhammontree)
 
 - 2026-08-12 — Sharpened the 7.2 lint gotcha into a real verification procedure: the default CLI is
   PHP 8.x, so `php -l` there proves nothing; use an actual 7.2 binary, add `-n` if its `php.ini` is

package/knowledge/1.0/apps/togadesk/INDEX.md

@@ -6,7 +6,7 @@
 | [Email-to-Ticket Intake (crons/tickets.php)](features/email-to-ticket-intake.md) | TOGa Desk ingests support email into tickets through a cron-driven IMAP poller (`crons/tickets.php`) plus a postfix pipe variant (`crons/pipe.php`). | crons/tickets.php, crons/tickets_prod.php, crons/pipe.php, desk/includes/classes/class.ticket.php, vendor/classes/class.imap.php |
 | [Field-Service Dispatch (central / repair orders)](features/field-service-dispatch.md) | The **central** subsystem is TOGa Desk's field-service dispatch domain: repair-order lifecycle, technician scheduling, onsite vs depot service, parts, and shipm | desk/includes/controllers/actions/central/, desk/includes/classes/class.repair.php, desk/includes/classes/class.repairhistory.php, desk/_/browser/datatable/central.php, desk/template/pages/central.php, desk/template/pages/central/view.php, desk/template/modals/central/addTracking.php, library/app/model/togadesk/repairorder.php, library/app/model/togadesk/repairordertracking.php, library/app/api/carrier/ups.php |
 | [Managed Service Order Create Flow (New MSO modal → tickets/add)](features/managed-service-order-create.md) | The **Managed Service Order (MSO) create flow** is how a TOGa Desk user manually creates a managed service order: pick an End User (or a Client Location), choos | desk/includes/controllers/actions/tickets/add.php, desk/template/modals/tickets/addNew.php, desk/template/pages/managed.php, desk/template/pages/getinfo.php, desk/template/footer.php, library/app/model/togadesk/repairorder.php |
-| [Ticket Email Notifications (notifications table)](features/notifications.md) | Which TOGa Desk emails fire for a given client is driven **entirely by data**, not code: the `TOGaDeskSupport.notifications` table holds one row per `(clientid, | desk/includes/classes/class.notification.php, crons/tickets.php, crons/tickets_prod.php |
+| [Ticket Email Notifications (notifications table)](features/notifications.md) | Which TOGa Desk emails fire for a given client is driven **entirely by data**, not code: the `TOGaDeskSupport.notifications` table holds one row per `(clientid, | desk/includes/classes/class.notification.php, desk/includes/functions.php, desk/includes/classes/class.ticket.php, crons/tickets.php, crons/tickets_prod.php |
 | [Per-Client Hostname Restriction (getRestrictedClient)](features/per-client-host-restriction.md) | TOGa Desk supports **per-client branded URLs** (e.g. | desk/includes/functions.php, desk/template/header.php, _/browser/datatable/tickets, desk/includes/classes/class.ticket.php, ebs/setup_phpini.php, ebs/http_to_https.php, desk/template/modals/kb/viewDocument.php, desk/template/modals/files/aws-view.php, desk/template/modals/files/contract-view.php, template/pages/documents/view.php, template/pages/tasks/view.php |
 | [REST API (RPC-over-POST) & API-Key Auth](features/rest-api.md) | TOGa Desk exposes a programmatic API at `desk/api/`. | desk/api/index.php, desk/api/resources/tickets.php, desk/api/resources/assets.php, desk/api/resources/authenticate.php, desk/includes/classes/class.apikey.php, desk/includes/functions.php |
 | [SMB Contract Editing & the clientMspId Corruption Trap](features/smb-contract-editing.md) | The SMB contracts page (`/desk/?route=toga/smbcontracts&togaClientId=<id>`) edits `TOGA_*.SMBContracts` rows via a modal. | desk/template/modals/toga/smbcontracts/smbContract.php, desk/includes/controllers/modals/toga/smbcontracts/smbContract.php, desk/includes/controllers/actions/toga/smbcontracts/smbContract.php, desk/includes/controllers/actions/toga/smbcontracts/edit.php |

package/knowledge/1.0/apps/togadesk/features/notifications.md

@@ -6,13 +6,16 @@ project: TOGa Desk
 client: shared
 type: feature
 status: active
-updated: 2026-06-24
-owners: ["ajean"]
+updated: 2026-08-20
+owners: ["ajean", "mhammontree"]
 files:
   - desk/includes/classes/class.notification.php
+  - desk/includes/functions.php
+  - desk/includes/classes/class.ticket.php
   - crons/tickets.php
   - crons/tickets_prod.php
 related:
+  - 1.0/apps/library/features/email-queue-attachments.md
   - 1.0/apps/togadesk/features/ticket-lifecycle.md
   - 1.0/apps/togadesk/features/email-to-ticket-intake.md
   - 1.0/apps/togadesk/architecture.md
@@ -55,6 +58,20 @@ nothing if the `STAFF_*` row is absent.
 - `TOGaDeskSupport.people_departments` — peopleid ↔ departmentid mapping.
 
 ## Gotchas / known issues
+- **Outbound attachments were dropped at the email queue, not in TOGa Desk.** `sendEmail()`
+  queues via `App_Email` into `Common.EmailsQueued`; until TRUE-80952 attachments were serialized
+  as local **paths** that do not exist on the worker host that drains the queue, and PHPMailer 5
+  silently sent the mail without them. Two earlier tickets (TRUE-75813, TRUE-79924) "fixed" the
+  upload path inside TOGa Desk — which was never the problem — and passed testing because
+  **alpha/beta/local run `test_mode = 1`, which disables queueing entirely and sends inline**.
+  Never validate an attachment/email change for TOGa Desk on a non-prod environment. See
+  [App_Email queued sending & attachments](../../library/features/email-queue-attachments.md).
+- **The AIG branch of `sendEmail()` never attaches anything (open bug, separate from TRUE-80952).**
+  `desk/includes/functions.php` ~L961-975 and the AIG arm of `addReply()` in
+  `class.ticket.php` build `App_Email_AigTemplates_SendAigSupportTicketReplyNotification` and pass
+  `$attachments` only to `logEmail()` — **`addAttachment()` is never called**. Staples Protection
+  ticket replies therefore lose attachments for a different reason, and the TRUE-80952 fix does
+  not help them. Needs its own ticket.
 - **Silent disable by omission.** A missing `(clientid, type)` row produces no email and no log
   line — diagnose "missing notification" reports by querying `notifications` for that client's
   rows first, before suspecting recipient flags or code.
@@ -70,6 +87,9 @@ nothing if the `STAFF_*` row is absent.
   notification still depends on a `STAFF_NEW_TICKET` row existing for the ticket's client.
 
 ## Change history
+- 2026-08-20 — TRUE-80952: recorded that outbound attachments were lost at the `App_Email` queue
+  (not in TOGa Desk), that non-prod `test_mode` disables queueing and yields false passes, and the
+  separate AIG-arm bug where `$attachments` never reaches `addAttachment()` (mhammontree)
 - 2026-06-24 — documented the `notifications` `(clientid, type)` data model and staff-recipient
   resolution; recorded the clientid-178 missing-`STAFF_*`-rows incident (ajean)
 

package/knowledge/1.0/standards/backend-php.md

@@ -5,11 +5,12 @@ project: Library
 client: shared
 type: standard
 status: active
-updated: 2026-08-14
+updated: 2026-08-20
 owners: [jcardinal, rgirish, mhammontree, ajean]
 files: []
 related:
   - ../apps/library/architecture.md
+  - ../apps/library/features/email-queue-attachments.md
   - ../apps/library/features/error-capture-1-0.md
   - ../../2.0/standards/backend-php.md
 ---
@@ -306,6 +307,14 @@ Two sanctioned patterns, in order of preference:
 
 Never leave either one in effect beyond the block that needs it.
 
+**Worked example — reading a file you are about to serialize (TRUE-80952).**
+`App_Email::embedAttachmentsForQueue()` wraps its `file_get_contents` in a local
+`set_error_handler` → `ErrorException` with a `finally` restore. Guarding with
+`is_file()`/`is_readable()` alone is **not** sufficient: the file can vanish between the
+check and the read (TOCTOU), and the resulting warning would `exit()` the
+`send_emails` cron **mid-loop**, stranding every queued email behind it. Any file read
+inside a loop that processes a queue takes pattern 1.
+
 ### Start the session before App_Error's fatal handler — and set the handler in code, not INI
 
 On EB PHP 8.5 / AL2023, the `.so` extension and cookie flags aside, two rules make Redis
@@ -501,6 +510,25 @@ Additional verified detail:
 > (~L112-123), `clients/prudential/profile.md` (~L75). Prefer the local-handler pattern
 > from the canonical section above for anything new.
 
+### Test mode silently disables queued email — non-prod cannot validate a queue bug
+
+`App_Email::setQueueForSending()` (`library/app/email.php` ~L161) is gated on
+`!App_Registry::inTestMode()`:
+
+    $this->queueForSending = (($bool && !App_Registry::inTestMode()) ? … : false);
+
+`inTestMode()` is `config['internal']['test_mode'] > 0`, and **every non-prod config is 1**
+(verified for togadesk: prod 0; alpha 1; beta 1; all `config.dev-*.ini` 1). So outside
+production `setQueueForSending(true)` collapses to `false` and the email is sent **inline**,
+off the local filesystem, by the same process that built it. Anything that only breaks on the
+queue path — a serialization gap, a resource the drain host cannot reach — **passes on
+alpha/beta/local**. TRUE-75813 and TRUE-79924 were both verified and closed this way while
+production stayed broken.
+
+Rule: a change to queued email is not verified until it is verified in production, or against
+a harness that exercises the serialize → drain boundary with the queueing host's local state
+removed. Treat a non-prod pass on queue code as no evidence at all.
+
 ## Security Best Practices
 
 ### Input Validation and Sanitization

package/knowledge/INDEX.md

@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 
 ## 1.0 framework
 
-- **library** (Library) _(framework core)_ — 18 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
+- **library** (Library) _(framework core)_ — 19 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
 - **worker** (Worker) — 25 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
 - **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
 - **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)

package/knowledge/sessions/2026-08-20-true-80952-queued-attachments-mhammontree.md

@@ -0,0 +1,88 @@
+---
+type: session
+slug: true-80952-queued-attachments
+title: TRUE-80952 outbound attachments lost at the email queue
+author: mhammontree
+repos: [library, togadesk]
+framework: "1.0"
+client: shared
+status: active
+created: 2026-08-20
+updated: 2026-08-20
+---
+
+# Session: true-80952-queued-attachments
+**Date:** 2026-08-20
+**Project/Repo:** library (1.0 core) + togadesk (1.0 app)
+**Task:** TRUE-80952 — investigate and fix why outbound attachments never arrive from TOGa Desk; root cause found at the email queue hop, fixed in `library/app/email.php`, committed but not deployed or verified in prod.
+
+---
+
+## What WORKED
+
+- **Root cause identified and traced end to end.** `App_Email::send()` queues an email as `base64_encode(serialize($this))` into `Common.EmailsQueued` and holds attachments **only as local filesystem PATHS**. The queue is drained by `worker/crons/notifications/infrastructure/send_emails.php` (scheduled `* * * * *`) on a **different host**, where the queueing app's `desk/uploads/` does not exist. PHPMailer 5's `AddAttachment()` (`library/phpmailer/class.phpmailer.php:1358`) catches the unreadable path as `STOP_CONTINUE` and, with `$exceptions` off, **swallows it and sends the email anyway** — body intact, attachment gone, nothing logged. Exact match for the reported symptom.
+- **Confirmed the togadesk side was already correct.** `class.notification.php:86-101` gathers files into `['path'=>..,'name'=>..]` with a `file_exists()` guard and logging; `functions.php:1077-1090` passes them to `addAttachment()`. Both work. The loss is one hop later.
+- **Fix implemented and committed** — `library/app/email.php`, +98/−2, commit `b3a9420d` on branch `TRUE-80952`. New private `embedAttachmentsForQueue()` reads each attachment into base64 bytes before serialization; `send()` delivers embedded entries via `AddStringAttachment()`; new constant `MAX_QUEUED_ATTACHMENT_BYTES = 7340032`.
+- **Regression suite: 26 assertions, all passing on PHP 7.2** — `C:\WWW\test\@Mark\TRUE-80952\test_queued_email_attachments.php`. Standalone (no DB, no SMTP): requires `library/app/email.php` directly and uses `ReflectionMethod` for the private method. The core test **deletes the source file after queueing** to simulate the drain host.
+- **Verified the suite genuinely fails pre-fix** — ran it against `git show HEAD:app/email.php` in a scratch tree; fatals with `ReflectionException: Method App_Email::embedAttachmentsForQueue() does not exist`. It is a real regression test, not one written to the new code.
+- **`php -l` clean on both PHP 7.2 (xampp_7_2_33) and PHP 8 (xampp_8x).** No typed properties introduced; file stays 7.2-compatible.
+- **php-reviewer agent run; both real findings fixed** (see Decisions). Its third point — that the base64 inflation comment said 3/4 — was **wrong**; the comment already reads "~4/3", which is correct. Left as-is.
+- **Empirically disproved the drain cron's exclusion filter.** `send_emails.php:34` has `AND EmailsQueued.serializedEmailObject NOT LIKE '%App_Email%'`. Ran a PHP round-trip: `base64_encode(serialize($appEmailObject))` never contains the literal `App_Email`. The filter matches nothing.
+- **Enumerated the blast radius across all 1.0 apps.** Only **two** call sites both queue and attach: togadesk's `sendEmail()`, and `worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php` (lines 44 + 82, bare-string form — same bug today, fixed by the same change). worker's other 16 `setQueueForSending` sites pass `false`; togaview's two queueing sites (`mvc/signup/post.php`, `mvc/reset_password_success/post.php`) have no attachments; `tools` and `walmarttechservices` never attach.
+- **Confirmed `Common.EmailsQueued.serializedEmailObject` is LONGTEXT** (developer checked) — no `dbchanges` migration needed.
+- **Knowledge captured and PUSHED to `_main`** — 5 docs, index regenerated (392 docs / 30 repos), mirrored to `C:\WWW\.claude\knowledge`.
+
+## What did NOT work — DO NOT RETRY THESE
+
+- **Testing this on alpha, beta, or a local laptop. It produces a FALSE PASS.** `App_Email::setQueueForSending()` (`library/app/email.php:161`) is gated on `!App_Registry::inTestMode()`:
+  `$this->queueForSending = (($bool && !App_Registry::inTestMode()) ? … : false);`
+  `inTestMode()` is `config['internal']['test_mode'] > 0`. Enumerated **every** togadesk config: **prod = 0; alpha = 1; beta = 1; and all sixteen `config.dev-*.ini` = 1** (bhavana-laptop, chadwimberly-mactop, davidfranks-laptop, hkurupati, jeffcardinal-alien, jeffcardinal-laptop, jeffcardinal-work, markhammontree-laptop, milindprabhakar-laptop, mkelly, nidhi-laptop, nkeshavamurthy-laptop, nmvenkatesha-laptop, pinalsoni-work, snaredla-laptop, tannercox-laptop). Outside prod the email is sent **inline**, PHPMailer reads the file off local disk, the attachment works, and `embedAttachmentsForQueue()` is **never called**. This is almost certainly why TRUE-75813 and TRUE-79924 were verified and closed while prod stayed broken.
+- **Fixing the upload path inside togadesk.** Three prior attempts did this — TRUE-75813 (`2700c28f`, `d01a21a3`), TRUE-79924 (`7c19371a`), and `4f3789a7` ("Changing the directory path", which corrected `UPLOAD_BASE_DIR` from `__DIR__.'/../uploads'` to `__DIR__.'/../../uploads'`). The path resolves correctly and `desk/uploads` exists. **The path was never the problem.** Do not re-open this line of investigation.
+- **Setting `test_mode = 0` on alpha to force the queue path — rejected, do not do this.** `test_mode` also drives App_Email's redirect to `config['email']['dev_email_to']`. Turning it off means **real email to real recipients** sitting in alpha's ticket data. The safe bypass is assigning the **public** property `$email->queueForSending = true` directly, which skips the setter's gate while leaving test mode on.
+- **Reproducing the bug locally even with queueing forced on.** Locally the queueing app and the drain share one filesystem, so `desk/uploads/...` still resolves and both old and new code pass. The source file must be **deleted between queue and drain** for the test to mean anything.
+- **Writing the PHP test file via a bash heredoc.** `cat > file <<'PHPEOF'` failed with `/usr/bin/bash: -c: line 163: unexpected EOF while looking for matching '`. Used the Write tool instead. Not worth re-attempting through bash.
+
+## Not tried yet (candidates for next session)
+
+- **Check `max_allowed_packet` on the prod Common cluster** — `SHOW VARIABLES LIKE 'max_allowed_packet'` via the TOGa Database Integration MCP, environment `legacy`. Rows can now carry ~9.8 MB of base64. MySQL 5.7 defaults to 4 MB, 8.0 to 64 MB. If below the budget the INSERT fails and surfaces as an `App_Query` exception on a live ticket reply — worse than a missing attachment.
+- **Write the CLI verification harness** (offered, not written): build an `App_Email` with an attachment, set the public `$email->queueForSending = true` to bypass the test-mode gate, `send()`, **rename the source file**, then unserialize the `EmailsQueued` row and dump `getAttachments()`. Proves both the queue and drain halves, changes no config, sends no mail.
+- **Confirm which git ref `setup_git_libraries.php` clones.** It is fetched from S3 (`asifiles.s3.us-west-2.amazonaws.com/system/setup_git_libraries.php`) by `togadesk/.ebextensions/020_setup_git_libraries.config` and cannot be read locally. If it clones `_main`, the fix needs a PR and merge before any deploy does anything.
+- **Prod verification** — after a ticket reply with an attachment, confirm the fresh `EmailsQueued` row's decoded payload contains `"data"` not `"path"`. This proves the fix independently of delivery, and the queue drains every minute. Use the TOGA Technology Helpdesk client (178), not a real client ticket.
+- **File two follow-up tickets** (both found this session, neither touched): the AIG arm never attaching, and the dead `NOT LIKE '%App_Email%'` filter.
+
+## Current file state
+
+| File | Status | Notes |
+|------|--------|-------|
+| `C:\WWW\library\app\email.php` | Modified, **committed** | `b3a9420d` on branch `TRUE-80952`, +98/−2. **Not pushed, not merged.** Adds `MAX_QUEUED_ATTACHMENT_BYTES`, `embedAttachmentsForQueue()`, `AddStringAttachment` handling, and the queue/inline fallback. |
+| `C:\WWW\test\@Mark\TRUE-80952\test_queued_email_attachments.php` | New, uncommitted | 26 assertions, passing on 7.2, confirmed failing pre-fix. Test repo is not branched — per-dev folder only. |
+| `knowledge/1.0/apps/library/features/email-queue-attachments.md` | Created, **pushed** | Full writeup: mechanism, failure, budget, warning guard, blast radius, deploy order, dead filter, open packet risk. |
+| `knowledge/1.0/apps/togadesk/features/notifications.md` | Updated, **pushed** | Loss-at-queue + non-prod false-pass gotchas; AIG arm never attaching. Owners unioned `[ajean, mhammontree]`. |
+| `knowledge/1.0/apps/test/features/static-no-db-regression-harness.md` | Updated, **pushed** | Reflection + deleted-source-file worked example. |
+| `knowledge/1.0/standards/backend-php.md` | Updated (⚠ ELEVATED), **pushed** | TRUE-80952 worked example on the warning-exit section; new subsection "Test mode silently disables queued email — non-prod cannot validate a queue bug". |
+| `C:\WWW\togadesk\**` | **Unchanged** | No togadesk code change was needed — it keeps calling `setQueueForSending(true)` and passing paths. |
+| `C:\WWW\worker\**` | **Unchanged** | Drain cron needs no change; it just calls `->send()` on the unserialized object. |
+
+## Decisions made
+
+- **Option B (embed bytes in `library`) over Option A (togadesk skips the queue when attachments present) or Option C (move uploads to S3).** B fixes the class of bug for every 1.0 app and every queue consumer. A was ~2 lines but leaves all other queued-attachment paths broken and puts SMTP inside a web request. C is correct long-term but far beyond this ticket.
+- **7 MB raw budget (`MAX_QUEUED_ATTACHMENT_BYTES = 7340032`) with an inline-send fallback.** Base64 inflates by 4/3, so ~9.8 MB encoded — just under the SES 10 MB message cap. Over budget, `send()` sends inline rather than queueing something that would be rejected downstream.
+- **Outbound-email-disabled + oversized attachments: log, do not force a send.** There is no inline fallback when sending is disabled, so the email queues with its paths and still loses the attachment on drain. Unfixable, but it must not be silent — silence was the original complaint. Raised by php-reviewer.
+- **Scoped `set_error_handler` (pattern 1 from the 1.0 back-end standard) rather than `App_Error::setThrowExceptionsEnabled(false)`.** Pattern 1 was chosen because the code needs catch-and-continue **per attachment**, not blanket suppression across the whole method. Raised by php-reviewer and it is the most important of its findings: an unguarded `file_get_contents` warning is fatal in 1.0 (`App_Error::handleError` → `handleException` → `exit`) and would kill the `send_emails` cron **mid-loop**, stranding every queued email behind it. `is_file()`/`is_readable()` alone cannot prevent it — TOCTOU.
+- **Deploy `worker` before `togadesk`.** The intermediate states are asymmetric: worker-new + togadesk-old is harmless (togadesk queues paths, worker still handles paths), but togadesk-new + worker-old **drops the attachment** because old drain code looks for `['path']` and an embedded entry has none. Rollback is the inverse: togadesk first, drain the queue empty, then worker.
+- **Did NOT touch `send_emails.php:34`'s dead `NOT LIKE '%App_Email%'` filter.** It excludes nothing today, but "fixing" it to work as apparently intended would **stop TOGa Desk mail from sending entirely**. Needs a decision from whoever wrote it.
+- **Did NOT touch the AIG email arm.** Separate pre-existing bug, separate ticket.
+- **Added `: bool` to the new private method** despite the file having no return types elsewhere — free, correct, and 7.2 supports it. Rejected the reviewer's framing that this was a violation; the coding-style rule covers public/protected only.
+
+## Blockers
+
+- **This cannot be validated outside production.** Every non-prod environment has `test_mode = 1`, which disables queueing entirely (see "What did NOT work"). Agreed with the developer that TRUE-80952 is a **production deployment and test**. The changes are in the deployment pipeline.
+- **`library` `b3a9420d` is committed but not pushed or merged**, and it is unknown whether prod pulls `_main` or a branch — so it is unconfirmed whether the pipeline will actually carry this fix.
+- **`max_allowed_packet` on the prod Common cluster is unchecked.** This is the one item that could make the deploy worse than the bug.
+
+## Exact next step
+
+> Run `SHOW VARIABLES LIKE 'max_allowed_packet'` against the prod **Common** cluster (TOGa Database Integration MCP, environment `legacy`) — get developer confirmation first, it is production. If the value is below ~10485760, lower `MAX_QUEUED_ATTACHMENT_BYTES` in `C:\WWW\library\app\email.php` to sit under it, amend/add a commit on branch `TRUE-80952`, and re-run `C:\WWW\test\@Mark\TRUE-80952\test_queued_email_attachments.php` (expect 26 passing) — all **before** the pipeline deploys.
+
+---
+_Saved by /session-save on 2026-08-20_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.621",
+  "version": "1.0.623",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",