toga-ai 1.0.622 → 1.0.623

package/knowledge/sessions/2026-08-20-true-80952-queued-attachments-mhammontree.md

@@ -0,0 +1,88 @@
+---
+type: session
+slug: true-80952-queued-attachments
+title: TRUE-80952 outbound attachments lost at the email queue
+author: mhammontree
+repos: [library, togadesk]
+framework: "1.0"
+client: shared
+status: active
+created: 2026-08-20
+updated: 2026-08-20
+---
+
+# Session: true-80952-queued-attachments
+**Date:** 2026-08-20
+**Project/Repo:** library (1.0 core) + togadesk (1.0 app)
+**Task:** TRUE-80952 — investigate and fix why outbound attachments never arrive from TOGa Desk; root cause found at the email queue hop, fixed in `library/app/email.php`, committed but not deployed or verified in prod.
+
+---
+
+## What WORKED
+
+- **Root cause identified and traced end to end.** `App_Email::send()` queues an email as `base64_encode(serialize($this))` into `Common.EmailsQueued` and holds attachments **only as local filesystem PATHS**. The queue is drained by `worker/crons/notifications/infrastructure/send_emails.php` (scheduled `* * * * *`) on a **different host**, where the queueing app's `desk/uploads/` does not exist. PHPMailer 5's `AddAttachment()` (`library/phpmailer/class.phpmailer.php:1358`) catches the unreadable path as `STOP_CONTINUE` and, with `$exceptions` off, **swallows it and sends the email anyway** — body intact, attachment gone, nothing logged. Exact match for the reported symptom.
+- **Confirmed the togadesk side was already correct.** `class.notification.php:86-101` gathers files into `['path'=>..,'name'=>..]` with a `file_exists()` guard and logging; `functions.php:1077-1090` passes them to `addAttachment()`. Both work. The loss is one hop later.
+- **Fix implemented and committed** — `library/app/email.php`, +98/−2, commit `b3a9420d` on branch `TRUE-80952`. New private `embedAttachmentsForQueue()` reads each attachment into base64 bytes before serialization; `send()` delivers embedded entries via `AddStringAttachment()`; new constant `MAX_QUEUED_ATTACHMENT_BYTES = 7340032`.
+- **Regression suite: 26 assertions, all passing on PHP 7.2** — `C:\WWW\test\@Mark\TRUE-80952\test_queued_email_attachments.php`. Standalone (no DB, no SMTP): requires `library/app/email.php` directly and uses `ReflectionMethod` for the private method. The core test **deletes the source file after queueing** to simulate the drain host.
+- **Verified the suite genuinely fails pre-fix** — ran it against `git show HEAD:app/email.php` in a scratch tree; fatals with `ReflectionException: Method App_Email::embedAttachmentsForQueue() does not exist`. It is a real regression test, not one written to the new code.
+- **`php -l` clean on both PHP 7.2 (xampp_7_2_33) and PHP 8 (xampp_8x).** No typed properties introduced; file stays 7.2-compatible.
+- **php-reviewer agent run; both real findings fixed** (see Decisions). Its third point — that the base64 inflation comment said 3/4 — was **wrong**; the comment already reads "~4/3", which is correct. Left as-is.
+- **Empirically disproved the drain cron's exclusion filter.** `send_emails.php:34` has `AND EmailsQueued.serializedEmailObject NOT LIKE '%App_Email%'`. Ran a PHP round-trip: `base64_encode(serialize($appEmailObject))` never contains the literal `App_Email`. The filter matches nothing.
+- **Enumerated the blast radius across all 1.0 apps.** Only **two** call sites both queue and attach: togadesk's `sendEmail()`, and `worker/crons/notifications/covid/send_covid_pending_vaccination_approval.php` (lines 44 + 82, bare-string form — same bug today, fixed by the same change). worker's other 16 `setQueueForSending` sites pass `false`; togaview's two queueing sites (`mvc/signup/post.php`, `mvc/reset_password_success/post.php`) have no attachments; `tools` and `walmarttechservices` never attach.
+- **Confirmed `Common.EmailsQueued.serializedEmailObject` is LONGTEXT** (developer checked) — no `dbchanges` migration needed.
+- **Knowledge captured and PUSHED to `_main`** — 5 docs, index regenerated (392 docs / 30 repos), mirrored to `C:\WWW\.claude\knowledge`.
+
+## What did NOT work — DO NOT RETRY THESE
+
+- **Testing this on alpha, beta, or a local laptop. It produces a FALSE PASS.** `App_Email::setQueueForSending()` (`library/app/email.php:161`) is gated on `!App_Registry::inTestMode()`:
+  `$this->queueForSending = (($bool && !App_Registry::inTestMode()) ? … : false);`
+  `inTestMode()` is `config['internal']['test_mode'] > 0`. Enumerated **every** togadesk config: **prod = 0; alpha = 1; beta = 1; and all sixteen `config.dev-*.ini` = 1** (bhavana-laptop, chadwimberly-mactop, davidfranks-laptop, hkurupati, jeffcardinal-alien, jeffcardinal-laptop, jeffcardinal-work, markhammontree-laptop, milindprabhakar-laptop, mkelly, nidhi-laptop, nkeshavamurthy-laptop, nmvenkatesha-laptop, pinalsoni-work, snaredla-laptop, tannercox-laptop). Outside prod the email is sent **inline**, PHPMailer reads the file off local disk, the attachment works, and `embedAttachmentsForQueue()` is **never called**. This is almost certainly why TRUE-75813 and TRUE-79924 were verified and closed while prod stayed broken.
+- **Fixing the upload path inside togadesk.** Three prior attempts did this — TRUE-75813 (`2700c28f`, `d01a21a3`), TRUE-79924 (`7c19371a`), and `4f3789a7` ("Changing the directory path", which corrected `UPLOAD_BASE_DIR` from `__DIR__.'/../uploads'` to `__DIR__.'/../../uploads'`). The path resolves correctly and `desk/uploads` exists. **The path was never the problem.** Do not re-open this line of investigation.
+- **Setting `test_mode = 0` on alpha to force the queue path — rejected, do not do this.** `test_mode` also drives App_Email's redirect to `config['email']['dev_email_to']`. Turning it off means **real email to real recipients** sitting in alpha's ticket data. The safe bypass is assigning the **public** property `$email->queueForSending = true` directly, which skips the setter's gate while leaving test mode on.
+- **Reproducing the bug locally even with queueing forced on.** Locally the queueing app and the drain share one filesystem, so `desk/uploads/...` still resolves and both old and new code pass. The source file must be **deleted between queue and drain** for the test to mean anything.
+- **Writing the PHP test file via a bash heredoc.** `cat > file <<'PHPEOF'` failed with `/usr/bin/bash: -c: line 163: unexpected EOF while looking for matching '`. Used the Write tool instead. Not worth re-attempting through bash.
+
+## Not tried yet (candidates for next session)
+
+- **Check `max_allowed_packet` on the prod Common cluster** — `SHOW VARIABLES LIKE 'max_allowed_packet'` via the TOGa Database Integration MCP, environment `legacy`. Rows can now carry ~9.8 MB of base64. MySQL 5.7 defaults to 4 MB, 8.0 to 64 MB. If below the budget the INSERT fails and surfaces as an `App_Query` exception on a live ticket reply — worse than a missing attachment.
+- **Write the CLI verification harness** (offered, not written): build an `App_Email` with an attachment, set the public `$email->queueForSending = true` to bypass the test-mode gate, `send()`, **rename the source file**, then unserialize the `EmailsQueued` row and dump `getAttachments()`. Proves both the queue and drain halves, changes no config, sends no mail.
+- **Confirm which git ref `setup_git_libraries.php` clones.** It is fetched from S3 (`asifiles.s3.us-west-2.amazonaws.com/system/setup_git_libraries.php`) by `togadesk/.ebextensions/020_setup_git_libraries.config` and cannot be read locally. If it clones `_main`, the fix needs a PR and merge before any deploy does anything.
+- **Prod verification** — after a ticket reply with an attachment, confirm the fresh `EmailsQueued` row's decoded payload contains `"data"` not `"path"`. This proves the fix independently of delivery, and the queue drains every minute. Use the TOGA Technology Helpdesk client (178), not a real client ticket.
+- **File two follow-up tickets** (both found this session, neither touched): the AIG arm never attaching, and the dead `NOT LIKE '%App_Email%'` filter.
+
+## Current file state
+
+| File | Status | Notes |
+|------|--------|-------|
+| `C:\WWW\library\app\email.php` | Modified, **committed** | `b3a9420d` on branch `TRUE-80952`, +98/−2. **Not pushed, not merged.** Adds `MAX_QUEUED_ATTACHMENT_BYTES`, `embedAttachmentsForQueue()`, `AddStringAttachment` handling, and the queue/inline fallback. |
+| `C:\WWW\test\@Mark\TRUE-80952\test_queued_email_attachments.php` | New, uncommitted | 26 assertions, passing on 7.2, confirmed failing pre-fix. Test repo is not branched — per-dev folder only. |
+| `knowledge/1.0/apps/library/features/email-queue-attachments.md` | Created, **pushed** | Full writeup: mechanism, failure, budget, warning guard, blast radius, deploy order, dead filter, open packet risk. |
+| `knowledge/1.0/apps/togadesk/features/notifications.md` | Updated, **pushed** | Loss-at-queue + non-prod false-pass gotchas; AIG arm never attaching. Owners unioned `[ajean, mhammontree]`. |
+| `knowledge/1.0/apps/test/features/static-no-db-regression-harness.md` | Updated, **pushed** | Reflection + deleted-source-file worked example. |
+| `knowledge/1.0/standards/backend-php.md` | Updated (⚠ ELEVATED), **pushed** | TRUE-80952 worked example on the warning-exit section; new subsection "Test mode silently disables queued email — non-prod cannot validate a queue bug". |
+| `C:\WWW\togadesk\**` | **Unchanged** | No togadesk code change was needed — it keeps calling `setQueueForSending(true)` and passing paths. |
+| `C:\WWW\worker\**` | **Unchanged** | Drain cron needs no change; it just calls `->send()` on the unserialized object. |
+
+## Decisions made
+
+- **Option B (embed bytes in `library`) over Option A (togadesk skips the queue when attachments present) or Option C (move uploads to S3).** B fixes the class of bug for every 1.0 app and every queue consumer. A was ~2 lines but leaves all other queued-attachment paths broken and puts SMTP inside a web request. C is correct long-term but far beyond this ticket.
+- **7 MB raw budget (`MAX_QUEUED_ATTACHMENT_BYTES = 7340032`) with an inline-send fallback.** Base64 inflates by 4/3, so ~9.8 MB encoded — just under the SES 10 MB message cap. Over budget, `send()` sends inline rather than queueing something that would be rejected downstream.
+- **Outbound-email-disabled + oversized attachments: log, do not force a send.** There is no inline fallback when sending is disabled, so the email queues with its paths and still loses the attachment on drain. Unfixable, but it must not be silent — silence was the original complaint. Raised by php-reviewer.
+- **Scoped `set_error_handler` (pattern 1 from the 1.0 back-end standard) rather than `App_Error::setThrowExceptionsEnabled(false)`.** Pattern 1 was chosen because the code needs catch-and-continue **per attachment**, not blanket suppression across the whole method. Raised by php-reviewer and it is the most important of its findings: an unguarded `file_get_contents` warning is fatal in 1.0 (`App_Error::handleError` → `handleException` → `exit`) and would kill the `send_emails` cron **mid-loop**, stranding every queued email behind it. `is_file()`/`is_readable()` alone cannot prevent it — TOCTOU.
+- **Deploy `worker` before `togadesk`.** The intermediate states are asymmetric: worker-new + togadesk-old is harmless (togadesk queues paths, worker still handles paths), but togadesk-new + worker-old **drops the attachment** because old drain code looks for `['path']` and an embedded entry has none. Rollback is the inverse: togadesk first, drain the queue empty, then worker.
+- **Did NOT touch `send_emails.php:34`'s dead `NOT LIKE '%App_Email%'` filter.** It excludes nothing today, but "fixing" it to work as apparently intended would **stop TOGa Desk mail from sending entirely**. Needs a decision from whoever wrote it.
+- **Did NOT touch the AIG email arm.** Separate pre-existing bug, separate ticket.
+- **Added `: bool` to the new private method** despite the file having no return types elsewhere — free, correct, and 7.2 supports it. Rejected the reviewer's framing that this was a violation; the coding-style rule covers public/protected only.
+
+## Blockers
+
+- **This cannot be validated outside production.** Every non-prod environment has `test_mode = 1`, which disables queueing entirely (see "What did NOT work"). Agreed with the developer that TRUE-80952 is a **production deployment and test**. The changes are in the deployment pipeline.
+- **`library` `b3a9420d` is committed but not pushed or merged**, and it is unknown whether prod pulls `_main` or a branch — so it is unconfirmed whether the pipeline will actually carry this fix.
+- **`max_allowed_packet` on the prod Common cluster is unchecked.** This is the one item that could make the deploy worse than the bug.
+
+## Exact next step
+
+> Run `SHOW VARIABLES LIKE 'max_allowed_packet'` against the prod **Common** cluster (TOGa Database Integration MCP, environment `legacy`) — get developer confirmation first, it is production. If the value is below ~10485760, lower `MAX_QUEUED_ATTACHMENT_BYTES` in `C:\WWW\library\app\email.php` to sit under it, amend/add a commit on branch `TRUE-80952`, and re-run `C:\WWW\test\@Mark\TRUE-80952\test_queued_email_attachments.php` (expect 26 passing) — all **before** the pipeline deploys.
+
+---
+_Saved by /session-save on 2026-08-20_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.622",
+  "version": "1.0.623",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",