toga-ai 1.0.1086 → 1.0.1087

package/knowledge/1.0/apps/togadesk/features/email-to-ticket-intake.md

@@ -11,9 +11,6 @@ owners: ["jcardinal", "dfranks", "mhammontree"]
 files:
   - crons/tickets.php
   - crons/tickets_prod.php
-  - crons/tickets_prod_mailbox.php
-  - crons/includes/ticket_attachments.php
-  - crons/recover_attachments.php
   - crons/pipe.php
   - desk/includes/classes/class.ticket.php
   - desk/includes/classes/class.user.php
@@ -39,22 +36,14 @@ TOGa Desk ingests support email into tickets through a cron-driven IMAP poller (
 - `Ticket::emailToTicket()` in `desk/includes/classes/class.ticket.php` — the parser/router.
 
 ### Flow
-1. **Bootstrap:** cron wrappers load the `library` framework + `App_Framework::cronInitialization()`, set `$env`, and register the IMAP (`Imap2`) helper. `$scriptpath` = `togadesk/desk`; autoload = `desk/vendor/autoload.php` (**no AWS SDK**) + `vendorClassAutoload` (`desk/vendor/classes/`). `tickets_prod.php` requires `tickets_prod_mailbox.php` for the **OAuth2/MSAL** O365 mailbox token.
+1. **Bootstrap:** cron wrappers load the `library` framework + `App_Framework::cronInitialization()`, set `$env`, and register the IMAP (`Imap2`) helper. `$scriptpath` = `togadesk/desk`; autoload = `desk/vendor/autoload.php` (**no AWS SDK**) + `vendorClassAutoload` (`desk/vendor/classes/`). `tickets_prod.php` additionally performs **OAuth2/MSAL** token acquisition for the O365 mailbox.
 2. **Poll:** connect to `outlook.office365.com:993` INBOX and loop for ~22 minutes (header comment suggests running every ~30 min), processing each message then moving it to the IMAP folder **"Trash"** (see Gotchas).
 3. **Parse & route (`emailToTicket()`):** extract from/to/cc/subject/body (with encoding fixes); match an existing ticket via `In-Reply-To` + `Message-ID` headers and **plus-addressing** (`support+<ticketnum>@…`). Matched ⇒ `addReply()`; otherwise ⇒ `add()`.
    - **Noise suppression (before any ticket/attachment is created):** both feeders drop known automated no-noise mail via the shared helper `Ticket::isSuppressedNoiseEmail(?string $from, ?string $subject): bool` (static, in `class.ticket.php`). It returns true only when the sender matches `/^no-?reply@/` **AND** the subject starts with `staples po ` — both required so a real person is never blocked, and the sender check is **domain-agnostic** (matches `noreply@togatech.com` / `noreply@goagilant.com` and survives future sender-domain flips; the Staples PO alert sender flipped `goagilant.com → togatech.com` on 2026-06-02). Wired into BOTH intake feeders: `crons/tickets.php` adds it as one more `&&`-clause in the existing IMAP intake guard (alongside the delivery-status / `MAILER-DAEMON` / Compass auto-reply filters); `crons/pipe.php` uses it as an early guard-and-`exit(0)` right after the `$from`/`$subject` parse. Each feeder logs a "suppressed" line. (TRUE-79975 — ~815 such alerts since Jun 2024, ~704 of them landing in the dept-1 "Support Desk" catch-all and being closed by hand.)
-4. **Side effects:** auto-create a user when the sender domain maps to an existing client; assign to a department via the department email lookup; save attachments (helpers in `crons/includes/ticket_attachments.php`) **straight into `desk/uploads`** — on the worker a symlink to the s3fs mount (`/var/app/asifiles/togadesk-prod/uploads`), so the write is the S3 write. Before writing it checks `is_link` + written size. **`App_Cloud` is not used** (PR #625, TRUE-82935). Blocked names (`TICKET_IMPORT_BLOCKED_FILE_PATTERN`: php*/cgi/sh/asp/jsp/html/htm/xhtml/shtml/svg/svgz/xml/xsl/xslt/xht/mht/mhtml/shtm/stm/ssi, `.htaccess`, `.user.ini`) are saved with `.txt` added. Optionally generate Talos summaries.
+4. **Side effects:** auto-create a user when the sender domain maps to an existing client; assign to a department via the department email lookup; save attachments **straight into `desk/uploads`** — on the worker a symlink to the s3fs mount (`/var/app/asifiles/togadesk-prod/uploads`), so the write is the S3 write. Before writing it checks `is_link` + written size. **`App_Cloud` is not used** (PR #625, TRUE-82935). Blocked names (`TICKET_IMPORT_BLOCKED_FILE_PATTERN` in `crons/tickets.php`: php*/cgi/sh/asp/jsp, html/htm, xht/xhtm/xhtml, shtm/shtml, stm, ssi, svg/svgz, xml, xsl/xslt, mht/mhtml, `.htaccess`, `.user.ini`) are saved with `.txt` added (widened TRUE-82935, PR #626). Optionally generate Talos summaries.
 5. **Post-pass:** `Ticket::processRules()` (escalation) and `Ticket::autoClose()` (SLA auto-close) run after the mailbox sweep.
 6. **On failure:** alert email to `serviceops@togatech.com`, also logged in legacy `Logs.Emails` (`subject LIKE 'TOGaDesk email import failed%'`) — the fastest way to read the cron's error (the cron has no stdout).
 
-### Attachment recovery (`crons/recover_attachments.php`, one-time, TRUE-82935)
-Re-saves attachments lost 2026-10-12 → 2026-10-19 from the originals in IMAP "Trash".
-- Read-only on the mailbox (`FT_PEEK`; `Imap::getAttachment()` has an optional `$peek` param).
-- Match: email `Message-ID` = `tickets_replies.emailMessageIdentifier` on exactly one ticket; duplicate Message-IDs in the folder skipped; `From` must equal the reply sender's `people.email`.
-- Re-saves missing bytes for existing `files` rows under the same name; inserts rows that were deleted; relinks inline `cid:` images.
-- Dry run by default; `--apply`, `--since`, `--id`, `--folder`, `--max-mb`, `--limit`. Refuses `--apply` unless `desk/uploads` is the S3 link.
-- Run on the worker between intake runs: `php /var/www/html/ontrack/crons/recover_attachments_prod.php`. Ship **after** the intake fix, as its own deploy (CTO decision).
-
 ### Data model
 Writes `tickets`, `tickets_replies`, attachment `files`; reads config from the `config` table (`tickets_defaultdepartment`, `tickets_defaultclient`, `auto_close_tickets`, `email_from_address`, `timezone`). Department routing keys on a `tickets_departments.email` `LIKE` match against the To/CC addresses; **anything unmatched falls through to config `tickets_defaultdepartment` (= dept 1 "Support Desk" catch-all) and `tickets_defaultclient`** — which is why unrouted/automated mail piles into "Support Desk". There is **no allowlist/blocklist config framework**: noise filtering is ad-hoc inline guards in the crons (the hardcoded Compass "Automatic reply:" filter and now `isSuppressedNoiseEmail`).
 
@@ -85,9 +74,9 @@ Ruled out as user-creation sources (do not re-chase):
 - **`$processName` guard:** `tickets.php` throws if executed directly instead of via an env wrapper — and `ENVIRONMENT`/`$env` must be set or it loads the wrong config.
 - **A togadesk merge does NOT update the cron.** The cron runs on the 1.0 **worker** (`/var/www/html/ontrack`, cloned from togadesk at worker deploy — NOT `/var/www/ontrack`). The `Worker` CodePipeline sources only `worker` + `library`; a togadesk merge runs `TOGaDesk-Production` (web envs) only. After any `crons/` change, **release the `Worker` pipeline by hand**. TRUE-82935: PR #625 merged 10-16 16:15, reached the cron only 10-19 13:10. See [worker architecture](../../worker/architecture.md).
 - **Never load a second composer autoloader in the cron as a fallback** (e.g. root `vendor/autoload.php` for the AWS SDK) — CTO decision TRUE-82935. Keep the cron off `App_Cloud`; any `App_Cloud` call there throws `Error` ([App_Cloud](../../library/features/app-cloud-aws-credentials.md)).
-- **Attachment outage 2026-10-12 → 10-19 (TRUE-82935).** 10-12 15:00 → 10-16 15:00: `files` rows written, S3 object missing (`copyFileToS3` threw). 10-16 15:16 → 10-19 13:10: PR #624 deleted the row on failure → no attachments at all. First good run 10-19 13:20 (files 615609-615611). Recovery: section above.
+- **Attachment outage 2026-10-12 → 10-19 (TRUE-82935).** 10-12 15:00 → 10-16 15:00: `files` rows written, S3 object missing (`copyFileToS3` threw). 10-16 15:16 → 10-19 13:10: PR #624 deleted the row on failure → no attachments at all. First good run 10-19 13:20 (files 615609-615611). Attachments lost 2026-10-12 15:00 → 2026-10-19 13:10 were **NOT recovered** (developer decision, TRUE-82935); the original emails remain in IMAP "Trash" if ever needed.
 - **Processed mail is moved, never purged.** `desk/vendor/classes/class.imap.php` `getTrash()` matches only `trash` / `inbox.trash` / `papierkorb`, else creates a custom folder **"Trash"** — originals stay recoverable there.
-- **OAuth2:** the prod O365 app secret has a hard expiry (~2026-12-20); rotate before then or intake silently stops. **Security, open:** the client secret for `support@togatech.com` is committed in `crons/tickets_prod_mailbox.php` (was `tickets_prod.php`) — rotate and move to config/SSM (value not recorded).
+- **OAuth2:** the prod O365 app secret has a hard expiry (~2026-12-20); rotate before then or intake silently stops. **Security, open:** the client secret for `support@togatech.com` is committed in `crons/tickets_prod.php` — rotate and move to config/SSM (value not recorded).
 - **Loop prevention** only excludes FROM==department-TO addresses; CC/reply-all across multiple department addresses can still loop.
 - **Auto-reply filters** are partly hardcoded (e.g. ignore `TechHub@compass-usa.com` "Automatic reply:" messages), and now the `isSuppressedNoiseEmail` Staples-PO filter — there is no config-driven allow/block list, so each new noise source is another inline guard.
 - **`Ticket::extractEmail()` returned an undefined offset on a no-match** (bare `$matches[0]`). With a sender-less/malformed email this codebase promotes the warning to a thrown error (`E_ALL` + `App_Error::handleError`), so an intake cron could **halt**. Hardened to `return $matches[0] ?? '';` (TRUE-79975) — important because `isSuppressedNoiseEmail` routes nullable/malformed sender input through `extractEmail`, putting this latent bug on the hot path.
@@ -95,7 +84,7 @@ Ruled out as user-creation sources (do not re-chase):
 - **`adminid` clobber on reply routing:** after `emailToTicket()` sets `$data['adminid']` from the FROM address (~line 1407), an asset lookup (~line 1437-1442) overwrites it with the replying user's asset's assigned admin. On a matched-ticket reply this makes a user's email look like a staff reply in `addReply()` (`isAdminReply=true`), so `Awaiting User` etc. never transition. The asset-derived adminid is for NEW-ticket auto-assign in `add()`, not reply classification. See the ticket-lifecycle doc.
 
 ## Change history
-- 2026-10-19 — Attachments saved straight into `desk/uploads` (s3fs), no `App_Cloud`; recovery CLI; worker-pipeline deploy gotcha; corrected live `class.imap.php` copy; wider blocked-file pattern (TRUE-82935) (jcardinal)
+- 2026-10-19 — Attachments saved straight into `desk/uploads` (s3fs), no `App_Cloud`; lost attachments not recovered; worker-pipeline deploy gotcha; corrected live `class.imap.php` copy; wider blocked-file pattern (TRUE-82935) (jcardinal)
 - 2026-09-25 — renamed product reference from "TogaIQ" to "Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)
 
 ## Related

package/knowledge/1.0/apps/worker/architecture.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: architecture
 status: active
-updated: 2026-10-12
+updated: 2026-10-19
 owners: [jcardinal, sking, bala]
 files:
   - worker/index.php
@@ -22,6 +22,7 @@ related:
   - ../dbchanges/workflows/authoring-and-shipping-sql-files.md
   - ./features/oneuptime-worker-uptime-monitoring.md
   - ../../2.0/apps/worker2/features/worker-fleet-role-assignment-monitor.md
+  - ../togadesk/features/email-to-ticket-intake.md
 ---
 
 The 1.0 `App_` background-job tier (~550 CRON scripts); open for how crons are scheduled, how the 7-worker role election works, and its silent-death failure modes.
@@ -32,7 +33,7 @@ The 1.0 `App_` background-job tier (~550 CRON scripts); open for how crons are s
 
 Production: **one Elastic Beanstalk environment running 7 instances**, each **self-electing a distinct role** (`notification`, `database`, `infrastructure`, `toga`, `togadesk`, `sync`, `catalog`) and running only that role's slice of the schedule. Platform: **PHP 7.2 on 64-bit Amazon Linux / 2.8.8**. This is the "legacy worker" from the 1.0 back-end standard's *Asynchronous / Background Processing* note; it builds on the `library` core.
 
-**Critical rules:** all classes use the `App_` prefix (1.0). Crons are plain PHP under `crons/`; the **schedule (`schedules/*.json`) is the source of truth** — a script not referenced by a schedule never runs. Client integrations live under `crons/toga2/<client>/`. Escape/cast all SQL input; never interpolate raw input. A worker that fails deploy-time role election runs **NO** crons while EB still reads healthy — never treat "EB green" as "fleet working"; check `Vision_Log.Workers` against the actual EB instance list.
+**Critical rules:** all classes use the `App_` prefix (1.0). Crons are plain PHP under `crons/`; the **schedule (`schedules/*.json`) is the source of truth** — a script not referenced by a schedule never runs. Client integrations live under `crons/toga2/<client>/`. Escape/cast all SQL input; never interpolate raw input. A worker that fails deploy-time role election runs **NO** crons while EB still reads healthy — never treat "EB green" as "fleet working"; check `Vision_Log.Workers` against the actual EB instance list. togadesk crons run from a deploy-time clone — a togadesk merge does not update them; release the `Worker` pipeline.
 
 ## How a job becomes a cron (the dispatch pipeline)
 
@@ -153,7 +154,7 @@ The worker box is assembled from several repos. `ebs/git.json` clones:
 | `library` | `/var/www/library` | 1.0 framework core (`_.php`, all `App_*`). |
 | `resources` | `…/resources` | Front-end assets (not used by crons). |
 | `dbchanges` | `/var/www/dbchanges` | DB migration scripts **+ their runner (`index.php`)**. Applied by the 2-min `crons/infrastructure/execute_dbchanges.php` cron on `_%` branches with a matching `config.<env>.ini`, and by `.ebextensions/030_dbchanges.config` on deploy. Authoring rules: [Authoring & Shipping a 1.0 dbchanges SQL File](../dbchanges/workflows/authoring-and-shipping-sql-files.md). |
-| `togadesk` | `…/ontrack` | TOGaDesk app — `cron.worker.togadesk.json` runs `../ontrack/crons/tickets_prod.php` and `monitoring_prod.php` **out of the sibling repo**. |
+| `togadesk` | `/var/app/ondeck/ontrack` → `/var/app/current/ontrack` = **`/var/www/html/ontrack`** (not `/var/www/ontrack`) | TOGaDesk app — `cron.worker.togadesk.json` runs `../ontrack/crons/tickets_prod.php` (from `/var/www/html/crons`) and `monitoring_prod.php` **out of the sibling repo**. Updated **only when the `Worker` pipeline runs** — see Gotchas. |
 | `_underscore` | `/var/www/_underscore` | **2.0 framework core** — present because `toga2/` client integrations reach into 2.0 (TOGa-2 / TOGaSupply). A genuine 1.0↔2.0 bridge inside a 1.0 app. |
 
 ## Deploy-time provisioning (`.ebextensions/` → `ebs/`)
@@ -167,6 +168,7 @@ Container commands set up the box per deploy: apache tuning, force HTTP→HTTPS,
 - **⚠ SECURITY — plaintext GitHub token in `crons/infrastructure/execute_dbchanges.php`.** A hardcoded GitHub personal access token sits in the config block near the top (~L14–18) and is interpolated into a `shell_exec()` git-clone command line, exposing it in the worker host's process listing. Treat as **compromised**: rotate, move to `config.<env>.ini`/env, stop passing it in argv. (No credential value recorded in the KB.)
 - **Schedules are the source of truth for what runs.** A `.php` under `crons/` that no schedule references is dormant (and may be `DOA/`). Set `active: 0` to disable without deleting.
 - **All cron paths are relative to `crons/`** except the `../ontrack/...` jobs that run from the sibling TOGaDesk repo.
+- **A togadesk merge does NOT reach the worker.** CodePipeline `Worker` sources only `worker` + `library` `_production`; togadesk is git-cloned at worker deploy. A togadesk merge runs `TOGaDesk-Production` (EB `agilant-togadesk-prod`, `odp-togadesk-prod`, `imhouston-togadesk-prod`) only. After any togadesk `crons/` change, **release the `Worker` pipeline by hand**. TRUE-82935: an email-intake fix merged 2026-10-16 reached the cron only 2026-10-19. Symlink `ontrack/desk/uploads` → `/var/app/asifiles/togadesk-prod/uploads` (`ebs/setup_symbolic_links.php` + `links.worker.json`). See [email intake](../togadesk/features/email-to-ticket-intake.md).
 - **Always call `App_Database::closeConnections()`** at the end; rely on `cronInitialization`'s overlap guard rather than rolling your own lock.
 - **Timezone is `America/Chicago`** everywhere; cron expressions are in that zone, but several `togadesk/` UMA jobs are commented with their UTC equivalents — read the `name` carefully.
 - **Two stale-worker cutoffs exist** (270 s in the heartbeat, 7 min in the deploy script) and the `$desiredWorkers` list is duplicated in `ebs/cron.worker.php` and `App_Worker` — keep them aligned when changing roles.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1086",
+  "version": "1.0.1087",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",