toga-ai 1.0.1174 → 1.0.1176
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/features/php-8-5-runtime-compatibility.md +3 -0
- package/knowledge/1.0/apps/test/features/compass-retrofix2-sql-generator.md +1 -0
- package/knowledge/1.0/apps/togadesk/INDEX.md +1 -1
- package/knowledge/1.0/apps/togadesk/architecture.md +5 -2
- package/knowledge/1.0/apps/togadesk/features/email-to-ticket-intake.md +13 -10
- package/knowledge/1.0/apps/togadesk/features/ticket-lifecycle.md +4 -3
- package/knowledge/2.0/apps/_underscore/features/apirequest-json-content-type.md +3 -2
- package/knowledge/2.0/apps/_underscore/features/note-record-authorship.md +3 -2
- package/knowledge/2.0/apps/_underscore/features/recursive-item-fulfillments.md +1 -0
- package/knowledge/2.0/apps/api2/features/api-key-authentication-and-scoping.md +6 -4
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
- package/knowledge/2.0/apps/worker2/features/inbound-email-ticket-importer.md +77 -0
- package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md +11 -10
- package/knowledge/2.0/standards/backend-php.md +8 -1
- package/knowledge/INDEX.md +1 -1
- package/knowledge/sessions/2026-10-27-nychh-jcardinal.md +18 -6
- package/package.json +1 -1
|
@@ -98,6 +98,8 @@ ext-imap is gone from PHP 8.4+, so on 8.5 hosts (worker1.5) every `imap2_*` call
|
|
|
98
98
|
|
|
99
99
|
`library/php-imap2/TOGA-PATCHES.md` lists every patch — re-apply after any re-vendor.
|
|
100
100
|
|
|
101
|
+
Upstream `javanile/php-imap2` is abandoned and its composer package pins `zbateson/mail-mime-parser` ^2.2 (CVE-2026-61815 header injection on write, CVE-2026-61816 parse DoS). Never add it to a 2.0 repo — worker2 uses `webklex/php-imap` ([inbound email importer](../../../../2.0/apps/worker2/features/inbound-email-ticket-importer.md), TRUE-82933). library still uses its vendored copy.
|
|
102
|
+
|
|
101
103
|
### Vendored PhpSpreadsheet + PHPMailer (TRUE-83159)
|
|
102
104
|
|
|
103
105
|
`PhpOffice/PhpSpreadsheet/` (early 1.x, no Composer) and `phpmailer/` 5.x are patched by hand to run on 8.5 with no deprecations or PHP 8 warnings, still valid on 7.2 (library `cc5705e6`). Decided (CTO agreed): patch in place, do not upgrade. New PhpSpreadsheet needs PHP 8.1+ (breaks the 7.2 apps); a second composer copy in worker1.5 clashes on the namespace. `PhpOffice/PhpSpreadsheet/PATCHES.md` lists every edit — re-apply after any re-vendor.
|
|
@@ -137,6 +139,7 @@ Verified: both NYC DOE reports + SMTP to a fake server on 7.2, 8.1, 8.5 gave byt
|
|
|
137
139
|
- **A library fix reaches worker1.5 only after a worker1.5 redeploy** ([worker1.5 architecture](../../worker1.5/architecture.md)).
|
|
138
140
|
|
|
139
141
|
## Change history
|
|
142
|
+
- 2026-10-27 — php-imap2 upstream abandoned + pins vulnerable zbateson ^2.2; worker2 moved to webklex (TRUE-82933) (jcardinal)
|
|
140
143
|
- 2026-10-27 — Example: `apple.php:74` warning stopped DOE ASN import 5 days; worker1.5 errors are in V2 `Logs.Issue` (TRUE-82940) (mhammontree)
|
|
141
144
|
- 2026-10-26 — `htmlEntity()` HTML-ENTITIES replaced byte-identically; `instance-data` dropped from error capture (log noise); `getInstanceId()` kept on purpose; worker1.5 sentry 4.34.0 (TRUE-83152, library PR #913, worker1.5 PR #15) (bala)
|
|
142
145
|
- 2026-10-23 — Vendored PhpSpreadsheet + phpmailer cleared for 8.5 (patched in place, PATCHES.md); phpexcel parse failure added to open list (TRUE-83159) (bala)
|
|
@@ -67,6 +67,7 @@ Walks each source IF up Agilant → ODP → Compass, emitting the mirror IFs.
|
|
|
67
67
|
- 2026-09-16 — Phase 0 CG merge and `runLevel()` isolation. (jcardinal)
|
|
68
68
|
|
|
69
69
|
## Related
|
|
70
|
+
- Rule: never cap or "fix" over-fulfilled quantities; agree with NetSuite — [backend-php § Synced data](../../../../2.0/standards/backend-php.md#synced-data--agree-with-the-source-even-when-illogical).
|
|
70
71
|
- [fix_item_fulfillment_chain — per-order tool](compass-fix-item-fulfillment-chain.md)
|
|
71
72
|
- [Test architecture](../architecture.md)
|
|
72
73
|
- [Compass order-lifecycle & data-integrity](../../../../clients/compass-usa/workflows/order-lifecycle-and-data-integrity.md)
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
| Doc | Summary |
|
|
4
4
|
|-----|---------|
|
|
5
5
|
| [TOGa Desk Architecture](architecture.md) | 1.0 (App_) support desk + field-service platform — staff at `/desk/`, public/client portal at `/`; open for the request pipeline, domain-class layer, MSP data m |
|
|
6
|
-
| [Email-to-Ticket Intake (crons/tickets.php)](features/email-to-ticket-intake.md) | TOGa Desk email→ticket intake
|
|
6
|
+
| [Email-to-Ticket Intake (crons/tickets.php)](features/email-to-ticket-intake.md) | TOGa Desk 1.0 email→ticket intake: IMAP poller `crons/tickets_prod.php` → `crons/tickets.php` → `Ticket::emailToTicket()`; open for routing/threading, noise sup |
|
|
7
7
|
| [Field-Service Dispatch (central / repair orders)](features/field-service-dispatch.md) | TOGa Desk field-service dispatch (the **central** domain): repair-order lifecycle, technician scheduling, onsite/depot, parts, and UPS shipment tracking; open f |
|
|
8
8
|
| [Managed Service Order Create Flow (New MSO modal → tickets/add)](features/managed-service-order-create.md) | TOGa Desk Managed Service Order (MSO) create flow: New MSO modal → `tickets/add` builds a `ManagedServiceOrder` + assets + a `RepairOrder` per asset; open for t |
|
|
9
9
|
| [Ticket Email Notifications (notifications table)](features/notifications.md) | TOGa Desk ticket email notifications are data-driven by `TOGaDeskSupport.notifications` rows keyed `(clientid, type)`; open when "staff never got notified" — us |
|
|
@@ -6,7 +6,7 @@ project: TOGa Desk
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-27
|
|
10
10
|
owners: ["mhammontree", "jcardinal", "snaredla"]
|
|
11
11
|
files:
|
|
12
12
|
- index.php
|
|
@@ -94,7 +94,7 @@ Classic **page ← data + modal + action** loop:
|
|
|
94
94
|
Crons live at the **repo root** (`crons/`), bootstrapped via the library framework + `App_Framework::cronInitialization()`, with per-environment wrappers (`*_prod.php` etc.):
|
|
95
95
|
- `tickets.php` — IMAP **email→ticket** intake from O365 (OAuth2 in prod). See the **email-to-ticket intake** feature doc.
|
|
96
96
|
- `monitoring.php` — `Monitoring::runAll()` host/port checks + up/down alerts.
|
|
97
|
-
- `pipe.php` — postfix stdin
|
|
97
|
+
- `pipe.php` — **dead** postfix stdin variant: undefined `$DBconfig`, loads the stale root `class.ticket.php`, root vendor has no zbateson; only a settings-page help text mentions it. Live intake: [email-to-ticket intake](features/email-to-ticket-intake.md). `general.php` — stub.
|
|
98
98
|
|
|
99
99
|
External integrations: **Talos** AI (ticket summaries / improved replies), **Syncro** PSA status push (when `tickets.referenceId='SYNCRO'`), **S3** (`asifiles`, `App_Cloud::copyFileToS3()`), **Sentry** (client-side JS), **PHPMailer** SMTP, and **NetSuite** customer id captured on SMB contracts. The `controllers/actions/toga/` and `controllers/data/toga/` subsystems bridge to the **TOGA** database (inventory items/levels/orders, SMB contracts, customer renewals).
|
|
100
100
|
|
|
@@ -163,3 +163,6 @@ Invariants:
|
|
|
163
163
|
- [Email-to-ticket intake](features/email-to-ticket-intake.md)
|
|
164
164
|
- [Ticket email notifications](features/notifications.md)
|
|
165
165
|
- [SMB contract editing](features/smb-contract-editing.md)
|
|
166
|
+
|
|
167
|
+
## Change history
|
|
168
|
+
- 2026-10-27 — `crons/pipe.php` marked dead; live intake is `tickets_prod.php` (TRUE-82933) (jcardinal)
|
|
@@ -6,12 +6,11 @@ project: TOGa Desk
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-27
|
|
10
10
|
owners: ["jcardinal", "dfranks", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- crons/tickets.php
|
|
13
13
|
- crons/tickets_prod.php
|
|
14
|
-
- crons/pipe.php
|
|
15
14
|
- desk/includes/classes/class.ticket.php
|
|
16
15
|
- desk/includes/classes/class.user.php
|
|
17
16
|
- desk/vendor/classes/class.imap.php
|
|
@@ -23,25 +22,25 @@ related:
|
|
|
23
22
|
- 1.0/apps/worker/architecture.md
|
|
24
23
|
- 1.0/apps/worker1.5/architecture.md
|
|
25
24
|
- 1.0/apps/library/features/php-8-5-runtime-compatibility.md
|
|
25
|
+
- 2.0/apps/worker2/features/inbound-email-ticket-importer.md
|
|
26
26
|
---
|
|
27
27
|
|
|
28
|
-
TOGa Desk email→ticket intake
|
|
28
|
+
TOGa Desk 1.0 email→ticket intake: IMAP poller `crons/tickets_prod.php` → `crons/tickets.php` → `Ticket::emailToTicket()`; open for routing/threading, noise suppression, or the sender auto-create of `people` rows. 2.0 replacement: [worker2 inbound importer](../../../../2.0/apps/worker2/features/inbound-email-ticket-importer.md).
|
|
29
29
|
|
|
30
30
|
## Summary
|
|
31
|
-
TOGa Desk ingests support email into tickets through
|
|
31
|
+
TOGa Desk 1.0 ingests support email into tickets through one live cron-driven IMAP poller (`crons/tickets_prod.php` wrapping `crons/tickets.php`). It funnels into `Ticket::emailToTicket()`, which threads replies onto existing tickets or creates new ones. Being replaced by the 2.0 [worker2 inbound importer](../../../../2.0/apps/worker2/features/inbound-email-ticket-importer.md) at the Desk cutover.
|
|
32
32
|
|
|
33
33
|
## How it works
|
|
34
34
|
|
|
35
35
|
### Entry points
|
|
36
|
-
- `crons/tickets.php` — the IMAP poll loop.
|
|
37
|
-
- `
|
|
38
|
-
- `Ticket::emailToTicket()` in `desk/includes/classes/class.ticket.php` — the parser/router.
|
|
36
|
+
- `crons/tickets.php` — the IMAP poll loop. Only the `tickets_prod.php` wrapper is live; it sets `$env`/`$processName` and `require`s it; the base script refuses to run unwrapped. `tickets_alpha.php`, `tickets_beta.php` and `crons/pipe.php` (postfix stdin variant) are **dead**.
|
|
37
|
+
- `Ticket::emailToTicket()` in `desk/includes/classes/class.ticket.php` (~L1450-1672) — the parser/router.
|
|
39
38
|
|
|
40
39
|
### Flow
|
|
41
40
|
1. **Bootstrap:** cron wrappers load the `library` framework + `App_Framework::cronInitialization()`, set `$env`, and register the IMAP (`Imap2`) helper. `$scriptpath` = `togadesk/desk`; autoload = `desk/vendor/autoload.php` (**no AWS SDK**) + `vendorClassAutoload` (`desk/vendor/classes/`). `tickets_prod.php` additionally performs **OAuth2/MSAL** token acquisition for the O365 mailbox.
|
|
42
|
-
2. **Poll:**
|
|
43
|
-
3. **Parse & route (`emailToTicket()`):** extract from/to/cc/subject/body (with encoding fixes)
|
|
44
|
-
- **Noise suppression (before any ticket/attachment is created):**
|
|
41
|
+
2. **Poll:** scheduled on worker1.5 every 15 min (:05/:20/:35/:50, `worker1.5/schedules/cron.worker.json:204`); connects to `outlook.office365.com:993` INBOX and long-polls up to 780s, processing each message then moving it to the IMAP folder **"Trash"** (see Gotchas).
|
|
42
|
+
3. **Parse & route (`emailToTicket()`):** extract from/to/cc/subject/body (with encoding fixes). Thread match, in order: **plus-addressing** `+<ticketnum>@` in To, then `Ticket #` in the subject. Matched ⇒ `addReply()`; otherwise ⇒ `add()`. `In-Reply-To` is captured but **never read**; there is **no Message-ID dedupe**.
|
|
43
|
+
- **Noise suppression (before any ticket/attachment is created):** `Ticket::isSuppressedNoiseEmail(?string $from, ?string $subject): bool` (static, `class.ticket.php`) is true only when the sender matches `/^no-?reply@/` (domain-agnostic) **AND** the subject starts with `staples po ` — both required so a real person is never blocked. `crons/tickets.php` adds it as one more `&&`-clause in the intake guard (with the delivery-status / `MAILER-DAEMON` / Compass auto-reply filters) and logs a "suppressed" line. (TRUE-79975 — ~815 such alerts since Jun 2024, mostly closed by hand in "Support Desk".)
|
|
45
44
|
4. **Side effects:** auto-create a user when the sender domain maps to an existing client; assign to a department via the department email lookup; save attachments **straight into `desk/uploads`** — on the worker a symlink to the s3fs mount (`/var/app/asifiles/togadesk-prod/uploads`), so the write is the S3 write. Before writing it checks `is_link` + written size. **`App_Cloud` is not used** (PR #625, TRUE-82935). Blocked names (`TICKET_IMPORT_BLOCKED_FILE_PATTERN` in `crons/tickets.php`: php*/cgi/sh/asp/jsp, html/htm, xht/xhtm/xhtml, shtm/shtml, stm, ssi, svg/svgz, xml, xsl/xslt, mht/mhtml, `.htaccess`, `.user.ini`) are saved with `.txt` added (widened TRUE-82935, PR #626). Optionally generate Talos summaries.
|
|
46
45
|
5. **Post-pass:** `Ticket::processRules()` (escalation) and `Ticket::autoClose()` (SLA auto-close) run after the mailbox sweep.
|
|
47
46
|
6. **On failure:** alert email to `serviceops@togatech.com`, also logged in legacy `Logs.Emails` (`subject LIKE 'TOGaDesk email import failed%'`) — the fastest way to read the cron's error (the cron has no stdout).
|
|
@@ -81,12 +80,16 @@ Ruled out as user-creation sources (do not re-chase):
|
|
|
81
80
|
- **Processed mail is moved, never purged.** `desk/vendor/classes/class.imap.php` `getTrash()` matches only `trash` / `inbox.trash` / `papierkorb`, else creates a custom folder **"Trash"** — originals stay recoverable there.
|
|
82
81
|
- **OAuth2:** the prod O365 app secret has a hard expiry (~2026-12-20); rotate before then or intake silently stops. **Security, open:** the client secret for `support@togatech.com` is committed in `crons/tickets_prod.php` — rotate and move to config/SSM (value not recorded).
|
|
83
82
|
- **Loop prevention** only excludes FROM==department-TO addresses; CC/reply-all across multiple department addresses can still loop.
|
|
83
|
+
- **Plus-address / `Ticket #` threading has no sender check** — anyone who knows a ticket number can add a reply to it.
|
|
84
|
+
- **Department `LIKE` match covers To AND CC, so a CC'd department address can override the To department.**
|
|
85
|
+
- **`deleteMessage` (move to Trash) result is unchecked** — a failed move re-imports the same message next run, and with no Message-ID dedupe it becomes a duplicate reply/ticket each run.
|
|
84
86
|
- **Auto-reply filters** are partly hardcoded (e.g. ignore `TechHub@compass-usa.com` "Automatic reply:" messages), and now the `isSuppressedNoiseEmail` Staples-PO filter — there is no config-driven allow/block list, so each new noise source is another inline guard.
|
|
85
87
|
- **`Ticket::extractEmail()` returned an undefined offset on a no-match** (bare `$matches[0]`). With a sender-less/malformed email this codebase promotes the warning to a thrown error (`E_ALL` + `App_Error::handleError`), so an intake cron could **halt**. Hardened to `return $matches[0] ?? '';` (TRUE-79975) — important because `isSuppressedNoiseEmail` routes nullable/malformed sender input through `extractEmail`, putting this latent bug on the hot path.
|
|
86
88
|
- **`class.imap.php` duplicate trap — `desk/vendor/classes/class.imap.php` is LIVE; root `vendor/classes/class.imap.php` is dead** (same direction as `class.ticket.php`). Both `vendorClassAutoload()` copies (`desk/includes/functions.php`, root `includes/functions.php`) load from `desk/vendor/classes/`, nothing `require`s the root copy, and only the desk copy has `getImapStream()` / `imap2_*` (verified 2026-10-19, TRUE-82935). The earlier KB claim that the root copy is live was wrong — TRUE-80116's edit + revert of the root copy had no effect on intake. Verify the right copy before calling a change "byte-identical".
|
|
87
89
|
- **`adminid` clobber on reply routing:** after `emailToTicket()` sets `$data['adminid']` from the FROM address (~line 1407), an asset lookup (~line 1437-1442) overwrites it with the replying user's asset's assigned admin. On a matched-ticket reply this makes a user's email look like a staff reply in `addReply()` (`isAdminReply=true`), so `Awaiting User` etc. never transition. The asset-derived adminid is for NEW-ticket auto-assign in `add()`, not reply classification. See the ticket-lifecycle doc.
|
|
88
90
|
|
|
89
91
|
## Change history
|
|
92
|
+
- 2026-10-27 — Corrected: threads on `+<ticket#>` then subject `Ticket #` only (In-Reply-To unread), no Message-ID dedupe, worker1.5 every 15 min with 780s long-poll; pipe.php/alpha/beta dead; new gotchas (sender check, CC dept override, unchecked move); linked 2.0 rewrite (TRUE-82933) (jcardinal)
|
|
90
93
|
- 2026-10-23 — worker1.5 / PHP 8.5 gotcha: intake needs library php-imap2 patches (TRUE-83148) (jcardinal)
|
|
91
94
|
- 2026-10-19 — Attachments saved straight into `desk/uploads` (s3fs), no `App_Cloud`; lost attachments not recovered; worker-pipeline deploy gotcha; corrected live `class.imap.php` copy; wider blocked-file pattern (TRUE-82935) (jcardinal)
|
|
92
95
|
- 2026-09-25 — renamed product reference from "TogaIQ" to "Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)
|
|
@@ -6,8 +6,8 @@ project: TOGa Desk
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: ["mhammontree", "dfranks"]
|
|
9
|
+
updated: 2026-10-27
|
|
10
|
+
owners: ["mhammontree", "dfranks", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- desk/includes/classes/class.ticket.php
|
|
13
13
|
- desk/includes/controllers/actions.php
|
|
@@ -41,7 +41,7 @@ All TOGa Desk ticket creation and reply handling funnels through `Ticket` in `de
|
|
|
41
41
|
### Reply entry points (FOUR doors — only two transition status)
|
|
42
42
|
There are **four** inbound reply paths, but the status-transition switch lives ONLY inside `Ticket::addReply()`. Any door that does not route through `addReply()` will not change status:
|
|
43
43
|
- **IMAP-poll cron** — `crons/tickets.php` → `emailToTicket()` → `addReply()`. ✅ transitions.
|
|
44
|
-
- **Mail pipe** — `crons/pipe.php:113` → `emailToTicket()` (legacy 6-arg call) → `addReply()`. ✅ transitions.
|
|
44
|
+
- **Mail pipe** — `crons/pipe.php:113` → `emailToTicket()` (legacy 6-arg call) → `addReply()`. ✅ transitions. **Dead** — not scheduled anywhere (verified TRUE-82933).
|
|
45
45
|
- **Desk REST API** — `desk/api/resources/ticket_replies.php` `'add'` → `addReply($data)` directly. ✅ transitions.
|
|
46
46
|
- **togaview customer portal** — the portal reply form (`action="/ticket"` + `newReplyBtn`, from the generic `common/togaview/ticket.php` and client variants e.g. `common/newcenturyholdingsllc/ticket.php`, `common/towfoundation/ticket.php`) POSTs to `togaview/mvc/ticket/post.php` `newReplyBtn` handler (~line 388). This handler inserts the reply row **directly** via `App_Model_TogaDesk_TicketReply` and **bypasses the `Ticket` class entirely** — it never updates `tickets.status`, `tickets_replies.newStatus`, or `tickets_history`. ❌ **no status transition.** This is the gap behind "a reply from outside Desk doesn't move the ticket."
|
|
47
47
|
|
|
@@ -76,6 +76,7 @@ When the caller supplies no status, a `switch ($ticket['status'])` decides. Patt
|
|
|
76
76
|
- Legacy `togadesk/includes/class.ticket.php` has active email-reopen logic; the live `desk/includes` version has it commented out — email replies never set status to "Reopened" via `emailToTicket()`.
|
|
77
77
|
|
|
78
78
|
## Change history
|
|
79
|
+
- 2026-10-27 — Marked `crons/pipe.php` mail-pipe door dead (TRUE-82933) (jcardinal)
|
|
79
80
|
- 2026-09-25 — renamed product reference from "TogaIQ" to "Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)
|
|
80
81
|
|
|
81
82
|
## Related
|
|
@@ -6,7 +6,7 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-27
|
|
10
10
|
owners: ["jcardinal", "ajean", "rgirish", "kyalamarthi", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/ApiRequest.php
|
|
@@ -102,7 +102,7 @@ case-insensitive scan of already-set headers, so a caller-supplied `Content-Type
|
|
|
102
102
|
`CURLOPT_SSL_VERIFYPEER` to `0` whenever `sslVerifyPeer` is null — despite the "AUTO based on
|
|
103
103
|
https" comment, `!null` is true, so an unset value never verifies, https included. New components
|
|
104
104
|
must call `setSslVerifyPeer(true)`. Changing the default touches every caller — needs a `cto`
|
|
105
|
-
opinion. 1.0 twin: [library architecture](../../../1.0/apps/library/architecture.md).
|
|
105
|
+
opinion. Backlog: TRUE-83327 (On Deck). 1.0 twin: [library architecture](../../../1.0/apps/library/architecture.md).
|
|
106
106
|
- **Multipart/file upload ALREADY WORKS — do not add an `ENCODE__MULTIPART` case (2026-09-04).**
|
|
107
107
|
The request-encoding switch (`ApiRequest.php:205-228`) has **no `default:` case**, so
|
|
108
108
|
`payloadEncoding: null` passes the payload through **untouched** and sets no `Content-Type`; `:236`
|
|
@@ -162,6 +162,7 @@ case-insensitive scan of already-set headers, so a caller-supplied `Content-Type
|
|
|
162
162
|
`dtStamp` (indexed) or it times out — `WHERE source = '…'` alone will not return.
|
|
163
163
|
|
|
164
164
|
## Change history
|
|
165
|
+
- 2026-10-27 — SSL-verify-off default re-confirmed; backlog TRUE-83327 (TRUE-82933) (jcardinal)
|
|
165
166
|
- 2026-10-05 — Added SSL-verify-off default (OPEN) and basic-auth-not-logged note, from TRUE-82185 (mhammontree)
|
|
166
167
|
- 2026-09-25 — renamed product reference from "TogaIQ" to "Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)
|
|
167
168
|
|
|
@@ -6,7 +6,7 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-27
|
|
10
10
|
owners: [apeterson, jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Model/Client/SalesOrderNote.php
|
|
@@ -62,7 +62,7 @@ Before building anything that shows or filters by note author:
|
|
|
62
62
|
|
|
63
63
|
## Gotchas
|
|
64
64
|
|
|
65
|
-
- **`contactId` on a `*Notes` model is an unwritten FK — treat it as always NULL.** Seven models, zero writers.
|
|
65
|
+
- **`contactId` on a `*Notes` model is an unwritten FK — treat it as always NULL.** Seven models, zero writers — one exception: `TicketNotes` written by the email importer (`Email Import` API, fixed uuid) carry the sender's `contactId` (since 2026-10-27, [email importer](../../worker2/features/inbound-email-ticket-importer.md)). Any other caller's `contactId` is dropped in `prePost`.
|
|
66
66
|
- **A notes list UI has no author to show.** Design for it, or the field silently renders blank for every row.
|
|
67
67
|
- **Exception — server-stamped author:** `CaseNotes`, `CustomerNotes`, `ContactNotes`, `IssuerNotes`, `CertificationNotes`, `AttributeNotes`, `UserNotes`, `LocationNotes`, `UnitInternalNotes`, `TicketDepartmentNotes` carry framework-stamped `createdByUserId` ([model-audit-user-stamps](./model-audit-user-stamps.md)); no manual stamping in `save()` any more, so the old "save() body copied 10 times" debt is gone for the author part. Each model still keeps its own NEEDS_USER and write-once checks. Copy that pattern for the real fix. See [Case status + stage gate](./case-status-and-stage-gate.md), [Customer related records](./customer-locations-files-notes.md), [Contact related records](./contact-notes-files-types.md), [Location / unit / department notes](./location-unit-department-notes-files.md).
|
|
68
68
|
- **Do not add author stamping to `UnitNotes`** — the BofA import writes it with no signed-in user and toga2-supply reads it by position. User-written asset notes use `UnitInternalNotes` ([why](./location-unit-department-notes-files.md#gotchas)).
|
|
@@ -71,6 +71,7 @@ Before building anything that shows or filters by note author:
|
|
|
71
71
|
- **`ServiceRequestNotes` keeps `dtStamp` AND has `dtCreated`.** `dtStamp` is the note time (SLA first response, Desk notes feed); `dtCreated` is row creation time. An integration passing an older `dtStamp` makes `dtStamp` earlier than `dtCreated`. Read `dtStamp` for note time.
|
|
72
72
|
|
|
73
73
|
## Change history
|
|
74
|
+
- 2026-10-27 — `TicketNotes.contactId` written for the email importer API only (TRUE-82933). (jcardinal)
|
|
74
75
|
- 2026-10-25 — 11 note-like tables now have full standard leading columns incl. dtUpdated/updatedByUserId; ServiceRequestNotes dtStamp vs dtCreated gotcha (TRUE-83181). (jcardinal)
|
|
75
76
|
- 2026-10-25 — Author now framework-stamped `createdByUserId` on all server-stamped note models; manual save() stamping removed (TRUE-83181). (jcardinal)
|
|
76
77
|
- 2026-10-19 — LocationNotes, UnitInternalNotes, TicketDepartmentNotes added (10 copies of save()); UnitNotes stays author-less (TRUE-82911). (jcardinal)
|
|
@@ -96,6 +96,7 @@ Engine is shared `_Model_Client_*` logic. Compass uses **empty pass-through subc
|
|
|
96
96
|
- **Performance:** every child write re-runs a full upstream walk (read-heavy, but each upstream record is written at most once — idempotent). Fine for normal fulfillment sizes.
|
|
97
97
|
|
|
98
98
|
## Related
|
|
99
|
+
- Rule: never cap or "fix" over-fulfilled quantities; agree with NetSuite — [backend-php § Synced data](../../../standards/backend-php.md#synced-data--agree-with-the-source-even-when-illogical).
|
|
99
100
|
- [isFulfillable Propagation Up the SO↔PO Chain](fulfillable-item-propagation.md) — a sibling engine that walks the **same** bridge topology to propagate a scalar item flag (rather than fulfillment rows) from the source item up to the client-facing copy.
|
|
100
101
|
- GroWrk transfer order flow plan: `clients/growrk/features/transfer-order-flow.md`.
|
|
101
102
|
- `_underscore` architecture (interceptors, `internalApiRequest`, `_Model` layer).
|
|
@@ -6,7 +6,7 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-27
|
|
10
10
|
owners: ["bala", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
@@ -52,13 +52,13 @@ A machine caller authenticates with `POST /v2/auth/api` using a `Client_<Client>
|
|
|
52
52
|
**A worker2 job calling api2 as an API record** (team rule: [machine callers use an API record, never a fake user](../../standards/framework-rules.md)). Reference: `worker2/Worker/Workflow/Run.php`.
|
|
53
53
|
|
|
54
54
|
1. Seed one `Client_<Client>.Apis` row per client (fixed `name`, random v4 `uuid` + secret, `isActive = 1`) and link it to a role through `Apis_Roles`. Set `maxAuthsPerHour` high (Workflow uses 1000): every queued job is a new process and signs in once.
|
|
55
|
-
2. In the job, call `_Database::registerClientDatabases` first, then load the row by `name` + `isActive`. Never take "the first active `Apis` row" — `Worker/Startech.php` and `Netsuite/SalesOrder.php` do, and break when a client adds a second key.
|
|
56
|
-
3. Call `_Component_Api_Toga::send(clientUuid, apiUuid, secret, ...)` → `POST /auth/api {client, api, secret}`.
|
|
55
|
+
2. In the job, call `_Database::registerClientDatabases` first, then load the row by `name` + `isActive`. Never take "the first active `Apis` row" — `Worker/Startech.php` and `Netsuite/SalesOrder.php` do, and break when a client adds a second key. **If model code grants the caller extra power, identify it by a fixed `Apis.uuid` (same uuid seeded in every client DB), never by `name`** — a name is editable, so any API could rename itself. Example: `_Model_Client_TicketNote::EMAIL_IMPORT_API_UUID` / `isEmailImportCaller()` ([email importer](../../worker2/features/inbound-email-ticket-importer.md)).
|
|
56
|
+
3. Call `_Component_Api_Toga::send(clientUuid, apiUuid, secret, ...)` → `POST /auth/api {client, api, secret}`. Tokens are cached per process, keyed by client uuid + API uuid (`tokenKey()`).
|
|
57
57
|
4. Result: the JWT carries `id.client.api.uuid`; roles = `Apis_Roles`; Activity shows `Apis.name` as the actor; `Logs_<Client>.Record` rows get `apiId`.
|
|
58
58
|
|
|
59
59
|
**What changes for an API caller.** api2 does not call `_Model::setRequestUserId()`, so `_Model::getRequestUserId()` is `null`. Every gate that needs a signed-in user behaves differently — check each one the job hits:
|
|
60
60
|
- `ServiceRequest` stage/reason rules and `ServiceRequestNote` `MESSAGE_NEEDS_USER` refuse or skip.
|
|
61
|
-
- `TicketNote` accepts an authorless note from an API caller (since 2026-10-22); it still refuses an unauthenticated request.
|
|
61
|
+
- `TicketNote` accepts an authorless note from an API caller (since 2026-10-22); it still refuses an unauthenticated request. `TicketNote` and `File` `prePost` keep a supplied `contactId` ONLY for the active `Email Import` API (fixed uuid); every other caller's is dropped.
|
|
62
62
|
- Author-stamped note models (`authorUserId`, see [note authorship](../../_underscore/features/note-record-authorship.md)) store `NULL`.
|
|
63
63
|
- Cross-client calls reject API callers.
|
|
64
64
|
|
|
@@ -77,6 +77,7 @@ A machine caller authenticates with `POST /v2/auth/api` using a `Client_<Client>
|
|
|
77
77
|
- **A shared role across keys means the key identifies the caller but does not limit it.** If several `Apis` rows point at the same roles, every one can do everything those roles allow, regardless of the integration's published documentation.
|
|
78
78
|
- **Never record a secret value in a doc or a committed SQL file.** Document only where it lives (the `Client_<Client>.Apis` row) and hand it over out of band.
|
|
79
79
|
- **⚠ Debug mode used to print the secret.** `_Component_Api_Toga` `authenticate` / `authenticateUser` dumped secrets, passwords and tokens in debug mode — removed 2026-10-22 (params now `#[\SensitiveParameter]`), and the `_ApiRequest` debug dump redacts `Authorization`. Still printed in debug: payloads, raw responses, `Cookie` and `X-Api-*` headers. Never share a debug-mode log.
|
|
80
|
+
- **Token cache was keyed by API uuid only (fixed 2026-10-27, TRUE-82933).** One process sending as the same Apis uuid in two client DBs reused the first client's token → writes landed in the wrong tenant. Now keyed by client uuid + API uuid; no signature change. Any new token cache must include the client.
|
|
80
81
|
- **Do not copy the `authority` field into caller documentation.** The response envelope's `authority` reports the **internal** server that handled the request (observed: `api1.togahub.com`, `api-writer.togahub.com`). Callers must always use `https://api.togahub.com/v2`.
|
|
81
82
|
|
|
82
83
|
## Related
|
|
@@ -88,4 +89,5 @@ A machine caller authenticates with `POST /v2/auth/api` using a `Client_<Client>
|
|
|
88
89
|
- [Creating worker actions](../../worker2/features/creating-worker-actions.md)
|
|
89
90
|
|
|
90
91
|
## Change history
|
|
92
|
+
- 2026-10-27 — Privileged machine identity = fixed `Apis.uuid`, never name; `Email Import` keeps note/file `contactId`; `_Component_Api_Toga` token cache keyed by client + API uuid (cross-tenant fix) (TRUE-82933). (jcardinal)
|
|
91
93
|
- 2026-10-22 — worker2 job → api2 as an `Apis` record (lookup by name, `maxAuthsPerHour`); API caller has no request user id; `TicketNote` accepts authorless API notes; debug dump no longer prints secrets (TRUE-82919). (jcardinal)
|
|
@@ -32,6 +32,7 @@
|
|
|
32
32
|
| [Error-Issue Auto-Resolution & Reopen (frequency-decay lifecycle)](features/error-issue-auto-resolution.md) | Issue-level OPEN→RESOLVED lifecycle on the escalation cron: auto-resolves an Issue after events decay to zero for a baseline-proportional quiet duration, auto-r |
|
|
33
33
|
| [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding: ingest an item catalog from an S3 CSV and enrich each item from the Etilize datasheet, writing directly to `DB_CLIENT`; open |
|
|
34
34
|
| [Etilize Item Translation Import](features/etilize-item-translation-import.md) | Imports non-English item + feature text from Etilize into the client's `ItemTranslations` (and feature sidecar tables), writing directly to `DB_CLIENT`; open wh |
|
|
35
|
+
| [Inbound email → Desk ticket importer (Email/Inbound, M365 IMAP)](features/inbound-email-ticket-importer.md) | 2.0 rewrite of 1.0 Desk email intake: worker2 reads M365 mailboxes over IMAP and writes tickets/notes/contacts/files through api2 as the per-client `Email Impor |
|
|
35
36
|
| [Logs retention cleanup worker (Infrastructure/Logs/Cleanup — Logs_<Tenant>.Api pruning)](features/logs-retention-cleanup.md) | Daily batched pruning of `Logs_<Tenant>.Api` (GET 30d, other methods 90d); open when changing log retention, adding a table to prune, or when an old Api log row |
|
|
36
37
|
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | DB-driven monitoring framework: one orchestrator applies anti-flap + notification rules to any registered "is X healthy?" child check; open when adding a monito |
|
|
37
38
|
| [Bidirectional NetSuite ↔ Forecast ↔ ClickUp opportunity stage + title sync — BUILT (stage map, echo guard, claim split, NS-owned title)](features/netsuite-clickup-opportunity-bidirectional-design.md) | Two-way NetSuite↔ClickUp opportunity STAGE sync + one-way NS→CU TITLE sync (shared `_Component_Opportunity_StageMap`, actor echo guard, claim split, stale-claim |
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Inbound email → Desk ticket importer (Email/Inbound, M365 IMAP)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-10-27
|
|
10
|
+
owners: ["jcardinal"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Email/Inbound.php
|
|
13
|
+
- worker2/Component/Email/Inbound/Imap/Imap.php
|
|
14
|
+
- worker2/Component/Email/Inbound/Parser/Parser.php
|
|
15
|
+
- worker2/Component/Email/Inbound/Rules/Rules.php
|
|
16
|
+
- worker2/Component/Email/Inbound/Sanitizer/Sanitizer.php
|
|
17
|
+
- worker2/Component/Email/Inbound/Desk/Desk.php
|
|
18
|
+
- worker2/Component/Email/Inbound/Api/Api.php
|
|
19
|
+
- worker2/Component/Secrets/Secrets.php
|
|
20
|
+
- dbchanges2/Core/2026-10-27a - TRUE-82933 InboundMailboxes.sql
|
|
21
|
+
- dbchanges2/Client/2026-10-27a - TRUE-82933 EmailImport.sql
|
|
22
|
+
- _underscore/Model/Client/TicketNote.php
|
|
23
|
+
- _underscore/Model/Core/InboundMailbox.php
|
|
24
|
+
related:
|
|
25
|
+
- ../../../1.0/apps/togadesk/features/email-to-ticket-intake.md
|
|
26
|
+
- ../../api2/features/api-key-authentication-and-scoping.md
|
|
27
|
+
- ../../_underscore/features/note-record-authorship.md
|
|
28
|
+
- ./creating-worker-actions.md
|
|
29
|
+
- ./oneuptime-worker2-monitoring.md
|
|
30
|
+
- ../../toga25-desk/architecture.md
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
2.0 rewrite of 1.0 Desk email intake: worker2 reads M365 mailboxes over IMAP and writes tickets/notes/contacts/files through api2 as the per-client `Email Import` API; OFF until the 1.0 → 2.0 Desk cutover. Open for routing, threading, sender trust, mailbox setup.
|
|
34
|
+
|
|
35
|
+
## Summary
|
|
36
|
+
|
|
37
|
+
Replaces 1.0 `togadesk` `crons/tickets_prod.php` ([1.0 intake](../../../1.0/apps/togadesk/features/email-to-ticket-intake.md)) with tenant-safe rules. Built TRUE-82933; **OFF** (`CronJobs` row and every `InboundMailboxes.isActive` = 0) until cutover. Mailbox/address seed is deferred to cutover (1.0 departments are not migrated — see Gotchas).
|
|
38
|
+
|
|
39
|
+
## How it works
|
|
40
|
+
|
|
41
|
+
**Jobs** (`Worker/Email/Inbound.php`):
|
|
42
|
+
1. `Email/Inbound/Poll` — cron, every minute. Queues one `Email/Inbound/Mailbox` job per active `Core.InboundMailboxes` row (param `inboundMailboxId` only).
|
|
43
|
+
2. `Email/Inbound/Mailbox` — `GET_LOCK('inbound-mailbox-<id>')` (one runner per mailbox), sign in, import INBOX oldest first for `CHUNK_TIME_BUDGET_SECONDS` (180), then `runTask` itself with ids + running counters only (self-chunking, see [creating worker actions](./creating-worker-actions.md)). JSON counts go to `WorkerJobs.output`.
|
|
44
|
+
|
|
45
|
+
**Mail access** (`Component/Email/Inbound/Imap`): M365 IMAP with OAuth2 client credentials (XOAUTH2, scope `https://outlook.office365.com/.default`) on `webklex/php-imap` at protocol level, TLS validated. Credentials: `_Component_Secrets::resolve(InboundMailboxes.secretRef)` → JSON `{tenantId, clientId, clientSecret}`. Parsing: `zbateson/mail-mime-parser` 4.x.
|
|
46
|
+
|
|
47
|
+
**Per message** (`Rules`, `Desk`, `Api`):
|
|
48
|
+
1. **Skip:** delivery status / `MAILER-DAEMON`, `Auto-Submitted` (not `no`), `X-MS-Exchange-Organization-SCL` ≥ 5, Staples-PO noise (`no-?reply@` + subject `staples po `), loop (From is one of our own addresses). Null `Return-Path` is still imported.
|
|
49
|
+
2. **Dedupe:** `TicketNotes.emailMessageId` (UNIQUE) — already imported → skip.
|
|
50
|
+
3. **Route:** To/CC addresses against `Core.InboundMailboxAddresses` (lowercase, UNIQUE). Exactly 1 client → that client + its department; 0 or >1 clients → mailbox `defaultClientId` / `defaultTicketDepartmentUuid`.
|
|
51
|
+
4. **Sender verified** = the FIRST `Authentication-Results` header (Exchange's, authserv `*.protection.outlook.com`) shows `dmarc=pass` or `compauth=pass`, OR `X-MS-Exchange-Organization-AuthAs: Internal` with a From domain of our own.
|
|
52
|
+
5. **Thread:** `In-Reply-To` / `References` matched to stored `emailMessageId` (verified senders only). Else `+<ticket#>@` in To or `Ticket #` in the subject — only when the sender is verified AND is the ticket's contact. Else a new ticket with a line noting the referenced number.
|
|
53
|
+
6. **Contact:** match/create only in the routed client. Unverified sender → `ContactEmailAddresses.isVerified = 0`; reuse an existing unverified row. Caps: 50 new contacts per run, 10 new tickets per sender.
|
|
54
|
+
7. **Write via api2** as `Apis` 'Email Import': `POST /tickets` → `/ticket-notes`; `/contacts`; `/files` → `/ticket-note-files`. The sender `contactId` is kept as note/file author (SLA first response counts) — see Gotchas.
|
|
55
|
+
8. **Attachments:** blocked extensions get `.txt` appended; Content-Type allowlist, else `application/octet-stream`; 25 MB total, 25 files, 200 MIME parts. HTML sanitized, remote images stripped, small footer images dropped.
|
|
56
|
+
9. **After:** success → move to `processedFolder`; failure → flag, then `failedFolder` (also after 2 fatal crashes on the same message).
|
|
57
|
+
|
|
58
|
+
**Schema:**
|
|
59
|
+
- Core `InboundMailboxes`: `clientId`, `defaultClientId`, `defaultTicketDepartmentUuid`, `isActive` (default 0 = the cutover switch), `provider` ENUM `MICROSOFT_365`, `emailAddress` UNIQUE, `secretRef` (FULL Secrets Manager ARN), `processedFolder`, `failedFolder`.
|
|
60
|
+
- Core `InboundMailboxAddresses`: `inboundMailboxId`, `clientId`, `ticketDepartmentUuid`, `isActive`, `emailAddress`.
|
|
61
|
+
- Core RecordFields 3572 `ticket-notes.emailMessageId`, 3573 `contact-email-addresses.isVerified`.
|
|
62
|
+
- Client: `TicketNotes.emailMessageId` UNIQUE; `ContactEmailAddresses.isVerified` default 1; `Apis` 'Email Import' with ONE fixed uuid `29990661-0293-448f-94d4-77e0133d7442` in every client DB (random secret per DB, stored only in the DB), on the existing API role, own `Email Import` capability + full grant chain.
|
|
63
|
+
|
|
64
|
+
**Tests:** Codeception unit Cests `worker2/tests/Unit/InboundEmail{Parser,Rules,Sanitizer}Cest.php`, synthetic `.eml` fixtures in `tests/Support/Data/inbound-email`.
|
|
65
|
+
|
|
66
|
+
## Gotchas
|
|
67
|
+
|
|
68
|
+
- **Identity = fixed `Apis.uuid`, never `Apis.name`.** `_Model_Client_TicketNote::EMAIL_IMPORT_API_UUID` / `isEmailImportCaller()` (active row only) is the ONLY caller whose supplied `contactId` is kept on `TicketNote` and `File` `prePost`. A name is editable, so any API could rename itself and forge authors.
|
|
69
|
+
- **Needs the `_Component_Api_Toga` token-cache fix** (cache keyed by client uuid + API uuid). Same Apis uuid in every client DB + the old uuid-only key = first client's token reused for the next client (cross-tenant writes). Ship `_underscore` with/before worker2. See [API-key auth](../../api2/features/api-key-authentication-and-scoping.md).
|
|
70
|
+
- **`secretRef` must be a FULL Secrets Manager ARN.** `_Component_Secrets::resolve` treats anything not starting `arn:aws:secretsmanager:` as an SSM parameter path. Secret name style: `<purpose>/<client>/<item>`.
|
|
71
|
+
- **M365 stored copies of forwarded/aliased mail have NO `Delivered-To` / `X-MS-Exchange-Organization-Original*` header** — route on To/CC. Only the FIRST `Authentication-Results` is Exchange's; later ones can be sender-forged. SCL `-1` = trusted, `1` = clean (verified on 40 prod messages, headers only, 2026-10-27).
|
|
72
|
+
- **1.0 departments are not in 2.0.** Prod `Client_True` has 0 `TicketDepartments`; only `Client_Elite` has `c_togadeskTicketDepartmentId` mappings. Seed mailboxes/addresses at cutover (Desk docs question `sf`). Unmatched-mail default = TOGA Technology "Support Desk" (1.0 dept 1) — NOT 1.0's `tickets_defaultclient=2` (Agilant).
|
|
73
|
+
- **Do not fold mailbox/identity config into a generic JSON config table** (CTO review, TRUE-82933): `Core.ClientApiIdentities` / `PartnerApiIdentities` have different unique keys, live cXML + 1.0 library readers, and per-field ACL would be lost. New meaning → new dedicated table. Hardening those tables: backlog TRUE-83314.
|
|
74
|
+
- **Never `javanile/php-imap2` in worker2.** Abandoned; pins `zbateson/mail-mime-parser` ^2.2 (CVE-2026-61815 header injection on write, CVE-2026-61816 parse DoS). worker2 uses `webklex/php-imap` 6.2.0 behind `_Component_Email_Inbound_Imap` (also used by the Compass `EmailedAsnImport` monitor).
|
|
75
|
+
|
|
76
|
+
## Change history
|
|
77
|
+
- 2026-10-27 — Built, OFF until cutover: Poll/Mailbox jobs, routing/threading/sender rules, Core mailbox tables, `Email Import` API, fixed-uuid author exception (TRUE-82933). (jcardinal)
|
|
@@ -401,7 +401,7 @@ pattern (first instance: `EmailedAsnImport()`, which intermittently paged on a t
|
|
|
401
401
|
**90s total-retry budget** — so total time can never approach the `maxExecutionTime` 120s watchdog even
|
|
402
402
|
if the socket-timeout knob is ineffective.
|
|
403
403
|
- **Catch `\Throwable`, not `Exception`,** in the loop so a non-`Exception` `Error` (e.g. from the
|
|
404
|
-
|
|
404
|
+
IMAP library) can't escape the method, skip the socket-timeout restore, or break this
|
|
405
405
|
worker action's "always return a string" invariant.
|
|
406
406
|
- **A delayed re-read must call `_Database::resetQueryCache()` first** — `_Query` caches SELECTs by SQL
|
|
407
407
|
text, so an identical second `_Query` is served from the cache and the "retry" sees the first result.
|
|
@@ -531,19 +531,19 @@ well under 120s.
|
|
|
531
531
|
|
|
532
532
|
### Mailbox monitoring (EmailedAsnImport) — IMAP XOAUTH2, not Graph
|
|
533
533
|
|
|
534
|
-
Use **IMAP with XOAUTH2** via
|
|
535
|
-
|
|
534
|
+
Use **IMAP with XOAUTH2** via `_Component_Email_Inbound_Imap` (`webklex/php-imap`, pure PHP, TLS
|
|
535
|
+
validated — shared with the [inbound email importer](./inbound-email-ticket-importer.md)), **not**
|
|
536
|
+
Microsoft Graph. `javanile/php-imap2` was removed from worker2 2026-10-27 (abandoned, pulls a vulnerable
|
|
537
|
+
`zbateson/mail-mime-parser` ^2.2; TRUE-82933). The Graph `messages/$count` approach returned HTTP 403 because the app lacks Graph
|
|
536
538
|
`Mail.Read` application consent and Entra consent cannot be obtained. IMAP XOAUTH2 uses the app's
|
|
537
539
|
existing `IMAP.AccessAsApp` permission (the 1.0 monitor already relies on it) — no new Entra consent.
|
|
538
540
|
|
|
539
|
-
- OAuth token scope: `https://outlook.office365.com/.default
|
|
540
|
-
- Then `
|
|
541
|
-
|
|
542
|
-
- Guarded by `function_exists('imap2_open')` so a missing library reports `status:error` rather than
|
|
543
|
-
throwing an uncatchable `Error`.
|
|
541
|
+
- OAuth token scope: `https://outlook.office365.com/.default` (token call uses `setSslVerifyPeer(true)`).
|
|
542
|
+
- Then `_Component_Email_Inbound_Imap::open($mailbox, $accessToken)` → `countMessages()` → `close()`.
|
|
543
|
+
No ext-imap needed.
|
|
544
544
|
- Requires the `[mailbox_compass_statuses]` config group (`tenant_id`, `client_id`, `client_secret`,
|
|
545
|
-
`mailbox`) in each `Config/*.ini
|
|
546
|
-
|
|
545
|
+
`mailbox`) in each `Config/*.ini`. The `client_secret` is a credential — it lives in the ini config
|
|
546
|
+
group, never in a doc.
|
|
547
547
|
|
|
548
548
|
### SFTP monitoring (phpseclib3)
|
|
549
549
|
|
|
@@ -658,6 +658,7 @@ production deploy.**
|
|
|
658
658
|
- [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
|
|
659
659
|
|
|
660
660
|
## Change history
|
|
661
|
+
- 2026-10-27 — `EmailedAsnImport` moved off php-imap2 to `_Component_Email_Inbound_Imap` (webklex), TLS now validated, no ext-imap (TRUE-82933). (jcardinal)
|
|
661
662
|
- 2026-10-27 — Added `OfficeDepotLostOrders` (TRUE-82736); legacy `Logs`/`Common` via `DB_VISION_LOGS`; per-run `resetQueryCache`; worker2 project status/severity ids + export fields. (amadda)
|
|
662
663
|
- 2026-10-27 — SFTP monitors: reuse `Sftp/Import` config + login when it already reads the SFTP; linked NYCDOE `StuckAsnFiles` (TRUE-82941). (mhammontree)
|
|
663
664
|
- 2026-10-26 — Added `CompassCanadaIncomingPurchaseOrders` (TRUE-82751, not yet deployed). (bala)
|
|
@@ -5,7 +5,7 @@ project: _Underscore
|
|
|
5
5
|
client: shared
|
|
6
6
|
type: standard
|
|
7
7
|
status: active
|
|
8
|
-
updated: 2026-10-
|
|
8
|
+
updated: 2026-10-27
|
|
9
9
|
owners: [jcardinal, mhammontree, dfranks, bala, ajean]
|
|
10
10
|
files: []
|
|
11
11
|
related:
|
|
@@ -1048,6 +1048,13 @@ See: 2.0/apps/worker2/features/cross-account-aws-access.md (role model),
|
|
|
1048
1048
|
- Planned (not built yet): the preferred date format and time format settings will travel the same path (token + audience + blox session store).
|
|
1049
1049
|
- Front-end conversion rule: [frontend.md § 25](frontend.md#25-time-zones--convert-from-the-storage-zone--must).
|
|
1050
1050
|
|
|
1051
|
+
## Synced data — agree with the source, even when illogical
|
|
1052
|
+
|
|
1053
|
+
- **Hard rule (jcardinal, TRUE-79669):** for synced data (NetSuite, other source systems) our DB must agree with the source. Agreement is the top priority.
|
|
1054
|
+
- Never cap, trim, round, or "fix" a quantity or value because it looks wrong (e.g. over-fulfilled lines). Bad source data stays bad in our copy.
|
|
1055
|
+
- Allowed: remove true duplicates of the SAME data (same shipment recorded twice) — that restores agreement.
|
|
1056
|
+
- Never guess data the source lacks (e.g. which line went in which box when the source has only header-level tracking).
|
|
1057
|
+
|
|
1051
1058
|
## Change history
|
|
1052
1059
|
|
|
1053
1060
|
- 2026-10-24 — Added "Time zones": store Chicago, zone is a setting with client/user override (TRUE-82916). (jcardinal)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -23,7 +23,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
23
23
|
## 2.0 framework
|
|
24
24
|
|
|
25
25
|
- **_underscore** (_Underscore) _(framework core)_ — 117 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
26
|
-
- **worker2** (Worker) —
|
|
26
|
+
- **worker2** (Worker) — 100 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
27
27
|
- **api2** (API) — 28 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
28
28
|
- **dbchanges2** (Database Changes) _(framework core)_ — 24 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
29
29
|
- **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
@@ -34,7 +34,7 @@ updated: 2026-10-27
|
|
|
34
34
|
- `worker1.5` sync_togasupply_hh.php: removed `inventoryAdjustmentStockLocationNetsuiteIds` (missing key = all locations). Committed d0c1f1a. Verified: every linked IA line in prod is at location 117, so no shares change and no share-runner re-run is needed.
|
|
35
35
|
- `library` app/api/toga2.php (syncSalesOrderFromNetsuite, existing-PO branch): SO line's own item wins over an item-group sibling when the PO has a line for it. Committed 715e7ecc. Re-synced SO 154937 (ns 4230372) and SO 154738 (ns 4223786): EIT0010893 L31 210-BFZY and L72 210-BGCU now linked and 0 on hand; "-A" lines 136/144-146 have no links.
|
|
36
36
|
- EIT0009416 fully matches NetSuite after Ops IAs #3141/#3142 (+IF #154769 on #3142), #3143/#3144 (210-AZBM), #3136/#3137 (210-BEQW-HHC).
|
|
37
|
-
- EIT0012615 matches
|
|
37
|
+
- EIT0012615 now matches (Ops moved the New York stock to Plainview).
|
|
38
38
|
- EIT0011189: IA #1611 unlinked; allocation runner left HPCLJEM528Z at ±2 (whole TO lines only).
|
|
39
39
|
- Excel for agreeing lines built: D:\TEMP\H&H\HH-Stock-Reconciliation-2026-10-26.xlsx (v1) and -v2.xlsx (adds ELM-resolved lines). Builder script lived in the session scratchpad (build_hh_xlsx.py) — dev said hold off on the Excel until discrepancies are settled.
|
|
40
40
|
|
|
@@ -47,15 +47,27 @@ updated: 2026-10-27
|
|
|
47
47
|
- Giving Ops actions for items where Supply and NetSuite already match (dev called this overcomplicating). Rule: if totals match, nothing to do.
|
|
48
48
|
- SuiteQL `custbody_stock_adjustment IS NULL` on the multi-select does not detect blank IAs — use BUILTIN.DF(...) per IA instead.
|
|
49
49
|
- Comparing NetSuite on-hand per item including "_" twin items separately (C31CJ57012 vs C31CJ57012_): add twins together before calling a mismatch.
|
|
50
|
+
- knowledge.js session-save rejects this file with "missing frontmatter: updated" when it has CRLF line endings — keep LF.
|
|
51
|
+
|
|
52
|
+
## Latest analysis (2026-10-27, after deploy + re-syncs) — 14 POs
|
|
53
|
+
Method: Supply on-hand per PO line (live code: no location, floored at 0) summed per item vs NetSuite inventorybalance at 117+173+202, "_" twins added.
|
|
54
|
+
- MATCH: EIT0012148, EIT0012673, EIT0012615, EIT0012867, EIT0012868 (AppleCare removed), EIT0012107 and EIT0011563 (own lines OK; AC785544 gap is on EIT0012372/EIT0012614).
|
|
55
|
+
- Ops already fixed: IA #978 linked (STK503RED/STM042RED now 115 = NS), IA #684 linked (EIT0010304), AB678031 now 858 = NS, 460-BCTK now matches.
|
|
56
|
+
- EIT0011733: K33821NA 129 vs 149 → Ops fill Stock Adjustment on IA #2555 (+20, 5/11/2026, not an old IA).
|
|
57
|
+
- EIT0012548: 520-BBNF 20 vs 60 → vendor PO #164473 receipt #59013 (40, 4/30/2026) into Plainview with no stocking order.
|
|
58
|
+
- EIT0012372: AC785544 → IA #2136 (memo SO 252514 = EIT0012614) lists IF #274364 (EIT0012372) + IF #289938 (EIT0012614); TOGa gives all 1800 to EIT0012372 (IA #3109 has NO AC785544 line). Ops: remove IF #274364 from IA #2136, then confirm whether EIT0012372's own 1800 (SO 235136) went into stock.
|
|
59
|
+
- COF0015668: 210-BFZY 268 vs 718 = customer returns RMA #1875 (250) + RMA #1876 (200) 3/28/2024; 210-BBCC 197 vs 202 = RMA #2742 (5, 4/30/2026). Supply does not count RMA receipts.
|
|
60
|
+
- EIT0011108: DTKN/64GB 78 vs 206 (Ops IA −128 unlinked for credit memo 8124100); K55888WW 6 vs 16 (vendor PO #161025, 10 into Processing 3/2026, no stocking order); K33821NA (IA #2555); DWM3770X +2 / DWMSC3780MUL +1 (SOs 169639/169656 ticked stocking, no IA → untick); PA03810-B055 L42 −1 / L85 +1 (allocation runner dry run); VB320-BL1313- B27-435 L83 −2 (stock belongs to EPC0002915 via old IA #1194); PA03795-B005 1 unit (847 vs 846) untraced.
|
|
61
|
+
- EIT0010893: 210-ASFS +2 / 210-AWKC +1 (SOs 169639/169656); 210-BCTU L54 −5 vs EIT0011933 +8, NS 3 (IA pair); 210-BHSM 40 vs 20 (old IAs #1224/#1226 + transfer +20 into ELM); 580-AJIS −10 (old IA #1000 = return on TO 164021); 210-BBCC (RMA #2742).
|
|
62
|
+
- EIT0012129: TD-LBK070VA-USB R 148 vs 182 (transfer #328 +34 from HQ, 8/12/2026); K33821NA; PA03795-B005 1 unit.
|
|
50
63
|
|
|
51
64
|
## Not tried yet (candidates for next session)
|
|
52
|
-
-
|
|
65
|
+
- Dev decision pending: count customer returns (RMA receipts) against the PO of the TO being returned (Dev), or Ops books returns as IA on the TO fulfillment.
|
|
53
66
|
- Rebuild the client Excel (receipts + deployments) once discrepancies are settled; reuse the receipts/deployments queries (filter by agreeing PO lines).
|
|
54
|
-
- Decide with dev: blank-Stock-Adjustment IAs → option 1 (list as known exceptions) vs option 2 (new IA pair +X linked / −X unlinked, old IA untouched).
|
|
67
|
+
- Decide with dev: blank-Stock-Adjustment IAs → option 1 (list as known exceptions) vs option 2 (new IA pair +X linked / −X unlinked, old IA untouched).
|
|
55
68
|
- Decide with dev: floor Nychh item-level `_qtyCommitted` at 0 (Model/Nychh/Item.php subtracts shipped from committed and can go negative, inflating available).
|
|
56
69
|
- Ops answers pending: SVX-VX-RPY is set up as InvtPart in NetSuite though Ops says licences are not inventory.
|
|
57
|
-
-
|
|
58
|
-
- Still unexplained: 520-BBNF (Supply 20 vs NetSuite 60); T-LBK462-BSB-R −25 on EIT0011108 / 210-BCTU −5 on EIT0010893 (cross-PO TO use → Ops IA pair, need the TO numbers).
|
|
70
|
+
- Trace PA03795-B005 1-unit gap; get TO numbers for 210-BCTU (EIT0010893 vs EIT0011933) IA pair.
|
|
59
71
|
|
|
60
72
|
## Current file state
|
|
61
73
|
| File | Status | Notes |
|
|
@@ -66,7 +78,7 @@ updated: 2026-10-27
|
|
|
66
78
|
| worker1.5 crons/toga2/netsuite/sync_togasupply_hh.php | committed d0c1f1a | IA location key removed |
|
|
67
79
|
| library app/api/toga2.php | committed 715e7ecc | own item before group sibling |
|
|
68
80
|
| dbchanges2 Client_Nychh/2026-10-16a, 2026-10-20a, 2026-10-20b | committed earlier | one-time cleanup SQL (ran) |
|
|
69
|
-
| All three app repos | pushed on TRUE-82604 |
|
|
81
|
+
| All three app repos | pushed on TRUE-82604, deployed | re-syncs of SO 154937/154738 done |
|
|
70
82
|
|
|
71
83
|
## Decisions made
|
|
72
84
|
- Location never decides H&H stock (dev + Ops). Rejected: hard-coded warehouse list; per-location "client stock" flag (not needed once location is out).
|
package/package.json
CHANGED