toga-ai 1.0.1085 → 1.0.1086
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/api/INDEX.md +5 -0
- package/knowledge/1.0/apps/api/features/talos-ticket-endpoint.md +45 -0
- package/knowledge/1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md +6 -2
- package/knowledge/1.0/apps/library/features/app-cloud-aws-credentials.md +5 -2
- package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md +2 -1
- package/knowledge/1.0/apps/togadesk/features/email-to-ticket-intake.md +27 -8
- package/knowledge/1.0/apps/togadesk/features/s3fs-asifiles-file-storage.md +7 -2
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +31 -2
- package/knowledge/INDEX.md +1 -1
- package/knowledge/clients/compass-usa/profile.md +2 -1
- package/knowledge/clients/office-depot/profile.md +6 -2
- package/knowledge/clients/true/profile.md +2 -0
- package/package.json +1 -1
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
# api (API 1.0) — 1.0 knowledge
|
|
2
|
+
|
|
3
|
+
| Doc | Summary |
|
|
4
|
+
|-----|---------|
|
|
5
|
+
| [Talos Ticket API (1.0 api POST/PUT /ticket) — Desk tickets + file uploads from Talos Voice](features/talos-ticket-endpoint.md) | 1.0 `api` route that Talos Voice calls to create (`POST /ticket`) and update (`PUT /ticket/{ticketNumber}`) TOGa Desk tickets and attach call files; open before |
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Talos Ticket API (1.0 api POST/PUT /ticket) — Desk tickets + file uploads from Talos Voice"
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: api
|
|
5
|
+
project: API 1.0
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-10-19
|
|
10
|
+
owners: ["jcardinal"]
|
|
11
|
+
files:
|
|
12
|
+
- api/_/app/route/ticket.php
|
|
13
|
+
- api/_/app/error.php
|
|
14
|
+
- api/ebs/create-s3fs-credentials-file.php
|
|
15
|
+
- api/ebs/mount-s3fs-folders.php
|
|
16
|
+
related:
|
|
17
|
+
- ../../togadesk/features/s3fs-asifiles-file-storage.md
|
|
18
|
+
- ../../library/features/app-cloud-aws-credentials.md
|
|
19
|
+
- ../../togadesk/features/email-to-ticket-intake.md
|
|
20
|
+
- ../../../../standalone/apps/voice-to-voice/features/ticket-api-integration.md
|
|
21
|
+
- ../../../../clients/office-depot/features/odp-tech-support.md
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
1.0 `api` route that Talos Voice calls to create (`POST /ticket`) and update (`PUT /ticket/{ticketNumber}`) TOGa Desk tickets and attach call files; open before touching its file-upload path or when Desk tickets show duplicate/raw-named attachments.
|
|
25
|
+
|
|
26
|
+
## Summary
|
|
27
|
+
|
|
28
|
+
Server side of the Talos ticket contract (client side: [ticket API integration](../../../../standalone/apps/voice-to-voice/features/ticket-api-integration.md)). Writes `TOGaDeskSupport` `tickets` / `tickets_replies` / `files` and stores file bytes on the s3fs mount of bucket `asifiles`. The `api` repo has **no composer** — never call `App_Cloud` from it.
|
|
29
|
+
|
|
30
|
+
## How it works
|
|
31
|
+
|
|
32
|
+
- `postTicket` / `putTicket` → `processFileUploads($files, $replyId, $ticketId)`. Talos sends base64 `{filename, content, mimetype}`; name is `basename()`'d.
|
|
33
|
+
- `assertStorageIsMounted()` runs **before any ticket/reply/file row is written**: compares `stat()` dev of `internal.storage_folder` vs its parent; same dev = mount down → `RuntimeException`. Skipped in dev mode.
|
|
34
|
+
- Per file: `fileAlreadySaved($replyId, $filename)` skips a name already on that reply (raw or `<id>-name`) — Talos re-sends. Then save `files` row → write `<storage_folder>/togadesk-<env>/uploads/<id>-<name>` (= `s3://asifiles/togadesk-<env>/uploads/<id>-<name>` via the mount) → rename row to `<id>-<name>`. Write failure (local error handler turns the warning into an exception) deletes the row.
|
|
35
|
+
- `_/app/error.php` wraps its own `App_Cloud` S3 copy in `try/catch (\Throwable)` so the `Logs.Errors` row is always written.
|
|
36
|
+
- s3fs auth = `-o iam_role=auto` (instance role); `ebs/create-s3fs-credentials-file.php` only deletes `/etc/passwd-s3fs`. See [asifiles s3fs mount](../../togadesk/features/s3fs-asifiles-file-storage.md).
|
|
37
|
+
|
|
38
|
+
## Gotchas
|
|
39
|
+
|
|
40
|
+
- **`App_Cloud` in api throws `Error`, not `Exception`** (no AWS SDK, see [App_Cloud](../../library/features/app-cloud-aws-credentials.md)). `catch (Exception)` misses it. TRUE-82935: `copyFileToS3` after the mount write threw → row never renamed, request failed, and `error.php` itself crashed so nothing reached `Logs.Errors`.
|
|
41
|
+
- **A failed PUT repeats hourly.** Talos re-sends at ~:00:49 and resets the reply timestamp each time → one more raw-named `session_report_*.txt` row per hour per ticket. By 2026-10-19: 4,219 rows (clientid 3 = 3,772, clientid 188 = 447). Cleanup: `dbchanges/TOGaDeskSupport/JC/2026-10-19.sql` — run only **after** the api fix is deployed, else rows come back.
|
|
42
|
+
- **Security, open (values not recorded):** `config.*.ini` sits in the web root with SES SMTP credentials and no deny rule; the old api s3fs IAM key is still in git history (key itself deleted 2026-10-12, TRUE-82705).
|
|
43
|
+
|
|
44
|
+
## Change history
|
|
45
|
+
- 2026-10-19 — Created: removed `copyFileToS3` from uploads (mount write is the S3 write), mount check before any row, per-reply duplicate guard, `error.php` Throwable guard, s3fs `iam_role=auto` (TRUE-82935, branch, not deployed) (jcardinal)
|
|
@@ -6,8 +6,8 @@ project: Database Changes
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: workflow
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: [sking]
|
|
9
|
+
updated: 2026-10-19
|
|
10
|
+
owners: [sking, jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- dbchanges/index.php
|
|
13
13
|
- dbchanges/Core/
|
|
@@ -106,6 +106,7 @@ The repo ships `config.alpha/beta/demo/hotfix/prod/stage/test/worker.ini`, there
|
|
|
106
106
|
## Gotchas
|
|
107
107
|
|
|
108
108
|
- **A comment-only chunk stops the whole queue** (error 1065 + `exit()` on the prod path). Your malformed file blocks unrelated teammates' migrations — a team-wide failure, not a local one.
|
|
109
|
+
- **A `--` comment line ending in `;` splits the file too** (the split is on `";\n"`, comments included). So verify SELECTs go in the header comment **without** a trailing `;` — e.g. `-- Verify: SELECT COUNT(*) FROM files WHERE ...` (TRUE-82935).
|
|
109
110
|
- **No `DELIMITER`, no stored routines, no explicit `COMMIT`.** See the guard-pattern section.
|
|
110
111
|
- **Error 1093** is why the preflight subquery must be wrapped in a derived table.
|
|
111
112
|
- **⚠ The repo does NOT describe the whole live schema — never conclude "that column/index does not exist" from `dbchanges` alone.** `Core.AdvanceShippingNoticeQueue.dedupeKey` and `.alertSentAt` exist in production (`dedupeKey` confirmed UNIQUE) but **no migration for them exists anywhere in this repo** — they were created from a `library` commit (TRUE-77676). Found 2026-09-16 because a reviewer read the repo and reached the wrong conclusion. Verify columns and indexes against the **database**.
|
|
@@ -114,6 +115,9 @@ The repo ships `config.alpha/beta/demo/hotfix/prod/stage/test/worker.ini`, there
|
|
|
114
115
|
- **Retry semantics are a feature**: because the data change and the `_dbchanges` row share a transaction, a failing file re-runs next cycle. Do not "help" it with a manual COMMIT.
|
|
115
116
|
- **⚠ SECURITY — hardcoded GitHub token.** `worker/crons/infrastructure/execute_dbchanges.php` contains a **plaintext GitHub personal access token** in the small config block near the top of the file (~L14–18), and interpolates it into a `shell_exec()` git-clone command line — which also exposes it in the worker host's process listing. Treat the token as **compromised**: rotate it, move it to config/env (`config.<env>.ini`), and pass it without embedding it in an argv string. No credential value is recorded here by design.
|
|
116
117
|
|
|
118
|
+
## Change history
|
|
119
|
+
- 2026-10-19 — Comment lines ending in `;` also split the file; verify SELECTs in the header without `;` (TRUE-82935) (jcardinal)
|
|
120
|
+
|
|
117
121
|
## Related
|
|
118
122
|
|
|
119
123
|
- [Worker architecture](../../worker/architecture.md) — the worker tier drives every dbchanges application.
|
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-19
|
|
10
10
|
owners: [jcardinal, mhammontree, bala, snaredla]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/cloud.php
|
|
@@ -71,17 +71,20 @@ CodePipeline `Worker` (502, us-west-2) sources `_production` of **both** `agilan
|
|
|
71
71
|
- **Triage: `Aws\S3\Exception\S3Exception` 403 `InvalidAccessKeyId` (or any S3 403) in a 1.0 app → check App_Cloud credentials and whether the current library is on that app's instances — not the app code.** Library is not in the app repo: each app's EB deploy clones it (`.ebextensions/020_setup_git_libraries.config`) and pages load it from the include path (`require_once('_.php')`). A library-only fix reaches an app only on that app's next successful deploy. TRUE-82712: TOGa Desk MSP redbox (blob-field model load — see [BLOBSTORAGE](mvc-page-pattern-and-app-skeleton.md#gotchas)) stopped once library + togadesk were released 2026-10-13; no togadesk change.
|
|
72
72
|
- **Fixed in TRUE-82705: shell injection in `copyFileToS3()`.** Both paths are now escaped with `addcslashes` (double quote, backslash, `$`, backtick) inside the double quotes, so an upload named like `$(cmd).pdf` (togaview ticket uploads build the S3 path from the file name) no longer runs `cmd`. Escape every new CLI-helper argument the same way, or use the SDK.
|
|
73
73
|
- **`getS3Objects()` returns `null` for both a failed CLI call and an empty prefix.** A caller that loops `['Contents']` without a null check raises a warning, which 1.0 `App_Error` turns into `exit`. When the prefix should be non-empty, throw on `null` (the ODP AS2 move cron does).
|
|
74
|
-
- **`copyFileToS3()` (and the other CLI-based calls) never report failure.** `shell_exec('aws s3 cp ...')` then `return true` — bad credentials = silent lost upload. Uploads made while the old key was dead (repair photos, KB files, `BLOBSTORAGE` saves via `App_Model::save()`, email-intake attachments) may be missing from S3. Open, not fixed.
|
|
74
|
+
- **`copyFileToS3()` (and the other CLI-based calls) never report failure.** `shell_exec('aws s3 cp ...')` then `return true` — bad credentials = silent lost upload. Uploads made while the old key was dead (repair photos, KB files, `BLOBSTORAGE` saves via `App_Model::save()`, email-intake attachments) may be missing from S3. Open, not fixed — CTO decision (TRUE-82935): fix it in library on its own ticket.
|
|
75
75
|
- `togadesk` `ebs/check_s3fs.php` + `ebs/register_instance_to_shared_application_load_balancer.php` now assume `WorkloadsRuntime` (TRUE-82718, PR #622, merged 2026-10-13); togaview fix is PR #307 (opened). The s3fs **mount** cannot assume a role — it uses the instance role directly: see [asifiles s3fs mount](../../togadesk/features/s3fs-asifiles-file-storage.md).
|
|
76
76
|
- **Every 1.0 app that bootstraps `library` needs `aws/aws-sdk-php` in its own `composer.json`** (since TRUE-82705). Without it any App_Cloud call fatals `Class 'Aws\Credentials\CredentialProvider' not found`. Hidden trigger: on prod/worker/bigworker, `App_Database::getDatabaseReaderHost()` (`library/app/database.php` ~156) calls `App_Cloud::getParameter('database_reader_endpoint')` whenever its cache file `databaseReadHost.txt` is older than 30-60 s. The fatal hits before the cache is rewritten, so the cache stays stale and **every** reader-DB request fails — the app looks fully down (TRUE-82793: Vision PunchOut for Canon / S&P Global via SAP Ariba returned HTTP 500). Fix = add `"aws/aws-sdk-php": "^3.328"` and pin `composer.lock` to worker's prod versions (aws-sdk-php 3.328.1, guzzle 7.9.2, promises 2.0.4, psr7 2.7.0, jmespath 2.8.0 — PHP 7.2 safe); EB runs `composer install --no-dev` on deploy (`vendor/` is gitignored).
|
|
77
77
|
- Has it (`_production`): worker, togadesk (`^3.337`), togaview, tools, toga (TRUE-82714, 63c5ed0; confirmed 2026-10-16), vision (TRUE-82793, merged + deployed 2026-10-15).
|
|
78
78
|
- store: fixed on branch (TRUE-82856, ae1861f4f — same pins plus aws-crt-php 1.2.7, psr/http-client 1.0.3, polyfill-mbstring 1.31.0; verified on PHP 7.2.33); needs merge + deploy. Its `.ebextensions/015_install_composer.config` already runs `composer install --no-dev`.
|
|
79
|
+
- **Missed by the audit (TRUE-82935): the 1.0 `api` repo** (no composer at all) and **togadesk's email cron** (`crons/tickets.php` loads only `desk/vendor/autoload.php`; the root togadesk composer with aws is not loaded there). Both stopped `App_Cloud`: [Talos ticket API](../../api/features/talos-ticket-endpoint.md), [email intake](../../togadesk/features/email-to-ticket-intake.md). Audit per **entry point**, not per repo: a CLI cron or sub-app can load a different autoloader than the web app.
|
|
80
|
+
- The missing class throws **`Error`, not `Exception`** — `catch (Exception)` around an App_Cloud call does not catch it; use `\Throwable` where a fallback is meant.
|
|
79
81
|
- Not changed on TRUE-82856: walmarttechservices — GitHub repo `agilantsolutions/walmarttechservices` no longer exists ("Repository not found", checked 2026-10-16); webhook — reported no longer active by the developer (2026-10-16). Removal / deprecation is left to the owners.
|
|
80
82
|
- Verify assume-role in CloudTrail (`AssumeRole` on `WorkloadsRuntime`, by session name: `library-app-cloud`, `setup-git-libraries`, `register-instance`, `worker-deploy`).
|
|
81
83
|
- Open, not fixed (developer owns): remove `AWS_ACCESS_KEY_ID`/`AWS_SECRET_KEY` from `agilant-worker` EB env properties.
|
|
82
84
|
- Every long-lived key still in code across repos (SES SMTP, s3fs, ALB scripts, other accounts): [audit 2026-10-13](../../../../2.0/apps/worker2/features/cross-account-aws-access.md#long-lived-keys-still-in-code-audit-2026-10-13).
|
|
83
85
|
|
|
84
86
|
## Change history
|
|
87
|
+
- 2026-10-19 — aws-sdk-php audit missed `api` + togadesk email cron; `Error` not `Exception` (TRUE-82935) (jcardinal)
|
|
85
88
|
- 2026-10-16 — aws-sdk-php: store fixed (TRUE-82856), vision deployed, walmarttechservices + webhook not changed (repo gone / reported inactive) (mhammontree)
|
|
86
89
|
- 2026-10-15 — aws-sdk-php requirement: reader-host crash path, per-app audit, vision fixed (TRUE-82793) (mhammontree)
|
|
87
90
|
- 2026-10-13 — togadesk ebs scripts converted (TRUE-82718); aws-sdk-php composer requirement (TRUE-82714) (mhammontree)
|
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-19
|
|
10
10
|
owners: ["dfranks", "jcardinal", "kyalamarthi", "ajean", "rgirish", "bala"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/netsuite/rest.php
|
|
@@ -72,6 +72,7 @@ In production (worker/EB) `send()` is fine — the Logs DB is reachable there.
|
|
|
72
72
|
- **Endpoint:** `POST {base}/query/v1/suiteql?limit=1000&offset=N`, body `{"q": "<sql>"}`.
|
|
73
73
|
- **`Prefer: transient` header is required** (avoids server-side cursor overhead / errors).
|
|
74
74
|
- **Pagination:** max `limit=1000`; the response carries `items[]` and a boolean **`hasMore`**. Loop `offset += 1000` until `hasMore` is false. The private `suiteqlListAll()` helper in `rest.php` does exactly this.
|
|
75
|
+
- **⚠ Paging must be safe (2026-10-19, TRUE-82938).** (1) Every paged query needs a **unique** `ORDER BY` (e.g. `ia.id`, tie-break `tl.id`) or offset paging can skip/repeat rows. (2) A 2xx page that is not an object or has no `items` array must **throw** (`requireSuiteqlPage()` in `suiteqlListAll` / `suiteqlListShimmed`) — a silent stop reads as "no rows" and drives wrong deletes in reconcile code.
|
|
75
76
|
- **Session timezone is `America/New_York`** — `CURRENT_DATE`/`CURRENT_TIMESTAMP`, `TO_TIMESTAMP()`, and `lastmodifieddate` all resolve in ET (−04:00/−05:00), distinct from the account's "Pacific" setting. Confirmed empirically: `SYSDATE` is −07:00 but the date columns are ET. **Build window bounds in ET.**
|
|
76
77
|
- **Date vs timestamp filters:** `trandate` is a **DATE** column — use `TO_DATE('YYYY-MM-DD', 'YYYY-MM-DD')`. `lastmodifieddate` is a **TIMESTAMP** — use `TO_TIMESTAMP('YYYY-MM-DD HH:MM:SS', 'YYYY-MM-DD HH24:MI:SS')`. `TO_DATE` rejects impossible calendar dates (e.g. `2026-06-31`) with an opaque HTTP 400 — validate dates before building SQL.
|
|
77
78
|
- **Oracle-flavored functions work:** `TO_DATE`, `TO_TIMESTAMP`, `TO_CHAR(trandate,'YYYY')` / `'YYYY-MM'` for bucketing, `NVL(...)`, and `BUILTIN.DF(field)` for a reference field's display value (e.g. `BUILTIN.DF(entity)` → customer name, `BUILTIN.DF(status)` → "Pending Fulfillment") — a cheap alternative to per-id GETs for display strings.
|
|
@@ -6,18 +6,24 @@ project: TOGa Desk
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-10-19
|
|
10
10
|
owners: ["jcardinal", "dfranks", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- crons/tickets.php
|
|
13
13
|
- crons/tickets_prod.php
|
|
14
|
+
- crons/tickets_prod_mailbox.php
|
|
15
|
+
- crons/includes/ticket_attachments.php
|
|
16
|
+
- crons/recover_attachments.php
|
|
14
17
|
- crons/pipe.php
|
|
15
18
|
- desk/includes/classes/class.ticket.php
|
|
16
19
|
- desk/includes/classes/class.user.php
|
|
17
|
-
- vendor/classes/class.imap.php
|
|
20
|
+
- desk/vendor/classes/class.imap.php
|
|
18
21
|
related:
|
|
19
22
|
- 1.0/apps/togadesk/architecture.md
|
|
20
23
|
- 1.0/apps/togadesk/features/ticket-lifecycle.md
|
|
24
|
+
- 1.0/apps/togadesk/features/s3fs-asifiles-file-storage.md
|
|
25
|
+
- 1.0/apps/library/features/app-cloud-aws-credentials.md
|
|
26
|
+
- 1.0/apps/worker/architecture.md
|
|
21
27
|
---
|
|
22
28
|
|
|
23
29
|
TOGa Desk email→ticket intake via IMAP poller (`crons/tickets.php`) + postfix pipe (`crons/pipe.php`), both into `Ticket::emailToTicket()`; open for routing/threading, noise suppression, or the sender auto-create of `people` rows.
|
|
@@ -33,12 +39,21 @@ TOGa Desk ingests support email into tickets through a cron-driven IMAP poller (
|
|
|
33
39
|
- `Ticket::emailToTicket()` in `desk/includes/classes/class.ticket.php` — the parser/router.
|
|
34
40
|
|
|
35
41
|
### Flow
|
|
36
|
-
1. **Bootstrap:** cron wrappers load the `library` framework + `App_Framework::cronInitialization()`, set `$env`, and register the IMAP (`Imap2`) helper. `
|
|
37
|
-
2. **Poll:** connect to `outlook.office365.com:993` INBOX and loop for ~22 minutes (header comment suggests running every ~30 min), processing each message then
|
|
42
|
+
1. **Bootstrap:** cron wrappers load the `library` framework + `App_Framework::cronInitialization()`, set `$env`, and register the IMAP (`Imap2`) helper. `$scriptpath` = `togadesk/desk`; autoload = `desk/vendor/autoload.php` (**no AWS SDK**) + `vendorClassAutoload` (`desk/vendor/classes/`). `tickets_prod.php` requires `tickets_prod_mailbox.php` for the **OAuth2/MSAL** O365 mailbox token.
|
|
43
|
+
2. **Poll:** connect to `outlook.office365.com:993` INBOX and loop for ~22 minutes (header comment suggests running every ~30 min), processing each message then moving it to the IMAP folder **"Trash"** (see Gotchas).
|
|
38
44
|
3. **Parse & route (`emailToTicket()`):** extract from/to/cc/subject/body (with encoding fixes); match an existing ticket via `In-Reply-To` + `Message-ID` headers and **plus-addressing** (`support+<ticketnum>@…`). Matched ⇒ `addReply()`; otherwise ⇒ `add()`.
|
|
39
45
|
- **Noise suppression (before any ticket/attachment is created):** both feeders drop known automated no-noise mail via the shared helper `Ticket::isSuppressedNoiseEmail(?string $from, ?string $subject): bool` (static, in `class.ticket.php`). It returns true only when the sender matches `/^no-?reply@/` **AND** the subject starts with `staples po ` — both required so a real person is never blocked, and the sender check is **domain-agnostic** (matches `noreply@togatech.com` / `noreply@goagilant.com` and survives future sender-domain flips; the Staples PO alert sender flipped `goagilant.com → togatech.com` on 2026-06-02). Wired into BOTH intake feeders: `crons/tickets.php` adds it as one more `&&`-clause in the existing IMAP intake guard (alongside the delivery-status / `MAILER-DAEMON` / Compass auto-reply filters); `crons/pipe.php` uses it as an early guard-and-`exit(0)` right after the `$from`/`$subject` parse. Each feeder logs a "suppressed" line. (TRUE-79975 — ~815 such alerts since Jun 2024, ~704 of them landing in the dept-1 "Support Desk" catch-all and being closed by hand.)
|
|
40
|
-
4. **Side effects:** auto-create a user when the sender domain maps to an existing client; assign to a department via the department email lookup; save attachments
|
|
46
|
+
4. **Side effects:** auto-create a user when the sender domain maps to an existing client; assign to a department via the department email lookup; save attachments (helpers in `crons/includes/ticket_attachments.php`) **straight into `desk/uploads`** — on the worker a symlink to the s3fs mount (`/var/app/asifiles/togadesk-prod/uploads`), so the write is the S3 write. Before writing it checks `is_link` + written size. **`App_Cloud` is not used** (PR #625, TRUE-82935). Blocked names (`TICKET_IMPORT_BLOCKED_FILE_PATTERN`: php*/cgi/sh/asp/jsp/html/htm/xhtml/shtml/svg/svgz/xml/xsl/xslt/xht/mht/mhtml/shtm/stm/ssi, `.htaccess`, `.user.ini`) are saved with `.txt` added. Optionally generate Talos summaries.
|
|
41
47
|
5. **Post-pass:** `Ticket::processRules()` (escalation) and `Ticket::autoClose()` (SLA auto-close) run after the mailbox sweep.
|
|
48
|
+
6. **On failure:** alert email to `serviceops@togatech.com`, also logged in legacy `Logs.Emails` (`subject LIKE 'TOGaDesk email import failed%'`) — the fastest way to read the cron's error (the cron has no stdout).
|
|
49
|
+
|
|
50
|
+
### Attachment recovery (`crons/recover_attachments.php`, one-time, TRUE-82935)
|
|
51
|
+
Re-saves attachments lost 2026-10-12 → 2026-10-19 from the originals in IMAP "Trash".
|
|
52
|
+
- Read-only on the mailbox (`FT_PEEK`; `Imap::getAttachment()` has an optional `$peek` param).
|
|
53
|
+
- Match: email `Message-ID` = `tickets_replies.emailMessageIdentifier` on exactly one ticket; duplicate Message-IDs in the folder skipped; `From` must equal the reply sender's `people.email`.
|
|
54
|
+
- Re-saves missing bytes for existing `files` rows under the same name; inserts rows that were deleted; relinks inline `cid:` images.
|
|
55
|
+
- Dry run by default; `--apply`, `--since`, `--id`, `--folder`, `--max-mb`, `--limit`. Refuses `--apply` unless `desk/uploads` is the S3 link.
|
|
56
|
+
- Run on the worker between intake runs: `php /var/www/html/ontrack/crons/recover_attachments_prod.php`. Ship **after** the intake fix, as its own deploy (CTO decision).
|
|
42
57
|
|
|
43
58
|
### Data model
|
|
44
59
|
Writes `tickets`, `tickets_replies`, attachment `files`; reads config from the `config` table (`tickets_defaultdepartment`, `tickets_defaultclient`, `auto_close_tickets`, `email_from_address`, `timezone`). Department routing keys on a `tickets_departments.email` `LIKE` match against the To/CC addresses; **anything unmatched falls through to config `tickets_defaultdepartment` (= dept 1 "Support Desk" catch-all) and `tickets_defaultclient`** — which is why unrouted/automated mail piles into "Support Desk". There is **no allowlist/blocklist config framework**: noise filtering is ad-hoc inline guards in the crons (the hardcoded Compass "Automatic reply:" filter and now `isSuppressedNoiseEmail`).
|
|
@@ -68,15 +83,19 @@ Ruled out as user-creation sources (do not re-chase):
|
|
|
68
83
|
|
|
69
84
|
## Gotchas
|
|
70
85
|
- **`$processName` guard:** `tickets.php` throws if executed directly instead of via an env wrapper — and `ENVIRONMENT`/`$env` must be set or it loads the wrong config.
|
|
71
|
-
- **
|
|
86
|
+
- **A togadesk merge does NOT update the cron.** The cron runs on the 1.0 **worker** (`/var/www/html/ontrack`, cloned from togadesk at worker deploy — NOT `/var/www/ontrack`). The `Worker` CodePipeline sources only `worker` + `library`; a togadesk merge runs `TOGaDesk-Production` (web envs) only. After any `crons/` change, **release the `Worker` pipeline by hand**. TRUE-82935: PR #625 merged 10-16 16:15, reached the cron only 10-19 13:10. See [worker architecture](../../worker/architecture.md).
|
|
87
|
+
- **Never load a second composer autoloader in the cron as a fallback** (e.g. root `vendor/autoload.php` for the AWS SDK) — CTO decision TRUE-82935. Keep the cron off `App_Cloud`; any `App_Cloud` call there throws `Error` ([App_Cloud](../../library/features/app-cloud-aws-credentials.md)).
|
|
88
|
+
- **Attachment outage 2026-10-12 → 10-19 (TRUE-82935).** 10-12 15:00 → 10-16 15:00: `files` rows written, S3 object missing (`copyFileToS3` threw). 10-16 15:16 → 10-19 13:10: PR #624 deleted the row on failure → no attachments at all. First good run 10-19 13:20 (files 615609-615611). Recovery: section above.
|
|
89
|
+
- **Processed mail is moved, never purged.** `desk/vendor/classes/class.imap.php` `getTrash()` matches only `trash` / `inbox.trash` / `papierkorb`, else creates a custom folder **"Trash"** — originals stay recoverable there.
|
|
90
|
+
- **OAuth2:** the prod O365 app secret has a hard expiry (~2026-12-20); rotate before then or intake silently stops. **Security, open:** the client secret for `support@togatech.com` is committed in `crons/tickets_prod_mailbox.php` (was `tickets_prod.php`) — rotate and move to config/SSM (value not recorded).
|
|
72
91
|
- **Loop prevention** only excludes FROM==department-TO addresses; CC/reply-all across multiple department addresses can still loop.
|
|
73
92
|
- **Auto-reply filters** are partly hardcoded (e.g. ignore `TechHub@compass-usa.com` "Automatic reply:" messages), and now the `isSuppressedNoiseEmail` Staples-PO filter — there is no config-driven allow/block list, so each new noise source is another inline guard.
|
|
74
93
|
- **`Ticket::extractEmail()` returned an undefined offset on a no-match** (bare `$matches[0]`). With a sender-less/malformed email this codebase promotes the warning to a thrown error (`E_ALL` + `App_Error::handleError`), so an intake cron could **halt**. Hardened to `return $matches[0] ?? '';` (TRUE-79975) — important because `isSuppressedNoiseEmail` routes nullable/malformed sender input through `extractEmail`, putting this latent bug on the hot path.
|
|
75
|
-
- **`class.imap.php` duplicate trap —
|
|
76
|
-
- Attachments are deleted locally immediately after the S3 upload with **no retry** if S3 fails.
|
|
94
|
+
- **`class.imap.php` duplicate trap — `desk/vendor/classes/class.imap.php` is LIVE; root `vendor/classes/class.imap.php` is dead** (same direction as `class.ticket.php`). Both `vendorClassAutoload()` copies (`desk/includes/functions.php`, root `includes/functions.php`) load from `desk/vendor/classes/`, nothing `require`s the root copy, and only the desk copy has `getImapStream()` / `imap2_*` (verified 2026-10-19, TRUE-82935). The earlier KB claim that the root copy is live was wrong — TRUE-80116's edit + revert of the root copy had no effect on intake. Verify the right copy before calling a change "byte-identical".
|
|
77
95
|
- **`adminid` clobber on reply routing:** after `emailToTicket()` sets `$data['adminid']` from the FROM address (~line 1407), an asset lookup (~line 1437-1442) overwrites it with the replying user's asset's assigned admin. On a matched-ticket reply this makes a user's email look like a staff reply in `addReply()` (`isAdminReply=true`), so `Awaiting User` etc. never transition. The asset-derived adminid is for NEW-ticket auto-assign in `add()`, not reply classification. See the ticket-lifecycle doc.
|
|
78
96
|
|
|
79
97
|
## Change history
|
|
98
|
+
- 2026-10-19 — Attachments saved straight into `desk/uploads` (s3fs), no `App_Cloud`; recovery CLI; worker-pipeline deploy gotcha; corrected live `class.imap.php` copy; wider blocked-file pattern (TRUE-82935) (jcardinal)
|
|
80
99
|
- 2026-09-25 — renamed product reference from "TogaIQ" to "Talos" (rebrand; TOGa IQ is now the separate `toga25-iq` app) (jcardinal)
|
|
81
100
|
|
|
82
101
|
## Related
|
|
@@ -6,8 +6,8 @@ project: TOGa Desk
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
10
|
-
owners: ["mhammontree", "snaredla"]
|
|
9
|
+
updated: 2026-10-19
|
|
10
|
+
owners: ["mhammontree", "snaredla", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- togadesk/ebs/mount-s3fs-folders.php
|
|
13
13
|
- togadesk/ebs/create-s3fs-credentials-file.php
|
|
@@ -17,6 +17,8 @@ files:
|
|
|
17
17
|
related:
|
|
18
18
|
- ../architecture.md
|
|
19
19
|
- ../../library/features/app-cloud-aws-credentials.md
|
|
20
|
+
- ../../api/features/talos-ticket-endpoint.md
|
|
21
|
+
- email-to-ticket-intake.md
|
|
20
22
|
---
|
|
21
23
|
|
|
22
24
|
How TOGa Desk reads stored files through the s3fs mount of bucket `asifiles` at `/var/app/asifiles` (instance role, no keys); open when a Desk file "is missing" or before touching the s3fs ebextension.
|
|
@@ -39,6 +41,9 @@ Every Desk file read from `/var/app/asifiles/...` (Knowledge Base docs `knowledg
|
|
|
39
41
|
- The mount hook uses the **Amazon Linux 1** hooks path (`/opt/elasticbeanstalk/hooks/...`) — it will not run on AL2023. Rewrite as a platform hook when migrating.
|
|
40
42
|
- **Open, not fixed — `template/pages/documents/viewdocument.php`:** `$_GET['id']` goes raw into the file path (path traversal, e.g. `../../../current/config.prod.ini`) and into unescaped SQL on `kb_versioning` and `kb_documents` (views UPDATE). A missing file red-boxes (`App_Error` exits on the warning) instead of a clean message. Fix: cast id to `(int)`; guard the read and show "document not available". Needs its own ticket.
|
|
41
43
|
- The KB path is hardcoded to `knowledgebase-prod` (env-based line commented out), so non-prod reads prod KB files.
|
|
44
|
+
- **Writing into the mount IS the S3 upload.** A file written under `/var/app/asifiles/<prefix>/` lands at `s3://asifiles/<prefix>/`; do not also `App_Cloud::copyFileToS3` it. Used by the email cron (worker `ontrack/desk/uploads` → `/var/app/asifiles/togadesk-prod/uploads`, `worker/ebs/setup_symbolic_links.php` + `links.worker.json`) and the 1.0 `api` Talos uploads ([Talos ticket API](../../api/features/talos-ticket-endpoint.md), same `iam_role=auto` setup since TRUE-82935). A write while the mount is down goes to local disk hidden under the mount point — check the mount first (`is_link`, or compare `stat()` dev with the parent).
|
|
45
|
+
- **Open, not fixed — `odp-togadesk-prod` (Office Depot) deploys fail.** TOGaDesk-Production deploys of PR #624 and #625 (2026-10-16) failed there: `command 04-make_s3fs in .ebextensions/006_mount-s3fs.config failed` (s3fs 1.84 source build: virtual memory exhausted). `agilant-togadesk-prod` and `imhouston-togadesk-prod` succeeded. ODP instances may run mixed versions. Needs its own ticket (bigger instance / swap, or install a prebuilt s3fs).
|
|
42
46
|
|
|
43
47
|
## Change history
|
|
48
|
+
- 2026-10-19 — Mount write = S3 upload (email cron, api); ODP s3fs build failure on deploy (TRUE-82935) (jcardinal)
|
|
44
49
|
- 2026-10-13 — Created: s3fs mount moved from the dead IAM key to `-o iam_role=auto`; check_s3fs/register scripts assume `WorkloadsRuntime` (TRUE-82718, togadesk PR #622) (snaredla, mhammontree)
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-10-
|
|
9
|
+
updated: 2026-10-19
|
|
10
10
|
owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
@@ -722,6 +722,34 @@ See the [tracking-number bridge doc](../../../2.0/apps/_underscore/features/trac
|
|
|
722
722
|
max-depth FK omission. Keep new fields explicit in the projection — a needed field left out of the
|
|
723
723
|
list silently reads `null`, not an error.
|
|
724
724
|
|
|
725
|
+
### ⚠ A serial moved between fulfillments in NetSuite → unit RENAME hit 1062 (Compass, 2026-10-19, TRUE-82938, library fix not deployed)
|
|
726
|
+
|
|
727
|
+
Symptom: ITEM_FULFILLMENTS `1-RUNNING`, cursor stuck `2026-10-19 11:08:58|0`, `Logs.Issue` 468
|
|
728
|
+
(clientId 2). Real error only in `Logs_Compass.Api`: `PUT /v2/units/<77337>` →
|
|
729
|
+
`Duplicate entry '2658-Q3GD-6HEC-WV4T' for key 'Units.itemId_serialNumber'`.
|
|
730
|
+
|
|
731
|
+
- **Cause (valid NetSuite state):** IF 7629593 (shipped serial `Q3GD-6HEC-WV4T` → unit 77217) was
|
|
732
|
+
deleted; IF 7636054 (TOGa 332224) had assignment 7661487 edited `Q3GD-T5KH-NATN` → `Q3GD-6HEC-WV4T`.
|
|
733
|
+
The update branch renamed the linked unit 77337 to a serial another unit (same item) already owned.
|
|
734
|
+
- **Fix (`syncItemFulfillmentFromNetsuite` update branch):** serial differs → look up the new serial
|
|
735
|
+
**within the linked unit's item** (the unique-key domain; optional `uuidItem` on
|
|
736
|
+
`getUnitBySerialNumber`, also used by create). Owner found → repoint the link
|
|
737
|
+
(`PUT /item-fulfillment-item-units/{uuid}` unit=owner) and act on the owner; none → rename (now
|
|
738
|
+
collision-free). Strict string compare in the repoint gate. Swaps/rotations self-heal in one run (a
|
|
739
|
+
temporary double link is allowed).
|
|
740
|
+
- **Stale-link cleanup per line:** delete links (tracking bridge rows first) whose non-empty
|
|
741
|
+
assignment id is gone from NetSuite for that line — also when the line has no serials
|
|
742
|
+
(`inventoryDetail` null = SuiteQL returned no assignment rows). Never ASSETTAG lines; links with an
|
|
743
|
+
empty assignment id untouched; throws if an assignment has no `internalId`.
|
|
744
|
+
- **Paged SuiteQL must be safe:** see [SuiteQL paging](../../library/features/netsuite-suiteql-api-reference.md#suiteql)
|
|
745
|
+
— a silent short page here would read as "no rows" and drive wrong deletes.
|
|
746
|
+
- **Verify after deploy:** ITEM_FULFILLMENTS climbs to `864000-IDLE` on its own; IF 332224 links to
|
|
747
|
+
unit 77217.
|
|
748
|
+
- **OPEN:** (a) the sync never removes IFs deleted in NetSuite (TOGa IF 332016 stale; unit 77337 /
|
|
749
|
+
`T5KH-NATN` left with no IF link); (b) item-receipt (~`toga2.php` L6306) and inventory-adjustment
|
|
750
|
+
(~L12120) update paths have the same serial-rename 1062 risk; (c) paged `rest.php` queries at ~L736,
|
|
751
|
+
910, 1166, 1513 still lack a unique tie-break.
|
|
752
|
+
|
|
725
753
|
### Single-tracking-number fan-out to items (318) and units (319) — corrected 2026-08-26
|
|
726
754
|
|
|
727
755
|
The earlier "maintains all three bridges" claim was **only true for 317 (header)**: in prod
|
|
@@ -1196,7 +1224,7 @@ same `Parameters` cursors: whoever writes last wins and the other's window is si
|
|
|
1196
1224
|
|
|
1197
1225
|
## Fix rules (team directives for any sync/reconcile change)
|
|
1198
1226
|
|
|
1199
|
-
1. **Self-heal to match NetSuite
|
|
1227
|
+
1. **Self-heal to match NetSuite — never leave a state that needs hand cleanup** (jcardinal, 2026-10-19). Re-processing a record must leave the DB matching the source, including deleting stale/duplicate rows and repointing links. Any valid NetSuite data state (serial moved, swapped, removed) is handled in code; throw only for real bugs (missing data, API/ACL errors). A one-time drain is allowed only after the code fix.
|
|
1200
1228
|
2. **Repair by code, never SQL.** Wrong sync data is fixed in sync code (self-heals on re-run), then cursors are rewound to replay. Also ship a single-order runner keyed by the NetSuite order number / internal id (not the Toga id). No data-repair or backfill SQL. Schema/metadata/ACL SQL in dbchanges2 is fine.
|
|
1201
1229
|
3. **Fail loud, fix upstream.** No `error_log()`-and-`continue` skips (worker stdout is discarded): throw so the section stays RUNNING and lands in `Logs.Event`. On a failed match, look higher up for the real cause (e.g. the REST shim returning bad lines) before adding any skip.
|
|
1202
1230
|
4. **Dependencies import first.** Import a record's parents before it (SO before IF/Invoice, PO before ItemReceipt, IF before InventoryAdjustment). If the NetSuite record declares a parent and we end with no real linked Toga parent, throw before creating. Never skip, orphan, or advance the cursor past it.
|
|
@@ -2168,6 +2196,7 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
|
|
|
2168
2196
|
[NetSuite Sync Alert Monitor](../../library/features/netsuite-sync-alert-monitor.md).
|
|
2169
2197
|
|
|
2170
2198
|
## Change history
|
|
2199
|
+
- 2026-10-19 — Compass ITEM_FULFILLMENTS froze on a serial moved between IFs in NetSuite (1062, Issue 468); update branch now repoints within the item + prunes stale assignment links; SuiteQL paging throws on bad pages + unique ORDER BY. Fix rule 1 widened (no hand cleanup). Library only, not deployed. TRUE-82938. (jcardinal)
|
|
2171
2200
|
- 2026-10-18 — NYCHH SALES_ORDERS froze on unknown warehouse NS location 1 (Issue 1041) after the 10-14 cursor rewind; fixed by adding the Location row, not the skip flag. Data only. TRUE-82859. (jcardinal)
|
|
2172
2201
|
- 2026-10-16 — NYCHH SALES_ORDERS froze on a TO line removed in NetSuite (FK 1451, Issue 445); TO-line delete now clears PO/SO links first. Library only, prod-verified. TRUE-82859. (jcardinal)
|
|
2173
2202
|
- 2026-10-09 — Customer PO duplicate lines: merge step (all TO clients) + SO-QTY opt-in (hh); [doc](../../library/features/customer-po-line-merge-and-quantity.md). Library + worker deploy together. TRUE-82606. (jcardinal)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -17,7 +17,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
17
17
|
- **store** (Store) — 4 doc(s) → [1.0/apps/store/INDEX.md](1.0/apps/store/INDEX.md)
|
|
18
18
|
- **tools** (Tools) — 25 doc(s) → [1.0/apps/tools/INDEX.md](1.0/apps/tools/INDEX.md)
|
|
19
19
|
- **vision** (Vision) — 0 doc(s) → [1.0/apps/vision/INDEX.md](1.0/apps/vision/INDEX.md)
|
|
20
|
-
- **api** (API 1.0) —
|
|
20
|
+
- **api** (API 1.0) — 1 doc(s) → [1.0/apps/api/INDEX.md](1.0/apps/api/INDEX.md)
|
|
21
21
|
|
|
22
22
|
## 2.0 framework
|
|
23
23
|
|
|
@@ -23,7 +23,7 @@ project: _Underscore
|
|
|
23
23
|
client: compass-usa
|
|
24
24
|
type: profile
|
|
25
25
|
status: active
|
|
26
|
-
updated: 2026-10-
|
|
26
|
+
updated: 2026-10-19
|
|
27
27
|
owners: [jcardinal, bala, tcox, apeterson, dfranks, akhokhani, ajean, rgirish]
|
|
28
28
|
files: []
|
|
29
29
|
related:
|
|
@@ -233,6 +233,7 @@ separate, related client (see its own profile).
|
|
|
233
233
|
the ⚠ rule that **deactivation is raw bulk SQL, so no model hook ever fires on it**.
|
|
234
234
|
|
|
235
235
|
## Notes
|
|
236
|
+
- **⚠ ITEM_FULFILLMENTS froze 2026-10-19 (Issue 468, 1062 on `Units.itemId_serialNumber`): a serial was moved between IFs in NetSuite** (IF 7629593 deleted, IF 7636054 re-serialed). Library fix (TRUE-82938) not yet deployed; verify IF 332224 → unit 77217 after. [Per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
|
|
236
237
|
- **Record numbers are Compass's own series, counted in order (TRUE-82715, 2026-10-18).** Compass checks its own order numbers for gaps; SA continues after SA139197. A gap = a failed create (check `Logs.Api` / `Logs.Issue`). [Record numbering](../../2.0/apps/_underscore/features/autoincrement-record-numbering.md).
|
|
237
238
|
- **⚠ Approval auto-cancel never fired for 926 legacy-migrated orders (TRUE-82868, 2026-10-16, not yet deployed).** Their admin approval has `dtDecision` NULL; the crons now age from `COALESCE(dtDecision, SalesOrders.dtCreated)`, reminders stop at 30 days, cancel starts after. [Approval reminder & auto-cancel crons](../../1.0/apps/worker/features/compass-manager-approval-reminder-emails.md).
|
|
238
239
|
- **⚠ SALES_ORDERS froze 2026-10-02 → 10-05 (Issue 940): closed-order delete matched the Office Depot twin SO** (dual-SO trap). Fixed in library with the per-client scope; open: higher-tier mirror fulfillments 91650/91651. Details: [closed-order rule](../../1.0/apps/library/features/netsuite-closed-order-rule.md).
|
|
@@ -4,12 +4,15 @@ framework: "standalone"
|
|
|
4
4
|
apps:
|
|
5
5
|
- voice-to-voice
|
|
6
6
|
- toga
|
|
7
|
+
- togadesk
|
|
8
|
+
- api
|
|
9
|
+
- dbchanges
|
|
7
10
|
project: TOGa Voice
|
|
8
11
|
client: office-depot
|
|
9
12
|
type: profile
|
|
10
13
|
status: active
|
|
11
|
-
updated: 2026-10-
|
|
12
|
-
owners: [akhokhani, mhammontree]
|
|
14
|
+
updated: 2026-10-19
|
|
15
|
+
owners: [akhokhani, mhammontree, jcardinal]
|
|
13
16
|
files: []
|
|
14
17
|
related:
|
|
15
18
|
- features/odp-tech-support.md
|
|
@@ -28,6 +31,7 @@ Office Depot is the production support client of the standalone Python/LiveKit T
|
|
|
28
31
|
## Platforms and knowledge
|
|
29
32
|
|
|
30
33
|
- [ODP support agent](features/odp-tech-support.md) owns ODP prompt, settings, billing tenant and deployment target.
|
|
34
|
+
- Talos call files reach Desk tickets (TOGaDesk clientid 3) through the 1.0 `api`: [Talos ticket API](../../1.0/apps/api/features/talos-ticket-endpoint.md). ODP's own Desk env `odp-togadesk-prod` has an open deploy failure: [asifiles s3fs mount](../../1.0/apps/togadesk/features/s3fs-asifiles-file-storage.md).
|
|
31
35
|
- [Voice architecture](../../standalone/apps/voice-to-voice/architecture.md) maps SIP, LiveKit, shared services and operational checks.
|
|
32
36
|
- [Ticket API](../../standalone/apps/voice-to-voice/features/ticket-api-integration.md) owns signing and client/department routing.
|
|
33
37
|
- [Post-call Lambda](../../standalone/apps/voice-to-voice/features/post-call-lambda.md) owns recordings/transcript attachments and its dated outage evidence.
|
package/package.json
CHANGED