toga-ai 1.0.1007 → 1.0.1009

package/knowledge/1.0/apps/library/features/app-cloud-aws-credentials.md

@@ -7,7 +7,7 @@ client: shared
 type: feature
 status: active
 updated: 2026-10-13
-owners: [jcardinal, mhammontree]
+owners: [jcardinal, mhammontree, bala]
 files:
   - library/app/cloud.php
   - library/app/systemmonitor/rdsproductioncluster.php
@@ -24,6 +24,8 @@ related:
   - ../../tools/workflows/deploy-to-elastic-beanstalk-al2023.md
   - ../architecture.md
   - mvc-page-pattern-and-app-skeleton.md
+  - ../../togaview/features/ticket-detail-page.md
+  - ../../../../clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md
 ---
 
 How every 1.0 AWS call (App_Cloud CLI + SDK, deploy scripts, `setup_git_libraries.php`) gets credentials by assuming `WorkloadsRuntime` from the EC2 instance role; open before touching 1.0 AWS access or EB deploy hooks.
@@ -62,11 +64,15 @@ CodePipeline `Worker` (502, us-west-2) sources `_production` of **both** `agilan
 - **CLI temp creds are visible in argv (`ps`).** Open follow-up: CLI profile with `credential_source=Ec2InstanceMetadata`.
 - **`setup_git_libraries.php` failing is silent.** When the key died its pipeline lookup failed and **library was not updated on deploy** — the old library kept running. If a library change "didn't deploy", check this script first. It was public-read with the key inside — a likely leak path; never put a credential in it.
 - **Triage: `Aws\S3\Exception\S3Exception` 403 `InvalidAccessKeyId` (or any S3 403) in a 1.0 app → check App_Cloud credentials and whether the current library is on that app's instances — not the app code.** Library is not in the app repo: each app's EB deploy clones it (`.ebextensions/020_setup_git_libraries.config`) and pages load it from the include path (`require_once('_.php')`). A library-only fix reaches an app only on that app's next successful deploy. TRUE-82712: TOGa Desk MSP redbox (blob-field model load — see [BLOBSTORAGE](mvc-page-pattern-and-app-skeleton.md#gotchas)) stopped once library + togadesk were released 2026-10-13; no togadesk change.
+- **Fixed in TRUE-82705: shell injection in `copyFileToS3()`.** Both paths are now escaped with `addcslashes` (double quote, backslash, `$`, backtick) inside the double quotes, so an upload named like `$(cmd).pdf` (togaview ticket uploads build the S3 path from the file name) no longer runs `cmd`. Escape every new CLI-helper argument the same way, or use the SDK.
+- **`getS3Objects()` returns `null` for both a failed CLI call and an empty prefix.** A caller that loops `['Contents']` without a null check raises a warning, which 1.0 `App_Error` turns into `exit`. When the prefix should be non-empty, throw on `null` (the ODP AS2 move cron does).
 - **`copyFileToS3()` (and the other CLI-based calls) never report failure.** `shell_exec('aws s3 cp ...')` then `return true` — bad credentials = silent lost upload. Uploads made while the old key was dead (repair photos, KB files, `BLOBSTORAGE` saves via `App_Model::save()`, email-intake attachments) may be missing from S3. Open, not fixed.
 - **Open, not fixed: `togadesk` and `togaview` `ebs/check_s3fs.php` + `ebs/register_instance_to_shared_application_load_balancer.php` still hardcode the old dead key** (verified on `origin/_production` 2026-10-13; TRUE-82705 did not convert them). Not on the request path; follow up with Jeff/DevOps.
 - Verify assume-role in CloudTrail (`AssumeRole` on `WorkloadsRuntime`, by session name: `library-app-cloud`, `setup-git-libraries`, `register-instance`, `worker-deploy`).
 - Open, not fixed (developer owns): remove `AWS_ACCESS_KEY_ID`/`AWS_SECRET_KEY` from `agilant-worker` EB env properties.
+- Every long-lived key still in code across repos (SES SMTP, s3fs, ALB scripts, other accounts): [audit 2026-10-13](../../../../2.0/apps/worker2/features/cross-account-aws-access.md#long-lived-keys-still-in-code-audit-2026-10-13).
 
 ## Change history
 - 2026-10-12 — Created: removed the static IAM key from all 1.0 code/config; App_Cloud, deploy scripts and `setup_git_libraries.php` assume `WorkloadsRuntime` (TRUE-82705) (jcardinal)
 - 2026-10-13 — S3 403 triage, silent `copyFileToS3` failure, unconverted togadesk/togaview ebs scripts (TRUE-82712) (mhammontree)
+- 2026-10-13 — Recorded the `copyFileToS3` shell-injection fix (addcslashes, TRUE-82705), the `getS3Objects` null ambiguity, and a link to the remaining-keys audit (bala)

package/knowledge/1.0/apps/togaview/features/ticket-detail-page.md

@@ -6,8 +6,8 @@ project: TOGa View
 client: shared
 type: feature
 status: active
-updated: 2026-09-16
-owners: ["mhammontree"]
+updated: 2026-10-13
+owners: ["mhammontree", "bala", "jcardinal"]
 files:
   - common/togaview/ticket.php
   - common/towfoundation/ticket.php
@@ -17,6 +17,7 @@ files:
 related:
   - 1.0/apps/togaview/features/login-flows.md
   - 1.0/apps/togaview/features/msp-dashboard.md
+  - 1.0/apps/library/features/app-cloud-aws-credentials.md
 ---
 
 Security of the TOGa View ticket detail page `common/*/ticket.php` and its THREE near-duplicate forks (IDOR ownership checks, int-cast, name-split guard, sqlProtect removal); open before touching any ticket.php fork.
@@ -75,6 +76,9 @@ Both RetailServices logins hardcode `$_SESSION['togadeskClientId'] = 11` (`mvc/l
 ### OPEN — portal-uploaded attachments can never be downloaded
 Upload writes to **S3** (`App_Cloud::copyFileToS3($tempPath, 'asifiles', '/togadesk-<env>/uploads/<name>')`, `mvc/ticket/post.php:160`; `files.file` stores only the bare filename), but the download handler reads the **local filesystem** at `$_SERVER['DOCUMENT_ROOT'] . $config['internal']['kb_documents_path'] . '/' . $file->file` — the **knowledge-base** document path, evidently copy-pasted from a KB handler and never adapted for S3. Fails **silently**: `file_exists()` fails → `routeTo(...&invalidFile=true)` → "Invalid File" toaster, not an error, which is why it went unnoticed. Same in all three forks. `App_Cloud::getFileFromS3()` (`library/app/cloud.php:540`) exists and is unused here. Verify against production (prod `DOCUMENT_ROOT/uploads` may be S3-synced) before sizing.
 
+### FIXED 2026-10-12 — attachment upload filename reached a shell command
+`mvc/ticket/post.php` (~`:158-160`) builds the S3 path from the uploaded file's name and passes it to `App_Cloud::copyFileToS3()`, which runs `shell_exec` inside double quotes. Until TRUE-82705 a file named like `$(cmd).pdf` ran `cmd` on the togaview host; `copyFileToS3()` now escapes both paths with `addcslashes`. Uploads authenticate through the `WorkloadsRuntime` role. Detail: [AWS credentials](../../library/features/app-cloud-aws-credentials.md).
+
 ### Per-developer config must carry the `kb_*` keys
 Attachment download reads `$config['internal']['kb_documents_path']`. A dev config missing the five `kb_*` keys (`kb_documents_path`, `kb_articles_path`, `kb_faqs_path`, `kb_tasks_path`, `kb_general_path`) — which prod/alpha/beta/stage and most dev configs carry — raises "Undefined index", **fatal** under `App_Error`. The active config is chosen by `getenv('ENVIRONMENT')`, set via Apache `SetEnv ENVIRONMENT` in `httpd.conf`. Copy the `kb_*` block when creating a new `config.dev-*.ini`.
 
@@ -83,6 +87,11 @@ Attachment download reads `$config['internal']['kb_documents_path']`. A dev conf
 - Pending as of 2026-06-12: MSP/CUSTOMER bypass for the ownership check (MSP session's `togadeskClientId` is the MSP's own client id — matches their customers' tickets since those carry the MSP's clientid, but verify when an MSP views a ticket whose clientid differs).
 - TRUE-80671 changes were **working-tree only** (branch `TRUE-80671`, nothing committed) at session end.
 
+## Change history
+- 2026-10-13 — Upload shell injection fixed in `copyFileToS3()` (TRUE-82705, jcardinal); uploads work again via the role. (bala)
+- 2026-10-12 — Recorded the open shell-injection path from the upload filename into `App_Cloud::copyFileToS3()`. (bala)
+
 ## Related
 - [Login flows](login-flows.md)
 - [MSP dashboard & ticket visibility](msp-dashboard.md)
+- [AWS credentials (WorkloadsRuntime role)](../../library/features/app-cloud-aws-credentials.md)

package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-12
+updated: 2026-10-13
 owners: [jcardinal, bala]
 files:
   - worker2/Component/Aws/Workloads/Workloads.php
@@ -23,13 +23,15 @@ related:
   - ./oneuptime-worker2-monitoring.md
   - ./elastic-beanstalk-health-monitor.md
   - ./worker-fleet-role-assignment-monitor.md
+  - ./committed-secrets-and-config-debt.md
+  - ../../../../clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md
 ---
 
 The `WorkloadsRuntime` role model (accounts, trusts, permissions — no IAM keys anywhere, 1.0 or 2.0) and worker2's `_Component_Aws_Workloads`; open before writing any code that touches AWS.
 
 ## Summary
 
-**No long-lived IAM keys anywhere, 1.0 or 2.0** (TRUE-82705, 2026-10-12 — the static `[cloud]` key leaked and was deleted). Every AWS caller uses the EB EC2 **instance role as its identity** and **assumes `WorkloadsRuntime`** via STS in the target account. This doc owns the role model; the per-stack helpers are:
+**No long-lived IAM keys in the `App_Cloud` / `_Cloud` / worker2 paths** (TRUE-82705, 2026-10-12 — the static `[cloud]` key leaked and was deleted). Some keys remain elsewhere in code: [audit 2026-10-13](#long-lived-keys-still-in-code-audit-2026-10-13). Every AWS caller uses the EB EC2 **instance role as its identity** and **assumes `WorkloadsRuntime`** via STS in the target account. This doc owns the role model; the per-stack helpers are:
 - worker2 SDK clients: `_Component_Aws_Workloads` (below).
 - 2.0 `_Cloud` (S3/SQS/SSM/CLI): [`_Cloud` credentials](../../_underscore/features/cloud-s3-helpers.md#credentials) — `VipSupport` and `ZoomInfo` use `_Cloud::getCredentialProvider()`.
 - 1.0 `App_Cloud` + deploy scripts: [1.0 AWS credentials](../../../../1.0/apps/library/features/app-cloud-aws-credentials.md).
@@ -64,6 +66,7 @@ IAM is **global**, so **one `WorkloadsRuntime` role per account** covers all reg
 
 **One broad role per account, on purpose** (developer decision 2026-10-12): no per-env roles, no resource scoping.
 - Permissions: `elasticbeanstalk:*`, `ec2:*`, `s3:*`, `sqs:*`, `cloudwatch:*`, `logs:*`, `ssm:*`, `autoscaling:*`, `elasticloadbalancing:*`. The 502 role also has `transfer:*`, `rds:*`, `ses:*`, `guardduty:*`, `codepipeline:*`, `route53:*`, `iam:PassRole` (what the deleted key could do).
+- **Open concern (security review, 2026-10-13):** the breadth (`iam:PassRole` on `*`, `s3:*`, `rds:*`, `ssm:*`, `transfer:*`, `ses:*`, `route53:*`) means any compromised app host gets all of it. Narrowing is not decided; the decision above stands until changed.
 - Trust (502 and 868 roles): **both** instance roles — 868 `ElasticBeanstalk-EC2-Instance-Profile` (all 2.0 prod envs) and 502 `aws-elasticbeanstalk-ec2-role` (shared by ~65 legacy EB envs). Each instance role has inline policy `AssumeWorkloadsRuntime` (`sts:AssumeRole` on `arn:aws:iam::*:role/WorkloadsRuntime`).
 - Default account when the caller names none = **502** (`_Cloud`, `App_Cloud`, deploy scripts). SQS: assume in the account in the queue URL (2.0 `WorkerProductionQueue` is in 868).
 - Verify in CloudTrail: `AssumeRole` on `WorkloadsRuntime`, filter by session name (`worker2-workloads`, `_underscore-cloud`, `library-app-cloud`, …).
@@ -127,7 +130,15 @@ The `agilant-as2` bucket (the Office Depot AS2/EDI drop bucket, `us-west-2`) is
 
 **IAM ARN gotcha (bucket-level vs object-level).** `s3:ListBucket` and `s3:GetBucketLocation` are **bucket-level** → the `Resource` must be the **bucket** ARN `arn:aws:s3:::agilant-as2` with **no** `/*`. `s3:DeleteObject` is **object-level** → the `Resource` must end in `/*` (scope it, e.g. `arn:aws:s3:::agilant-as2/OUTBOX/processed/*`). Putting `ListBucket` on the `/*` ARN is the classic mistake and returns exactly this AccessDenied.
 
-First S3 consumer of this pattern: the [S3 folder retention cleanup action](./s3-folder-retention-cleanup.md).
+First S3 consumer of this pattern: the [S3 folder retention cleanup action](./s3-folder-retention-cleanup.md). Also: [`Monitor/Compass/OfficeDepotInboundFiles`](../../../../clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md) reads CloudWatch `AS2-Inbound` and S3 in 502 via `_Cloud::getCredentialProvider(awsAccountId: '502614707982')`.
+
+### Long-lived keys still in code (audit 2026-10-13)
+Default remote branches, all repos. Locations only; never record a value.
+- **Live, in use:** the SES SMTP key (`ses-smtp-user`, the **only active IAM key left in 502**) in `library/app/email.php` and `_underscore/Email.php` (plus test scripts). Plan: SES API `SendRawEmail` via the role, or SMTP creds in SSM.
+- **Deleted key, still read at deploy / new-instance boot:** s3fs mounts (`toga`, `togaview`, `vision`, `worker` `ebs/` create/mount-s3fs-credentials-file scripts) → `s3fs -o iam_role=auto` plus an S3 policy on the EC2 role (s3fs 1.84 is IMDSv1 only); togaview ALB registration script (alpha/beta).
+- **23 legacy EB envs** still carry the deleted key in environment properties. Remove only **after** the s3fs fix; the worker mount script exits when the property is missing.
+- **Other accounts:** Talos (654) key in `worker2` / `tools` `production.ini` (also in the [committed-secrets register](./committed-secrets-and-config-debt.md)); CloudFront/test (975) key in `api2` `beta.ini`, the api2 ALB script, and the tools CloudFront page → `getCredentialProvider(<account>)` plus trust/permissions in that account.
+- **Dead, delete:** `[email]` SMTP creds in ~178 config ini files (only reader `library/app/DOAemail.php`, no callers); `worker1.5` `[aws]`; unscheduled worker DOA crons; `vision/sendtestemail.php` (web-reachable); unused webhook/tools s3fs scripts.
 
 ## Gotchas
 
@@ -141,6 +152,7 @@ First S3 consumer of this pattern: the [S3 folder retention cleanup action](./s3
 - **`$environments` must be typed `object|array`, not `array`.** The cron dispatcher casts only the top level of `CronJobs.parameters` to an array, so this nested JSON **object** arrives as a PHP `stdClass` and a strict `array` type throws a `TypeError` at dispatch. Declare it `object|array` and `(array)`-cast at the top. General rule for any object-valued cron param — see [creating-worker-actions.md](./creating-worker-actions.md).
 
 ## Change history
+- 2026-10-13 — Added the remaining long-lived keys audit, the role-breadth concern, and the `OfficeDepotInboundFiles` consumer of 502 (bala)
 - 2026-10-12 — Static `[cloud]` key deleted everywhere (TRUE-82705); this doc now owns the role model for 1.0 + 2.0: broad role per account, trusts both instance roles, default account 502, SQS account from queue URL (jcardinal)
 
 ## Related

package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md

@@ -6,7 +6,7 @@ project: Worker
 client: shared
 type: feature
 status: active
-updated: 2026-10-06
+updated: 2026-10-13
 owners: ["jcardinal", "mhammontree", "bala", "rgirish"]
 files:
   - worker2/Worker/Monitor/Compass.php
@@ -36,6 +36,7 @@ related:
   - ../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
   - ../../../../clients/wje/features/oneuptime-monitors.md
   - ../../../clients/compass-canada/features/oneuptime-po-transmissions-to-vendors-monitor.md
+  - ../../../../clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md
 ---
 
 OneUptime push-metric monitor pattern for worker2 (2.0): a self-contained cron action decides pass/fail and POSTs a string token to a OneUptime Incoming Request monitor; open when building/tuning a worker2 monitor, its thresholds, or the OneUptime import JSON.
@@ -46,7 +47,7 @@ A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, p
 `App_SystemMonitor_Compass` monitors. A worker2 cron action runs a self-contained check, decides
 pass/fail itself, and POSTs a JSON metric body to a OneUptime "Incoming Request" monitor — replacing
 the 1.0 tier's email alerts. The class is `abstract class _Worker_Monitor_Compass` and now carries a
-**suite of 13 monitors** covering the Compass USA / Office Depot integration (plus one Compass Canada monitor) (S3, DB, mailbox, and
+**suite of 14 monitors** covering the Compass USA / Office Depot integration (plus one Compass Canada monitor) (S3, DB, mailbox, and
 NetSuite SuiteQL checks). Designed to be **client-agnostic** — a reusable template for future monitors and clients.
 
 Distinct from the DB-driven email-orchestrator [Monitoring Framework](./monitoring-framework.md)
@@ -74,12 +75,18 @@ disallows `const` at function scope), plus a `$push` closure using `_ApiRequest`
 method, so the class scales cleanly as monitors accumulate. The metric POST runs with **logging
 disabled** and **`throwExceptionsOnFailure` disabled**, so a failed ping never fails the worker job.
 
-### The 13 monitors
+### The 14 monitors
 
 Each measures one Compass/Office Depot (or Compass Canada) integration metric and pushes a decided alarm token.
 
 - **OfficeDepotEdiImportQueue** (S3) — counts files >10 min old in `s3://agilant-as2/OfficeDepot/`,
-  excluding `OUTBOX/`, `SENT/` and directory placeholders.
+  excluding `OUTBOX/`, `SENT/` and directory placeholders. **Blind when the AS2 mover stops** (folder
+  stays empty, monitor stays green: 2026-10-10 outage); `OfficeDepotInboundFiles` covers that step.
+- **OfficeDepotInboundFiles** (CloudWatch + S3, added 2026-10-13) — files received on the Office Depot
+  AS2 agreement that were never moved out of `agilant-as2/processed/`, plus business-hours silence.
+  `Core.CronJobs` id 93, `*/5`. First monitor to read CloudWatch Logs and to use
+  `_Cloud::getCredentialProvider(awsAccountId:)` directly. Details:
+  [Compass: Office Depot inbound files monitor](../../../../clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md).
 - **CompassSalesOrderTransmission** (DB) — approved SOs (both approval stages) with
   `c_dtTransmittedToCompass` NULL (not yet transmitted to MITS). Alarms on **either** `pendingCount > 10`
   **OR** `oldestPendingAgeMinutes > 180` (3h), so a single order stuck too long pages even under the
@@ -442,11 +449,11 @@ Testing **recovery** is the step people skip and the most valuable one:
    credential**; never log it and never record its value in a doc.
 3. Provision the OneUptime monitor by importing the monitor's OneUptime resource-export JSON (authored
    under a local staging dir, e.g. `d:\STAGE` — these are import artifacts, not repo files) and wiring
-   **only the body-`Contains` criteria** at this point. **There is no `worker2/monitors/` folder** — the
-   create-monitor skill's instruction to write the import JSON there is wrong; that path does not exist
-   in the repo. The pre-existing in-transit monitor JSON lives outside the repo in a local staging
-   folder. These artifacts are **not** version-controlled, so a monitor's exact criteria live only in
-   OneUptime and in this doc. The **push URL does not exist until the monitor is saved** — OneUptime
+   **only the body-`Contains` criteria** at this point. **Never commit an import JSON to worker2**
+   (developer rule, 2026-10-13: the `OfficeDepotInboundFiles` JSON was removed from PR #192 before
+   merge). There is no `worker2/monitors/` folder; the create-monitor skill's instruction to write the
+   JSON there is wrong. Hand the JSON over outside git. A monitor's exact criteria therefore live only
+   in OneUptime and in its KB doc. The **push URL does not exist until the monitor is saved** — OneUptime
    generates it, it cannot be pre-derived, so the monitor must exist before the worker code can be
    finished.
 4. Add the `Core.CronJobs` row: `action = 'Monitor/<Class>/<Method>'`, schedule `*/5 * * * *`,
@@ -592,6 +599,7 @@ production deploy.**
 - [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
 
 ## Change history
+- 2026-10-13 — Added `OfficeDepotInboundFiles` (cron 93, CloudWatch + S3) after `OfficeDepotEdiImportQueue` stayed green through the 2026-10-10 AS2 outage; import JSONs are never committed to worker2. (bala)
 - 2026-10-05 — Added `CompassCanadaPurchaseOrderTransmissionsToVendors` (cron 87), the first monitor reading `Client_CompassCanada`. (bala)
 - 2026-10-01 — Added the `resetQueryCache()` re-read trap to the in-run retry pattern; WJE link now points at the 3-monitor suite. (mhammontree)
 - 2026-10-01 — Clarified worker2 reads `TOGaDeskSupport` (`DB_TOGADESK1`) — only `Common` is unreachable; added `Monitor/Wje`. (mhammontree)

package/knowledge/INDEX.md

@@ -22,7 +22,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 94 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 85 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 86 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 22 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/compass-usa/INDEX.md

@@ -23,6 +23,7 @@
 | [Compass: ODP / Strategic Systems ASN CSV email import (and the scientific-notation tracking-number loss)](features/odp-strategic-systems-asn-csv-import.md) | 1.0 | The 1.0 worker cron that reads Office Depot / Strategic Systems ASN CSVs from email, why it routes by column count, and why a tracking number can arrive as `8.7 |
 | [Compass: MA refresh-order monitor (Monitor/Compass/MaRefreshOrderExceptions)](features/oneuptime-ma-refresh-order-monitor.md) | 2.0 | worker2 OneUptime push monitor counting Compass MA employee-refresh orders whose ODP SO has no NetSuite order yet; open when working on Compass MA monitoring or |
 | [Compass: NetSuite duplicate sales-order monitor (Monitor/Compass/NetsuiteDuplicateSalesOrders)](features/oneuptime-netsuite-duplicate-sales-orders-monitor.md) | 2.0 | worker2 OneUptime push monitor that queries NetSuite (SuiteQL) for Office Depot POs with more than one open NetSuite sales order; open when working on Compass d |
+| [Compass: Office Depot inbound AS2 files monitor (Monitor/Compass/OfficeDepotInboundFiles)](features/oneuptime-office-depot-inbound-files-monitor.md) | 2.0 | worker2 OneUptime push monitor that checks Office Depot 850/997 files are arriving over AS2 and are moved out of `agilant-as2/processed/` to the import folder; |
 | [Compass USA — Fulfilled vs Partially Fulfilled (order total kept; per-line rule built then rejected)](features/order-fulfillment-status-per-line.md) | 2.0 | Why Compass USA shows Fulfilled on partly-shipped orders and why the per-line fix was built, measured, then reverted; open before touching Compass USA `_status` |
 | [People File Import admin page (tools /compass/people-file-import → worker2)](features/people-file-import-admin-page.md) | 1.0 | Browser page that lists the Compass PEOPLE files on SFTP and runs the importer on demand; open before changing the page or the importer's listing/guard code. |
 | [Compass PEOPLE-File User Lifecycle (duplicate accounts, reactivation grace window, raw-SQL deactivation)](features/people-file-user-lifecycle.md) | 2.0 | The nightly Compass PEOPLE-file cron's Users-row lifecycle (insert/reactivate/deactivate), why it creates duplicate accounts, and its raw-SQL/index traps; open |
@@ -41,7 +42,7 @@
 | [Granting a Compass user the same toga25-supply navigation as another user (role vs. SurfaceOverride)](workflows/granting-navigation-access.md) | 2.0 | How to give a Compass user another user's toga25-supply menu items via role-scoped SurfaceOverrides (not ACL); open for "same nav as user Y" and "user sees a bl |
 | [Granting a Compass user access to a bundle (persona grant) and the SuperUser role](workflows/granting-persona-bundle-access.md) | 2.0 | The five-table procedure to give a Compass user sight of a kit (persona + bundle + items + assortments + user) plus the SuperUser role; open for any "give user |
 | [Compass Office Depot Duplicate PO-Line Cleanup (dual-catalog SKU)](workflows/odp-duplicate-po-line-cleanup.md) | 1.0 | Why the NetSuite fulfillment-SO sync duplicated Office Depot PO lines on a dual-catalog SKU, the forward fix, and the one-time cleanup migration; open when inve |
-| [Recovering a Lost Compass ODP EDI 850 Import (re-drop from Logs.FileLog)](workflows/odp-edi-import-recovery.md) | 1.0 | Runbook to recover a Compass ODP EDI 850 import that cron 3a lost by deleting the S3 file after a failure; open when an ODP order never reached NetSuite and the |
+| [Recovering Lost Compass ODP EDI 850 Imports (re-drop from Logs.FileLog, move stuck AS2 files, manual SQL import)](workflows/odp-edi-import-recovery.md) | 1.0 | Runbook to recover Compass ODP EDI 850 imports: re-drop an x12 that cron 3a deleted after a failure, move AS2 files stuck in `processed/` when nothing reaches 3 |
 | [Compass ODP Order Pipeline to NetSuite (numbered worker crons)](workflows/odp-order-pipeline-to-netsuite.md) | 1.0 | The end-to-end Compass ODP order → NetSuite pipeline through the numbered 1.0 worker crons, plus the "this order never reached NetSuite" diagnostic method; open |
 | [Compass Order Lifecycle & Data-Integrity Invariants](workflows/order-lifecycle-and-data-integrity.md) | 2.0 | The expected raw-data shape of a Compass order at every link/ASN/IF level, plus detect-and-repair sweeps; open when diagnosing or retro-fixing Compass order dat |
 | [Comparing Compass Persona/Catalogue Effects across prod and dev-sandbox](workflows/persona-population-env-comparison.md) | 2.0 | Full-population diff technique to prove what a Compass persona/catalogue change does across prod vs dev-sandbox; open before validating any Compass visibility c |

package/knowledge/clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md

@@ -0,0 +1,58 @@
+---
+title: "Compass: Office Depot inbound AS2 files monitor (Monitor/Compass/OfficeDepotInboundFiles)"
+framework: "2.0"
+repo: worker2
+project: Worker
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-10-13
+owners: ["bala"]
+files:
+  - worker2/Worker/Monitor/Compass.php
+  - _underscore/Cloud.php
+related:
+  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
+  - ../workflows/odp-order-pipeline-to-netsuite.md
+  - ../workflows/odp-edi-import-recovery.md
+  - ../../../2.0/apps/worker2/features/cross-account-aws-access.md
+  - ../profile.md
+---
+worker2 OneUptime push monitor that checks Office Depot 850/997 files are arriving over AS2 and are moved out of `agilant-as2/processed/` to the import folder; open when it alarms, when tuning it, or when an ODP order never reached cron 3a.
+
+## Summary
+`Monitor/Compass/OfficeDepotInboundFiles` (worker2 `_Worker_Monitor_Compass`, TRUE-82728, PR #192, on `_production` and deployed 2026-10-13). `Core.CronJobs` **id 93**, `*/5 * * * *`. Status-page name **"Incoming Office Depot Vendor POs (Received and Moved to Import Folder)"**. Follows the shared [OneUptime push-metric pattern](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md); the push URL is an in-method `$ONEUPTIME_URL` local and is a credential (not recorded here).
+
+Why it exists: `OfficeDepotEdiImportQueue` only counts aged files in `OfficeDepot/`. In the 2026-10-10 outage the Lambda that fills `OfficeDepot/` stopped, so that folder stayed **empty** and the monitor stayed **green** while 52 POs piled up in `processed/`. Lesson: monitor the **input** step, not only the queue the next step reads ([pattern rule](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#watch-the-step-before-the-log-row-exists)).
+
+## How it works
+AWS account toga-legacy `502614707982`, `us-west-2`, credentials from `_Cloud::getCredentialProvider(awsAccountId: '502614707982')`: worker2 runs in 868 and assumes `WorkloadsRuntime` in 502 ([role model](../../../2.0/apps/worker2/features/cross-account-aws-access.md#agilant-as2-is-in-the-legacy-account-502614707982--s3-work-on-it-is-cross-account)).
+1. **Received:** CloudWatch Logs `filterLogEvents` on log group `AS2-Inbound`, stream `a-904effd552db4348b` (the OfficeDepotProd agreement), pattern `{ $.internalState = "PROCESSED" }`, last **72 h** (spans a weekend). Each event's `filePath` is the S3 key in `agilant-as2` (`processed/<uuid>.<msgid>…uschecom.net`).
+2. **Moved:** takes the **20 newest** files older than 15 min and runs S3 `doesObjectExistV2` on each. Still in `processed/` = never moved (the Lambda / stop-gap cron moves it out).
+3. **Silence:** only in business hours (weekdays 08:00-17:00 `America/Chicago`), minutes since the later of the last file or 08:00 today.
+
+| Signal | DEGRADED | HIGH |
+|---|---|---|
+| a checked file still in `processed/` | >= 15 min old | oldest >= 30 min |
+| business-hours silence | >= 150 min | >= 180 min |
+
+Calibration: the mover runs every 5 min and a normal move finishes in under 8 min. Worst in-hours weekday gap measured Sep 1 to Oct 13 2026 was **144 min**; about **156 files per weekday**, weekends 2-5.
+
+Payload: `status`, `alarm` (`OK`/`DEGRADED`/`HIGH`), `reason`, `minutesSinceLastFile`, `filesReceivedLast24Hours`, `stuckFileCount`, `oldestStuckMinutes`, `checkedFileCount`, `businessSilentMinutes`, `isBusinessHours`, the four thresholds, `checkedAtUtc`.
+
+First live run 2026-10-13: last file 20 min ago, 214 in 24 h, 0 of 20 stuck, `alarm: OK`.
+
+## Gotchas
+- **⚠ There is no `status: "error"` push.** No try/catch (developer standard), so an AWS failure (role, CloudWatch, S3) throws, nothing is pushed, and OneUptime's **"not received in 15 min"** criterion turns it red. A red monitor with no recent body = the checker died, not a backlog; read the worker2 `WorkerJobs` row for the exception.
+- **Not covered:** weekend and holiday silence never alarms; only the 20 newest files are checked, so an old file stuck behind 20 good ones is missed.
+- Reads the **prod** agreement stream only. The OfficeDepotTest agreement also writes to `/agilant-as2` and is ignored.
+- The OneUptime import JSON is **not** in git. It was removed from the PR before merge; import JSONs are handed over outside the repo ([runbook](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#provisioning-a-monitor-runbook)).
+
+## Change history
+- 2026-10-13 — Created: CloudWatch AS2-Inbound + S3 existence check, business-hours silence alarm; added after the 2026-10-10 Lambda outage left `OfficeDepotEdiImportQueue` green (TRUE-82728). (bala)
+
+## Related
+- [OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md)
+- [Compass ODP order pipeline: inbound AS2 path](../workflows/odp-order-pipeline-to-netsuite.md#inbound-as2-path--how-an-850-reaches-officedepot-verified-2026-10-12)
+- [Recovering stuck ODP AS2 files](../workflows/odp-edi-import-recovery.md)
+- [Cross-account AWS access (WorkloadsRuntime)](../../../2.0/apps/worker2/features/cross-account-aws-access.md)

package/knowledge/clients/compass-usa/workflows/odp-edi-import-recovery.md

@@ -1,16 +1,17 @@
 ---
-title: Recovering a Lost Compass ODP EDI 850 Import (re-drop from Logs.FileLog)
+title: Recovering Lost Compass ODP EDI 850 Imports (re-drop from Logs.FileLog, move stuck AS2 files, manual SQL import)
 framework: "1.0"
 repo: worker
 project: Worker
 client: compass-usa
 type: workflow
 status: active
-updated: 2026-09-16
+updated: 2026-10-13
 owners: ["dfranks", "bala"]
 files:
   - worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php
   - worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php
+  - worker/crons/toga2/compass/workflow/move_office_depot_as2_files_to_import_folder.php
   - worker/schedules/cron.worker.sync.json
 related:
   - clients/compass-usa/features/odp-edi-file-retention-and-retry.md
@@ -19,8 +20,9 @@ related:
   - ../../../2.0/apps/api2/features/api-payload-interceptors.md
   - ../../../2.0/apps/api2/features/request-logging.md
   - clients/compass-usa/profile.md
+  - clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md
 ---
-Runbook to recover a Compass ODP EDI 850 import that cron 3a lost by deleting the S3 file after a failure; open when an ODP order never reached NetSuite and there is a zero-item or missing ODP SO.
+Runbook to recover Compass ODP EDI 850 imports: re-drop an x12 that cron 3a deleted after a failure, move AS2 files stuck in `processed/` when nothing reaches 3a, or build the order by SQL when 3a cannot run; open when ODP orders are missing in Supply or never reached NetSuite.
 
 ## Summary
 How to recover a Compass **Office Depot EDI 850** import that failed partway — the case where cron **3a** created the ODP SalesOrder header, the follow-up item write died (e.g. an api2 fatal), and 3a then **deleted the S3 object**, leaving a zero-item order and nothing to retry. The recovery works because the raw x12 is archived in `Logs.FileLog` and **3a is fully idempotent**: re-uploading the file makes it repair the items, the ODP PO, and both bridges. Verified end-to-end in **production** on 2026-08-04 for ODP PO `41608261-1135` / ODP SO `475390815001`.
@@ -38,17 +40,19 @@ The diagnostic shape that identifies a destroyed import (total-loss case, no ord
 
 That combination means the 850 arrived, was archived, and then died — the file is already gone from S3 and the file-log copy is the only one left.
 
+**Different case — no ODP POs arriving at all** (nothing reaches 3a, so there is nothing in `Logs.FileLog` either): see [AS2 files stuck in `processed/`](#other-case--as2-files-stuck-in-processed-3as-inbox-is-empty).
+
 ## Steps
 
 ### Step 1 — retrieve the archived x12
 The raw EDI is on the **legacy-core cluster**, table **`Logs.FileLog`**:
 - `job = 'ODP_EDI'`
-- `fileName = '<odpPoNumber>.x12'` — note **`.x12`**, even though the inbound S3 filename is `EDI_<odpPoNumber>.txt`.
+- `fileName = '<odpPoNumber>.x12'` — note **`.x12`**, even though the live inbound S3 filename is the AS2 name `<uuid>.<msgid>.net`.
 
 **Query it bounded by `job` + `fileTimestamp`.** An unbounded `fileName LIKE` scan times out on this table. Extract with `mysql --raw --batch --skip-column-names`, then **strip the trailing newline the client appends** so the byte count matches the stored `LENGTH(fileData)` exactly. A byte-count mismatch means a corrupted payload — do not upload it.
 
 ### Step 2 — know that S3 has nothing left
-3a **deletes the S3 object after processing**, so a failed import leaves no file to retry. (After the conditional-delete change ships this is only true of a *successful* import.) Recovery is a **re-upload**, to the **root** of the prefix:
+3a **deletes the S3 object after processing**, so a failed import leaves no file to retry. (After the conditional-delete change ships this is only true of a *successful* import.) Recovery is a **re-upload**, to the **root** of the prefix (3a reads any file name there):
 ```
 s3://agilant-as2/OfficeDepot/EDI_<odpPoNumber>.txt
 ```
@@ -63,15 +67,56 @@ s3://agilant-as2/OfficeDepot/EDI_<odpPoNumber>.txt
 ### Step 5 — let cron 5 transmit to NetSuite
 Cron 5 (`*/10`) picks the now-complete ODP SO up and stamps `c_dtTransmittedToNetsuite` + `c_netsuiteInternalSalesOrderId`. It excludes `number LIKE 'MA%'`. Its stamps landing is the confirmation the recovery finished.
 
+### Other case — AS2 files stuck in `processed/` (3a's inbox is empty)
+The files reach S3 but never get moved to `OfficeDepot/`, so 3a has nothing to import. Path and component ids: [Inbound AS2 path](odp-order-pipeline-to-netsuite.md#inbound-as2-path--how-an-850-reaches-officedepot-verified-2026-10-12). Production example 2026-10-10 → 10-12: 52 POs stuck.
+
+**Shape:**
+- `Logs.FileLog` (`job = 'ODP_EDI'`) shows **zero 850s AND zero 997s** for the period. 997s arrive all day, so zero of both means nothing reaches 3a.
+- 3a's rows in legacy `Common.CronJobExecutions` show short runs (~17 s) = it ran on an empty inbox.
+
+**Alarm:** [`Monitor/Compass/OfficeDepotInboundFiles`](../features/oneuptime-office-depot-inbound-files-monitor.md) goes DEGRADED/HIGH when a received file is still in `processed/` after 15/30 min.
+
+**Check, in order** (account toga-legacy, `us-west-2`):
+1. `s3://agilant-as2/processed/` — recent `*.net` files piling up = the move step stopped.
+2. CloudWatch `/aws/lambda/rename_edi_files` — a healthy move logs `Renamed object from ... to ...`. Also the Lambda `Invocations` metric.
+3. The AS2 inbound log group (`AS2-Inbound…`) — confirms ODP is still sending.
+4. If re-saving the S3 event notification fails with `Unable to validate the following destination configurations`, S3 can no longer invoke the Lambda (Lambda suspended or its permission gone).
+
+**Recover:** move the stuck `processed/*.net` files, **filtered by date window**, to the root of `OfficeDepot/` with the same name. Never move the 2024 test files in `processed/`. 3a skips POs that already exist, so replaying a file that was already imported is safe. The stop-gap cron `move_office_depot_as2_files_to_import_folder.php` does this every 5 minutes (live since 2026-10-12; the Lambda is still suspended as of 2026-10-13); turn it off once AWS restores the Lambda.
+
+### Last resort — manual import by SQL (3a cannot run)
+Prefer re-dropping the file (Steps 1-4): 3a also writes the `Logs_Compass` `Api` / `Record` / `RecordField` audit rows and an SQS event, which plain SQL does not. If the order must exist now, write exactly what 3a's api2 calls write for one SA/MR 850.
+
+3a calls api2 with the **"Agilant"** key (`Client_Compass.Apis` id 1). The Compass `PurchaseOrder` prePost/postPost hooks are registered **only for apiId 2** (MITS Service Hub), so they do **not** run for 3a. Only the `SalesOrders` postPost runs.
+
+| Table | What 3a writes |
+|---|---|
+| `Addresses` x2 | Separate ship-to for the SO and for the PO. CREATE policy (never reuse); `stateId` by `States.code`; zip 9 digits, no dash; addressee = N1 name; attention = N2. |
+| `SalesOrders` | number = `REF~QC`; `customerId` 1, `urgencyId` 2, `shippingMethodId` 9 for TD5 `ND` / 8 for `ST`; shipping 0, tax 0, `isQuote` 0. |
+| `SalesOrderItems` | `_status` NEW, `isOpen` 1. |
+| `Approvals` | One row from the SO postPost: `approvalTemplateId` 1, `recordId` 14, `primaryKeyId` = ODP SO id. No `ApprovalDecisions`, no email. |
+| `PurchaseOrders` | number = `BEG`; `customerId` 1, `vendorId` 1; `_total` = SUM(qty x cost); `c_mitsSalesOrder` = `REF~QC`; `c_refQc` = `REF~CO`. |
+| `PurchaseOrderItems` | `vendorItemId` = lowest `VendorItems.id` for `vendorId` 1 + the line's `vendorPartNumber`. |
+| Links | `SalesOrders_PurchaseOrders` (ODP SO → ODP PO), `PurchaseOrders_SalesOrders` (Compass PO → ODP SO), `SalesOrderItems_PurchaseOrderItems` (ODP SOI → ODP POI), `PurchaseOrderItems_SalesOrderItems` (Compass POI → ODP SOI). |
+
+After the SQL import:
+- If the file later reaches 3a, 3a sees the PO number exists and **silently skips** it (no email, no reject 855), logs the file and deletes it.
+- Cron 4 still sends the accept 855, because `dtAcknowledged` stays NULL.
+- Cron 5 picks up the ODP SO as normal. A later **extra** `PurchaseOrders_SalesOrders` link to a `customerId` 3 SO comes from the NetSuite sync after cron 5, not from 3a. Expected, not a SQL mistake.
+
 ## Gotchas
 - **⚠ `Logs_Compass.Api` is NOT where these failures show up.** The failing api2 calls 3a makes (`GET /items`, `GET /vendor-items`, `GET /purchase-orders`) were **absent** from the per-client `Logs_Compass.Api` log in the 2026-08-27 incident — only the surrounding successful `POST /v2/auth/api` and `GET /v2/clients` calls appear. Do not read an empty `Logs_Compass.Api` as "3a never ran." The reliable trail is **`Logs.Emails`** (subject `Office Depot PO Import error: <PO>`) plus **`Logs.FileLog`** (`job = 'ODP_EDI'`).
 - **⚠ Re-uploading before fixing the root cause destroys the file again** (3a deletes on the way through). Step 3 is not optional. This is exactly what the conditional-delete change removes — until it deploys, assume one attempt per recovered file.
 - **⚠ A recovered order waits another 30 minutes.** Cron 5's release timer runs from the Office Depot PO's `dtCreated`, i.e. **when we imported the 850** — so re-importing restarts it. See the release gate in the [ODP pipeline doc](odp-order-pipeline-to-netsuite.md).
-- **⚠ The archive filename extension differs from the S3 one** — `.x12` in `Logs.FileLog`, `EDI_<po>.txt` in S3. Searching for the wrong one finds nothing.
+- **⚠ The archive filename differs from the S3 one** — `<po>.x12` in `Logs.FileLog`, `<uuid>.<msgid>.net` in S3. Searching for the wrong one finds nothing.
 - **Don't search for the ODP SalesOrder by the `BEG` number.** `BEG` is the ODP *PO*; the SO number is `REF~QC`. Full segment map in the [ODP pipeline doc](odp-order-pipeline-to-netsuite.md).
 - **`Logs.FileLog` has limited retention** (~30 days) — beyond that there is no archived x12 and the recovery must come from ODP re-sending.
 - A **fully** rolled-back import leaves no row anywhere and no API log row, so it will not show up in an orphan sweep at all; only the error tracker's per-event detail can enumerate those.
 
+## Change history
+- 2026-10-13 — Added the manual-SQL-import section (exact rows 3a's api2 calls write, hooks that do not run for apiId 1); linked the inbound files monitor; Lambda still suspended, stop-gap cron live. (bala)
+- 2026-10-12 — Added the "AS2 files stuck in `processed/`" case (Lambda `rename_edi_files` stopped 2026-10-10, 52 POs); corrected the inbound S3 filename to `<uuid>.<msgid>.net`. (bala)
+
 ## Related
 - [ODP EDI File Retention & Import Retry](clients/compass-usa/features/odp-edi-file-retention-and-retry.md)
 - [ODP order pipeline to NetSuite](clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md)

package/knowledge/clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md

@@ -6,7 +6,7 @@ project: Worker
 client: compass-usa
 type: workflow
 status: active
-updated: 2026-10-12
+updated: 2026-10-13
 owners: ["rgirish", "bala", "dfranks", "jcardinal", "mhammontree"]
 files:
   - worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
@@ -33,6 +33,8 @@ related:
   - clients/compass-usa/profile.md
   - clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
   - 1.0/apps/library/features/netsuite-soap-toolkit-search.md
+  - 1.0/apps/library/features/app-cloud-aws-credentials.md
+  - clients/compass-usa/features/oneuptime-office-depot-inbound-files-monitor.md
 ---
 The end-to-end Compass ODP order → NetSuite pipeline through the numbered 1.0 worker crons, plus the "this order never reached NetSuite" diagnostic method; open for any ODP-to-NetSuite tracing or cron-mechanics question.
 
@@ -45,7 +47,8 @@ Commerce → Compass SalesOrder            (SalesOrders.customerId = 2)
   [cron 1_transmit_compass_sales_orders_to_mits]  → MITS (API)
 MITS → PurchaseOrder                      (PurchaseOrders.vendorId = 1 = OFFICE DEPOT; POST /v2/purchase-orders; PO number e.g. 50305906-1)
   [cron 2_transmit_mits_purchase_orders_to_vendors] → ODP (cXML; email fallback per integration)   [sets PurchaseOrders.dtSubmitted]
-ODP → 850 EDI dropped to S3 (AS2)         (bucket agilant-as2, prefix OfficeDepot/)
+ODP → 850 over AS2 → Transfer Family     (s3://agilant-as2/processed/)
+  [S3 event → Lambda rename_edi_files]    → moves *.net to s3://agilant-as2/OfficeDepot/   (see "Inbound AS2 path")
   [cron workflow/3a_import_office_depot_purchase_orders  — NOT edi/1, see Gotchas]
                                                 → creates ODP SalesOrder (customerId = 1); deletes the S3 file after processing
 ODP SalesOrder (customerId = 1)
@@ -69,6 +72,20 @@ ODP SalesOrder (customerId = 1)
 
 The **live** Office Depot EDI flow is entirely under `worker/crons/toga2/compass/workflow/`: **3a** import (every 5 min), **4** transmit-PO-acknowledgements (every 15 min), **5** create-NetSuite-SO, **6** generate-ASN.
 
+### Inbound AS2 path — how an 850 reaches `OfficeDepot/` (verified 2026-10-12)
+AWS account toga-legacy (`502614707982`), `us-west-2`:
+1. ODP sends 850s and 997s over **AS2** to Transfer Family server `s-54dad64b145d417a9` (our AS2 id `AGILANT-B2B-EDI`), agreement `a-904effd552db4348b` "OfficeDepotProd", BaseDirectory `/agilant-as2`.
+2. Files land at `s3://agilant-as2/processed/<uuid>.<msgid>.net`, plus `.json` and `.mdn` receipt files.
+3. An S3 event notification on `agilant-as2` (prefix `processed/`, Put + Post) invokes Lambda **`rename_edi_files`** (python3.12). It moves only `*.net` files to `s3://agilant-as2/OfficeDepot/<same name>` and leaves `.json` / `.mdn` behind.
+4. Cron 3a (`*/5`) reads `OfficeDepot/`, imports, logs to legacy `Logs.FileLog` (`job = 'ODP_EDI'`), deletes the file.
+- **The Lambda source is in no repo** (built in the console). Read it there.
+- **Not in the live path:** Transfer Family workflow `w-f8951fd4252a5b890` "OfficeDepot" (a COPY step to `OfficeDepot/`) is attached to the server OnUpload but has **never run** (no executions in a year of server logs). Do not debug it.
+- Agreement `a-f1a87e1edc5c442fa` "OfficeDepotTest" also writes to `/agilant-as2`. Canon AS2 uses a separate bucket, `agilant-canon-as2`.
+- `processed/` also holds old **2024 test `.net` files**. Never move them; any bulk move must filter by date.
+- **Stop-gap cron (2026-10-12):** `workflow/move_office_depot_as2_files_to_import_folder.php` (`*/5`, scheduled right after 3a) does the Lambda's job: moves `*.net` from `processed/` to `OfficeDepot/`, only files modified after `2026-10-10 12:30 UTC` and older than 120 s, and throws if the S3 list returns `null`. Committed on `worker` `_production` (`b3f57866`) and live; it authenticates through the `WorkloadsRuntime` role ([AWS credentials](../../../1.0/apps/library/features/app-cloud-aws-credentials.md)), first clean run 2026-10-12 17:00 CDT after the library deploy. **The Lambda is still suspended by AWS as of 2026-10-13** (last log 2026-10-10 12:29 UTC). **Turn the cron off (`"active": 0`) once AWS restores the Lambda.**
+- **Monitored by** [`Monitor/Compass/OfficeDepotInboundFiles`](../features/oneuptime-office-depot-inbound-files-monitor.md) (cron 93): alarms when a received file is not moved out of `processed/`, or on business-hours silence. `OfficeDepotEdiImportQueue` cannot see this step.
+- Diagnosing and recovering stuck files: [Recovering a lost ODP EDI import](odp-edi-import-recovery.md#other-case--as2-files-stuck-in-processed-3as-inbox-is-empty).
+
 ### 3a is idempotent — re-dropping an EDI file is safe
 Every step of 3a is gated by a `totalRecordCount == 0` existence check, so a re-drop produces **no duplicates and backfills whatever is missing**:
 | Step | Guard | Behavior on re-run |
@@ -148,8 +165,7 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
 - **To confirm a PO actually reached ODP:** check `PurchaseOrders.dtSubmitted` is set **AND** that downstream ODP SO(s) exist via the `PurchaseOrders_SalesOrders` join — their existence proves ODP received the PO and 850'd back.
 
 ## Gotchas
-
-- **⚠ AS2 files reach `OfficeDepot/` only via Lambda `rename_edi_files`.** AS2 drops land in `agilant-as2/processed/`; the Lambda moves the EDI file to `OfficeDepot/` for 3a. AWS suspended the Lambda after the 2026-10 IAM key leak (not firing since 2026-10-10) → ODP POs stopped importing. **Stop-gap:** cron `worker/crons/toga2/compass/workflow/move_office_depot_as2_files_to_import_folder.php` (`*/5`) does the same move (only `.net` files, newer than 2026-10-10 12:30 UTC, older than 120 s). **Disable it once the Lambda is restored.** Symptom of a dead mover: files pile up in `processed/`, `OfficeDepot/` stays empty, no new ODP SalesOrders.
+- **⚠ 2026-10-10 → 10-12: ODP POs stopped arriving because the Lambda stopped, not 3a.** An attacker used a leaked long-lived IAM key; AWS suspended services including Lambda, so `rename_edi_files` stopped at **07:29 CDT Sat 2026-10-10** and **52 POs** piled up in `processed/`. 67 `.net` files (850s + 997s) were moved by hand on 2026-10-12; 3a created all 52 POs (0 rejects) and cron 5 sent them to NetSuite. The leaked key was deleted the same day; 3a and the move cron now run on the `WorkloadsRuntime` role ([AWS credentials](../../../1.0/apps/library/features/app-cloud-aws-credentials.md)). All stuck POs imported. The `OfficeDepot/` backlog monitor stayed green throughout (its folder was empty); the [inbound files monitor](../features/oneuptime-office-depot-inbound-files-monitor.md) now covers that gap. Symptom of a dead mover: files pile up in `processed/`, `OfficeDepot/` stays empty, no new ODP SalesOrders.
 - **⚠ `edi/1_download_edi_s3_create_po_toga.php` is DEAD CODE (correction 2026-08-04).** It appears in **no** schedule file. The scheduled entry *named* "Download EDI From S3 Files & Create PO - EDI Office Depot" (`*/5 * * * *`, `worker/schedules/cron.worker.sync.json` `:254-259`) actually points at **`toga2/compass/workflow/3a_import_office_depot_purchase_orders.php`**. The two scripts have **different idempotency and different API call shapes** — `edi/1`'s create branch POSTs the SO with *nested* items and a `customer.name`, while **3a POSTs the header alone (uuids) and then POSTs the items in a separate request**. Reasoning from `edi/1` gives wrong answers. **General rule: never infer which script runs from its filename or folder** — verify against `worker/schedules/cron.*.json`, and note the schedule entry's **name can be actively misleading**. The folders `compass/edi/`, `compass/DOA_edi_odp_flow/`, and `compass/DOA_Scripts/` are **legacy / inactive**; `compass/edi/3_send_edi_855(reject).php` is an abandoned sketch using **non-standard** reject codes (`BAK01=A1` / `BAK02=RE`) — **do not** follow it.
 - **⚠ The `OfficeDepot/` listing makes 3a runs overlap (NOT fixed).** An instrumented prod run 2026-08-25 showed the S3 listing returning **138,472 objects** and taking **32 seconds** before any PO was processed, each PO then ~40 seconds. A file-bearing run blows past the `*/5` schedule and the following runs are **silently skipped** by `App_Framework::exitIfProcessRunning()` — leaving **no `CronJobExecutions` row at all**, which reads as "the cron never fired". The object count is driven by `OfficeDepot/SENT/` and `OfficeDepot/OUTBOX/` accumulating: 3a skips those prefixes when *processing* but still *lists* them. Fix candidates (separate work): prune `SENT/`, or list with a tighter prefix. See [Tracing a worker cron run in production](../../1.0/apps/worker/workflows/tracing-a-worker-cron-run-in-production.md).
 - **⚠ `REF~QC` — not `BEG` — is the value to search on for the ODP SalesOrder.** Searching `PurchaseOrders` / `SalesOrders` for the `BEG` number finds nothing for the SO and wastes time.
@@ -181,7 +197,9 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
 - **Editing cron recipient lists — a grep hit is not a live hit.** Email recipients are hardcoded in the cron scripts, and **several dormant copies of the same script exist** under `crons/toga2/compass/edi/` and `crons/toga2/compass/DOA_edi_odp_flow/`. Before changing a recipient list, cross-check the file against `worker/schedules/*.json` for a referencing entry whose `active` is not `0`. Only the scheduled copies matter.
 
 ## Change history
+- 2026-10-13 — Stop-gap move cron live on the role (Lambda still suspended); all stuck POs imported; linked the new `OfficeDepotInboundFiles` monitor. (bala)
 - 2026-10-12 — Lambda `rename_edi_files` suspended after the IAM key leak; documented the stop-gap mover cron (TRUE-82705) (jcardinal)
+- 2026-10-12 — Documented the inbound AS2 path (Transfer Family → `processed/` → Lambda `rename_edi_files` → `OfficeDepot/`), the 2026-10-10 Lambda outage (52 POs stuck), and the stop-gap move cron `b3f57866`. (bala)
 - 2026-10-08 — Cron 5 duplicate-proof send (`fd426d9b`): externalId `toga2-compass-odso-<ODP SO id>` + lookup before/after `add()`, PO fallback for older orders; removed the shutdown alert (false alarm on SA138624) and the try/catch around `add()`; chose add+lookup over upsert (CTO). Added no-beta-testing and ops-email gotchas. (bala)
 - 2026-09-25 — Corrected the fix description (forked stamp via `queryForked`, no `keepAlive`; stamp unconfirmed); widened duplicate scope to 22 POs Sep 15–22; flagged 9 all-Billed POs as open; recorded monitor-only decision and the `NetsuiteDuplicateSalesOrders` backstop. (rgirish)
 - 2026-09-22 — Root-caused the duplicate NetSuite SOs (15 across 11 POs, Sep 21-22) as the dead-MySQL-connection stamp race, not a Toga duplicate; fixed cron 5 with a forked stamp + a `register_shutdown_function` alert (`worker` `afce32ca`). Recorded that the Sep 21 try/catch is dead code and that `fe4d110d` re-enabling the cron is what surfaced the latent bug. (rgirish)

package/knowledge/clients/prudential/features/order-email-pdf-attachments.md

@@ -6,7 +6,7 @@ project: Worker
 client: prudential
 type: client-feature
 status: active
-updated: 2026-10-05
+updated: 2026-10-12
 owners: ["bala"]
 files:
   - worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php
@@ -58,16 +58,16 @@ Schedules in `worker/schedules/cron.worker.sync.json`. All `active: 1` except `t
 All three live under `https://external-client-files.s3.us-west-2.amazonaws.com/prudential/`. Two are **renamed on attach**:
 | S3 object | Attached to the email as | Size |
 |---|---|---|
-| `EquipMe-Win11-Sep_2026.pdf` (since 2026-10-05; replaced `EquipMe-Win11-v1.pdf`) | `Getting_Started_Guide_Windows11.pdf` | 2,735,276 B |
+| `EquipMe-Win11-Sep_2026_1.1.pdf` (since 2026-10-12; history: `v1` → `Sep_2026` → `Sep_2026_1.1`) | `Getting_Started_Guide_Windows11.pdf` | 2,762,485 B |
 | `2026_Aug_PrudentialMacSetupGuide_ZCC.pdf` | `PrudentialMacSetupGuide.pdf` | 7,543,154 B |
 | `Prudential_Instruction-2Page-V1.pdf` | (unchanged) | 100,224 B |
 
 A new-hire email (Win11 + Mac) is roughly **14 MB** once base64-encoded. Grepping for the attached-as filename will not find the S3 object, search the S3 basename.
 
-**The attached name never changes, so it does not tell you the version.** The email shows `Getting_Started_Guide_Windows11.pdf` and the cron caches to `$cacheFolder/Getting_Started_Guide_Windows11.pdf` whatever URL was fetched. Check the byte size instead (Sep 2026 = 2,735,276; old v1 = 773,591). The old `EquipMe-Win11-v1.pdf` and the S3 object `Getting_Started_Guide_Windows11.pdf` are byte-identical (both the old guide).
+**The attached name never changes, so it does not tell you the version.** The email shows `Getting_Started_Guide_Windows11.pdf` and the cron caches to `$cacheFolder/Getting_Started_Guide_Windows11.pdf` whatever URL was fetched. Check the byte size instead (Sep_2026_1.1 = 2,762,485; Sep_2026 = 2,735,276; v1 = 773,591). The old `EquipMe-Win11-v1.pdf` and the S3 object `Getting_Started_Guide_Windows11.pdf` are byte-identical (both the old guide).
 
 ### Swapping the Windows 11 guide: every hardcoded spot
-There is **no setting or table** for the guide URL. It is a literal in each file below, and all must change together:
+Recurring task: Prudential sends a new EquipMe Windows 11 PDF from time to time (content or phone-number fixes). There is **no setting or table** for the guide URL. It is a literal in **12 `$pdfUrl2` lines across 5 worker files plus 2 store links**, and all must change together:
 | File | Where |
 |---|---|
 | `worker/crons/toga2/prudential/generate_sales_and_purchase_orders_from_service_requests.php` | `$pdfUrl2` in the order-created branches New (1), Refresh (3), and `else` (Breakfix/Swap/Label). Reclaim (5) attaches only `Prudential_Instruction-2Page-V1.pdf`. |
@@ -77,7 +77,14 @@ There is **no setting or table** for the guide URL. It is a literal in each file
 | `worker/crons/toga2/prudential_beta/generate_sales_and_purchase_orders_from_service_requests.php` | Beta copy of order-created. |
 | `store/common/prudential/home.php` (~L244-245) | Equip Me home page "Document" announcement: `$link` ("Click here") and `$win10Link` ("click here") both point at the guide. The Mac link (`$macLink`) is separate. |
 
-Last swap 2026-10-05 (to the Sep 2026 build guide): New, Refresh, Breakfix/Swap/Label emails and the Equip Me page; Reclaim stays excluded.
+Procedure:
+1. Get the new PDF onto `s3://external-client-files/prudential/` under a **new** basename (never overwrite: the old name is what is live until deploy).
+2. `curl -sI <new url>` must return **200**. A new upload can be private: on 2026-10-12 `EquipMe-Win11-Sep_2026_1.1.pdf` returned **403** while the old file returned 200. Fix the object ACL/policy before deploying (no AWS writes from Claude).
+3. Branch from `_production` in **both** `worker` and `store`.
+4. One global find-replace of the old basename in `worker/crons/toga2/prudential*` and `store/common/prudential/home.php` covers every spot. `grep` the old name after: expect 0 hits; new name: 12 in worker, 2 in store.
+5. Leave the Mac guide (`$pdfUrl4`, `2026_Aug_PrudentialMacSetupGuide_ZCC.pdf`) alone unless it is the one being swapped. Reclaim stays excluded.
+
+Why step 2 matters: the crons fetch through `safeFetchPdf()`, which returns `false` on a 403, so a private file **silently drops the Windows 11 attachment** (and its sentence) from every email; nothing errors. The store link opens an S3 `AccessDenied` page.
 
 ### The verbiage has to match what actually attached
 Every attach is guarded by `if ($pdfGuideX !== false)`, but until 2026-08-31 the `$attachmentText` paragraph that *describes* the attachments was unconditional. A failed S3 fetch produced an email whose copy pointed the recipient at guides not on the message — and it always said "guides" plural even when a single guide attached. The rule now, in every branch touched: **derive the sentence from the guards, not from the intent.**
@@ -130,3 +137,9 @@ No committed harness. Build a throwaway `test_<x>_emails.php` next to the cron,
 - [transmit-ordershipped-email](transmit-ordershipped-email.md)
 - [Prudential profile](../profile.md)
 - [App_Email Side Effects & Test Mode](../../../1.0/apps/library/features/email-test-mode-and-write-free-sends.md)
+
+## Change history
+- 2026-10-12 — Win11 guide swapped `Sep_2026` → `Sep_2026_1.1` (phone number change); added swap procedure + S3 403 check (bala)
+- 2026-10-05 — Win11 guide swapped `v1` → `Sep_2026` in all 12 cron spots + Equip Me page (bala)
+- 2026-08-31 — attachment sentence derived from what actually attached (bala)
+- 2026-08-27 — one attachment rule across all stages; `safeFetchPdf()` in every cron (bala)

package/knowledge/sessions/2026-10-13-TRUE-82718-mhammontree.md

@@ -0,0 +1,61 @@
+---
+type: session
+slug: TRUE-82718
+title: Desk KB document red box — s3fs mount uses deleted IAM key
+author: mhammontree
+repos: [togadesk, toga]
+framework: "1.0"
+client: true
+status: active
+created: 2026-10-13
+updated: 2026-10-13
+---
+
+# Session: TRUE-82718
+**Date:** 2026-10-13
+**Project/Repo:** togadesk (1.0)
+**Task:** Investigate the TOGa Desk red box when opening Knowledge Base documents (`template/pages/documents/viewdocument.php` line 9: `file_get_contents(/var/app/asifiles/knowledgebase-prod/general/1741): failed to open stream: No such file or directory`). Investigation only; the fix is handed to Srija.
+
+---
+
+## What WORKED
+- **Proved the file is not missing.** Mark found `asifiles/knowledgebase-prod/general/1741` in S3. So the data is fine; the server cannot read it.
+- **Found how Desk reads KB files.** `/var/app/asifiles` is an s3fs mount of bucket `asifiles` on each EB server (`.ebextensions/006_mount-s3fs.config` → `ebs/create-s3fs-credentials-file.php` writes `/etc/passwd-s3fs`, then `ebs/mount-s3fs-folders.php` builds the mount command into the EB deploy hook `appdeploy/enact/04_s3fs.sh`).
+- **Root cause (confirmed by Mark in IAM).** `ebs/create-s3fs-credentials-file.php` hardcodes IAM key `AKIAJAGE…` (full id in the file). In IAM (Mark's view) only users `jhammontree` and `ses-smtp-user.20240731-170420` exist. Neither owns that key, so it was removed in the TRUE-82705 key cleanup (2026-10-12). The s3fs mount cannot log in, so every file under `/var/app/asifiles` (KB docs, ticket uploads, depot repair files) reads as "No such file".
+- **Second dead key.** `ebs/check_s3fs.php` (cron every 3 min, ends a server whose mount dropped) and `ebs/register_instance_to_shared_application_load_balancer.php` hardcode the deleted key `AKIAIPDI…` (the TRUE-82705 key). So the auto-repair also fails.
+- **Reviewed Srija's toga fix, commit `79336c1f01b5d632d146bc23f6debc93009b3cec`** (on toga `_production`, `_demo`, `TRUE-82714`): adds `-o iam_role=auto` to `ebs/mount-s3fs-folders.php`, and changes `ebs/create-s3fs-credentials-file.php` to drop the key and delete any old `/etc/passwd-s3fs`. toga's pre-change copies of both files are identical to togadesk's today (`diff --strip-trailing-cr` = same). So it applies to togadesk as-is.
+- **Reviewed Srija's composer commit `63c5ed07f76fc1962d035bf83d7751fb773c4fd6`** (toga, TRUE-82714): adds `aws/aws-sdk-php ^3.328` (locks 3.337.3). **togadesk does NOT need it.** togadesk's `composer.json` has had `aws/aws-sdk-php ^3.337` since `f7611893` (TRUE-57575, 2025-02-14), and `composer.lock` pins 3.337.3. Deploy runs `composer install` (`.ebextensions/015_install_composer.config`), and `_/app/framework.php:17-18` loads `vendor/autoload.php`.
+
+## What did NOT work — DO NOT RETRY THESE
+- Nothing failed. Do not look for a missing S3 object or a stale DB row: the file exists in S3, and the cause is the mount login.
+
+## Not tried yet (candidates for next session)
+- Check that the TOGa Desk EB environment uses the same EC2 instance profile as toga, and that the role has access to `asifiles` (`s3:ListBucket` on the bucket; `GetObject`/`PutObject`/`DeleteObject`/`PutObjectAcl` on objects — ACL because the mount uses `default_acl=public-read`). If Desk's servers are in a different account than 502614707982, a bucket policy is also needed.
+- Confirm toga uploads work after Srija's deploy. That proves `iam_role=auto` works with s3fs 1.84 on this old Amazon Linux 1 platform.
+- On a Desk prod server after deploy: `ls /var/app/asifiles/knowledgebase-prod/general/1741`.
+- togaview has the same dead keys in all four `ebs/` files and needs the same fix (separate ticket).
+
+## Current file state
+| File | Status | Notes |
+|------|--------|-------|
+| (none) | unchanged | No code changed this session. togadesk is on branch `TRUE-82718` (created by Mark), clean. |
+
+## Decisions made
+- **Srija does the togadesk fix** (Mark, 2026-10-13). She has it ready. This session made no code change.
+- **Scope of the togadesk fix**, as agreed in review:
+  1. Srija's s3fs change from toga `79336c1` (`mount-s3fs-folders.php` + `create-s3fs-credentials-file.php`).
+  2. `ebs/check_s3fs.php`: replace the deleted key with the `WorkloadsRuntime` assume-role pattern (see `tools/ebs/check_s3fs.php`). toga has no copy of this file, so Srija's commit does not cover it.
+  3. `ebs/register_instance_to_shared_application_load_balancer.php`: copy toga's fix `21b142d` (TRUE-82705).
+  4. `template/pages/documents/viewdocument.php`: cast `$_GET['id']` to int before the file path and the SQL. Today `?id=../../../current/config.prod.ini` can read any server file, and the SQL is not escaped. Also show "document not available" instead of the red box.
+- **Composer change not needed for togadesk** (AWS SDK already present, same version).
+- **Rejected:** deleting `create-s3fs-credentials-file.php` (Srija's version keeps it and removes `/etc/passwd-s3fs`, so `006_mount-s3fs.config` keeps working). Rejected: making a new long-lived IAM key (TRUE-82705 removed all keys on purpose).
+- **Order:** the role must have `asifiles` access before the togadesk deploy, or the mount still fails.
+
+## Blockers
+- none for this session. The fix is with Srija.
+
+## Exact next step
+> Ask Srija for her togadesk commit SHA, then review it against the 4-item scope in "Decisions made". Check especially items 2 (`check_s3fs.php`) and 4 (`viewdocument.php` int cast), which are not in her toga commit.
+
+---
+_Saved by /session-save on 2026-10-13_

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.1007",
+  "version": "1.0.1009",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",