toga-ai 1.0.1001 → 1.0.1002
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: session
|
|
3
|
+
slug: odp-as2-aws-key-outage
|
|
4
|
+
title: Office Depot POs not importing (AS2 Lambda stopped, then AWS key deleted)
|
|
5
|
+
author: bala
|
|
6
|
+
repos: [worker, library]
|
|
7
|
+
framework: "1.0"
|
|
8
|
+
client: compass-usa
|
|
9
|
+
status: active
|
|
10
|
+
created: 2026-10-12
|
|
11
|
+
updated: 2026-10-12
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# Session: odp-as2-aws-key-outage
|
|
15
|
+
**Date:** 2026-10-12
|
|
16
|
+
**Project/Repo:** worker + library (1.0)
|
|
17
|
+
**Task:** Find why Compass Office Depot vendor POs stopped coming into Supply after Sat 2026-10-10 07:23, recover the stuck POs, and replace the deleted AWS key that the 1.0 Office Depot crons depend on.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## What WORKED
|
|
22
|
+
- Found the real inbound path. Office Depot sends over AS2 to AWS Transfer server `s-54dad64b145d417a9` (toga-legacy 502614707982, us-west-2), agreement `a-904effd552db4348b` "OfficeDepotProd". Files land in `s3://agilant-as2/processed/<uuid>.<msgid>.net`. An S3 event rule (prefix `processed/`, Put+Post) calls Lambda `rename_edi_files`, which moves `*.net` to `s3://agilant-as2/OfficeDepot/`. Cron `worker/crons/toga2/compass/workflow/3a_import_office_depot_purchase_orders.php` imports from there.
|
|
23
|
+
- Proved the Lambda stopped. Its CloudWatch log ends at Sat 2026-10-10 07:29 CDT. The Invocations metric was 285-753 per weekday, then 0. Errors and Throttles were 0.
|
|
24
|
+
- Moved the stuck files by hand (`aws s3 mv` from `processed/` to `OfficeDepot/`, only `.net` files after 2026-10-10 12:30 UTC). Moved 67 files with 0 failures at about 13:20 CDT. 3a imported all 52 stuck POs into prod `Client_Compass.PurchaseOrders` (0 over-quantity rejects), and later 58 including new arrivals. Cron 5 pushed 57 of 58 to NetSuite by 14:42. The last one, SA138891 (ODP PO 42185598-1214), was eligible but not yet picked at that time.
|
|
25
|
+
- At 15:37 CDT, moved 39 more files to `OfficeDepot/` with `scripts/odp_move_stuck_as2_files.sh --once`. They are waiting there and CANNOT import until AWS access is restored (see Blockers).
|
|
26
|
+
- Wrote the stop-gap cron `worker/crons/toga2/compass/workflow/move_office_depot_as2_files_to_import_folder.php` plus a `*/5` entry in `worker/schedules/cron.worker.sync.json`. php-reviewer found nothing blocking, and the null-list throw was added after review. Per capture, it is committed in worker `_production` as `b3f57866`.
|
|
27
|
+
- Found why the cron fails on the box: at 15:00:28-15:00:31 CDT Jeff deleted IAM user `jcardinal` and its access key. That is the key hardcoded in `library/app/cloud.php` (static props, the `$preCommands` export, and the SDK `credentials` arrays). Error on the box: `InvalidAccessKeyId ... The AWS Access Key Id you provided does not exist in our records.`
|
|
28
|
+
- Confirmed 1.0 email still sends after 15:00. Legacy `Logs.Emails` shows 249 successful sends from 15:00 to 15:38, so SES SMTP was not affected.
|
|
29
|
+
|
|
30
|
+
## What did NOT work — DO NOT RETRY THESE
|
|
31
|
+
- Re-applying the same S3 event config (`aws s3api put-bucket-notification-configuration` on `agilant-as2`) failed with `InvalidArgument: Unable to validate the following destination configurations`. That proves S3 cannot call the Lambda. Nothing was changed.
|
|
32
|
+
- `aws lambda add-permission` on `rename_edi_files` failed with `AccessDeniedException`. Role `GoAgilant-Developers` has no Lambda rights. `get-function-configuration`, `get-function` and `list-functions` are also denied.
|
|
33
|
+
- Running `php 3a_import_office_depot_purchase_orders.php` by hand on the worker box exits silently. This is expected: the scheduled run (PID 1364) was already running and `cronInitialization` exits when another copy runs.
|
|
34
|
+
- Asking Jeff for a replacement key: refused. The key leaked via GitHub configs and an attacker used it on 2026-10-10. AWS suspended services (Lambda among them). The company is on SSO and will not issue long-lived keys again.
|
|
35
|
+
- The Transfer Family workflow `w-f8951fd4252a5b890` "OfficeDepot" (COPY to `OfficeDepot/`) is NOT the live path. It has no executions in a year of server logs.
|
|
36
|
+
|
|
37
|
+
## Not tried yet (candidates for next session)
|
|
38
|
+
- Ask Jeff to create a dedicated instance role for EB env `agilant-worker` (instance `i-0208ceac48ce0f343`, currently on the shared `aws-elasticbeanstalk-ec2-role`). Keep the 3 EB managed policies. Add S3 on `agilant-as2` (+ `asifiles`), `transfer:StartFileTransfer` on connectors `c-0fc19b6317af42f69` (ODP) and `c-392be4fc627a47a19` (Canon), logs `DescribeLogStreams/CreateLogStream/PutLogEvents` on `AS2-*`, `sqs:SendMessage` on the 2.0 worker queue (account 654654170868), and `ec2:AssociateAddress/CreateTags` for the worker EIP. NO `ssm:PutParameter`.
|
|
39
|
+
- Change `library/app/cloud.php` to the default AWS credential chain: remove the static key props, make `$preCommands` only `unset` lines, and drop the `credentials` arrays from the 5 SDK clients. Remove keys from the worker/tools config ini files and `worker/ebs/cron.worker.php`. Run php-reviewer + cso.
|
|
40
|
+
- Fix shell injection in `App_Cloud::copyFileToS3` (`library/app/cloud.php` ~377). It is reached from `togaview/mvc/ticket/post.php` ~137-160 with the upload filename unescaped. Use `escapeshellarg()` or the SDK.
|
|
41
|
+
- Check whether 2.0 is also broken. The same deleted key is in `api2/Config/production.ini:87`, `worker2/Config/production.ini:85,145` and `saml/Config/production.ini:37`. Check the SQS `NumberOfMessagesSent` metric for `WorkerProductionQueue` (toga-production, us-east-1) since 2026-10-12 20:00 UTC. Not checked because toga-production SSO had expired.
|
|
42
|
+
- Find ODP PO `42187275-6869` (Compass PO 50315219-1, ODP SO 485188717) in the files waiting in `OfficeDepot/` or `processed/`. The developer asked, then said "wait", so it was not searched.
|
|
43
|
+
- Publish the `/capture` docs. They are written in `toga-tech/knowledge` but held because the team repo has someone else's unpushed changes.
|
|
44
|
+
|
|
45
|
+
## Current file state
|
|
46
|
+
| File | Status | Notes |
|
|
47
|
+
|------|--------|-------|
|
|
48
|
+
| worker/crons/toga2/compass/workflow/move_office_depot_as2_files_to_import_folder.php | created | Stop-gap move cron. Committed b3f57866 on _production per capture. Fails until AWS access is restored. Disable when the Lambda is back. |
|
|
49
|
+
| worker/schedules/cron.worker.sync.json | modified | Added `*/5` entry for the move cron, after 3a |
|
|
50
|
+
| C:/WWW/scripts/odp_move_stuck_as2_files.sh | created (outside repos) | Local helper using the developer SSO profile. Loop every 5 min, or `--once`. |
|
|
51
|
+
| toga-tech/knowledge (5 docs) | written, NOT published | New `1.0/apps/library/features/app-cloud-aws-credentials.md`. Updated the compass-usa ODP pipeline and recovery workflows, `error-capture-1-0.md` and the togaview `ticket-detail-page.md`. Architecture section skipped by the developer. |
|
|
52
|
+
|
|
53
|
+
## Decisions made
|
|
54
|
+
- **Replace keys with per-environment instance roles + the default credential chain** in `App_Cloud` (and 2.0 `_underscore/Cloud.php`). Rationale: nothing to leak, credentials expire, least privilege. The CTO agreed. cso agreed with the design but blocks rollout until the other committed secrets (DB passwords, NetSuite key, payment keys, SSO signing keys) are rotated. Rejected: a new long-lived key (DevOps refuses, and it is the same risk), widening the shared `aws-elasticbeanstalk-ec2-role` (about 60 envs would get EDI access), and per-cron credentials (duplicates the shared class).
|
|
55
|
+
- **Put the stop-gap cron in `crons/toga2/compass/workflow/` with no step number.** It sits beside `send_as2_outbox.php` (the outbound AS2 plumbing). It is temporary, not a business step.
|
|
56
|
+
- **Disable the infra-automation crons** (RDS scaler, terminate, failover) until reviewed. They needed admin powers.
|
|
57
|
+
|
|
58
|
+
## Blockers
|
|
59
|
+
- No AWS access for any 1.0 App_Cloud call since 2026-10-12 15:00 CDT. 3a cannot import, `send_as2_outbox.php` cannot send ACKs/ASNs, and the move cron cannot run. Waiting on Jeff to create the instance role.
|
|
60
|
+
- Lambda `rename_edi_files` is suspended by AWS after the breach. Jeff is waiting for AWS to restore it.
|
|
61
|
+
- 39+ Office Depot files are waiting in `s3://agilant-as2/OfficeDepot/`, and new ones keep landing in `processed/`.
|
|
62
|
+
|
|
63
|
+
## Exact next step
|
|
64
|
+
> Ask Jeff whether the `agilant-worker` instance role is created. Then edit `library/app/cloud.php` to use the default credential chain (remove the key statics, the `$preCommands` export and the SDK `credentials` arrays), run php-reviewer + cso, and deploy library + worker. After that, confirm 3a imports the files waiting in `s3://agilant-as2/OfficeDepot/` (new `ODP_EDI` rows in legacy `Logs.FileLog`).
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
_Saved by /session-save on 2026-10-12_
|
package/package.json
CHANGED