toga-ai 1.0.808 → 1.0.810
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md +60 -1
- package/knowledge/2.0/apps/_underscore/features/cloud-s3-helpers.md +1 -0
- package/knowledge/2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md +19 -2
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
- package/knowledge/2.0/apps/worker2/features/cross-account-aws-access.md +40 -1
- package/knowledge/2.0/apps/worker2/features/s3-folder-retention-cleanup.md +130 -0
- package/knowledge/INDEX.md +1 -1
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-15
|
|
10
10
|
owners: ["dfranks", "bala", "jcardinal", "mhammontree", "snaredla", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/netsuite/common_sync_togasupply.php
|
|
@@ -1270,6 +1270,50 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
|
|
|
1270
1270
|
Note `SalesOrderItems.lineNumber` is a **`smallint`**, so the parked base must stay within smallint
|
|
1271
1271
|
range for the client's line counts (headroom 1000 is safe at observed volumes).
|
|
1272
1272
|
|
|
1273
|
+
- **⚠ A NetSuite-removed line that still holds an UPSTREAM customer-PO link blocked the renumber and
|
|
1274
|
+
re-froze Compass SALES_ORDERS (fixed 2026-09-15, `library` only, Compass-gated).** Compass USA's
|
|
1275
|
+
SALES_ORDERS was stuck at `1-RUNNING` (`Logs.Issue` 749, clientId 2) throwing *"unresolvable
|
|
1276
|
+
NetSuite<->DB lineNumber conflict … reconcile the orphaned/undeletable row"* whenever a
|
|
1277
|
+
NetSuite-removed line sat on a `lineNumber` a surviving line had to move onto. Root cause: the
|
|
1278
|
+
**existing** stale-line prune (`toga2.php` ~L1582) runs **AFTER** the renumber **and** skips any
|
|
1279
|
+
line carrying a PO link (it cannot delete a PO-linked line), so those removed-but-PO-linked lines
|
|
1280
|
+
survived and occupied a needed target number — and the renumber threw first anyway.
|
|
1281
|
+
- **Fix — a Compass-gated prune BEFORE the renumber** in `syncSalesOrderFromNetsuite`. For each SO
|
|
1282
|
+
line whose `c_netsuiteLineUniqueKey` is **non-empty** and **absent** from the current NetSuite
|
|
1283
|
+
line set (a line NetSuite removed) that was **never fulfilled and never invoiced**: delete its
|
|
1284
|
+
Agilant-side SO↔PO **bridge LINK** rows in **both** directions (routes
|
|
1285
|
+
`/purchase-order-item-sales-order-items/{uuid}` and `/sales-order-item-purchase-order-items/{uuid}`),
|
|
1286
|
+
then DELETE the `SalesOrderItem` (`/sales-order-items/{uuid}`), then drop it from the in-memory
|
|
1287
|
+
`$salesOrderItemsInToga`. It **NEVER** deletes/modifies the `PurchaseOrder`/`PurchaseOrderItem`
|
|
1288
|
+
itself (the Compass **customer** order is off-limits) and never touches another order record.
|
|
1289
|
+
- **Guardrails (owner decision 2026-09-15; cto AGREE; php-reviewer clean; cso SAFE TO SHIP):**
|
|
1290
|
+
(a) non-empty key **absent from NetSuite** only — an empty key was never matched to NetSuite, so
|
|
1291
|
+
treating it as "removed" is a false positive; (b) never a shipped/invoiced line (checked against
|
|
1292
|
+
the fulfillment/invoice maps built ~L990-1032); (c) **Compass-gated** (`CLIENT_UUID_COMPASS`);
|
|
1293
|
+
(d) **re-check that one line's fulfillment/invoice status immediately before the deletes** to
|
|
1294
|
+
close a TOCTOU gap (the maps are a start-of-run snapshot); (e) the final `SalesOrderItem` DELETE
|
|
1295
|
+
stays **throwing** (fail-loud) if any unhandled attachment remains — mirrors the PO-line cleanup
|
|
1296
|
+
in `syncPurchaseOrderFromNetsuite` (~L2793-2806) and the mirror-NetSuite decision (2026-09-02).
|
|
1297
|
+
- **Every FK child of `Client_Compass.SalesOrderItems` blocks a delete.** All are `NO ACTION` or
|
|
1298
|
+
`RESTRICT`, and InnoDB treats `NO ACTION` as `RESTRICT`: `ItemFulfillmentItems`, `InvoiceItems`,
|
|
1299
|
+
`PurchaseOrderItems_SalesOrderItems`, `SalesOrderItems_PurchaseOrderItems`,
|
|
1300
|
+
`SalesOrderItems_CommittedUnits`, `SalesOrderItems_Tags`, `SalesOrderItems`
|
|
1301
|
+
(`parentSalesOrderItemId` kit children), `SalesOrderItems_TransferOrderItems`,
|
|
1302
|
+
`TransferOrderItems_SalesOrderItems`. This is why the prune must clear the bridge links first and
|
|
1303
|
+
only ever touches never-fulfilled/never-invoiced lines.
|
|
1304
|
+
- **Proven (evidence):** NetSuite sales order 271929 (internal id 6926956), customer "6096 ODP
|
|
1305
|
+
Veyer (B2B)"; NetSuite reorganized it to 9 lines and removed 6 kitting lines (lineUniqueKeys
|
|
1306
|
+
20024694/695/696/702/703/704) that never shipped and were never invoiced but each still held an
|
|
1307
|
+
upstream customer-PO link — those 6 blocked the renumber.
|
|
1308
|
+
- **⚠ OPEN, NOT FIXED — a DUPLICATE `SalesOrders` record on one NetSuite internal id.** `SalesOrders`
|
|
1309
|
+
number **467435086001** shares NetSuite internal id **6926956** with order 271929 (two TOGa
|
|
1310
|
+
records for one NetSuite sales order; 467435086001 has 12 keyless lines with downstream vendor-PO
|
|
1311
|
+
links, none shipped). The main SO lookup in `syncSalesOrderFromNetsuite` (~L946-974) filters
|
|
1312
|
+
Compass by `locationId 615286` / `customerId 3`, which currently narrows to 271929, so the prune
|
|
1313
|
+
only ever touches 271929 — but **two records on one NetSuite id is an anomaly that could make the
|
|
1314
|
+
lookup bind to the wrong record on other orders.** Needs its own ticket. Deploy: `library` only;
|
|
1315
|
+
the worker redeploys to pick it up (worker clones library on deploy).
|
|
1316
|
+
|
|
1273
1317
|
- **⚠ Canon + Endeavor Health ITEM_RECEIPTS froze because `fetchPurchaseOrderById` never populated the
|
|
1274
1318
|
PO vendor (deployed/verified 2026-08-29).** The PO shim's `->entity` was always null (its SuiteQL
|
|
1275
1319
|
SELECT omitted `entity`, unlike `listPurchaseOrders`), so the fail-loud missing-entity guard in
|
|
@@ -1581,6 +1625,21 @@ library (or vice versa) crashes GroWrk and Adyen on their next sync run.
|
|
|
1581
1625
|
|
|
1582
1626
|
## Change history
|
|
1583
1627
|
|
|
1628
|
+
- 2026-09-15 — **Compass SALES_ORDERS unfrozen: a Compass-gated prune runs BEFORE the renumber.**
|
|
1629
|
+
`syncSalesOrderFromNetsuite` was throwing *"unresolvable NetSuite<->DB lineNumber conflict"*
|
|
1630
|
+
(`Logs.Issue` 749, clientId 2) because a NetSuite-removed line that still held an upstream
|
|
1631
|
+
customer-PO link survived the existing stale-line prune (which runs after the renumber and skips
|
|
1632
|
+
PO-linked lines) and occupied a `lineNumber` a surviving line needed. Added a prune BEFORE the
|
|
1633
|
+
renumber that, for each never-fulfilled/never-invoiced line whose non-empty `c_netsuiteLineUniqueKey`
|
|
1634
|
+
is absent from NetSuite, deletes its SO↔PO bridge links in both directions then the `SalesOrderItem`
|
|
1635
|
+
— never touching the `PurchaseOrder`/`PurchaseOrderItem` (Compass customer order). Guardrails:
|
|
1636
|
+
absent-from-NetSuite + non-empty key only, never shipped/invoiced, Compass-gated, immediate
|
|
1637
|
+
TOCTOU re-check, final DELETE stays fail-loud. Recorded that every FK child of
|
|
1638
|
+
`Client_Compass.SalesOrderItems` is `NO ACTION`/`RESTRICT` (all block a delete), and an OPEN
|
|
1639
|
+
duplicate-`SalesOrders`-record anomaly (467435086001 shares NetSuite internal id 6926956 with
|
|
1640
|
+
271929). Proven on NetSuite SO 271929 (6 removed kitting lines blocked the renumber). `library`
|
|
1641
|
+
only, no SQL; worker redeploys to pick it up. cto AGREE; php-reviewer clean; cso SAFE TO SHIP.
|
|
1642
|
+
(jcardinal)
|
|
1584
1643
|
- 2026-09-13 — **ITEM_FULFILLMENTS unfrozen for NYCHH: five stacked fulfillment-reconcile bugs fixed
|
|
1585
1644
|
(library, deployed + verified in prod).** (1) **Unit duplication from a truncated nested GET** — the
|
|
1586
1645
|
existing-units dedup map was built from the nested `itemFulfillmentItemUnits` of one
|
package/knowledge/2.0/apps/_underscore/features/sales-order-purchase-order-bridge-direction.md
CHANGED
|
@@ -6,8 +6,8 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: [apeterson, bala]
|
|
9
|
+
updated: 2026-09-15
|
|
10
|
+
owners: [apeterson, bala, jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Model/Client/PurchaseOrders/SalesOrder.php
|
|
13
13
|
- _underscore/Model/Client/SalesOrders/PurchaseOrder.php
|
|
@@ -40,6 +40,18 @@ one returns `200` with an empty array, which reads like a permission or data bug
|
|
|
40
40
|
Read the name as **"`<source>`\_`<thing created from it>`"**. The *first* table name is the
|
|
41
41
|
origin; the second is what was generated from it.
|
|
42
42
|
|
|
43
|
+
**The line-item bridges follow the same rule (same name order, same directions):**
|
|
44
|
+
|
|
45
|
+
| Table | Direction | Meaning | API route |
|
|
46
|
+
|---|---|---|---|
|
|
47
|
+
| `PurchaseOrderItems_SalesOrderItems` | **UPSTREAM** | an SO line created **FROM** a customer PO line | `purchase-order-item-sales-order-items` |
|
|
48
|
+
| `SalesOrderItems_PurchaseOrderItems` | **DOWNSTREAM** | a vendor PO line created **FROM** an SO line | `sales-order-item-purchase-order-items` |
|
|
49
|
+
|
|
50
|
+
The 1.0 Compass NetSuite sync deletes these **item-level** link rows in **both** directions when it
|
|
51
|
+
prunes a NetSuite-removed SO line — see
|
|
52
|
+
[NetSuite → TOGa Supply per-client sync](../../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md)
|
|
53
|
+
(2026-09-15).
|
|
54
|
+
|
|
43
55
|
## How it works
|
|
44
56
|
|
|
45
57
|
- Both `Records` rows are `aclDatabase = CLIENT`, `childPolicy = MATCH_UPSERT`, so both are
|
|
@@ -122,6 +134,11 @@ everywhere. Present is not populated.
|
|
|
122
134
|
collapses many POs without multiplying rows.
|
|
123
135
|
|
|
124
136
|
## Change history
|
|
137
|
+
- 2026-09-15 — Added the **line-item** bridge parallel: `PurchaseOrderItems_SalesOrderItems`
|
|
138
|
+
(upstream, route `purchase-order-item-sales-order-items`) and `SalesOrderItems_PurchaseOrderItems`
|
|
139
|
+
(downstream, route `sales-order-item-purchase-order-items`) follow the same name-order rule as the
|
|
140
|
+
header bridges. Confirmed while building the Compass 1.0 sync's pre-renumber prune, which deletes
|
|
141
|
+
both item-level link directions for a NetSuite-removed SO line. (jcardinal)
|
|
125
142
|
- 2026-08-26 — Added the **write** side: the two directions are populated by two different
|
|
126
143
|
pipelines (2.0 `_Trait_Netsuite_SalesOrder` writes **upstream** customer POs from `otherRefNum`;
|
|
127
144
|
the 1.0 `syncPurchaseOrderFromNetsuite` cron writes **downstream** vendor POs), and the api2 child
|
|
@@ -46,6 +46,7 @@
|
|
|
46
46
|
| [Platform Cache Cleanup (_Worker_Platform_Cache — Clean + Truncate)](features/platform-cache-cleanup.md) | `_Worker_Platform_Cache` owns maintenance of the shared **Cache** cluster that backs api2's [multi-client data retrieval](../../api2/features/cross-client-data- |
|
|
47
47
|
| [QA/QC Branch Automation (auto-mirror + rebuild-from-ledger revert)](features/qa-qc-branch-automation.md) | Two CTO-approved (AGREE) pieces of GitHub branch automation, centralized in **worker2** rather than per-repo GitHub Actions. |
|
|
48
48
|
| [QA/QC Review Pipeline (ClickUp review chain, QC batch + defect/client-review)](features/qa-qc-review-pipeline.md) | The ClickUp-driven review + promotion chain for the platform-wide QA/QC dev process (per the approved `QA-QC Plan.txt`), moving a task through **Development → Q |
|
|
49
|
+
| [S3 folder retention cleanup worker action (CleanS3Folder)](features/s3-folder-retention-cleanup.md) | `_Worker_Infrastructure_Aws_Cleanup::CleanS3Folder` (action path `Infrastructure/Aws/Cleanup/CleanS3Folder`) deletes S3 objects under **one folder** (key prefix |
|
|
49
50
|
| [Service Request → Sales Order → Purchase Order generation (Sync/ServiceRequest)](features/service-request-sales-order-generation.md) | `_Worker_Sync_ServiceRequest` turns a **Service Request into a Sales Order, and then into one Purchase Order per vendor**, for **any** tenant. |
|
|
50
51
|
| [SSO Stability Monitor (Monitor/Operations/SsoStability)](features/sso-stability-monitor.md) | `_Worker_Monitor_Operations::SsoStability(): string` is the **reporter half** of the SAML uptime monitor. |
|
|
51
52
|
| [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Receives inbound webhook events from Startech (Easeedesk) and creates or updates the corresponding ticket in TOGA 2.0. |
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-14
|
|
10
10
|
owners: [jcardinal, bala]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Component/Aws/Workloads/Workloads.php
|
|
@@ -171,8 +171,40 @@ that account is the
|
|
|
171
171
|
`agilant-worker-alpha` and `agilant-worker-beta` as **separate EB environments that have no role
|
|
172
172
|
registry at all**. An account-level EC2 scan would sweep them in and misreport them.
|
|
173
173
|
|
|
174
|
+
### `agilant-as2` is in the LEGACY account 502614707982 — S3 work on it is cross-account
|
|
175
|
+
|
|
176
|
+
The `agilant-as2` bucket (the Office Depot AS2/EDI drop bucket, `us-west-2`) is **not** in the
|
|
177
|
+
868 production hub where worker2 runs. Verified via `aws s3api list-buckets` per profile: only
|
|
178
|
+
account **502614707982** lists it. So any S3 work on `agilant-as2` is **cross-account**.
|
|
179
|
+
|
|
180
|
+
**The fix (same model as this component):** pass the bucket's **owner** account id as
|
|
181
|
+
`awsAccountId` so the worker assumes `WorkloadsRuntime` **in that account** and touches the
|
|
182
|
+
bucket **same-account**. Then an identity policy on the owner account's `WorkloadsRuntime`
|
|
183
|
+
role is enough — **no bucket policy is needed**. Every `WorkloadsRuntime` trusts the 868
|
|
184
|
+
instance role, so assuming into 502 from 654 works.
|
|
185
|
+
|
|
186
|
+
**Diagnostic tell — the AccessDenied wording tells you the account.** The SDK error
|
|
187
|
+
`AccessDenied ... is not authorized to perform: s3:ListBucket ... because no resource-based
|
|
188
|
+
policy allows the s3:ListBucket action` is AWS's **cross-account** signal. A same-account
|
|
189
|
+
identity-policy gap instead says *"no identity-based policy"*. Seeing **"no resource-based
|
|
190
|
+
policy"** means the resource lives in another account — assume into the owner account.
|
|
191
|
+
|
|
192
|
+
**IAM ARN gotcha (bucket-level vs object-level).** `s3:ListBucket` and
|
|
193
|
+
`s3:GetBucketLocation` are **bucket-level** → the `Resource` must be the **bucket** ARN
|
|
194
|
+
`arn:aws:s3:::agilant-as2` with **no** `/*`. `s3:DeleteObject` is **object-level** → the
|
|
195
|
+
`Resource` must end in `/*` (scope it, e.g. `arn:aws:s3:::agilant-as2/OUTBOX/processed/*`).
|
|
196
|
+
Putting `ListBucket` on the `/*` ARN is the classic mistake and returns exactly this
|
|
197
|
+
AccessDenied.
|
|
198
|
+
|
|
199
|
+
First S3 consumer of this pattern: the
|
|
200
|
+
[S3 folder retention cleanup action](./s3-folder-retention-cleanup.md).
|
|
201
|
+
|
|
174
202
|
## Gotchas / known issues
|
|
175
203
|
|
|
204
|
+
- **`agilant-as2` is cross-account (owner 502614707982), not in the 868 hub.** Assume into the
|
|
205
|
+
owner account; the `AccessDenied ... no resource-based policy` wording is the cross-account
|
|
206
|
+
tell, and `ListBucket`/`GetBucketLocation` need the bare bucket ARN while `DeleteObject`
|
|
207
|
+
needs the `/*` ARN. See [the section above](#agilant-as2-is-in-the-legacy-account-502614707982--s3-work-on-it-is-cross-account).
|
|
176
208
|
- **Do not assume the 1.0 worker fleet is in the 868 production hub.** `AWS_WORKER_QUEUE_URL`
|
|
177
209
|
(654654170868) is the SQS queue only; the instances are in the legacy account 502614707982,
|
|
178
210
|
`us-west-2`. See [the section above](#which-account-the-10-worker-fleet-lives-in-easy-and-expensive-to-get-wrong).
|
|
@@ -191,6 +223,13 @@ registry at all**. An account-level EC2 scan would sweep them in and misreport t
|
|
|
191
223
|
see [creating-worker-actions.md](./creating-worker-actions.md).
|
|
192
224
|
|
|
193
225
|
## Change history
|
|
226
|
+
- 2026-09-14 — Recorded that **`agilant-as2` is owned by the legacy account 502614707982**
|
|
227
|
+
(`us-west-2`), not the 868 production hub — so S3 work on it is cross-account: pass the
|
|
228
|
+
owner account id as `awsAccountId` and no bucket policy is needed. Added the diagnostic tell
|
|
229
|
+
(`AccessDenied ... no resource-based policy` = cross-account; *no identity-based policy* =
|
|
230
|
+
same-account) and the IAM ARN gotcha (`ListBucket`/`GetBucketLocation` need the bare bucket
|
|
231
|
+
ARN, `DeleteObject` needs the `/*` ARN). Found while building the
|
|
232
|
+
[S3 folder retention cleanup action](./s3-folder-retention-cleanup.md). (jcardinal)
|
|
194
233
|
- 2026-08-24 — Pinned down **which account the 1.0 `worker` fleet actually runs in**:
|
|
195
234
|
**502614707982** (`us-west-2`, EB env `agilant-worker`), **not** the 654654170868 production hub
|
|
196
235
|
that `App_Worker::AWS_WORKER_QUEUE_URL` implies — that account holds only the SQS queue the 1.0
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: S3 folder retention cleanup worker action (CleanS3Folder)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-14
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Infrastructure/Aws/Cleanup.php
|
|
13
|
+
- dbchanges2/Core/2026-09-14a - Insert S3 Folder Cleanup CronJob.sql
|
|
14
|
+
related:
|
|
15
|
+
- ./creating-worker-actions.md
|
|
16
|
+
- ./cross-account-aws-access.md
|
|
17
|
+
- ../../_underscore/features/cloud-s3-helpers.md
|
|
18
|
+
- ./platform-cache-cleanup.md
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Summary
|
|
22
|
+
|
|
23
|
+
`_Worker_Infrastructure_Aws_Cleanup::CleanS3Folder` (action path
|
|
24
|
+
`Infrastructure/Aws/Cleanup/CleanS3Folder`) deletes S3 objects under **one folder** (key
|
|
25
|
+
prefix) that are older than a retention window. **Both the folder and the days-to-keep are
|
|
26
|
+
cron parameters**, so one action + one `Core.CronJobs` row cleans any folder with **no code
|
|
27
|
+
change**. First use case: keep 30 days of `s3://agilant-as2/OUTBOX/processed/`.
|
|
28
|
+
|
|
29
|
+
> This is the **S3** folder cleanup. It is separate from the DB "Cleanup cron" — see
|
|
30
|
+
> [platform-cache-cleanup](./platform-cache-cleanup.md) for that one.
|
|
31
|
+
|
|
32
|
+
## Key files / entry points
|
|
33
|
+
|
|
34
|
+
- `worker2/Worker/Infrastructure/Aws/Cleanup.php` — `_Worker_Infrastructure_Aws_Cleanup::CleanS3Folder`.
|
|
35
|
+
- `dbchanges2/Core/2026-09-14a - Insert S3 Folder Cleanup CronJob.sql` — the seed
|
|
36
|
+
`Core.CronJobs` row (seed rows live in dbchanges2, **never** a `.sql` in worker2).
|
|
37
|
+
|
|
38
|
+
## How it works
|
|
39
|
+
|
|
40
|
+
1. Parses an `s3://bucket/prefix/` URI into bucket + prefix.
|
|
41
|
+
2. Assumes credentials via
|
|
42
|
+
`_Component_Aws_Workloads::assumeCredentials($awsAccountId)` and builds **one**
|
|
43
|
+
`Aws\S3\S3Client` reused for both list and delete. It uses **no static keys** and does
|
|
44
|
+
**not** use the `_Cloud` static-key helpers — this is a role-based cross-account path (see
|
|
45
|
+
[cross-account-aws-access](./cross-account-aws-access.md)).
|
|
46
|
+
3. Paginates `listObjectsV2`; keeps only keys whose `LastModified` is older than
|
|
47
|
+
`now - daysToKeep`.
|
|
48
|
+
4. Batch-deletes via `deleteObjects` (max 1000 keys per call).
|
|
49
|
+
5. Returns a JSON summary on **every** path: `bucket`, `prefix`, `region`, `daysToKeep`,
|
|
50
|
+
`cutoffUtc`, `dryRun`, `scanned`, `eligible`, `deleted`, `failed`, `failedSample`,
|
|
51
|
+
`complete` — a proof-of-work return, per the
|
|
52
|
+
[creating-worker-actions](./creating-worker-actions.md) null-return rule.
|
|
53
|
+
|
|
54
|
+
### Self-chunking — carry the S3 list position, not a page number
|
|
55
|
+
|
|
56
|
+
A non-scheduler action is hard-capped at **300s** (the watchdog only honours
|
|
57
|
+
`CronJobs.maxExecutionTime` for scheduler-fired jobs, and it does not stop the process — see
|
|
58
|
+
[creating-worker-actions](./creating-worker-actions.md#-the-real-time-budget-is-300s--and-for-a-hand-triggered-action-it-cannot-be-raised)).
|
|
59
|
+
The action works a ~240s slice, then re-enqueues itself with `_Worker::runTask(self::ACTION, …)`.
|
|
60
|
+
|
|
61
|
+
The cursor it carries in the `resume` param is the **S3 `continuationToken`**, not a page
|
|
62
|
+
counter:
|
|
63
|
+
|
|
64
|
+
- **Why a keyset-style cursor is required:** S3 lists keys in **name order, not date order**.
|
|
65
|
+
A chunk can burn its whole 240s budget paging through recent keys before it reaches any old
|
|
66
|
+
object, and delete nothing that pass. Re-listing from page 1 each chunk (an earlier draft
|
|
67
|
+
did this) then **stalls the sweep forever with no alert** — every chunk re-scans the same
|
|
68
|
+
front of the list. Carrying the `continuationToken` guarantees the sweep advances through
|
|
69
|
+
the whole prefix.
|
|
70
|
+
- **Backstop:** `MAX_CHUNK_ATTEMPTS = 100`.
|
|
71
|
+
- **Single-attempt hand-off:** `runTask` mints a fresh uuid per call, so a *retried* hand-off
|
|
72
|
+
would fork the chain — do not retry it (same rule as
|
|
73
|
+
[creating-worker-actions](./creating-worker-actions.md#do-not-retry-the-chunk-hand-off--a-failed-hand-off-must-break-the-chain)).
|
|
74
|
+
- **Normalize `resume` with a full round trip** — `json_encode` then `json_decode(true)` —
|
|
75
|
+
because the dispatcher casts only the top level of parameters, so a nested object arrives as
|
|
76
|
+
`stdClass`.
|
|
77
|
+
|
|
78
|
+
### Destructive-delete safety guards (reviewed by CTO + cso)
|
|
79
|
+
|
|
80
|
+
This deletes production S3, so the action refuses to run unless the target is provably safe:
|
|
81
|
+
|
|
82
|
+
- Refuses a missing `s3://` scheme.
|
|
83
|
+
- Refuses an **empty prefix** (never a whole-bucket sweep).
|
|
84
|
+
- Forces the prefix to end in `/`.
|
|
85
|
+
- Refuses a prefix shallower than `MIN_PREFIX_DEPTH = 2` folder levels — so a mistyped row
|
|
86
|
+
cannot hit a top-level or a live feed like `OfficeDepot/`.
|
|
87
|
+
- `daysToKeep` must be `1..3650`.
|
|
88
|
+
- Skips the folder-placeholder key.
|
|
89
|
+
- Validates `awsAccountId` against a **3-account allowlist** (`654654170868` prod,
|
|
90
|
+
`502614707982` legacy, `975050298201` non-prod) **before** assume-role, because
|
|
91
|
+
WorkloadsRuntime reaches all three with broad permissions.
|
|
92
|
+
- Logs each failed `deleteObjects` key — `Errors[]` returns HTTP 200 and does **not** throw on
|
|
93
|
+
per-key failure — and carries a capped `failedSample`.
|
|
94
|
+
|
|
95
|
+
## Scheduling
|
|
96
|
+
|
|
97
|
+
The seed `Core.CronJobs` row is
|
|
98
|
+
`dbchanges2/Core/2026-09-14a - Insert S3 Folder Cleanup CronJob.sql`:
|
|
99
|
+
|
|
100
|
+
- action `Infrastructure/Aws/Cleanup/CleanS3Folder`
|
|
101
|
+
- schedule `0 2 * * *` (daily, Central), `maxExecutionTime` 300
|
|
102
|
+
- parameters
|
|
103
|
+
`{"folder":"s3://agilant-as2/OUTBOX/processed/","daysToKeep":30,"awsAccountId":"502614707982","awsRegion":"us-west-2"}`
|
|
104
|
+
- guarded `INSERT ... SELECT ... FROM DUAL WHERE NOT EXISTS`, fresh random v4 uuid.
|
|
105
|
+
|
|
106
|
+
`awsAccountId` is the **legacy** account `502614707982` because `agilant-as2` lives there — see
|
|
107
|
+
[cross-account-aws-access](./cross-account-aws-access.md). Ship the worker2 code deploy and
|
|
108
|
+
this dbchanges2 migration **together**.
|
|
109
|
+
|
|
110
|
+
## Client variations
|
|
111
|
+
|
|
112
|
+
None — shared infrastructure, not client-specific.
|
|
113
|
+
|
|
114
|
+
## Gotchas / known issues
|
|
115
|
+
|
|
116
|
+
- **S3 lists by name, not date** — the whole reason the cursor must be a `continuationToken`
|
|
117
|
+
and not a page number (see self-chunking above).
|
|
118
|
+
- `deleteObjects` returns HTTP 200 with per-key `Errors[]` — it does not throw on a partial
|
|
119
|
+
failure, so you must read `Errors[]` yourself.
|
|
120
|
+
- `agilant-as2` is in **us-west-2** and is owned by a **different AWS account** — pass the
|
|
121
|
+
right `awsRegion` and `awsAccountId` (see related docs).
|
|
122
|
+
|
|
123
|
+
## Change history
|
|
124
|
+
- 2026-09-14 — Created. Added the generic `CleanS3Folder` action (folder + days-to-keep are
|
|
125
|
+
cron params, one row per folder, no code change), its self-chunking sweep carrying the S3
|
|
126
|
+
`continuationToken` (S3 lists by name not date, so a page-number cursor stalls the sweep),
|
|
127
|
+
and the destructive-delete guards (allowlist accounts, min prefix depth, prefix must end in
|
|
128
|
+
`/`, never whole-bucket). Seed cron row in dbchanges2 `Core/2026-09-14a`. (jcardinal)
|
|
129
|
+
</content>
|
|
130
|
+
</invoke>
|
package/knowledge/INDEX.md
CHANGED
|
@@ -19,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
19
19
|
## 2.0 framework
|
|
20
20
|
|
|
21
21
|
- **_underscore** (_Underscore) _(framework core)_ — 83 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
22
|
-
- **worker2** (Worker) —
|
|
22
|
+
- **worker2** (Worker) — 69 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
23
23
|
- **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
24
24
|
- **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
25
25
|
- **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
package/package.json
CHANGED