toga-ai 1.0.343 → 1.0.345

package/knowledge/2.0/apps/_underscore/features/cloud-s3-helpers.md

@@ -7,11 +7,12 @@ client: shared
 type: feature
 status: active
 updated: 2026-07-14
-owners: ["jcardinal"]
+owners: ["jcardinal", "bala"]
 files:
   - _underscore/Cloud.php
 related:
   - ../../worker2/features/oneuptime-worker2-monitoring.md
+  - ../../worker2/features/compass-vip-support-importer.md
 ---
 
 ## Summary
@@ -58,8 +59,17 @@ None — shared core helper.
   do not add your own `ContinuationToken` loop on top.
 - A bucket outside the config-default us-east-1 (e.g. `agilant-as2` in us-west-2) throws
   `AuthorizationHeaderMalformed` unless you pass `$awsRegion`.
+- **`getFileFromS3` (the GET method) is single-region.** It builds the `S3Client` with region
+  `_Config::cloud('aws_region')` (the worker's own region) and only swallows HTTP 404. Reading
+  an object from a bucket in a **different** region raises a `301 PermanentRedirect`, which it
+  re-throws. To read a cross-region (or public) S3 object, use a direct HTTPS download instead,
+  or construct an `S3Client` with the bucket's own region. (Unlike `getS3Objects()`, `getFileFromS3`
+  has no region-override parameter.)
 
 ## Change history
+- 2026-07-15 — Documented that `getFileFromS3` is single-region (client built with the worker's
+  own `aws_region`, only swallows 404, re-throws `301 PermanentRedirect` for cross-region reads);
+  use a direct HTTPS download for cross-region/public objects. (bala)
 - 2026-07-14 — `getS3Objects()` gained an optional `?string $awsRegion = null` 3rd param
   (falls back to `_Config::cloud('aws_region')`); needed because `agilant-as2` is in
   us-west-2, not the config-default us-east-1. Additive — 2-arg callers unchanged. (jcardinal)

package/knowledge/2.0/apps/worker2/INDEX.md

@@ -11,6 +11,7 @@
 | [ClickUp Rich-Text Custom Fields via Quill Delta (API)](features/clickup-richtext-api.md) | ClickUp custom text fields (type `text` and long-text) support rich formatting only through a **Quill Delta** written to the undocumented `value_richtext` key o | test/@dave/clickup_md2delta.js, .claude/skills/plan-ticket/scripts/clickup.js |
 | [ClickUp Subtask Activity → Parent Opportunity/Epic Comments](features/clickup-subtask-activity.md) | Surfaces **subtask** progress, completion, and discussion on the top-level **Opportunity** or **Epic** it rolls up to, so a deal/project owner sees activity whe | worker2/Worker/Clickup/Subtask.php, worker2/Worker/Clickup.php, dbchanges2/Team/2026-07-14a - Add ClickupSubtaskActivity ledger.sql |
 | [ClickUp Work Type Automation (Committed / Conditional / Stretch)](features/clickup-work-type-automation.md) | The ClickUp webhook handler (`_Worker_Clickup`) automatically maintains each task's **Work Type** custom field — `Committed`, `Conditional`, or `Stretch` — base | worker2/Worker/Clickup.php, worker2/Tests/Worker/ClickupWorkTypeTest.php |
+| [Compass VIP Support Importer (worker2)](features/compass-vip-support-importer.md) | A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP user's support technician by setting `Users.c_supportedByUserId` in `Clie | worker2/Worker/Client/Compass/VipSupport.php |
 | [Creating Worker Actions](features/creating-worker-actions.md) | How to add a new callable Worker action — a PHP class whose `public static` methods are invoked as background jobs (via webhook, cron, or `_Worker::runTask()`). | worker2/Worker/, worker2/Controller/Index.php, _underscore/Worker.php |
 | [Elite Freshservice Sync (worker2)](features/elite-freshservice-sync.md) | `_Worker_Elite` processes Freshservice webhook events and syncs them into TOGA 2. | worker2/Worker/Elite.php, worker2/Config/dev-kmaramreddy-laptop.ini |
 | [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding from an S3 CSV plus an Etilize re-pull. | worker2/Worker/Etilize/Items.php |

package/knowledge/2.0/apps/worker2/features/compass-vip-support-importer.md

@@ -0,0 +1,158 @@
+---
+title: Compass VIP Support Importer (worker2)
+framework: "2.0"
+repo: worker2
+project: Worker
+client: compass-usa
+type: client-feature
+status: active
+updated: 2026-07-15
+owners: [bala]
+files:
+  - worker2/Worker/Client/Compass/VipSupport.php
+related:
+  - ./creating-worker-actions.md
+  - ../../../clients/compass-usa/profile.md
+  - ../../_underscore/features/cloud-s3-helpers.md
+---
+
+## Summary
+
+A worker2 action that ingests Compass's quarterly VIP spreadsheet and assigns each VIP
+user's support technician by setting `Users.c_supportedByUserId` in `Client_Compass`. It is
+invoked **manually via Postman** (no cron). Given a file, it stages one update per matching
+VIP row, applies them in a single transaction, and returns a JSON report of what was updated
+and — with a per-row reason — what was skipped.
+
+- **Action:** `Client/Compass/VipSupport/ProcessVipFile`
+- **Params:** `{"filePath": "<local path OR http(s) URL>"}`
+
+## Key files / entry points
+
+- `worker2/Worker/Client/Compass/VipSupport.php` — `ProcessVipFile(string $filePath): string`.
+
+## How it works
+
+1. **Locate the input.** `filePath` may be a local filesystem path **or** an `http(s)` URL.
+   A URL (e.g. a public S3 object) is downloaded via cURL to a temp file first (see
+   *Remote input* below); a local path is used as-is.
+2. **Find the header row.** Compass now prefixes the sheet with a **title-banner row above
+   the header**, so the reader does not assume row 1. `findHeaderRowNumber()` scans for the
+   row whose first cell equals `PERS. NO.` and uses that as the header; the banner above it
+   is ignored.
+3. **Validate headers.** The header row is checked against the 12 `EXPECTED_HEADERS`
+   (`PERS. NO.`, `FIRST NAME`, `LAST NAME`, `USERNAME`, `EMAIL ADDRESS`, `POSITION`,
+   `COST CTR`, `SECTOR`, `COUNTRY`, `LOCATION`, `SUPPORT TECH`, `Support Tech Email`).
+   A missing/reordered column throws.
+4. **Load users.** `loadUserMaps()` loads **every** user (active + inactive) from
+   `Client_Compass.Users` into two maps keyed by lowercased `c_hrEmpUsername` and `email`,
+   carrying the `isActive` flag. `ORDER BY isActive DESC` means an active row wins a
+   duplicate key, and a skip can distinguish "inactive" from "not in the DB".
+5. **Evaluate each data row.** For every row below the header, `evaluateVipRow()` (guard
+   clauses / early return — no `continue`) returns either a staged update
+   `{id, c_supportedByUserId}` or a structured skip `{row, user, reason}`.
+6. **Stop at the first fully-empty row.** `isEntirelyEmptyRow()` (all 12 columns empty) marks
+   the end of the list; the loop breaks and reports `stoppedAtRow`. Rows below it (including a
+   trailing legend/note row) are never processed. A *partial* row (only a support tech, or a
+   note in one column) is **not** a stop.
+7. **Apply.** Staged updates are applied in batches inside one transaction
+   (`buildBatchUpdateSql` — a `CASE id WHEN … THEN …` update); rollback on any failure.
+8. **Clean up the source.** On a successful run (and unless `TEST_MODE`): a local input file
+   is unlinked; a URL input has its **source S3 object deleted** so an input file (which may
+   contain PII) does not linger (see *Source cleanup* below). The downloaded temp copy is
+   always removed in the `finally`.
+9. **Report.** Returns JSON `{updated, skipped, stoppedAtRow, sourceDeleted, skippedRows[]}`.
+   There is no echo logging — the returned JSON is the entire output.
+
+### Matching keys
+
+- VIP identity → `Users.c_hrEmpUsername` (matched against the file's `USERNAME` column).
+- Support technician → `Users.email` (matched against the file's `Support Tech Email` column).
+
+### Skip reasons
+
+Each `skippedRows[]` entry carries the **real spreadsheet row number** (`$row->getRowIndex()`,
+so it lines up with what you see in Excel — not an internal counter offset by the banner/header)
+plus a reason: VIP not in the DB · VIP inactive · VIP has no username in the file (with an
+`(and no support tech email)` note when applicable) · row has no VIP identity to match ·
+support tech email not in the DB · support tech inactive.
+
+### Remote input
+
+`isRemoteUrl()` detects an `http(s)` URL; `downloadToTempFile()` fetches it via cURL
+(`DOWNLOAD_TIMEOUT_SECONDS = 60`, `FAILONERROR`, follow redirects) to a temp file **kept with
+an `.xlsx` suffix** so PhpSpreadsheet's `IOFactory` resolves the reader by extension. The temp
+file is **always removed in a `finally` block**. A local input file is unlinked after a
+successful run (unless `TEST_MODE`).
+
+### Source cleanup
+
+After a successful run, `deleteRemoteSourceIfS3()` deletes the **source** S3 object behind a
+URL input so the input file (which may contain PII) does not linger. `parseS3Url()` extracts
+the bucket, key, **and region** from either a virtual-hosted (`<bucket>.s3.<region>.amazonaws.com`)
+or path-style (`s3.<region>.amazonaws.com/<bucket>/<key>`) HTTPS URL; region defaults to
+`us-east-1` when the URL omits it. The result JSON's `sourceDeleted` field reports the outcome.
+Key decisions:
+
+- **Uses the AWS SDK `S3Client->deleteObject` directly** (`use Aws\S3\S3Client`), **not**
+  `_Cloud::deleteFileFromS3`. That `_underscore` helper shells out via
+  `shell_exec("aws s3 rm …")`, and passing a user-supplied, URL-derived bucket/key into a shell
+  is a command-injection risk (the security rule forbids `shell_exec` with user input). The SDK
+  call avoids the shell entirely.
+- **Client built with the bucket's own region** parsed from the URL, so a cross-region bucket
+  is deleted correctly — the same single-region constraint that breaks `_Cloud::getFileFromS3`
+  on cross-region reads applies to any S3 client that assumes the config-default region.
+- **Best-effort — never throws.** The whole call is wrapped in try/catch; the import already
+  committed its DB updates, so a delete failure (e.g. a missing `s3:DeleteObject` permission) is
+  logged and surfaced as `sourceDeleted: "delete failed: …"` rather than failing the run.
+- **Only on success**, and **skipped in `TEST_MODE`** (so the source is kept for re-runs) and
+  for non-S3 URLs / local paths (`sourceDeleted: "skipped — not an S3 URL"`).
+
+## Client variations
+
+Compass-only. Some spreadsheet rows reference users that belong to a **different Compass tenant
+DB** and are inactive in `Client_Compass` — those rows are skipped with the "not active"
+reason, which is expected, not an error.
+
+## Gotchas / known issues
+
+- **PhpSpreadsheet `RowIterator` rewinds on `foreach`.** `RowIterator::rewind()` resets to
+  `startRow`, and `foreach` always rewinds. An earlier version advanced past the header with
+  manual `next()`/`current()`, but the subsequent `foreach` rewound the iterator, so the header
+  row was silently reprocessed as a (skipped) data row. Fix: use explicit ranges —
+  `getRowIterator($headerRowNum, $headerRowNum)` for the header and
+  `getRowIterator($headerRowNum + 1)` for data.
+- **The file format changes.** The current Compass standard is **12 columns** with a banner row
+  above the header. The previous format was 14 columns (header on row 1) with `Support Tech
+  First`/`Support Tech Last` columns; removing those moved `Support Tech Email` from column N
+  (index 13) to column L (index 11). If Compass changes the layout again, adjust
+  `EXPECTED_COL_COUNT`, `EXPECTED_HEADERS`, `IDX_SUPPORT_EMAIL`, and the `'A'..'L'` cell
+  iterators together.
+- **Why a URL is preferred over a `/tmp` path in production.** The prod EB worker tier runs
+  **more than one instance** and SQS distributes jobs across them, so a file placed in `/tmp`
+  on one instance is invisible to the instance that actually runs the job — a local path fails
+  intermittently with "file not found". A URL fetched at runtime works regardless of which
+  instance runs the job. (Running-app path is `/var/app/current`, even though an SSH session on
+  the AL2023 box shows `/var/www/html`.)
+- **Cross-region / public S3 reads.** `_Cloud::getFileFromS3` is single-region and cannot read
+  a bucket in another region — this importer downloads the object over plain HTTPS instead. See
+  [_Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md).
+
+## Change history
+- 2026-07-15 — On a successful URL-input run, `ProcessVipFile` now deletes the source S3 object
+  (new `deleteRemoteSourceIfS3()` / `parseS3Url()`) so a PII input file does not linger, and
+  returns a `sourceDeleted` field. Uses the AWS SDK `S3Client->deleteObject` directly (bucket's
+  own region from the URL), **not** `_Cloud::deleteFileFromS3`, whose `shell_exec("aws s3 rm …")`
+  would put URL-derived bucket/key into a shell (command-injection risk). Best-effort (never
+  throws), success-only, skipped in `TEST_MODE`. (bala)
+- 2026-07-15 — Adapted to Compass's new 12-column format (banner row above the header;
+  `Support Tech First`/`Support Tech Last` removed, so `Support Tech Email` moved col N→L);
+  added `findHeaderRowNumber()` scanning for `PERS. NO.` and switched to explicit
+  `getRowIterator` ranges to fix the RowIterator-rewind bug that silently reprocessed the
+  header as a data row. (bala)
+- 2026-07-15 — `ProcessVipFile` now accepts an `http(s)` URL (downloaded via cURL to a temp
+  `.xlsx`, always removed in `finally`), because the prod EB worker runs multiple instances and
+  a `/tmp` file on one is invisible to the instance that runs the job. (bala)
+- 2026-07-15 — Added structured per-row skip reporting (real spreadsheet row numbers; reasons
+  distinguishing inactive vs. not-in-DB), all-users-with-`isActive` load, and stop-at-first-
+  empty-row; `ProcessVipFile` returns JSON and echo logging was removed. (bala)

package/knowledge/INDEX.md

@@ -18,7 +18,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
 ## 2.0 framework
 
 - **_underscore** (_Underscore) _(framework core)_ — 32 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
-- **worker2** (Worker) — 29 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
+- **worker2** (Worker) — 30 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
 - **api2** (API) — 10 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
 - **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
 - **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)

package/knowledge/clients/compass-usa/profile.md

@@ -15,13 +15,14 @@ project: _Underscore
 client: compass-usa
 type: profile
 status: active
-updated: 2026-07-08
+updated: 2026-07-15
 owners: [jcardinal, bala, tcox, apeterson]
 files: []
 related:
   - features/asn-to-item-fulfillment.md
   - features/cost-centers.md
   - workflows/cross-kit-bundle-corruption.md
+  - ../../2.0/apps/worker2/features/compass-vip-support-importer.md
   - ../../2.0/apps/toga2-commerce/features/expedited-shipping-gating.md
   - ../../2.0/apps/toga2-commerce/features/cart-bundle-submission-and-identity.md
   - ../../2.0/apps/_underscore/features/item-fulfillment-stage-lifecycle-and-order-status.md
@@ -71,6 +72,9 @@ separate, related client (see its own profile).
 - [Cost Centers (Unit Locations, numeric-only)](features/cost-centers.md) — what a Compass
   "cost center" actually is (a Unit Location keyed by `c_erpSystemEntityId`), and the
   numeric-only selection/import/data policy (2026-07-06).
+- [VIP Support Importer (worker2)](../../2.0/apps/worker2/features/compass-vip-support-importer.md)
+  — manual (Postman) worker2 action that reads Compass's quarterly VIP spreadsheet and sets
+  `Users.c_supportedByUserId` (assigned support tech) per VIP.
 
 ## Notes
 - **Order status is shipped-only (2026-06-30).** Compass imports all IF stages

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toga-ai",
-  "version": "1.0.343",
+  "version": "1.0.345",
   "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
   "keywords": [
     "claude",