toga-ai 1.0.1079 → 1.0.1081

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (27) hide show
  1. package/knowledge/1.0/apps/worker/features/compass-ma-sales-order-exception-report.md +4 -1
  2. package/knowledge/1.0/apps/worker/features/staples-asn-856-outbound.md +4 -3
  3. package/knowledge/1.0/apps/worker/features/staples-cxml-order-import.md +8 -1
  4. package/knowledge/1.0/apps/worker/workflows/running-a-worker-cron-locally.md +3 -2
  5. package/knowledge/2.0/apps/api2/features/request-logging.md +3 -0
  6. package/knowledge/2.0/apps/saml/INDEX.md +1 -0
  7. package/knowledge/2.0/apps/saml/workflows/diagnosing-failed-saml-logins.md +54 -0
  8. package/knowledge/2.0/apps/toga2-commerce/INDEX.md +1 -0
  9. package/knowledge/2.0/apps/toga2-commerce/features/error-codes-and-sentry-signals.md +61 -0
  10. package/knowledge/2.0/apps/toga2-commerce/features/sso-redirect-and-session-gating.md +15 -4
  11. package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
  12. package/knowledge/2.0/apps/worker2/architecture.md +3 -2
  13. package/knowledge/2.0/apps/worker2/features/committed-secrets-and-config-debt.md +5 -2
  14. package/knowledge/2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md +79 -0
  15. package/knowledge/INDEX.md +3 -3
  16. package/knowledge/clients/compass-usa/INDEX.md +1 -0
  17. package/knowledge/clients/compass-usa/features/oneuptime-ma-refresh-order-monitor.md +2 -2
  18. package/knowledge/clients/compass-usa/features/people-file-import-admin-page.md +14 -2
  19. package/knowledge/clients/compass-usa/features/people-file-user-lifecycle.md +6 -2
  20. package/knowledge/clients/compass-usa/features/saml-sso.md +13 -10
  21. package/knowledge/clients/compass-usa/workflows/sso-login-troubleshooting.md +48 -0
  22. package/knowledge/clients/nycdoe/features/asn-sftp-ingestion-2-0.md +15 -8
  23. package/knowledge/clients/nycdoe/features/servicenow-integration.md +6 -3
  24. package/knowledge/clients/nycdoe/profile.md +4 -4
  25. package/knowledge/clients/staples/features/oneuptime-sftp-folder-monitors.md +19 -2
  26. package/knowledge/clients/staples/profile.md +3 -1
  27. package/package.json +1 -1
@@ -14,7 +14,10 @@ files:
14
14
  - worker/crons/toga2/netsuite/common_sync_togasupply.php
15
15
  - library/app/api/netsuite/rest.php
16
16
  - library/app/api/toga2.php
17
+ - library/app/error/capture.php
17
18
  related:
19
+ - ../../library/features/error-capture-1-0.md
20
+ - ../../../standards/backend-php.md
18
21
  - ../workflows/tracing-a-worker-cron-run-in-production.md
19
22
  - ../workflows/running-a-worker-cron-locally.md
20
23
  - ../../library/features/netsuite-closed-order-rule.md
@@ -25,7 +28,7 @@ related:
25
28
  - ../../library/features/email-test-mode-and-write-free-sends.md
26
29
  ---
27
30
 
28
- Worker cron emailing ops the "Compass Refresh Exception Report" of `MA%` Compass sales orders whose ODP NetSuite order isn't keyed yet; open for the two-hop bridge query, the synced-copy and NetSuite closed/cancelled lookups that drop rows, the accepted no-reopen scope, the NULL-safe stage filter, and the "prove the run not the email" rule.
31
+ Worker cron emailing ops the "Compass Refresh Exception Report" of `MA%` Compass sales orders whose ODP NetSuite order isn't keyed yet; open for the two-hop bridge query, the synced-copy and NetSuite closed/cancelled lookups that drop rows, the accepted no-reopen scope, the NetSuite-down-still-emails exception, the NULL-safe stage filter, and the "prove the run not the email" rule.
29
32
 
30
33
  ## Summary
31
34
  A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%`
@@ -6,8 +6,8 @@ project: Worker
6
6
  client: staples
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-10-14
10
- owners: [mhammontree]
9
+ updated: 2026-10-19
10
+ owners: [mhammontree, bala]
11
11
  files:
12
12
  - worker/crons/sync/staples/staples_asn_netsuite.php
13
13
  - dbchanges/Common/MH/2026-09-30.sql
@@ -69,8 +69,9 @@ Expected (2026-10-01 run): first run 101 rows, 90 matched as already sent by the
69
69
  - **Old design (pre-TRUE-82180) — why ASNs were late and doubled.** It searched only fulfillments modified in the last 3600 s, so each shipment got one chance; a skipped one went out only when someone edited the fulfillment (days/weeks late, ~half of ASNs). Dedupe checked only `/archive` by prefix `856_<PO>_<n>` where `n` restarted every run, and every fulfillment on the SO was re-sent when one changed.
70
70
  - **Do not use `Common.CronJobExecutions` or `Logs.FileLog` as a watermark/dedupe source** — both are too big and unindexed for per-run lookups (see [Cron execution monitoring](../../library/features/cron-execution-monitoring.md) and the FileLog note below). That is why the dedicated table exists.
71
71
  - **`Logs.FileLog` has no index on `job` / `fileName`** (verified locally) and `Logs` is ~400 GB — never `LIKE`-scan its BLOB per record; it cannot be pulled locally in full.
72
- - **Hardcoded credentials (parked).** The SFTP password and the cXML SharedSecret are hardcoded in `staples_asn_netsuite.php`; they should move to config.
72
+ - **Hardcoded credentials (parked).** The SFTP password and the cXML SharedSecret are hardcoded in `staples_asn_netsuite.php`; they should move to config. The same SFTP login is in two other places; rotate them together — see the [Staples profile](../../../clients/staples/profile.md).
73
73
 
74
74
  ## Change history
75
+ - 2026-10-19 — TRUE-82897: linked the hardcoded SFTP password to the shared rotate-together note on the Staples profile. No code changed. (bala)
75
76
  - 2026-10-01 — First doc. TRUE-82180 rewrite: `StaplesAdvanceShipNotices` queue with 7-day lookback, shipping-hash dedupe, per-row retry/FAILED, 24 h alert, test mode never uploads, old-job match for pre-go-live shipments. Committed worker `cf5198a27`, dbchanges `8b1f7cb` on branch TRUE-82180; not deployed as of 2026-10-01. (mhammontree)
76
77
  - 2026-10-14 — Added "Checking a missing-ASN report" (TRUE-82655 investigation: all reported ASNs were sent, client checked before Shipped). (mhammontree)
@@ -6,13 +6,14 @@ project: Worker
6
6
  client: staples
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-10-16
9
+ updated: 2026-10-19
10
10
  owners: [jcardinal, mhammontree, bala]
11
11
  files:
12
12
  - worker/crons/sync/staples/sync_staples_cxml.php
13
13
  related:
14
14
  - clients/staples/profile.md
15
15
  - 1.0/apps/worker/features/netsuite-sales-order-sales-rep-sourcing.md
16
+ - 2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md
16
17
  ---
17
18
 
18
19
  Hourly cron importing Staples cXML POs from SFTP into NetSuite Sales Orders (writes 855 ack back); open for the per-file flow, saved-search-8412 Location resolution, new-account onboarding, how to diagnose an order that never appeared (incl. SOs deleted by hand after import), and the silent add-failure trap.
@@ -81,6 +82,10 @@ This also separates the two failure modes: FileLog is written before the NetSuit
81
82
 
82
83
  **The `salesRep` assignment on this cron is dead code.** `$salesOrder->salesRep = 'Aaron Myall';` (~line 138) assigns a plain **string** to a field NetSuite types as a **RecordRef**; NetSuite silently discards it. SO **286680** came out with **Paulina Szeliga**, parent 30057's rep, because `entity` is always the parent. Also, the `//sales rep` comment above the `custbody_ctc_inside_sales_transaction` block is **wrong** — that block sets **Inside Sales Support** (hardcoded Nicholas Vance, internalId 497). Full mechanics in [NetSuite Sales Order Sales Rep Sourcing](netsuite-sales-order-sales-rep-sourcing.md).
83
84
 
85
+ **The SFTP box is shared.** `sftp.goagilant.com` is the same single server as NYCDOE's `ftp.asisystem.com` and Compass's PEOPLE file; it bans by source IP. 1.0 uses its own IP, so this cron kept working in the Oct 17-19 2026 worker2 ban. See [Shared SFTP server](../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md).
86
+
87
+ **Hardcoded SFTP password (open).** The password is committed at ~line 20; it should move to config and be rotated together with its other two copies — see the [Staples profile](../../../clients/staples/profile.md).
88
+
84
89
  **Recommended hardening (NOT yet implemented):**
85
90
  - Add the missing `else` to log the NetSuite `statusDetail` and email add failures (mirror the existing "missing items" email), leaving the file for retry.
86
91
  - Pre-guard: if `$internalClientLocationId` is NULL, skip the doomed add and alert "Staples customer `<CustomerID>` not set up in NetSuite", naming the exact id.
@@ -91,6 +96,8 @@ This also separates the two failure modes: FileLog is written before the NetSuit
91
96
  - [NetSuite Sales Order Sales Rep Sourcing](netsuite-sales-order-sales-rep-sourcing.md)
92
97
 
93
98
  ## Change history
99
+ - 2026-10-19 — TRUE-82932: linked the shared SFTP server doc (same box as NYCDOE + Compass). (bala)
100
+ - 2026-10-19 — TRUE-82897: recorded the committed SFTP password as an open security gap (move to config + rotate). This cron kept reading `/in` while the worker2 monitor was blocked, see the [monitor doc](../../../clients/staples/features/oneuptime-sftp-folder-monitors.md). No code changed. (bala)
94
101
  - 2026-10-16 — TRUE-82884: 37 of 70 "missing" 10/8 orders were created then deleted by hand (no hardware lines). Added the deleted-SO diagnostic (855 present + SO missing, `tranId` gap method, deleted-records report), SOAP lookup, FileLog columns, and the no-hardware-check gotcha. No code changed. (mhammontree)
95
102
  - 2026-09-18 — Production debug of account `0ZZZMICR` (Ingram Micro) confirmed the blank `custentity_customer_location` root cause end to end. Added the diagnostic steps (FileLog `fileData LIKE`, success-only 855/archive, SuiteQL on `otherrefnum`, saved search 8412), the multi-select single-entry + parent-Location-36 setup rule, the duplicate external-customer-id failure mode, and real retry counts (435KNN x674, order lost; 955KVL x28, dedupe held). No code changed. (bala)
96
103
  - 2026-08-21 — Recorded that `$salesOrder->salesRep = 'Aaron Myall'` is dead code and that the adjacent `//sales rep` comment actually describes Inside Sales Support (Nicholas Vance, 497). Removed a departing employee from the sibling cron's recipient list. No behavior change. (mhammontree)
@@ -55,7 +55,8 @@ For logic-only tests with no bootstrap, use the
55
55
 
56
56
  ### Step 3 — stub the outside world
57
57
  - **NetSuite:** a mock REST server with `php -S 127.0.0.1:<port>`. It must also answer
58
- `/services/rest/auth/oauth2/v1/token`, not just the SuiteQL endpoint.
58
+ `/services/rest/auth/oauth2/v1/token`, not just the SuiteQL endpoint. Test the failure path
59
+ too: have the mock return 500, then stop it (unreachable).
59
60
  - **Email:** declare a stub `App_Email_Agilant` class **before** `require '_.php'` so the
60
61
  autoloader never loads the real one; the stub records what would have been sent.
61
62
  - **Sentry:** block outbound HTTPS, e.g. `HTTPS_PROXY` pointed at a dead local port with
@@ -84,4 +85,4 @@ hits an `Iterator` return-type deprecation, which `App_Error` turns into an exit
84
85
 
85
86
  ## Change history
86
87
  - 2026-10-16 — Created from the cron 7 local test harness (prod-pointing configs, Sentry, SMTP,
87
- PHP 7.2, Windows `:` file names). (bala, TRUE-82888)
88
+ PHP 7.2, Windows `:` file names, NetSuite 500/unreachable test). (bala, TRUE-82888)
@@ -227,9 +227,12 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
227
227
  - **Query prod `Logs.Api` by narrowing on `dtStamp` (indexed) first**, collect ids, then fetch `mediumtext` payloads by id with `LEFT()`/`SUBSTRING()`. Never start from a `LIKE` over `requestPayload`/`responsePayload`.
228
228
  - **Never copy logged rows into committed artifacts** — they carry live Bearer tokens and PII.
229
229
  - **Timing: `Logs_*.Api.dtStamp` is the request START time; `executionTimeSeconds` misses part of the cost.** Measured 2026-10-16 (Prudential, TRUE-82596): every GET ~1.1s `executionTimeSeconds` (even `/announcements` with 3 rows), plus ~1.5–1.9s more between back-to-back calls outside that timer; PUT/POST ~0.2s. Likely api2 bootstrap and/or the synchronous Logs write — open follow-up. To measure a caller's real wait, diff consecutive `dtStamp`s, not `executionTimeSeconds`.
230
+ - **⚠ A row under a user's `userId` is not proof that user's browser sent it.** View-as / delegator sessions (`POST /v2/auth/delegator`) and staff password logins as the user all log under the TARGET user's `userId`. To trace the user's own browser, filter on their `X-Forwarded-For` value inside `requestHeaders`, inside a `dtStamp` range (a week-wide `LIKE` timed out at 300 s), and check User-Agent before concluding "they logged in". `dtStamp` is server-local time (-05:00).
231
+ - **A browser "Network Error" leaves no row at all** — the request never reached api2. See [commerce Sentry signals](../../toga2-commerce/features/error-codes-and-sentry-signals.md).
230
232
  - **`Logs_<Client>.Api` is pruned (once TRUE-81902 deploys)** — GET rows after 30 days, writes after 90. A missing older row is not proof the request never happened. See [Logs retention cleanup](../../worker2/features/logs-retention-cleanup.md).
231
233
 
232
234
  ## Change history
235
+ - 2026-10-16 — Gotchas: delegator/staff logins log under the target `userId` (trace a browser by XFF in `requestHeaders`); `dtStamp` is server-local; Network Errors leave no row. (bala)
233
236
  - 2026-10-16 — Timing gotcha: `dtStamp` = request start; GETs ~1.1s logged + ~1.5–1.9s unlogged between calls. (rgirish)
234
237
  - 2026-10-05 — Recorded the beta `originalUserId` outage: an undeclared model property on the deployed `_Model_Client_Logs_Api` 500s every client-logged request (639, incl. `/v2/auth/api`); the column migration does not fix it and 401s stay healthy throughout. (bala)
235
238
  - 2026-09-25 — Linked the `Logs_<Client>.Api` retention job (GET 30d / writes 90d). (rgirish)
@@ -8,5 +8,6 @@
8
8
  | [SAML gateway per-environment hosts & redirect safety (samlHostUrl / safeRedirectUrl)](features/gateway-environments-and-redirect-safety.md) | Which SAML gateway host serves which environment, and how `/acs` decides where it is allowed to send the login handoff. |
9
9
  | [SAML Uptime Healthcheck Endpoint (/healthcheck)](features/healthcheck-endpoint.md) | Secret-gated `GET /healthcheck` synthetic-SSO probe on the saml gateway; open when working on SSO uptime monitoring or the probe's seeds/guardrails. |
10
10
  | [Rate SAML User Provisioning](features/rate-user-provisioning.md) | How a Rate SSO login finds or just-in-time provisions the user record chain in `Client_Rate`; open when touching Rate SSO or its user/customer/contact models. |
11
+ | [Diagnosing failed SAML logins on saml-production (where the evidence is, and how to decode /acs)](workflows/diagnosing-failed-saml-logins.md) | Where to find evidence of a failed SSO login on `saml-production`, which has no log streaming, and how to read a failed `POST /acs` from its byte size; open whe |
11
12
  | [Onboarding a New SSO Client with Just-In-Time User Provisioning](workflows/onboarding-a-new-sso-client.md) | Procedure to add a new client to the 2.0 saml gateway when its users don't pre-exist and must be created on first login (JIT provisioning); open when onboarding |
12
13
  | [Rotating the SAML SP Certificate](workflows/rotating-the-saml-sp-certificate.md) | Procedure to rotate the TOGa SAML Service Provider (SP) cert + private key; open when a SP cert is expiring, a partner needs a public cert, or you're editing th |
@@ -0,0 +1,54 @@
1
+ ---
2
+ title: Diagnosing failed SAML logins on saml-production (where the evidence is, and how to decode /acs)
3
+ framework: "2.0"
4
+ repo: saml
5
+ project: SAML SSO Gateway
6
+ client: shared
7
+ type: workflow
8
+ status: active
9
+ updated: 2026-10-16
10
+ owners: ["bala"]
11
+ files:
12
+ - saml/Controller/Index.php
13
+ - _underscore/Model/Compass/ClientAuthentication.php
14
+ related:
15
+ - ../architecture.md
16
+ - ../features/downstream-integration-contract.md
17
+ - ../features/healthcheck-endpoint.md
18
+ - ../../../../clients/compass-usa/features/saml-sso.md
19
+ - ../../../../clients/compass-usa/workflows/sso-login-troubleshooting.md
20
+ ---
21
+
22
+ Where to find evidence of a failed SSO login on `saml-production`, which has no log streaming, and how to read a failed `POST /acs` from its byte size; open when a user says "SSO does not let me in".
23
+
24
+ ## Summary
25
+
26
+ `saml-production` (EB env, us-east-1, account 654654170868) has almost no observability:
27
+ - **No CloudWatch log streaming** (no `/aws/elasticbeanstalk/saml*` log groups).
28
+ - **ALB access logs disabled.**
29
+ - **Instances are not SSM-managed** (no Session Manager shell).
30
+
31
+ Only two sources exist: the EB **Request logs** bundle, and the Sentry project **`saml`**.
32
+
33
+ ## Steps
34
+
35
+ 1. **Sentry `saml` first.** Every exception from `getAuthenticatedSsoUser()` is captured. Compass failures arrive as `User not found: <nameId>` (from `_underscore/Model/Compass/ClientAuthentication.php`). Search the user's NameID (for Compass, their `c_hrEmpUsername`).
36
+ 2. **EB console → `saml-production` → Logs → Request logs → Full logs.** The bundle holds Apache httpd `access_log` / `error_log` only, about 3.5 days of history. It does **not** contain PHP `error_log()` output. The client IP is the ALB IP (no XFF) and there is no username, so match by time.
37
+ 3. **Decode each `POST /acs` in `access_log` by status and body size:**
38
+
39
+ | Response | Meaning |
40
+ |---|---|
41
+ | `302` | success, browser sent back to the app |
42
+ | not 302, 31 bytes | `Invalid SAMLResponse (Code = N)` |
43
+ | not 302, `16 + len(nameId) + 26` bytes | `User not found: <nameId>User Authentication Failed` |
44
+
45
+ 4. **Compare to the baseline.** 11-14 Oct 2026: 861 `/acs`, 839 success, 22 failures. A user with no failed `/acs` in their window never got to the gateway; look upstream (browser, network, commerce sign-in screen).
46
+
47
+ ## Gotchas
48
+
49
+ - **Git Bash mangles `/aws/...` CLI arguments** into Windows paths. Prefix the command with `MSYS_NO_PATHCONV=1` (e.g. for `aws logs describe-log-groups --log-group-name-prefix /aws/...`).
50
+ - **The bundle rolls over in ~3.5 days.** Pull it the same day the user reports the problem.
51
+ - **No PHP logs on disk to read.** Anything the controller only `error_log()`s is lost; only what reaches Sentry survives.
52
+
53
+ ## Change history
54
+ - 2026-10-16 — Created: saml-production has no CloudWatch / ALB logs / SSM; evidence = EB request-log bundle + Sentry `saml`; `/acs` byte-size decode; Oct 2026 baseline. TRUE-82716 investigation. (bala)
@@ -12,6 +12,7 @@
12
12
  | [Catalog cache freshness — the persisted query cache, per-query staleTime overrides, and focus refetch](features/catalog-cache-freshness.md) | The persisted React Query cache (`localStorage["commerce"]`), the per-query `staleTime` overrides that served prices up to 24h old, focus refetch, how to opt a |
13
13
  | [Category Tile Order (AssortmentItems.sortOrder) — merchandising a storefront category](features/category-tile-sort-order.md) | Storefront category tile order is one DB column (`AssortmentItems.sortOrder` ASC) with a NULL-sorts-first trap; open when asked to reorder items on a category p |
14
14
  | [Client Fields — per-tenant / language / role content & config](features/client-fields.md) | Per-tenant/language/role JSON config that drives almost all commerce UI text, fields, and login fetches (Layer A global + Layer B FIELDS registry); open before |
15
+ | [Commerce error screens (CODE E0-1) & Sentry signals (Network Error vs 401)](features/error-codes-and-sentry-signals.md) | What a commerce "CODE: E0-1" screen means, how to tell an API failure from a render crash, and how to read commerce Sentry "Network Error" vs 401 issues; open w |
15
16
  | [Config-Driven Expedited Shipping Gating (Cart)](features/expedited-shipping-gating.md) | Expedited shipping options are config-gated to show only when the cart holds a computer kit; open when touching cart shipping-method options or the gate config. |
16
17
  | [Filter / Search-Results Page & the Two Search Entry Points](features/filter-search-results-page.md) | Two search entry points (Home in-place vs Header→/filter) render the same card via different code paths and FIELDS files; open for any search-results or kit-car |
17
18
  | [Force Logout on Deployment (DeploymentGuard) — commerce](features/force-logout-on-deployment.md) | Signs every user out when a deploy bumps a Core parameter, after showing a notice; open for surprise logouts, "why wasn't I logged out?", a repeating "new versi |
@@ -0,0 +1,61 @@
1
+ ---
2
+ title: Commerce error screens (CODE E0-1) & Sentry signals (Network Error vs 401)
3
+ framework: "2.0"
4
+ repo: toga2-commerce
5
+ project: TOGa Commerce
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-10-16
10
+ owners: ["bala"]
11
+ files:
12
+ - toga2-commerce/src/components/ErrorMessages/components/FiveHundredContent.tsx
13
+ - toga2-commerce/src/components/ErrorMessages/components/UserSettingsError.tsx
14
+ - toga2-commerce/src/pages/Home/viewModel/useHomeViewModel.ts
15
+ - toga2-commerce/src/routes.tsx
16
+ - toga2-commerce/src/api/axiosInstance.ts
17
+ related:
18
+ - ./sso-redirect-and-session-gating.md
19
+ - ../architecture.md
20
+ - ../../api2/features/request-logging.md
21
+ - ../../../../clients/compass-usa/workflows/sso-login-troubleshooting.md
22
+ ---
23
+
24
+ What a commerce "CODE: E0-1" screen means, how to tell an API failure from a render crash, and how to read commerce Sentry "Network Error" vs 401 issues; open when a user sends an E0-1 screenshot or "the site will not load".
25
+
26
+ ## Summary
27
+
28
+ `CODE: E0-1` is a hardcoded string, not a server code. It does **not** identify the failing call. The screenshot tells you which of two cases it is:
29
+
30
+ | Screen | Cause |
31
+ |---|---|
32
+ | Header + "Welcome, <name>" banner still visible | Home's bundles and/or assortments query failed (API failure) |
33
+ | No header | `AuthLayout` route `errorElement` caught a render crash |
34
+
35
+ In commerce Sentry, `AxiosError: Network Error` means **the browser never reached api2** (no status code). A status in the message (e.g. `401`) means api2 answered.
36
+
37
+ ## How it works
38
+
39
+ - `FiveHundredContent.tsx:76` renders the literal `CODE: E0-1`.
40
+ - **Case (a), API failure:** `useHomeViewModel.ts` (~L299-331) sets the page error when `assortmentsError` and/or `bundlesError` is set. An error with no status (a string, or a Network Error) defaults to `500`, so it renders the 500 screen inside the layout. Header and welcome banner stay.
41
+ - **Case (b), crash:** `routes.tsx:124`, `AuthLayout`'s `errorElement` is `<ErrorMessage errorType={500}/>`. It replaces the whole layout, so no header.
42
+ - `UserSettingsError.tsx` builds other codes as `E<context>-<setting><op>-<status>`; with no error recorded it also falls back to `E0-1`.
43
+
44
+ ### Sentry issues to know (project `commerce`)
45
+
46
+ | Issue | Message | Meaning |
47
+ |---|---|---|
48
+ | COMMERCE-21 | `Error fetching assortments (status AxiosError: Network Error)` | Home assortments call never reached api2 → E0-1 case (a) |
49
+ | COMMERCE-22 | `Error fetching bundles (status AxiosError: Network Error)` | same, bundles |
50
+ | COMMERCE-1S | `Error validating domain (status AxiosError: Network Error)` | user typed an email on a password form shown because the SSO lookup failed (see [SSO sign-in](./sso-redirect-and-session-gating.md)) |
51
+ | — | `Failed to fetch refresh token` (`axiosInstance.ts:166`) | refresh call failed; session ends |
52
+
53
+ ## Gotchas
54
+
55
+ - **⚠ A Network Error leaves NOTHING in `Logs_<Client>.Api`.** The request never arrived. An empty API log for the user's time window is expected, not proof they were not using the site.
56
+ - **Search commerce Sentry by `user.ip`.** The Sentry user is often only the IP; `user.email` / `user.id` are not set.
57
+ - **Quick reachability test for an end user:** open `https://api1.togahub.com` in their browser. A reachable API answers JSON `{"status":401,"error":{"code":"EV-2"...}` (no `/v2` path needed). A block page or timeout means their network blocks it. Also test `https://api-writer.togahub.com` (login) and `https://saml.togahub.com` (SSO). Blocks can be intermittent (ISP or corporate filter), so one good test does not clear it.
58
+ - **⚠ Open (not ticketed) — expired sessions land on E0-1 instead of sign-in.** Commerce Sentry, 14 days to 2026-10-14: `Failed to fetch refresh token` 682 events / 576 IPs, `Error fetching assortments (401)` 648 / 581 IPs, `Error fetching bundles (401)` 589 / 531 IPs. The Home 401s show the E0-1 page rather than sending the user back to sign in.
59
+
60
+ ## Change history
61
+ - 2026-10-16 — Created: E0-1 is hardcoded (API failure vs crash by the header), Network Error = never reached api2, Sentry issue map, `user.ip` search, end-user reachability test, open 401 → E0-1 issue. TRUE-82716 investigation. (bala)
@@ -6,13 +6,15 @@ project: TOGa Commerce
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-09-25
10
- owners: [tcox, apeterson, rgirish]
9
+ updated: 2026-10-16
10
+ owners: [tcox, apeterson, rgirish, bala]
11
11
  files:
12
12
  - toga2-commerce/src/lib/session.ts
13
13
  - toga2-commerce/src/hooks/useAuthenticationFlow.ts
14
14
  - toga2-commerce/src/hooks/useClientSsoUrl.ts
15
+ - toga2-commerce/src/pages/Login/LoginPage.tsx
15
16
  - toga2-commerce/src/pages/Login/view/SsoSignIn.tsx
17
+ - toga2-commerce/src/lib/signInMethod.ts
16
18
  - toga2-commerce/src/api/axiosInstance.ts
17
19
  - toga2-commerce/src/contexts/AuthContext.tsx
18
20
  - toga2-commerce/src/lib/authStorage.ts
@@ -20,6 +22,7 @@ files:
20
22
  related:
21
23
  - ./catalog-cache-freshness.md
22
24
  - ./view-as-session-tokens.md
25
+ - ./error-codes-and-sentry-signals.md
23
26
  - ../workflows/unit-testing-vitest.md
24
27
  - ../architecture.md
25
28
  - ../../toga25-supply/features/sso-redirect-and-session-gating.md
@@ -39,9 +42,10 @@ One predicate answers "is the user signed in?": **`hasSession()` in `src/lib/ses
39
42
  | Piece | File | Question |
40
43
  |---|---|---|
41
44
  | axios request interceptor | `src/api/axiosInstance.ts` | user Bearer or public token? |
42
- | `useAuthenticationFlow` | `src/hooks/useAuthenticationFlow.ts` | signed in → nothing; else `/login` |
45
+ | `useAuthenticationFlow` | `src/hooks/useAuthenticationFlow.ts` | signed in → nothing; else `/` |
43
46
  | `useClientSsoUrl` | `src/hooks/useClientSsoUrl.ts` | is this tenant SSO, and what URL? |
44
47
  | `LoginPage` | `src/pages/Login/LoginPage.tsx` | render `SsoSignIn` or the credential form |
48
+ | `signInMethod` | `src/lib/signInMethod.ts` | where does sign-out send the user? |
45
49
 
46
50
  ## How it works
47
51
 
@@ -53,7 +57,13 @@ One predicate answers "is the user signed in?": **`hasSession()` in `src/lib/ses
53
57
 
54
58
  `useClientSsoUrl` is a React Query (so the auth flow and the login page share one cached answer): `GET /domains` joined to `ClientAuthentications` + `Environments`, matching `Domains.domain = "<protocol>//<hostname>"` and `Environments.slug` = build `MODE` uppercased, requiring `ClientAuthentications.type === "SSO"`.
55
59
 
56
- Unauthenticated visitors always go to **`/login`**. `/login` renders `SsoSignIn` for an SSO tenant, else the credential form. Copy comes from `Login/viewModel/FIELDS/LOGINDUMMYFIELDS.json`. SSO tenants **no longer auto-jump to the IdP** — the user sees a page and clicks. Every logout path already lands on `/login`, so sign-out needs no extra SSO code.
60
+ Unauthenticated visitors go to **`/`**, the sign-in entry point (`routes.tsx` `PrivateRoute` / `AuthRedirect`). `/` runs the lookup and renders `SsoSignIn` for an SSO tenant, else the credential form. **`/login` is the credential form always** (`isCredentialFormOnly`, lookup disabled), for internal password sign-in on an SSO tenant; it is not linked anywhere. Copy comes from `Login/viewModel/FIELDS/LOGINDUMMYFIELDS.json`. SSO tenants **do not auto-jump to the IdP**; the user clicks.
61
+
62
+ **A failed lookup silently shows the credential form.** `useClientSsoUrl` uses `retry: false`; on any error `isSsoClient` is false and `LoginPage` (L46-61, L172) renders the generic email form, with no warning. The lookup failure itself is **not** sent to Sentry. If the user then submits an email, `validateEmailDomain` fails (`useLoginPageViewModel.ts` ~L48) → Sentry `Error validating domain (status AxiosError: Network Error)` (COMMERCE-1S, extra `Validate domain API Error`) and the user sees "Error validating domain. Please try again." So **COMMERCE-1S on url `/` = an SSO-tenant user was shown the password form and tried it**, usually because their network cannot reach the API (see [error screens & Sentry signals](./error-codes-and-sentry-signals.md)). Possible improvement, not built: show a "can't connect" message when the lookup errors.
63
+
64
+ ### Sign-out destination (`signInMethod`)
65
+
66
+ Sign-out (`AccountPage` → `logout()` → `navigate(takeSignInEntryPath())`) returns the user to the screen they signed in on. `sessionStorage.signInMethod` is set to `CREDENTIALS` on any credential-form sign-in and `SSO` on the SAML handoff. `CREDENTIALS` → `/login`; `SSO` or empty → `/`. Reading it clears it. So a developer who signed in on `/login` lands back on `/login`; an SSO user lands on `/`. Expected (TRUE-81991), not a bug. sessionStorage because it survives `clearAuthStorage()` and dies with the tab.
57
67
 
58
68
  ### Single sign-out state: `AUTH_STORAGE_KEYS` + `resetClientState()`
59
69
 
@@ -86,6 +96,7 @@ api2 **rotates** the refresh token. Concurrent 401s each spent a token the other
86
96
  - **No per-session key prefix, and tokens stay in `localStorage`.** Renaming every zustand `persist` name silently orphans existing users' state on deploy and touches every call site; supply's cheaper answer is plain `localStorage.clear()` on logout since every key commerce writes is session state. httpOnly cookies would need backend work plus CSRF protection that does not exist — api2 is Bearer-only. Matches `2.0/standards/frontend.md` §12.
87
97
 
88
98
  ## Change history
99
+ - 2026-10-16 — Corrected entry point (`/` resolves the tenant, `/login` is credential-only); documented the silent credential-form fallback on a failed SSO lookup (= COMMERCE-1S) and the `signInMethod` sign-out destination. TRUE-82716 investigation. (bala)
89
100
  - 2026-09-25 — Recorded that `GET /domains` is cached (`Core.Records` id 1, `ttlCache = 15`), the likely reason freshly inserted SSO rows still fall through to `/login` (rgirish)
90
101
  - 2026-09-25 — Deep links now survive sign-in: `PrivateRoute` saves before redirecting, one consumer per path, `onSuccess` navigates removed; header/footer logos point at `/home`. Known gap: the saml gateway always returns to `/`. TRUE-81991. (apeterson)
91
102
  - 2026-09-22 — Public token moved out of `localStorage`; `hasSession()` + identity epoch + `AUTH_STORAGE_KEYS` added; SSO sign-in screen replaces the auto IdP jump; open-redirect and SSO-URL-scheme holes closed; single-flight refresh. TRUE-81991 (`e546db3d`). (apeterson)
@@ -55,6 +55,7 @@
55
55
  | [S3 folder retention cleanup worker action (CleanS3Folder)](features/s3-folder-retention-cleanup.md) | Cron-parameterized worker action that deletes S3 objects under one folder older than a retention window; open when adding/tuning an S3 retention sweep or its de |
56
56
  | [The two sandbox-dev worker2 environments (and how `_underscore` reaches a worker box)](features/sandbox-dev-worker-environments.md) | Two Elastic Beanstalk environments both call themselves "sandbox-dev" worker, they run **different `ENVIRONMENT` values**, they clone **different `_underscore` |
57
57
  | [Service Request → Sales Order → Purchase Order generation (Sync/ServiceRequest)](features/service-request-sales-order-generation.md) | Worker `_Worker_Sync_ServiceRequest` that turns a Service Request → Sales Order → one PO per vendor for any tenant; open for the SR→SO→PO chain, its interceptor |
58
+ | [Shared SFTP server (sftp.goagilant.com) — one box, source-IP ban, worker2 egress IP](features/shared-sftp-server-source-ip-ban.md) | One Azure SFTP box serves Staples, Compass and NYCDOE, and it bans by source IP; all worker2 traffic shares one IP, so one noisy worker2 job takes every worker2 |
58
59
  | [SSO Stability Monitor (Monitor/Operations/SsoStability)](features/sso-stability-monitor.md) | Reporter cron that GETs the saml `/healthcheck` and pushes an SSO heartbeat to a OneUptime monitor; open when working on the SSO uptime probe, its tokens, or it |
59
60
  | [Startech Entitlement Provisioning Monitor (Monitor/Startech/EntitlementProvisioning)](features/startech-entitlement-provisioning-monitor.md) | OneUptime monitor that every new entitlement on a Startech-trait client got a 2xx `/sku` call; open when it alarms or when a new client adopts `_Trait_Startech_ |
60
61
  | [Startech Webhook Handler (worker2)](features/startech-webhook-handler.md) | Inbound Startech (Easeedesk) webhook handler that creates/updates the matching TOGA 2.0 ticket; open for the webhook flow, its payload, or ticket-type/stage res |
@@ -6,8 +6,8 @@ project: Worker
6
6
  client: shared
7
7
  type: architecture
8
8
  status: active
9
- updated: 2026-10-06
10
- owners: [jcardinal, ajean, rgirish]
9
+ updated: 2026-10-19
10
+ owners: [jcardinal, ajean, rgirish, bala]
11
11
  files:
12
12
  - worker2/Controller/Index.php
13
13
  - worker2/Worker/
@@ -46,6 +46,7 @@ Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** app that pro
46
46
  | SQS `WebhookFallback` | S3 references for webhook payloads that failed the MySQL insert. |
47
47
  | S3 `agilant-worker-fallback` | Failed webhook payloads (`webhook-fallback/{uuid}.json`, 7-day lifecycle). |
48
48
  | API Gateway `WorkerWebhookIngestion` | `webhook.togahub.com` → Lambda. Lambda Function URLs are blocked by an AWS Org SCP, so API Gateway is the only public entry point. |
49
+ | Outbound (egress) IP | All prod outbound traffic leaves from **`34.232.23.158`** (same IP is the `sourceIp` on api2 `Logs_<Tenant>.Api` rows for worker2-made calls). Prod instances have **no public IP**. They run in the shared subnet `subnet-0e135d4093ab60b2f` (`true-production-private-web-1c`, VPC `vpc-08030daf1c2dad280`), owned by the network AWS account `339712893798`. Its `0.0.0.0/0` route goes to NAT `nat-0e2c874a7a3740e7a`, which is **not visible** from the `toga-production` SSO profile (`describe-nat-gateways` returns NotFound); changing it needs network-account access (Jeff). Any third party that allowlists worker2 needs this IP. |
49
50
 
50
51
  ## No SQL migration files in this repo
51
52
 
@@ -6,8 +6,8 @@ project: Worker
6
6
  client: shared
7
7
  type: feature
8
8
  status: active
9
- updated: 2026-10-12
10
- owners: ["kyalamarthi", "ajean", "mhammontree", "jcardinal"]
9
+ updated: 2026-10-19
10
+ owners: ["kyalamarthi", "ajean", "mhammontree", "jcardinal", "bala"]
11
11
  files:
12
12
  - worker2/.gitignore
13
13
  - worker2/Worker/Team/Github.php
@@ -47,6 +47,8 @@ the publish on a credential literal.
47
47
  | TOGa Talos (`api.togaiq.com`) API key — **old** release-notes key | git history only: `worker2/Worker/Team/Sprint.php` before PR #179 (still visible at commit `8613ace`) | Code no longer uses it, but it was committed, so it **must be revoked** in TOGa Talos. Open until revoked. |
48
48
  | TOGa Talos API key — **current** (`[talos] api_key`) | `worker2/Config/production.ini` `[talos]` group (~L230) | Shared by release notes and the transcript pipeline. Committed like every `Config/*.ini` (see Summary). |
49
49
  | AWS access key (`[talos]` group) | `worker2/Config/*.ini` `[talos]` | Open — not part of TRUE-82705; rotate + replace with the instance role. |
50
+ | Compass PEOPLE-file SFTP password (`cgsftp`) | `worker2/Worker/Client/Compass/PeopleFile.php` constant `SFTP_PASSWORD` | Open — move to Secrets Manager, rotate. |
51
+ | Other accounts on the **same** SFTP server (1.0 copies) | NYCDOE `FTP_PASSWORD` in `library/app/asnprocessor/{apple,lenovo,lexmark,acer}.php`; Staples in `worker/crons/sync/staples/{sync_staples_cxml.php,staples_asn_netsuite.php}` (worker2's copy is the `[sftp_staples]` ini group) | Open — move to Secrets Manager, rotate. The 2.0 NYCDOE import already reads `sftp/nycdoe/<mfr>`; rotate each account in all its places at once. Server: [Shared SFTP server](./shared-sftp-server-source-ip-ban.md). |
50
52
  | PayPal webhook signature verification **off** in production | `worker2/Config/production.ini:182` | Impact already written up — a caller who knows the URL can create a paid SalesOrder. See [Rate PayPal subscription webhook](../../../../clients/rate/features/paypal-subscription-purchase-webhook.md). |
51
53
 
52
54
  ### Secrets stored at runtime in `Core.WorkerJobs` (not committed, same exposure)
@@ -95,6 +97,7 @@ the publish on a credential literal.
95
97
  - [Rate PayPal Subscription Purchase & Webhook Pipeline](../../../../clients/rate/features/paypal-subscription-purchase-webhook.md)
96
98
 
97
99
  ## Change history
100
+ - 2026-10-19 — Added the shared SFTP server passwords: Compass `PeopleFile.php` (worker2) and the 1.0 NYCDOE/Staples copies (TRUE-82898). (bala)
98
101
  - 2026-10-12 — Moved the `[cloud]` AWS key to Fixed (deleted, TRUE-82705); listed the `[talos]` AWS key as still open (jcardinal)
99
102
  - 2026-10-06 — WorkerJobs runtime exposures: trace arguments (`zend.exception_ignore_args` devops option), NYCDOE FTP passwords in old job params (mhammontree)
100
103
  - 2026-10-01 — TOGa Talos release-notes key moved to `[talos] api_key`; old key listed for revocation (ajean)
@@ -0,0 +1,79 @@
1
+ ---
2
+ title: "Shared SFTP server (sftp.goagilant.com) — one box, source-IP ban, worker2 egress IP"
3
+ framework: "2.0"
4
+ repo: worker2
5
+ project: Worker
6
+ client: shared
7
+ type: feature
8
+ status: active
9
+ updated: 2026-10-19
10
+ owners: ["bala"]
11
+ files:
12
+ - worker2/Worker/Sftp/Import.php
13
+ - worker2/Worker/Monitor/Staples.php
14
+ - worker2/Worker/Client/Compass/PeopleFile.php
15
+ - library/app/asnprocessor.php
16
+ related:
17
+ - ../../../../clients/nycdoe/features/asn-sftp-ingestion-2-0.md
18
+ - ../../../../clients/nycdoe/features/servicenow-integration.md
19
+ - ../../../../clients/staples/features/oneuptime-sftp-folder-monitors.md
20
+ - ../../../../clients/compass-usa/features/people-file-import-admin-page.md
21
+ - ./oneuptime-worker2-monitoring.md
22
+ - ./committed-secrets-and-config-debt.md
23
+ ---
24
+
25
+ One Azure SFTP box serves Staples, Compass and NYCDOE, and it bans by source IP; all worker2 traffic shares one IP, so one noisy worker2 job takes every worker2 SFTP job down. Open when any worker2 SFTP job fails at once, or before adding an SFTP job.
26
+
27
+ ## Summary
28
+
29
+ - `sftp.goagilant.com` and `ftp.asisystem.com` are the **same machine**: `13.90.137.176`, an **Azure VM, not AWS**.
30
+ - Accounts on it: `staplessftp` (Staples), `cgsftp` (Compass PEOPLE file), `DOE_Apple` / `DOE_Lenovo` / `DOE_Lexmark` (NYCDOE vendor ASN feeds, `Core.CronJobs` ids 89-92; `DOE_Lexmark` is the CDW account).
31
+ - **No failover, and no documented owner.** Who runs the box and who has admin access is written down nowhere. An earlier outage (2026-08-15/16) has no recorded cause or fix either. Unblocking an IP needs that person.
32
+ - **All worker2 prod outbound traffic leaves from one IP, `34.232.23.158`.** Network path (subnet, NAT, owning account): [worker2 architecture, AWS infrastructure](../architecture.md#aws-infrastructure).
33
+ - Our AWS tools cannot see or change the SFTP server's firewall (it is on Azure).
34
+ - **1.0 worker leaves from `34.214.204.95`** (per-role Elastic IP, see [1.0 worker architecture](../../../../1.0/apps/worker/architecture.md)).
35
+ - The server **bans by source IP before login**. A ban on `34.232.23.158` breaks every worker2 SFTP job for every client, while 1.0 keeps working.
36
+
37
+ ## How it works
38
+
39
+ ### Recognize a ban (not a credential or network problem)
40
+
41
+ | Signature (phpseclib3) | Meaning |
42
+ |---|---|
43
+ | `Error reading SSH identification string; are you sure you're connecting to an SSH server?`, fails in ~0 s | Full IP ban. Not a timeout, not a bad password. |
44
+ | `phpseclib3\Exception\ConnectionClosedException` on some logins | Partial throttling, the step before a full ban. |
45
+
46
+ Quick check: worker2 jobs for **all** accounts fail together while 1.0 jobs on the same server succeed.
47
+
48
+ ### Check it fast (read-only)
49
+ 1. `Logs.Issue`: errors from `PeopleFile.php` `connectSFTP` or `Sftp/Import` (`SFTP connect/login failed for ftp.asisystem.com`).
50
+ 2. `Core.WorkerJobs`: action `Sftp/Import/RunForFile` grouped by day, count + `isSuccess`. Healthy is a few hundred a day; thousands means the login storm is back.
51
+ 3. Compass only: `Logs_Compass.Email` subject `COMPASS PEOPLE File Import - <file>`. No email for a day = the import never ran (it throws on the SFTP connect, before the email step).
52
+
53
+ ### Incident 2026-10-17 → 2026-10-19 (TRUE-82898)
54
+
55
+ 1. **Cause.** NYCDOE gated delete (TRUE-80519) went live ~2026-10-11. Every 5 min, 1.0 `legacy_import_asn.php` calls `App_ASNProcessor::releaseSourceFile()` for **every** file on the SFTP, already-logged ones included. Each call queues worker2 `Sftp/Import/RunForFile`, which opens **its own SFTP login**. Files 2.0 keeps (parse `failed`, or `not_matched_kept`) are never deleted, so they are re-queued forever. Detail: [NYCDOE 2.0 ASN SFTP ingestion](../../../../clients/nycdoe/features/asn-sftp-ingestion-2-0.md).
56
+ 2. **Growth.** Logins went 825/day (Oct 11) → 2,883/day (Oct 16), ~10 at the same moment. The server dropped some from 2026-10-17 00:35 CT and banned `34.232.23.158` fully from 02:55 CT. Failed `RunForFile`: Oct 17 2,820 (342 ok), Oct 18 3,456, Oct 19 1,404 by 11:00. The failures kept the ban alive.
57
+ 3. **Impact.** Staples monitors reported `sftp_read_failed` (they were right, no change needed). Compass PEOPLE file missed Oct 17, 18, 19. NYCDOE 2.0 import failed.
58
+ 4. **Recovery 2026-10-19.** worker2 PR #189 (TRUE-82654) deployed 10:42 CT → 9 failed NYCDOE `Files` rows reset → IP unblocked on the server ~11:05 CT → all 9 parsed `succeeded` and deleted by 11:10 → Staples monitors OK from 11:05 → Compass files 17/18/19 imported by 3 manual runs (one file per run, see [People File Import admin page](../../../../clients/compass-usa/features/people-file-import-admin-page.md)).
59
+
60
+ Order matters: stop the login storm **before** asking for the unblock, or the ban comes back.
61
+
62
+ **Still open after recovery.** Right after the unblock the 1.0 side was still queuing **144 `RunForFile` jobs per hour**. Until 1.0 stops re-queuing kept files (or sends one job per run, not one per file), expect the ban to return.
63
+
64
+ ## Gotchas
65
+
66
+ - **Every new worker2 SFTP job shares the ban risk of all the others.** Budget logins: one login per run, not per file; never re-queue a job that logs in for a file that will never change state.
67
+ - **Retries count as failed logins.** While banned, every retry extends the ban. Do not add retry loops around a login refusal (see [SFTP monitoring](./oneuptime-worker2-monitoring.md#sftp-monitoring-phpseclib3)).
68
+ - **Credentials:** several account passwords are hardcoded in git (1.0 and worker2). Locations only: [worker2 secrets register](./committed-secrets-and-config-debt.md). The 2.0 NYCDOE import reads Secrets Manager `sftp/nycdoe/<mfr>`.
69
+
70
+ ### Follow-ups (open, not done)
71
+ - `RunForFile`: skip the SFTP login for a path already known `failed` / not matched.
72
+ - One SFTP login per run instead of per file.
73
+ - Alert on files kept on the SFTP > 1 day (`not_matched_kept` is silent today).
74
+ - Add the worker2 IP to the server's never-block list.
75
+ - Find and record who owns the server and has admin access.
76
+
77
+ ## Change history
78
+ - 2026-10-19 — TRUE-82932: no owner/admin recorded, Aug 15/16 outage unexplained, CronJobs ids 89-92, daily fail counts, 144/hour still queued after the unblock, read-only check steps. No code changed. (bala)
79
+ - 2026-10-19 — Created from the 2026-10-17/19 worker2 IP ban (TRUE-82898): shared box facts, egress IPs, ban signature, cause, recovery order, follow-ups. (bala)
@@ -22,15 +22,15 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
22
22
  ## 2.0 framework
23
23
 
24
24
  - **_underscore** (_Underscore) _(framework core)_ — 110 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
25
- - **worker2** (Worker) — 90 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
25
+ - **worker2** (Worker) — 91 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
26
26
  - **api2** (API) — 27 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
27
27
  - **dbchanges2** (Database Changes) _(framework core)_ — 22 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
28
28
  - **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
29
- - **saml** (SAML SSO Gateway) — 9 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
29
+ - **saml** (SAML SSO Gateway) — 10 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
30
30
  - **apiproxy** (API Proxy) — 1 doc(s) → [2.0/apps/apiproxy/INDEX.md](2.0/apps/apiproxy/INDEX.md)
31
31
  - **toga2-view** (TOGa View Frontend) — 12 doc(s) → [2.0/apps/toga2-view/INDEX.md](2.0/apps/toga2-view/INDEX.md)
32
32
  - **toga2-hub** (TOGa Hub) — 2 doc(s) → [2.0/apps/toga2-hub/INDEX.md](2.0/apps/toga2-hub/INDEX.md)
33
- - **toga2-commerce** (TOGa Commerce) — 30 doc(s) → [2.0/apps/toga2-commerce/INDEX.md](2.0/apps/toga2-commerce/INDEX.md)
33
+ - **toga2-commerce** (TOGa Commerce) — 32 doc(s) → [2.0/apps/toga2-commerce/INDEX.md](2.0/apps/toga2-commerce/INDEX.md)
34
34
  - **toga25-supply** (TOGa 2.5 Supply) — 26 doc(s) → [2.0/apps/toga25-supply/INDEX.md](2.0/apps/toga25-supply/INDEX.md)
35
35
  - **toga25-iq** (TOGa IQ) — 1 doc(s) → [2.0/apps/toga25-iq/INDEX.md](2.0/apps/toga25-iq/INDEX.md)
36
36
  - **toga25-desk** (TOGa Desk 2.5) — 1 doc(s) → [2.0/apps/toga25-desk/INDEX.md](2.0/apps/toga25-desk/INDEX.md)
@@ -48,3 +48,4 @@
48
48
  | [Compass Order Lifecycle & Data-Integrity Invariants](workflows/order-lifecycle-and-data-integrity.md) | 2.0 | The expected raw-data shape of a Compass order at every link/ASN/IF level, plus detect-and-repair sweeps; open when diagnosing or retro-fixing Compass order dat |
49
49
  | [Comparing Compass Persona/Catalogue Effects across prod and dev-sandbox](workflows/persona-population-env-comparison.md) | 2.0 | Full-population diff technique to prove what a Compass persona/catalogue change does across prod vs dev-sandbox; open before validating any Compass visibility c |
50
50
  | [Compass USA Persona Refactor — the three-way persona split, migration & verification (TRUE-80672)](workflows/persona-refactor-migration.md) | 2.0 | Splits Compass USA's catch-all persona **1 ("All")** **three ways**, so the persona contents read the way Compass thinks about them. |
51
+ | [Compass user can't see the SSO sign-in page or gets CODE E0-1 (troubleshooting)](workflows/sso-login-troubleshooting.md) | 2.0 | Step-by-step check for one Compass USA user who sees the password form instead of "Sign in with SSO", or a CODE E0-1 screen; open on any single-user Compass sig |
@@ -44,7 +44,7 @@ One `COUNT`/`MAX` query, mirroring cron 7's filters exactly so the monitor and t
44
44
  - Walks the same two-hop bridge chain as cron 7: Compass SO (`number LIKE 'MA%'`) → `SalesOrders_PurchaseOrders` → Compass PO → `PurchaseOrders_SalesOrders` → **Office Depot SO** (`customerId = CUSTOMER_ID__OFFICE_DEPOT`) → `SalesOrders_PurchaseOrders` → Office Depot PO.
45
45
  - Requires `OfficeDepotSalesOrders.c_netsuiteInternalSalesOrderId IS NULL` and the same NULL-safe stage exclusion `(salesOrderStageId IS NULL OR salesOrderStageId NOT IN (8, 9))`.
46
46
  - Cron 7's **synced-copy check** (does an **Agilant** SO, `customerId = 3`, exist against the same Office Depot PO number?) is reproduced set-based as a `NOT EXISTS` subquery. Cron 7 self-heals those rows by stamping `c_netsuiteInternalSalesOrderId`; the monitor must not count them or it would alarm on orders that are already fine.
47
- - Cron 7's **NetSuite closed/cancelled lookup** (TRUE-82888) is **not** reproduced: the monitor does not call NetSuite. A row closed in NetSuite keeps counting until cron 7's next run (hourly, 8am-4pm Central) sets ODP SO stage 9/8, so one closed after 4pm counts overnight.
47
+ - Cron 7's **NetSuite closed/cancelled lookup** (TRUE-82888) is **not** reproduced: the monitor does not call NetSuite. A row closed in NetSuite keeps counting until a cron 7 run (hourly, 8am-4pm Central) whose NetSuite lookup succeeds sets ODP SO stage 9/8, so one closed after 4pm counts overnight. A failed lookup is caught and that run writes no stages, so the count does not drop for it.
48
48
 
49
49
  Pushed body: `stuckOrders`, `oldestStuckHours`, `lateOrders` (count past the DEGRADED age), all four thresholds echoed back, plus `alarm` / `reason`. A DB failure pushes `status:"error"`, `error:"db_query_failed"` so "we cannot see" never reads as healthy.
50
50
 
@@ -87,4 +87,4 @@ The sibling Compass monitors ship 6 criteria, including a DEGRADED incident at *
87
87
  - [Compass USA profile](../profile.md)
88
88
 
89
89
  ## Change history
90
- - 2026-10-16 — Noted the monitor does not mirror cron 7's new NetSuite closed/cancelled lookup, so NetSuite-closed rows count until cron 7 runs. (bala, TRUE-82888)
90
+ - 2026-10-16 — Noted the monitor does not mirror cron 7's new NetSuite closed/cancelled lookup, so NetSuite-closed rows count until a cron 7 run with a working NetSuite lookup. (bala, TRUE-82888)
@@ -6,8 +6,8 @@ project: Tools
6
6
  client: compass-usa
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-09-23
10
- owners: [mhammontree]
9
+ updated: 2026-10-19
10
+ owners: [mhammontree, bala]
11
11
  files:
12
12
  - tools/_/app/compass/peoplefile.php
13
13
  - tools/mvc/compass/people-file-import/get.php
@@ -18,6 +18,7 @@ related:
18
18
  - ../../../1.0/apps/tools/features/worker2-job-bridge.md
19
19
  - ../../../1.0/apps/tools/features/compass-user-persona-admin.md
20
20
  - ../../../2.0/apps/worker2/features/stranded-worker-jobs.md
21
+ - ../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md
21
22
  ---
22
23
 
23
24
  Browser page that lists the Compass PEOPLE files on SFTP and runs the importer on demand; open before changing the page or the importer's listing/guard code.
@@ -51,6 +52,16 @@ Reference: prod `Core.CronJobs` id **8** = "Compass - People File Import", sched
51
52
  `maxExecutionTime` 3600, action `Client/Compass/PeopleFile/ProcessPeopleFiles`. A job enqueued
52
53
  from this page is **not** scheduler-fired, so it gets the 300s action cap, not 3600.
53
54
 
55
+ ### Catching up after missed days
56
+ Each run imports **one** `PEOPLE*.txt` (oldest first by mtime), deletes it from the SFTP, and
57
+ emails `COMPASS PEOPLE File Import - <file>` (queued in `Logs_Compass.Email`) to
58
+ devteam@togatech.com and vburks@togatech.com. So after N missed days:
59
+ 1. Run the import once per waiting file, **one after another** (never two at once).
60
+ 2. After each run, confirm the email for that file before the next run.
61
+ 3. Stop when the listing is empty. Example: 2026-10-19, files for Oct 17/18/19 needed 3 runs
62
+ after the [shared SFTP server](../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md)
63
+ banned worker2.
64
+
54
65
  ## Gotchas
55
66
 
56
67
  - **A failed run does not mean nothing happened.** The importer is not all-or-nothing — see
@@ -59,4 +70,5 @@ from this page is **not** scheduler-fired, so it gets the 300s action cap, not 3
59
70
  - **Persona-gated to Development Team.** It runs a production import; do not widen it casually.
60
71
 
61
72
  ## Change history
73
+ - 2026-10-19 — Catch-up procedure (one run per waiting file, check the import email) after Oct 17–19 were missed in the worker2 SFTP IP ban (TRUE-82898). (bala)
62
74
  - 2026-09-23 — Created with the page (TRUE-82250, tools 0cc6f99 / worker2 a81ad6c, branch TRUE-82250): shared `gatherPeopleFiles`, new `ListFiles` action, in-flight guard, and a JSON summary return on every path. (mhammontree)
@@ -6,12 +6,14 @@ project: Worker
6
6
  client: compass-usa
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-10-06
9
+ updated: 2026-10-19
10
10
  owners: [bala, ajean, mhammontree]
11
11
  files:
12
12
  - worker2/Worker/Client/Compass/PeopleFile.php
13
13
  related:
14
14
  - ./persona-model-and-levy-gating.md
15
+ - ../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md
16
+ - ./people-file-import-admin-page.md
15
17
  - ../../../2.0/apps/_underscore/features/client-user-contactid-required.md
16
18
  - ../../../2.0/apps/_underscore/features/per-client-database-connections.md
17
19
  - ../../../2.0/apps/worker2/workflows/running-worker2-locally.md
@@ -116,7 +118,8 @@ The insert-time hook designed 2026-08-12 was built and grew into a reusable engi
116
118
 
117
119
  - **⚠ A whole tenant run is ONE transaction, and US runs first — so a US exception means Canada never executes at all.** Any throw rolls back everything for that run: inserts, updates, reactivations, **deactivations**, personas, roles and approval reassignment. The 2026-09-22 FK failure blocked both tenants for a day this way. Rollback is clean (proven: a run that died on `Users_Personas` after the `Users` insert left 0 rows behind), and the exception aborts **before** the SFTP delete, so the input file survives and the next run reprocesses it.
118
120
  - **⚠ Read `LAST_INSERT_ID()` on the write connection, never through `_Query`.** `_Query` routes SELECTs to the read host, and `LAST_INSERT_ID()` is per connection — so it would return `0`, which the `contactId` FK now rejects. `insertContact()` uses `mysqli_query` + `mysqli_insert_id` on the transaction connection, throws if the id is `<= 0`, and calls `_Database::resetQueryCache($dbName)` by hand because it bypasses `_Query`'s automatic cache reset. Framework detail: [Per-client database connections](../../../2.0/apps/_underscore/features/per-client-database-connections.md).
119
- - **A "successful" run is not proof a file was processed.** The SFTP picker is FIFO (oldest first) and handles **one** file per run, and an **empty SFTP directory is a silent success** — the job just returns. The PEOPLE file is a full snapshot of the active roster (80,805 usernames vs 75,133 active users on 2026-09-23), not a delta.
121
+ - **A "successful" run is not proof a file was processed.** The SFTP picker is FIFO (oldest first) and handles **one** file per run, and an **empty SFTP directory is a silent success** — the job just returns. The PEOPLE file is a full snapshot of the active roster (80,805 usernames vs 75,133 active users on 2026-09-23), not a delta. After N missed days, catch-up needs N runs: [admin page](./people-file-import-admin-page.md#catching-up-after-missed-days).
122
+ - **No summary email = the import never ran.** `connectSFTP()` (~line 181) throws before any DB work or the email step. `Error reading SSH identification string` there is the shared SFTP server banning worker2's IP, not a bad password. That box also serves Staples + NYCDOE, and NYCDOE load can lock Compass out (Oct 17-19 2026, TRUE-82932). Check steps + cause: [Shared SFTP server](../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md).
120
123
  - **⚠ A stale local client DB silently invalidates a test of this cron.** Local `Client_Compass` had `contactId` NULLABLE, so the *old broken code passed locally* — a NULL FK is allowed; only prod's NOT NULL turns the missing column into a `0`. Local also lacked persona 35 and still called persona 1 "All". Drop and reload both tenants from prod, then re-verify the constraint survived the copy, before trusting any result. Local-run mechanics: [Running worker2 locally](../../../2.0/apps/worker2/workflows/running-worker2-locally.md).
121
124
 
122
125
  ## Related
@@ -130,6 +133,7 @@ The insert-time hook designed 2026-08-12 was built and grew into a reusable engi
130
133
  - [Users.email as login identity](../../../2.0/apps/_underscore/features/user-email-as-identity-and-password-reset.md)
131
134
 
132
135
  ## Change history
136
+ - 2026-10-19 — TRUE-82932: files for Oct 17/18/19 missed while the shared SFTP server banned worker2; no email = import never ran; linked the shared-server doc. No code changed. (bala)
133
137
  - 2026-10-06 — Noted cart Add New User as a second source of duplicate `Users` rows (now checks email too, TRUE-82423). (bala)
134
138
  - 2026-09-23 — Recorded that the import is **not** all-or-nothing (post-commit email + SFTP delete run outside the tenant transaction; job 1076884 failed yet kept 460 US / 26 CA users). TRUE-82230 confirmed in prod: 543 `Client_Compass` users created, **zero** null-or-0 `contactId`, two manual ACTION runs ~18s each — needed because the 09:15 cron row (job 1784367) was stranded. (mhammontree)
135
139
  - 2026-09-23 — **Fixed the `Users.contactId` FK write order (TRUE-82230).** Jeff's `NOT NULL` + `Users_ibfk_1` on both tenants broke the importer from 2026-09-22 09:15 (last good run 2026-09-21); the whole run rolled back and Canada never executed. New `insertContactsForNewUsers()` creates the Contact before the `Users` batch and puts `contactId` in the INSERT column list; existing users missing a contact keep the create-then-UPDATE path. Also recorded: `LAST_INSERT_ID()` must be read on the write connection (not `_Query`); `TEST_MODE` vs `DRY_RUN` neither being safe alone; the FIFO/one-file/empty-is-silent-success facts; and that a stale local DB hides this class of bug. Verified against prod-reloaded local schemas: 80 new US / 17 new CA users, 0 with null-or-0 `contactId`. (mhammontree)
@@ -6,8 +6,8 @@ project: SAML SSO Gateway
6
6
  client: compass-usa
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-09-25
10
- owners: ["rgirish"]
9
+ updated: 2026-10-16
10
+ owners: ["rgirish", "bala"]
11
11
  files:
12
12
  - _underscore/Model/Compass/ClientAuthentication.php
13
13
  - saml/Controller/Index.php
@@ -17,6 +17,8 @@ related:
17
17
  - ../../../2.0/apps/saml/features/acs-assertion-trust-gaps.md
18
18
  - ../../compass-canada/profile.md
19
19
  - ../../../2.0/apps/toga2-commerce/features/sso-redirect-and-session-gating.md
20
+ - ../../../2.0/apps/saml/workflows/diagnosing-failed-saml-logins.md
21
+ - ../workflows/sso-login-troubleshooting.md
20
22
  ---
21
23
 
22
24
  How Compass users are matched from an Entra SAML assertion, the hardcoded `@togatech.com` test bypass that is live in production, and what it takes to turn SSO on for a Compass tenant in a non-production environment.
@@ -35,7 +37,9 @@ so enabling a second Compass tenant on an environment is a data change, not a co
35
37
  ### Identity mapping
36
38
 
37
39
  `_underscore/Model/Compass/ClientAuthentication.php` — the NameID is looked up against
38
- `Users.c_hrEmpUsername`.
40
+ `Users.c_hrEmpUsername` **AND `isActive = 1`**. No match throws `User not found: <nameId>`, which
41
+ lands in Sentry project `saml` (see [diagnosing failed SAML logins](../../../2.0/apps/saml/workflows/diagnosing-failed-saml-logins.md)).
42
+ An inactive user and a wrong username fail the same way.
39
43
 
40
44
  ### ⚠ The `@togatech.com` bypass (live in production)
41
45
 
@@ -78,13 +82,11 @@ Canada sandbox-client SSO as done.
78
82
 
79
83
  ### How the frontend finds the SSO config
80
84
 
81
- `toga2-commerce/src/hooks/useAuthenticationFlow.ts` (~lines 88-134) calls `GET /domains` filtered on
82
- **both** its own origin **and** `Environments.slug` = its own Vite build `MODE`, joining
83
- `ClientAuthentications` on `clientId` + `environmentId`. If `type == 'SSO'` it calls the scripted
84
- API `/client-authentications/singleSignOnServiceUrl` and redirects.
85
-
86
- **A build whose `MODE` does not match the environment slug falls through to `/login` with no
87
- error.** That is the first thing to check on "SSO just shows the password form".
85
+ Owned by [commerce SSO sign-in](../../../2.0/apps/toga2-commerce/features/sso-redirect-and-session-gating.md)
86
+ (`useClientSsoUrl`: `GET /domains` on origin + `Environments.slug` = build `MODE`). **A `MODE`
87
+ mismatch, a missing row, or a failed lookup (e.g. the user's network blocks the API) all show the
88
+ password form with no error.** For a single user's report, follow
89
+ [Compass SSO login troubleshooting](../workflows/sso-login-troubleshooting.md).
88
90
 
89
91
  ## Gotchas
90
92
 
@@ -97,4 +99,5 @@ error.** That is the first thing to check on "SSO just shows the password form".
97
99
  one tenant breaks both.
98
100
 
99
101
  ## Change history
102
+ - 2026-10-16 — Mapper also requires `isActive = 1`; misses land in Sentry `saml`. Frontend section now links to the commerce doc (the old `useAuthenticationFlow` redirect description was stale). (bala)
100
103
  - 2026-09-25 — Created; documented the `c_hrEmpUsername` mapper, the hardcoded `@togatech.com` → user 4273 bypass live in production, the Compass_Canada sandbox-client SSO enablement SQL (rows inserted, frontend still on `/login`, `ttlCache` suspected), and how the commerce frontend resolves an SSO config (rgirish)
@@ -0,0 +1,48 @@
1
+ ---
2
+ title: Compass user can't see the SSO sign-in page or gets CODE E0-1 (troubleshooting)
3
+ framework: "2.0"
4
+ repo: toga2-commerce
5
+ project: TOGa Commerce
6
+ client: compass-usa
7
+ type: workflow
8
+ status: active
9
+ updated: 2026-10-16
10
+ owners: ["bala"]
11
+ files:
12
+ - toga2-commerce/src/hooks/useClientSsoUrl.ts
13
+ - toga2-commerce/src/components/ErrorMessages/components/FiveHundredContent.tsx
14
+ - _underscore/Model/Compass/ClientAuthentication.php
15
+ related:
16
+ - ../profile.md
17
+ - ../features/saml-sso.md
18
+ - ../../../2.0/apps/toga2-commerce/features/sso-redirect-and-session-gating.md
19
+ - ../../../2.0/apps/toga2-commerce/features/error-codes-and-sentry-signals.md
20
+ - ../../../2.0/apps/saml/workflows/diagnosing-failed-saml-logins.md
21
+ - ../../../2.0/apps/api2/features/request-logging.md
22
+ ---
23
+
24
+ Step-by-step check for one Compass USA user who sees the password form instead of "Sign in with SSO", or a CODE E0-1 screen; open on any single-user Compass sign-in report.
25
+
26
+ ## Summary
27
+
28
+ Most single-user reports are **the user's network blocking the API**, not their account. A blocked API makes the SSO lookup fail (password form shown) and Home's calls fail (E0-1). Prove the account works, then prove the network.
29
+
30
+ Compass facts: DB `Client_Compass`, `Core.Clients` id 2 (`Compass_Usa`), prod `ClientAuthentications` id 1 (type `SSO`). Compass SSO logins land in `Logs_Compass.Api` as `POST /v2/auth/encrypted-user-uuid`, about 200-290 distinct users a day in Oct 2026.
31
+
32
+ ## Steps
33
+
34
+ 1. **Find the user** in `Client_Compass.Users`. There is no username column; the SSO key is `c_hrEmpUsername`. Check `isActive` and roles. Inactive = SSO will fail with "User not found".
35
+ 2. **Look for their own SSO logins** in `Logs_Compass.Api` (`POST /v2/auth/encrypted-user-uuid`) and note their `X-Forwarded-For`. Rows under their `userId` from view-as or staff password logins are not theirs; see the trap in [request logging](../../../2.0/apps/api2/features/request-logging.md).
36
+ 3. **Sentry `saml`:** search `User not found: <c_hrEmpUsername>`. A hit = gateway reached, account mismatch or inactive.
37
+ 4. **Sentry `commerce`:** search by `user.ip`. `Network Error` = never reached api2; `401` = session expired. Map in [error screens & Sentry signals](../../../2.0/apps/toga2-commerce/features/error-codes-and-sentry-signals.md).
38
+ 5. **Network Error → test their network.** Have the user open `https://api1.togahub.com` (should show a JSON 401) and try another network or VPN. If their ISP or company blocks it, their IT must allow `api1.togahub.com`, `api-writer.togahub.com` and `saml.togahub.com`.
39
+ 6. **Prove the account itself works:** sign in as the user through `/login` (credential form).
40
+
41
+ ## Gotchas
42
+
43
+ - **The block can be intermittent.** User 14131 (Oct 2026): her home internet provider blocked the API from 12 Oct; it worked on VPN. Network Errors also showed on 5 Oct, but 8 Oct worked.
44
+ - **A screenshot with the header and "Welcome" banner = an API failure**, not a crash.
45
+ - **Nothing in `Logs_Compass.Api` for her time window is expected** when the cause is Network Error.
46
+
47
+ ## Change history
48
+ - 2026-10-16 — Created from the TRUE-82716 investigation (user's ISP blocked the API; SSO page and E0-1 both caused by it). (bala)
@@ -6,8 +6,8 @@ project: Worker
6
6
  client: nycdoe
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-10-15
10
- owners: [mhammontree, jcardinal]
9
+ updated: 2026-10-19
10
+ owners: [mhammontree, jcardinal, bala]
11
11
  files:
12
12
  - worker2/Worker/Sftp/Import.php
13
13
  - worker2/Worker/Sftp/Parse.php
@@ -24,13 +24,14 @@ related:
24
24
  - ../../../2.0/apps/worker2/features/codeception-integration-tests.md
25
25
  - ../../../2.0/apps/_underscore/features/cli-script-bootstrap.md
26
26
  - ../../../2.0/apps/_underscore/features/case-status-and-stage-gate.md
27
+ - ../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md
27
28
  ---
28
29
 
29
30
  2.0 rewrite of DOE's vendor-SFTP ASN intake (TRUE-80519 Phases 1–2): per-manufacturer fetch to `Files`, header-validated parse into one Case per customer PO + one ServiceRequest per line; open for any 2.0 DOE ASN/SFTP work.
30
31
 
31
32
  ## Summary
32
33
 
33
- The 2.0 replacement for 1.0's `legacy_import_asn.php` + `asnprocessor/*` (see [ServiceNow / ASN Integration](servicenow-integration.md) for the 1.0 pipeline, still live). **ASN-first:** nothing is created in 2.0 except from an ingested ASN file. Phase 1 (fetch) and Phase 2 (parse) are built and verified locally against real sample files for all 4 manufacturers; Gated delete (1.0 hands each file to 2.0) is built, not deployed. Phase 3 (NetSuite SO, freeze/growth lineage) is not built. Not scheduled in prod.
34
+ The 2.0 replacement for 1.0's `legacy_import_asn.php` + `asnprocessor/*` (see [ServiceNow / ASN Integration](servicenow-integration.md) for the 1.0 pipeline, still live). **ASN-first:** nothing is created in 2.0 except from an ingested ASN file. Phase 1 (fetch) and Phase 2 (parse) are built for all 4 manufacturers. **Gated delete (1.0 hands each file to 2.0) is live in prod since ~2026-10-11** (shadow mode: the 4 `Sftp/Import/Run` crons stay `isActive=0`; 2.0 runs only through `RunForFile`). Phase 3 (NetSuite SO, freeze/growth lineage) is not built.
34
35
 
35
36
  ## How it works
36
37
 
@@ -46,9 +47,11 @@ The 2.0 replacement for 1.0's `legacy_import_asn.php` + `asnprocessor/*` (see [S
46
47
  | Lenovo | own 30-col layout (not CDW) | `LENOVO_HEADER` | Ship Date `M/D/YYYY H:MM:SS AM/PM` (date part) |
47
48
  | Lexmark | CDW 47-col | `LEXMARK_HEADER` | `Order_Date` `YYYY/MM/DD` |
48
49
  | Acer | CDW 47-col, 3 names differ from Lexmark | `ACER_HEADER` | `YYYY/MM/DD` |
49
- | Apple | own | `APPLE_HEADER` | `order_date` `M/D/YY` or `M/D/YYYY` |
50
+ | Apple | own; 50 cols since Apple added a trailing `Email` column (Oct 2026, was 49) | `APPLE_HEADER` | `order_date` `M/D/YY` or `M/D/YYYY` |
50
51
 
51
- ### Gated delete (BUILT 2026-10-06, branch `TRUE-80519`, NOT deployed)
52
+ **CDW file names do not tell you the layout** — an `acer`-named file arrived with the Lexmark header. This and the Apple 50th column failed Parse until worker2 PR #189 (TRUE-82654, deployed 2026-10-19). The Lexmark CronJobs row (id 92) `fileNamePattern` is `DOE-CDW-*.csv` (dbchanges2 PR #515; already set in prod before the PR merged).
53
+
54
+ ### Gated delete (LIVE in prod since ~2026-10-11)
52
55
  **Rule (Mark 2026-09-25, Jeff approved flow 2026-10-06): 1.0 never deletes a DOE ASN file; 2.0 deletes it only after it is imported and parsed.** Cutover = flip cron jobs 1.0 → 2.0; Phase 3 (NetSuite) later. 1.0 side: Stage 1b gated mode in [ServiceNow / ASN Integration](servicenow-integration.md).
53
56
 
54
57
  1. **1.0 queues `Sftp/Import/RunForFile(clientId, username, filePath)`** in place of its delete — no password in SQS.
@@ -69,7 +72,7 @@ The 2.0 replacement for 1.0's `legacy_import_asn.php` + `asnprocessor/*` (see [S
69
72
 
70
73
  **Before cutover (open):**
71
74
  - `Run()` has no retry for a lost Parse or lost `DeleteSource` job (only `RunForFile` re-drives). A row stuck `processing` turns `failed` only on the next 1.0 trigger after 30 min.
72
- - **Manual ASN files** are uploaded to the Lenovo folder (e.g. `WR260197522-RPL Manual ASN.csv`, 2026-09-14). 2.0 KEEPS them (no pattern) and imports nothing — decide how 2.0 handles manual files.
75
+ - **Manual ASN files** are uploaded to the Lenovo folder (e.g. `WR260197522-RPL Manual ASN.csv`, 2026-09-14) and the CDW folder (`ICC ASN.csv`, Acer header, PO S202669520 — 1.0 imported it as queue rows 557487/557488). 2.0 KEEPS them (no pattern) and imports nothing — decide how 2.0 handles manual files. **Do not rename one to make 2.0 match it:** 1.0 then re-processes and re-emails it (see Stage 1b in [ServiceNow / ASN Integration](servicenow-integration.md)).
73
76
  - Separate tickets (not done): SSH host-key pinning in `Sftp/Import` `connect()`; rotate the FTP passwords old `DeleteSftpFile` jobs likely stored in `Core.WorkerJobs.parameters` (see [worker2 secrets register](../../../2.0/apps/worker2/features/committed-secrets-and-config-debt.md)).
74
77
 
75
78
  Schema (`dbchanges2/Client_Nycdoe/2026-10-06a - NYCDOE SFTP ASN import schema.sql`, statements in run order): Files `c_status` (ENUM `pending`/`processing`/`succeeded`/`failed`/`released`) + non-unique index on `Files.checksum`/`c_manufacturer`/`c_dtProcessed`; ServiceRequestStages seed (New/Scheduled/On Hold/Delivered/Cancelled); ServiceRequests fields; Cases `c_customerPurchaseOrder`; Cases `longDescription`.
@@ -79,7 +82,7 @@ Lenovo row (Core file) carries `retiredFileNamePatterns`. The two files replace
79
82
  Tests: `ParseCest.php` (all 4 mfrs, header rejection, comma repair, claim, kept `failed` status) + `ImportDecisionCest.php` (RunForFile decisions, path check, retired names) — see [worker2 Codeception integration tests](../../../2.0/apps/worker2/features/codeception-integration-tests.md).
80
83
 
81
84
  ## Activation / deployment runbook (TRUE-80519)
82
- PRs open 2026-10-06: library, worker2, dbchanges2, _underscore (worker optional for Part A).
85
+ Parts A and B are done in prod (gated mode live ~2026-10-11). Kept for rollback and the failed-file steps.
83
86
 
84
87
  **Part A — inert (no 1.0 behavior change):**
85
88
  1. Run `Client_Nycdoe/2026-10-06a - NYCDOE SFTP ASN import schema.sql` on the NYCDOE client DB.
@@ -97,12 +100,15 @@ PRs open 2026-10-06: library, worker2, dbchanges2, _underscore (worker optional
97
100
 
98
101
  **Rollback:** remove `asn_delete_mode` (or set `immediate`), deploy worker.
99
102
 
100
- **Failed file:** investigate (Tools → Errors / `Logs.Issue`), set `Files.c_status='released'`, queue `Sftp/Import/DeleteSource {clientId: 38, filesUuid}`.
103
+ **Failed file** (`failed` rows are kept forever — `decideLedgeredFileAction` returns KEEP; nothing retries them). Investigate (Tools → Errors / `Logs.Issue`), then either:
104
+ 1. **Re-parse** (only after the parser fix is live, else it fails again): give a dbchanges2 file that sets `c_status='pending'` and `dtUpdated = NOW() - INTERVAL 1 HOUR` (past `STALE_PARSE_SECONDS` = 1800 s; the explicit value beats `ON UPDATE CURRENT_TIMESTAMP`), guarded by `c_status='failed' AND uuid IN (...)`. The next 1.0 trigger's `RunForFile` re-queues Parse at once. Example: `dbchanges2/Client_Nycdoe/2026-10-19a - Requeue failed NYCDOE ASN files for parse.sql` (9 files, all `succeeded` in prod 2026-10-19).
105
+ 2. **Drop it:** set `Files.c_status='released'`, queue `Sftp/Import/DeleteSource {clientId: 38, filesUuid}`.
101
106
 
102
107
  **Later (not this deploy):** cutover = turn the 4 crons on + remove NYCDOE processors from 1.0; decide manual ASN file handling; Phase 3 NetSuite; separate tickets for host-key check + FTP password rotation (see *Before cutover* above).
103
108
 
104
109
  ## Gotchas
105
110
 
111
+ - **⚠ Every kept file costs one SFTP login every 5 min, forever.** 1.0 re-releases every file still on the SFTP on every run (already-logged ones too), and each `RunForFile` opens its own login. `failed` and `not_matched_kept` files never leave, so logins grew to 2,883/day and the shared SFTP server banned the worker2 IP 2026-10-17 → 19 (Staples + Compass broke too). Clear kept files fast; open fixes (skip login for known kept paths, one login per run, alert on files kept > 1 day) are in [Shared SFTP server](../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md).
106
112
  - **Direct `_Model` writes need every `c_` field declared as a real property** — `_Model::__set()` throws `There is no field called 'c_…'` otherwise. The "c_ needs no declaration" behavior is the /v2 API engine's, not `_Model`'s. Hence `_Model_Nycdoe_File` / `_Model_Nycdoe_Case` / `_Model_Nycdoe_ServiceRequest`.
107
113
  - **1.0 `asnprocessor/lenovo.php` is not a source of truth for today's Lenovo layout.** Its 47-col mapping is stale; the real file is 30 cols (confirmed on 3 live samples). Verify every header and date format against a **real current sample file**, never the 1.0 code — wrong Lenovo header, Lexmark date, and Apple 2-digit year (`'25'` → year `0025`) were all caught this way.
108
114
  - **Acer and Lexmark share column positions, not header names** (Acer `First_Name_Shipping`/`Last_Name_Shipping`/`SerialNo` vs Lexmark `Contact_Name_Shipping`/`Email_Shipping`/`Svc_Tag_ID`). Keep separate header constants. Lenovo stays its own path (not a CDW supplier).
@@ -116,6 +122,7 @@ PRs open 2026-10-06: library, worker2, dbchanges2, _underscore (worker optional
116
122
  - **Parsed Cases get no stage; `_status` is derived** (`open`, or `in-progress` once an owner is set) since TRUE-82615. Direct model writes skip the api2 set-stage gate. Never set a stage to mean "open" — see [Case status + stage gate](../../../2.0/apps/_underscore/features/case-status-and-stage-gate.md).
117
123
 
118
124
  ## Change history
125
+ - 2026-10-19 — Gated delete marked live (~10-11); kept-file login storm caused the worker2 SFTP IP ban; failed-file re-parse procedure; Apple 50-col + CDW name/header mismatch fixed (PR #189); manual `ICC ASN.csv`. (bala)
119
126
  - 2026-10-15 — Gotcha: parsed Cases now derive status (no stage) after the shared Case model change. (jcardinal)
120
127
  - 2026-10-06 — 9 dbchanges2 files merged into one Client_Nycdoe + one Core file; runbook Part A updated (mhammontree)
121
128
  - 2026-10-06 — Activation/deployment runbook (Part A inert, Part B shadow mode, rollback, failed-file release) (mhammontree)
@@ -6,8 +6,8 @@ project: Worker
6
6
  client: nycdoe
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-10-16
10
- owners: [mhammontree, sking]
9
+ updated: 2026-10-19
10
+ owners: [mhammontree, sking, bala]
11
11
  files:
12
12
  - worker/crons/sync/nycdoe/import_asn.php
13
13
  - worker/crons/sync/nycdoe/import_inc.php
@@ -114,7 +114,8 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘ [UNIQUE ded
114
114
  - **Stage 1a (API):** vendor detected by string match on `u_asset_mfg` (apple=1, acer=2, lenovo=3, lexmark=4; unknown → Lexmark). Serial 'S'-prefix stripped for Lexmark/Lenovo. Ship-to resolved via `App_Api_NYCDOEV2::resolveShippingAddress()`; on failure a tracking `NYCDOETicket` is recorded but **ASN creation is skipped**. `dedupeKey = vendorId|PO|DOE-part|serialNumber`. Dedup is SELECT-before-INSERT with per-row try/catch (unique index is the race backstop).
115
115
  - **Stage 1b (SFTP):** the only source of **non-serialized** items (in practice the Lexmark feed). Serial-less lines of the same PO+part are summed per file into one queue row with `dedupeKey = vendorId|PO|DOE-part|<file token>`. Dedup is INSERT + catch "Duplicate entry" (race-safe).
116
116
  - **Per file, in order** (`App_ASNProcessor_{Lenovo,Apple,Lexmark}::processFiles()`): read over plain FTP (`App_FTP`: `ftp_connect`/`ftp_rawlist`/`ftp_get`) → parse → cache to disk → email → save legacy `Logs.FileLog` (jobs `ASN_LENOVO` / `ASN_APPLE` / `ASN_LEXMARK`; the row means "done") → `App_ASNProcessor::releaseSourceFile()`. A file is never released before it is logged.
117
- - **`asn_delete_mode` (`[nycdoe]` in `worker/config.worker.ini`; built 2026-10-06 on `TRUE-80519`, NOT deployed):** missing/`immediate` = original behavior — queue `App_Worker::runTask('Utilities/Sftp/DeleteFile/DeleteSftpFile')`, worker2 `Worker/Utilities/Sftp/DeleteFile.php` deletes over SFTP (phpseclib3), delete gated only on the disk-cache write, not a good parse. ⚠ That path sends the SFTP password **inside the SQS message** (source: `FTP_*` constants in `asnprocessor/*.php`) — see the credentials gotcha. `gated` = queue 2.0 `Sftp/Import/RunForFile` (clientId/username/filePath, no password); 2.0 deletes only after its own import+parse — see [2.0 ASN SFTP Ingestion](asn-sftp-ingestion-2-0.md) (Gated delete). Gated mode also skips files already in `Logs.FileLog` (`isFileAlreadyLogged()`: job + fileName + fileTimestamp via `App_Date::convertToSQLDate`, same as `App_Model`) but still re-releases them. **Activate:** set the ini value + worker deploy. Default config verified to behave exactly as before (offline under PHP 7.2; SQL against legacy prod).
117
+ - **File names barely matter to 1.0.** The processors read **every** file in the folder regardless of name. The name is used only for the `Logs.FileLog` "already seen" check (name + timestamp) and in the `dedupeKey` of serial-less rows (`legacy_import_asn.php`). So **renaming a file makes 1.0 re-process and re-email it.**
118
+ - **`asn_delete_mode` (`[nycdoe]` in `worker/config.worker.ini`; `gated` LIVE in prod since ~2026-10-11, TRUE-80519):** missing/`immediate` = original behavior — queue `App_Worker::runTask('Utilities/Sftp/DeleteFile/DeleteSftpFile')`, worker2 `Worker/Utilities/Sftp/DeleteFile.php` deletes over SFTP (phpseclib3), delete gated only on the disk-cache write, not a good parse. ⚠ That path sends the SFTP password **inside the SQS message** (source: `FTP_*` constants in `asnprocessor/*.php`) — see the credentials gotcha. `gated` = queue 2.0 `Sftp/Import/RunForFile` (clientId/username/filePath, no password); 2.0 deletes only after its own import+parse — see [2.0 ASN SFTP Ingestion](asn-sftp-ingestion-2-0.md) (Gated delete). Gated mode also skips files already in `Logs.FileLog` (`isFileAlreadyLogged()`: job + fileName + fileTimestamp via `App_Date::convertToSQLDate`, same as `App_Model`) but still re-releases them — every file, every 5-min run, one worker2 SFTP login each. Files 2.0 keeps are re-released forever; this got the worker2 IP banned on the shared SFTP server 2026-10-17 (see the login-storm gotcha in [2.0 ASN SFTP Ingestion](asn-sftp-ingestion-2-0.md#gotchas)). **Activate:** set the ini value + worker deploy. Default config verified to behave exactly as before (offline under PHP 7.2; SQL against legacy prod).
118
119
  - **No Acer processor runs.** `asnprocessor/acer.php` is dead code (not called from `legacy_import_asn.php`, never appends to `$output`, never deletes). Acer files do arrive — via CDW on the shared CDW/Lexmark SFTP account — and Lexmark `processFiles()` has **no file-name filter**, so 1.0 processes every Acer file **as Lexmark** (Lexmark vendorId). Proof: 458 `ASN_LEXMARK` FileLog rows with `acer` in `fileName` (2025-08-18 → 2026-09-25), 0 `ASN_ACER`. Any 1.0-vs-2.0 count must compare 1.0 Lexmark against 2.0 Lexmark + Acer.
119
120
  - **Vendors upload and forget** — they never re-send or notice if a file sits on the SFTP (Mark, 2026-09-25). A deleted-but-unparsed file is gone unless recovered from `Logs.FileLog`.
120
121
  - **Stage 2:** queue → ASN/Items/Units/TrackingNumbers in a transaction. Serialized: one Unit per serial. Non-serialized: the file's quantity is a **cumulative** total — `unsent item qtyOrder = cumulative − SUM(committed qtyOrder)`; post-freeze growth becomes a **fresh item row**, a decrease fires a discrepancy email and changes nothing. Exactly one Unit per serial-less item row regardless of quantity. Failed rows retry 5× then alert `devteam@goagilant.com`.
@@ -176,6 +177,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
176
177
  5. With no staging environment, verify with a standalone arithmetic simulation plus a read of the consumer query; `php -l` every touched file.
177
178
 
178
179
  ## Gotchas
180
+ - **⚠ OPEN BUG (found 2026-10-19, not fixed) — 1.0 Lexmark/CDW dates save as `1969-12-31`.** CDW sends `YYYY/MM/DD`, but `library/app/asnprocessor/lexmark.php` reads dates with `App_Date::convertMDYYYYtoSQLDate`. All 5,244 of 5,244 `App_ASNProcessor_Lexmark` queue rows since 2026-09-01 have `dateShipped` (and other dates) = 1969-12-31. 2.0 Parse reads `YYYY/MM/DD` correctly. Never trust a 1.0 Lexmark/Acer ship date.
179
181
  - **⚠ OUTAGE (fixed TRUE-82735, not deployed 2026-10-14) — the RITM and POD crons never finished for months.** Signs: `Common.CronJobExecutions.dtCheckOut` NULL on every run, zero ServiceNow calls. RITM cron: Feb–Mar 2026 refactors dropped the per-MSO inner loop → `Logs.Issue` 1R *"Undefined variable: row2"* every run since at least 2026-05-07 (last sane version `ce98cd2b5`). POD cron: the file query had **no `ORDER BY`**, so every run hit the same poison file (RITM1387575, already closed → empty-body 200 on `-7` → `Logs.Issue` 1S *"Empty response body from API (HTTP 200)"*), died, never set `dtTransmitted`, and re-uploaded that POD every 5 min; 6,718 files stuck, last sent 2026-09-04. Lesson: an unordered work queue + one poison row + a run-killing throw = a silent total outage; order the queue, isolate per row, and capture.
180
182
 
181
183
  - **⚠ OPEN (parked, handed to Skyler 2026-10-16) — Stage 5 repair-order loop on RITM1320328.** MSO 317327 / PO `WR260255743` / ASN 24959 item 51224 (qty 282, 283 units, `installTicketStatus` `pending`, 18 units with no `togadeskRepairOrderId`). Since June `3_create_installation_ticket.php` creates a new INSTALLATION repair order every few hours: 637 ROs (1 COMPLETE, 23 ORDER_ASSIGNED_AWAITING_SCHEDULING, 613 ORDER_CREATED_AWAITING_REVIEW), all on serial `NXJJEAA0015281EAF72P00`. Each pass also PATCHes the RITM to `-12`, which ServiceNow refuses (empty-body 200). Stop lever: cancel `AdvanceShippingNoticeItems.installTicketStatus` with a reason, per `dbchanges/Core/SK/2026-07-30-dedupe-asn-install-units.sql` — a business call. Root cause in code not confirmed.
@@ -286,6 +288,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
286
288
  - Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
287
289
 
288
290
  ## Change history
291
+ - 2026-10-19 — Gated delete live since ~10-11 and its re-release login storm (worker2 SFTP IP ban); file names matter to 1.0 only for FileLog/dedupeKey (rename = re-email); open Lexmark 1969-12-31 date bug. (bala)
289
292
  - 2026-10-16 — TRUE-82776: link to the update-rejection monitor; parked RITM1320328 repair-order loop and POD blank-ticket 400s. (mhammontree)
290
293
  - 2026-10-14 — TRUE-82735: RITM + POD outbound crons revived (per-MSO loop, own-RITM match, state ownership, POD ordering/isolation), library RITM filter + location lookup fixed, backlog stamps, months-long outage gotcha. Not deployed. (mhammontree)
291
294
  - 2026-10-06 — Dormant DOA files moved to `crons/sync/DOA/` and 4 unused scripts deleted on worker branch TRUE-80519 (effective on merge) (mhammontree)
@@ -13,8 +13,8 @@ project: Worker
13
13
  client: nycdoe
14
14
  type: profile
15
15
  status: active
16
- updated: 2026-10-16
17
- owners: [mhammontree, sking]
16
+ updated: 2026-10-19
17
+ owners: [mhammontree, sking, bala]
18
18
  files: []
19
19
  related:
20
20
  - features/servicenow-integration.md
@@ -30,7 +30,7 @@ NYC DOE — TOGA client whose **live** integration runs in the **1.0 worker tier
30
30
 
31
31
  ## Platforms & data
32
32
  - **1.0 worker tier** — all live business logic. Local mirror tables in `db_common` (`NYCDOETickets`, `NYCDOELocations`, `AdvanceShippingNotice*`).
33
- - **2.0 (`worker2` + `_underscore` + `dbchanges2`, tenant `Client_Nycdoe`)** — (1) the OneUptime liveness probe `Monitor/Nycdoe/ServiceNowEndpointHealth` (`worker2/Worker/Monitor/Nycdoe.php`, TRUE-80587, live) and the [ServiceNow update-rejection monitor](features/oneuptime-servicenow-update-rejections-monitor.md) (TRUE-82776, ships inactive); (2) the ASN rewrite `Sftp/Import` → `Sftp/Parse` → Case/ServiceRequest (Phases 1–2 built, not deployed) — see [2.0 ASN SFTP Ingestion](features/asn-sftp-ingestion-2-0.md). Entity map: `Case` = 1.0 managed_service_order, `ServiceRequest` = 1.0 repair_order; DOE does not use `Ticket`. Client models: `_underscore/Model/Nycdoe/`. 2.0 does **not** write to ServiceNow yet — the 1.0 RITM/POD crons keep sending until 2.0 takes over, then 1.0 is turned off.
33
+ - **2.0 (`worker2` + `_underscore` + `dbchanges2`, tenant `Client_Nycdoe`)** — (1) the OneUptime liveness probe `Monitor/Nycdoe/ServiceNowEndpointHealth` (`worker2/Worker/Monitor/Nycdoe.php`, TRUE-80587, live) and the [ServiceNow update-rejection monitor](features/oneuptime-servicenow-update-rejections-monitor.md) (TRUE-82776, ships inactive); (2) the ASN rewrite `Sftp/Import` → `Sftp/Parse` → Case/ServiceRequest (Phases 1–2; gated delete live in prod since ~2026-10-11, scheduled crons still off) — see [2.0 ASN SFTP Ingestion](features/asn-sftp-ingestion-2-0.md). Entity map: `Case` = 1.0 managed_service_order, `ServiceRequest` = 1.0 repair_order; DOE does not use `Ticket`. Client models: `_underscore/Model/Nycdoe/`. 2.0 does **not** write to ServiceNow yet — the 1.0 RITM/POD crons keep sending until 2.0 takes over, then 1.0 is turned off.
34
34
  - **TOGa Desk:** DOE is client id **16** (`db_togadesk`) — repair orders, managed service orders, the "#received / #ordered" Receiving Summary UI.
35
35
  - **NetSuite:** Sales Orders, PO reconciliation, item receipts, Item Fulfillments, invoices (SuiteTalk toolkit in `library`).
36
36
  - **`dbchanges` (1.0):** DOE schema + **data-repair** changes ship here (`dbchanges/Core/` targets the legacy/V1 `Core` schema). Stranded-ASN recoveries are data-only repairs — see the [1.0 dbchanges authoring workflow](../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
@@ -41,7 +41,7 @@ NYC DOE — TOGA client whose **live** integration runs in the **1.0 worker tier
41
41
  | ServiceNow prod (live) | `https://supportadmin.schools.nyc` (stage `nycdoedev3.service-now.com`) | `App_Api_NYCDOEV2` — all live INC/RITM GET/PATCH under `/api/nycd3/`, incl. disposals; V1 `App_Api_NYCDOE` uses the same prod host (stage `supportadminstg.schools.nyc`) |
42
42
  | ServiceNow legacy | `https://nycdoeprod.service-now.com` (stage `nycdoedev2`) | only the dormant `download_tickets*.php` — not live |
43
43
  | DOE EDI SFTP | `transfer.schools.nyc` (SSH key `doedimedu.pem`) | EDI 850 in, invoices out |
44
- | Vendor SFTP feeds | Apple / Lexmark / Lenovo (Lenovo: `ftp.asisystem.com`) | legacy ASN file ingestion |
44
+ | Vendor SFTP feeds | `ftp.asisystem.com` = `sftp.goagilant.com`, logins `DOE_Apple` / `DOE_Lenovo` / `DOE_Lexmark` (CDW) | ASN files, read by 1.0 and (gated delete) 2.0. Shared box with Staples + Compass — see [Shared SFTP server](../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md) |
45
45
 
46
46
  ## API clients
47
47
  - `App_Api_NYCDOE` (`library/app/api/nycdoe.php`) — V1 client (legacy ticket downloaders).
@@ -6,7 +6,7 @@ project: Worker
6
6
  client: staples
7
7
  type: client-feature
8
8
  status: active
9
- updated: 2026-10-06
9
+ updated: 2026-10-19
10
10
  owners: ["bala", "jcardinal"]
11
11
  files:
12
12
  - worker2/Worker/Monitor/Staples.php
@@ -17,9 +17,10 @@ related:
17
17
  - ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
18
18
  - ../profile.md
19
19
  - ../../../1.0/apps/worker/features/staples-cxml-order-import.md
20
+ - ../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md
20
21
  ---
21
22
 
22
- Two worker2 OneUptime push monitors watching the Staples SFTP `/in` and `/out` for stuck files (also an SFTP connectivity check); open for folder semantics, thresholds, and the cron registration.
23
+ Two worker2 OneUptime push monitors watching the Staples SFTP `/in` and `/out` for stuck files (also an SFTP connectivity check); open for folder semantics, thresholds, the cron registration, and diagnosing a blind monitor (worker2 IP blocked vs SFTP down).
23
24
 
24
25
  ## Summary
25
26
 
@@ -70,8 +71,20 @@ Both actions retry the SFTP connect+list inside one run (shared [in-run retry](.
70
71
  - `$sftp->disconnect()` in `finally`; error push includes `attempts`.
71
72
  - Worst case ~75s SFTP + 10s push, under the 120s `maxExecutionTime`.
72
73
 
74
+ ### Diagnosing a blind monitor (`status:error`, `error=sftp_read_failed`)
75
+
76
+ OneUptime shows the existing error push as "checker blind". It already alerts on its own; no code change needed. To tell "Staples SFTP is down" from "only worker2 is blocked":
77
+
78
+ 1. **worker2 side** — `Core.WorkerJobs` output for `action LIKE 'Monitor/Staples/%'`, grouped by result. Find the last good run and the first failing one, and read the error text.
79
+ 2. **1.0 side** — legacy `Logs.FileLog`, `job = 'STAPLES_IMPORT_ORDER'` (time column is `dtStamp`; there is no `dtCreated`; mind the [FileLog size note](../../../1.0/apps/worker/features/staples-asn-856-outbound.md#gotchas)). The 1.0 import cron uses the **same host, port 22, and user**. Rows still arriving = the server and the login are fine.
80
+ 3. **Read the error:**
81
+ - `Error reading SSH identification string; are you sure you're connecting to an SSH server?` on every attempt, each failing at once (~10s for 3 attempts) = TCP connects but the server closes it **before the SSH banner**. With the 1.0 cron still working, that is the shared SFTP server **banning worker2's IP** (`34.232.23.158`). The cause is usually another worker2 job's login storm, not Staples. Server, IPs and recovery order: [Shared SFTP server](../../../2.0/apps/worker2/features/shared-sftp-server-source-ip-ban.md).
82
+ - A single `fsockopen(): ... Connection timed out` = transient; the in-run retry covers it.
83
+ - Login refused (`ERROR_CODE_LOGIN_FAILED`) = credentials; check the `[sftp_staples]` group.
84
+
73
85
  ## Gotchas
74
86
 
87
+ - **Blind monitor while the 1.0 cron still works = worker2's IP is blocked, not a server outage.** Both use the same host and user; only the source IP differs (worker2 NAT vs the 1.0 worker). See *Diagnosing a blind monitor* above.
75
88
  - **phpseclib3 `rawlist` returns arrays, and its SFTP type constants are not loaded** — see the shared doc's [SFTP monitoring gotchas](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md#sftp-monitoring-phpseclib3). Getting this wrong silently makes the aged-file count always 0, so the monitor never alarms and looks healthy.
76
89
  - The `[sftp_staples]` password and the OneUptime push URLs are credentials — reference their location, never their value.
77
90
 
@@ -80,3 +93,7 @@ Both actions retry the SFTP connect+list inside one run (shared [in-run retry](.
80
93
  - [OneUptime push-metric monitors for 2.0 workers](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — the shared pattern, payload contract, and runbook.
81
94
  - [Staples profile](../profile.md)
82
95
  - [Staples cXML order import (1.0)](../../../1.0/apps/worker/features/staples-cxml-order-import.md) — the inbound pipeline these monitors watch.
96
+
97
+ ## Change history
98
+ - 2026-10-19 — TRUE-82898: IP ban confirmed and fixed (cause: NYCDOE gated-delete login storm; unblocked ~11:05 CT, monitors OK). Network facts moved to the shared SFTP server doc. Monitors were correct; no code change. (bala)
99
+ - 2026-10-19 — TRUE-82897: from 2026-10-17 02:55 CT every run failed with "Error reading SSH identification string" while the 1.0 import cron (same host and user) kept reading `/in`. Inferred an IP-level block of worker2's NAT IP on the Azure SFTP server (not yet confirmed by the server owner). Added the blind-monitor diagnosis and network facts. No code changed. (bala)
@@ -11,7 +11,7 @@ project: Worker
11
11
  client: staples
12
12
  type: profile
13
13
  status: active
14
- updated: 2026-10-01
14
+ updated: 2026-10-19
15
15
  owners: [jcardinal, bala, mhammontree]
16
16
  files:
17
17
  - worker/crons/sync/staples/sync_staples_cxml.php
@@ -36,4 +36,6 @@ Staples is a **headless** integration client — no UI, Worker 1.0 only, dependi
36
36
 
37
37
  **Onboarding rule (load-bearing):** any new Staples buying account must be set up in NetSuite — as a child customer of 30057, with its **External Customer ID** = the Staples `CustomerID` **and** exactly one **Customer Location** (a child of Location 36 "New York : New York Warehouse : Staples") — *before* its first order arrives. Miss either and inbound orders for that account silently jam on SFTP `/in`, retrying hourly with nothing logged. Full setup rule and the diagnostic steps are in the [cXML import feature doc](../../1.0/apps/worker/features/staples-cxml-order-import.md).
38
38
 
39
+ **SFTP credential (security gap, open):** one Staples SFTP login (`sftp.goagilant.com`) is used in three places. worker2 reads it from the `[sftp_staples]` config group. Both 1.0 crons have the password **hardcoded and committed** (`sync_staples_cxml.php` ~line 20, `staples_asn_netsuite.php` ~line 11). Treat it as exposed: move both crons to config, then rotate, and update all three places in the same change or the other two break.
40
+
39
41
  **Sales rep on Staples sales orders:** the SO `entity` is always parent **30057**, so NetSuite sources **30057's** rep (currently Paulina Szeliga, 50695) regardless of which child account bought — see [NetSuite Sales Order Sales Rep Sourcing](../../1.0/apps/worker/features/netsuite-sales-order-sales-rep-sourcing.md).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "toga-ai",
3
- "version": "1.0.1079",
3
+ "version": "1.0.1081",
4
4
  "description": "TOGA Technology Team Claude Knowledge System — shared AI coding harness with skills, knowledge base CLI, and project installer for Claude Code.",
5
5
  "keywords": [
6
6
  "claude",