toga-ai 1.0.898 → 1.0.900
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md +6 -4
- package/knowledge/2.0/apps/_underscore/features/core-domains-app-host-registry.md +18 -2
- package/knowledge/2.0/apps/api2/features/request-logging.md +5 -1
- package/knowledge/2.0/apps/saml/INDEX.md +2 -0
- package/knowledge/2.0/apps/saml/features/acs-assertion-trust-gaps.md +80 -0
- package/knowledge/2.0/apps/saml/features/gateway-environments-and-redirect-safety.md +159 -0
- package/knowledge/2.0/apps/toga2-commerce/features/sso-redirect-and-session-gating.md +6 -2
- package/knowledge/2.0/apps/worker2/INDEX.md +1 -0
- package/knowledge/2.0/apps/worker2/architecture.md +5 -3
- package/knowledge/2.0/apps/worker2/features/logs-retention-cleanup.md +84 -0
- package/knowledge/2.0/apps/worker2/features/oneuptime-worker2-monitoring.md +14 -5
- package/knowledge/INDEX.md +2 -2
- package/knowledge/clients/compass-canada/profile.md +4 -2
- package/knowledge/clients/compass-usa/INDEX.md +2 -0
- package/knowledge/clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md +66 -0
- package/knowledge/clients/compass-usa/features/saml-sso.md +100 -0
- package/knowledge/clients/compass-usa/profile.md +3 -1
- package/knowledge/clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md +12 -5
- package/knowledge/clients/nycdoe/INDEX.md +1 -1
- package/knowledge/clients/nycdoe/features/hold-status-sync.md +20 -6
- package/knowledge/clients/nycdoe/features/servicenow-integration.md +3 -1
- package/knowledge/clients/nycdoe/profile.md +2 -3
- package/knowledge/clients/true/profile.md +25 -2
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: ["rgirish", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/database.php
|
|
@@ -18,6 +18,7 @@ related:
|
|
|
18
18
|
- ../../standards/backend-php.md
|
|
19
19
|
- ../../apps/worker/features/netsuite-togasupply-per-client-sync.md
|
|
20
20
|
- ../../../clients/compass-usa/workflows/odp-order-pipeline-to-netsuite.md
|
|
21
|
+
- ../../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
|
|
21
22
|
---
|
|
22
23
|
|
|
23
24
|
How a 1.0 mysqli connection dies during a long external call ("MySQL server has gone away"), why the built-in 2006 reconnect never fired, and the three helpers (`keepAlive`, `reconnect`, `queryForked`) that fix it; open before writing any cron that does a slow HTTP/SOAP call between two DB writes.
|
|
@@ -86,10 +87,10 @@ It returns **`false`** (rather than throwing) when the link has no config entry,
|
|
|
86
87
|
|
|
87
88
|
When a cron creates a record in an external system and then stamps our DB, a dead connection at the stamp step produces a **silent duplicate**. `register_shutdown_function()` is the **only** code that still runs after `App_Error::handleException()` calls `exit`, so it is the only place an alert can fire.
|
|
88
89
|
|
|
89
|
-
The
|
|
90
|
+
The alert pattern as shipped in Compass ODP cron 5 (`worker` commit `afce32ca`). Cron 5 has **no `keepAlive()` call** — it stamps via `App_Database::queryForked($sql, $clientToga2DatabaseLink)` (corrected 2026-09-25):
|
|
90
91
|
|
|
91
92
|
1. **Arm** an `$unstampedNetsuiteOrder` array immediately *before* the external `add()`.
|
|
92
|
-
2. **Clear** it after
|
|
93
|
+
2. **Clear** it right after the stamp is handed to `queryForked()`, and also when the remote system *rejects* the order (nothing was created, so nothing is orphaned). **⚠ The forked write is not confirmed** (code comment says so): a failed fork leaves the order sent, unstamped, and the alarm already off → possible duplicate on the next run. Cron 5 relies on the [NetSuite duplicate SO monitor](../../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md) as backstop. If you need a confirmed stamp, clear only after a synchronous write succeeds.
|
|
93
94
|
3. In the shutdown handler, if it is still armed, email devteam.
|
|
94
95
|
4. **Reconnect `db_common` and `db_logs` first — NOT the client link.** `App_Email::send()` reads the suppression list on **db_common** (`App_Model_Common_EmailSuppressionList`, `App_Model_Common_EmailsQueued`) and writes its log row on **db_logs** (`App_Model_Logs_Email`). Reconnecting the client link buys nothing for the email.
|
|
95
96
|
5. Wrap it in `catch (Throwable)` falling back to `error_log()`, which needs no database at all.
|
|
@@ -100,7 +101,7 @@ The working pattern (Compass ODP cron 5, `worker` commit `afce32ca`):
|
|
|
100
101
|
- **⚠ A `catch` around a DB call proves nothing about whether an alert will fire.** Verify against the actual email/log table, not the code. Thirty-three failures produced zero emails.
|
|
101
102
|
- **⚠ Kit / bundle orders are the ones that break.** Nothing about a part number causes this — the slow path does. On Compass ODP, `PC-KIT-OPT2` (10 expanded lines) accounted for 10 of 11 duplicated POs simply because kits are the orders whose `add()` runs past 180 s. Read a part-number correlation as "this is the slow path", not "this SKU is special".
|
|
102
103
|
- **⚠ Reconnecting the wrong link is a silent no-op.** Emailing needs `db_common` + `db_logs`. A handler that only reconnects the client link will still die trying to send.
|
|
103
|
-
- **⚠ Turning a latent version of this bug ON at volume is a separate risk.** The Compass race existed from 2026-09-15 but the cron was `"active": 0`, so it barely fired. Flipping it to `1` on 2026-09-21 produced
|
|
104
|
+
- **⚠ Turning a latent version of this bug ON at volume is a separate risk.** The Compass race existed from 2026-09-15 but the cron was `"active": 0`, so it barely fired. Flipping it to `1` on 2026-09-21 produced most of the duplicates (22 POs affected Sep 15–22 in total). When re-enabling a long-dormant cron that writes to an external system, check its stamp path first.
|
|
104
105
|
- **`keepAlive()` after the call must sit OUTSIDE the `try`/`catch`** and be guarded on the response being non-null, so a rejected call does not run it against state you have not checked.
|
|
105
106
|
|
|
106
107
|
## Decisions
|
|
@@ -110,4 +111,5 @@ The working pattern (Compass ODP cron 5, `worker` commit `afce32ca`):
|
|
|
110
111
|
- `library/app/error.php:5-8` has `IMMEDIATELY_TERMINATE_INSTANCE_IF_ERROR_STRING_CONTAINS`, matched **on substring before severity**. A recoverable warning could terminate an EC2 instance.
|
|
111
112
|
|
|
112
113
|
## Change history
|
|
114
|
+
- 2026-09-25 — Corrected the cron 5 pattern: it stamps via `queryForked` with no `keepAlive`, and disarms the alarm before the forked write is confirmed; noted the monitor backstop. (rgirish)
|
|
113
115
|
- 2026-09-22 — Doc created. Root-caused the "gone away" class: warning fires at `mysqli_select_db` before the 2006 check, `databaseClose()` never cleared `$connections`, `$link` captured above the retry loop. Added `App_Database::reconnect()` + `queryForked()` (`library` commit `60fdb888`) and documented the shutdown-handler alert pattern. Decided against raising `wait_timeout` from 180 s. (rgirish)
|
|
@@ -6,8 +6,8 @@ project: _Underscore
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["bala", "apeterson"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["bala", "apeterson", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Model/Core/Domain.php
|
|
13
13
|
- api2/Component/Api/CrossClient/CrossClient.php
|
|
@@ -18,6 +18,7 @@ related:
|
|
|
18
18
|
- ../../toga2-supply/workflows/client-host-scoping.md
|
|
19
19
|
- ../../api2/features/encrypted-user-uuid-auth-handoff.md
|
|
20
20
|
- ../../../../clients/compass-usa/features/approval-decision-flow.md
|
|
21
|
+
- ../../saml/features/gateway-environments-and-redirect-safety.md
|
|
21
22
|
---
|
|
22
23
|
|
|
23
24
|
`Core.Domains` is the authoritative host↔client↔app↔env registry; a host is NOT derivable from env slug or api host — read the table. Open before building an app link.
|
|
@@ -137,6 +138,21 @@ app-specific lookup it will happily return a commerce domain for a supply link.*
|
|
|
137
138
|
origin string (scheme included) against `Core.Domains` for that client + environment. Do not
|
|
138
139
|
assume a sibling client's working row proves the host is registered for yours.
|
|
139
140
|
|
|
141
|
+
- **⚠ A row in this table is now an AUTHORIZATION, not just a lookup.** As of 2026-09-25 the SAML
|
|
142
|
+
gateway's `/acs` treats "a `Core.Domains` row exists, owned by this client + environment" as
|
|
143
|
+
permission to deliver the `?saml=` login handoff to that host — there is deliberately no separate
|
|
144
|
+
allowlist (an allowlist would have blocked 24 of the 48 live SSO domains). So **write access to
|
|
145
|
+
`Core.Domains` is control over where a credential payload lands.** `domain` is a plain
|
|
146
|
+
`FIELD_CHAR` with no validation and no `isActive` column, and it already holds `http://localhost`,
|
|
147
|
+
two ngrok URLs, and a malformed `'https://\040store.gamma.togacommerce.com'`. The gateway
|
|
148
|
+
structurally validates and **rebuilds** the URL via
|
|
149
|
+
`_Model_Core_ClientAuthentication::safeRedirectUrl()` before redirecting; see
|
|
150
|
+
[SAML gateway per-environment hosts & redirect safety](../../saml/features/gateway-environments-and-redirect-safety.md).
|
|
151
|
+
Related: in `Client_True` the role named "Base" maps to `coreRoleId` 3 (Super User), so ~2,826
|
|
152
|
+
users can `POST`/`PUT` `/domains`.
|
|
153
|
+
- **`/domains` responses are cached — `Core.Records` id 1 has `ttlCache = 15`.** A freshly inserted
|
|
154
|
+
row is not visible to a frontend immediately; an SSO lookup can keep returning the old answer.
|
|
155
|
+
|
|
140
156
|
## Related
|
|
141
157
|
|
|
142
158
|
- [Environment configuration & provisioning](../../api2/workflows/environment-configuration-and-provisioning.md)
|
|
@@ -6,7 +6,7 @@ project: API
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: ["mhammontree", "dfranks", "bala", "jcardinal", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- api2/Component/Api/V2/V2.php
|
|
@@ -21,6 +21,7 @@ related:
|
|
|
21
21
|
- ./v2-deadlock-retry.md
|
|
22
22
|
- ./environment-variable-drives-underscore-branch.md
|
|
23
23
|
- ../../_underscore/features/error-reporting-issue-event.md
|
|
24
|
+
- ../../worker2/features/logs-retention-cleanup.md
|
|
24
25
|
- ../../_underscore/features/order-for-delegation.md
|
|
25
26
|
- ../../../../clients/aig/features/entitlement-intake.md
|
|
26
27
|
- ../../../../clients/prudential/features/service-request-auth-rejection-alert.md
|
|
@@ -215,8 +216,10 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
|
|
|
215
216
|
- **Decode an `EO-1` via `error.id` = `<Issue.reference>-<Event.eventNumber>`** — search `Issue.reference` on the part BEFORE the dash; read `Issue.trace` (full), not the 255-char-truncated `errorMessage`.
|
|
216
217
|
- **Query prod `Logs.Api` by narrowing on `dtStamp` (indexed) first**, collect ids, then fetch `mediumtext` payloads by id with `LEFT()`/`SUBSTRING()`. Never start from a `LIKE` over `requestPayload`/`responsePayload`.
|
|
217
218
|
- **Never copy logged rows into committed artifacts** — they carry live Bearer tokens and PII.
|
|
219
|
+
- **`Logs_<Client>.Api` is pruned (once TRUE-81902 deploys)** — GET rows after 30 days, writes after 90. A missing older row is not proof the request never happened. See [Logs retention cleanup](../../worker2/features/logs-retention-cleanup.md).
|
|
218
220
|
|
|
219
221
|
## Change history
|
|
222
|
+
- 2026-09-25 — Linked the `Logs_<Client>.Api` retention job (GET 30d / writes 90d). (rgirish)
|
|
220
223
|
- 2026-09-22 — Log-save now saves/restores `_Database` error state in `try/finally`; its own 1062s were disarming the unique-race retry on `/v2/service-requests`. Also recorded the `_production`-branch missing-method hazard and the rejected null-`transactionId` finding. (rgirish)
|
|
221
224
|
|
|
222
225
|
- 2026-09-21 — Added `Logs_<Client>.Api.originalUserId` (delegation audit: NULL = none, never equal to `userId`) and the rule that outbound `direction = OUT` rows stamp neither user column. (bala)
|
|
@@ -230,6 +233,7 @@ Both a **fallback recovery path** and a **scope diagnostic**. In the TRUE-80179
|
|
|
230
233
|
- [V2 deadlock retry](./v2-deadlock-retry.md)
|
|
231
234
|
- [ENVIRONMENT drives the `_underscore` branch](./environment-variable-drives-underscore-branch.md)
|
|
232
235
|
- [Error Reporting — Issue/Event Capture](../../_underscore/features/error-reporting-issue-event.md)
|
|
236
|
+
- [Logs retention cleanup](../../worker2/features/logs-retention-cleanup.md)
|
|
233
237
|
- [AIG entitlement intake](../../../../clients/aig/features/entitlement-intake.md)
|
|
234
238
|
- [Prudential service-request auth-rejection alert](../../../../clients/prudential/features/service-request-auth-rejection-alert.md)
|
|
235
239
|
- [Compass MITS PO → SO item linking](../../../../clients/compass-usa/features/mits-po-to-so-item-linking.md)
|
|
@@ -3,7 +3,9 @@
|
|
|
3
3
|
| Doc | Summary |
|
|
4
4
|
|-----|---------|
|
|
5
5
|
| [saml Architecture](architecture.md) | Gateway internals of the `saml` SAML 2.0 SP; open when working on `/acs`, `/meta`, cert/decryption, or deploy of the SSO gateway. |
|
|
6
|
+
| [/acs assertion trust gaps — what the gateway does NOT verify](features/acs-assertion-trust-gaps.md) | The exact list of SAML checks `/acs` skips today, with evidence, so nobody assumes the gateway proves who asserted an identity. |
|
|
6
7
|
| [SAML Downstream Integration Contract](features/downstream-integration-contract.md) | Integration recipe for a downstream TOGa app to authenticate users via the SAML gateway (`saml.togahub.com`); open when wiring a new SSO-consuming app. |
|
|
8
|
+
| [SAML gateway per-environment hosts & redirect safety (samlHostUrl / safeRedirectUrl)](features/gateway-environments-and-redirect-safety.md) | Which SAML gateway host serves which environment, and how `/acs` decides where it is allowed to send the login handoff. |
|
|
7
9
|
| [SAML Uptime Healthcheck Endpoint (/healthcheck)](features/healthcheck-endpoint.md) | Secret-gated `GET /healthcheck` synthetic-SSO probe on the saml gateway; open when working on SSO uptime monitoring or the probe's seeds/guardrails. |
|
|
8
10
|
| [Rate SAML User Provisioning](features/rate-user-provisioning.md) | How a Rate SSO login finds or just-in-time provisions the user record chain in `Client_Rate`; open when touching Rate SSO or its user/customer/contact models. |
|
|
9
11
|
| [Onboarding a New SSO Client with Just-In-Time User Provisioning](workflows/onboarding-a-new-sso-client.md) | Procedure to add a new client to the 2.0 saml gateway when its users don't pre-exist and must be created on first login (JIT provisioning); open when onboarding |
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: /acs assertion trust gaps — what the gateway does NOT verify
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: saml
|
|
5
|
+
project: SAML SSO Gateway
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- saml/Controller/Index.php
|
|
13
|
+
- _underscore/String.php
|
|
14
|
+
- _underscore/Model/True/ClientAuthentication.php
|
|
15
|
+
related:
|
|
16
|
+
- ../architecture.md
|
|
17
|
+
- ./gateway-environments-and-redirect-safety.md
|
|
18
|
+
- ./downstream-integration-contract.md
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
The exact list of SAML checks `/acs` skips today, with evidence, so nobody assumes the gateway proves who asserted an identity. Open before trusting a gateway login, or when picking up the signature-verification work.
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
`/acs` accepts an assertion it has **not proven came from the expected IdP**, and its RelayState is
|
|
26
|
+
encrypted but **not authenticated**. Neither is fixed. Both need their own tickets. Everything
|
|
27
|
+
below was verified by reading `saml/Controller/Index.php` on 2026-09-25, not inferred.
|
|
28
|
+
|
|
29
|
+
## How it works — what is actually checked
|
|
30
|
+
|
|
31
|
+
The live `/acs` path does exactly this: decrypt RelayState → check version + TTL → parse the
|
|
32
|
+
`SAMLResponse` → check `STATUS_SUCCESS` → decrypt the assertion → map identity. That is all.
|
|
33
|
+
|
|
34
|
+
### Gap 1 — no IdP signature verification, and no SAML condition checks
|
|
35
|
+
|
|
36
|
+
`saml/Controller/Index.php` lines ~306-405 are **one comment block**, and the only
|
|
37
|
+
`XMLSecurityDSig::verify()` call in the file lives inside it. Grep confirms **zero** occurrences
|
|
38
|
+
anywhere outside that comment of:
|
|
39
|
+
|
|
40
|
+
- `Audience` — nothing checks the assertion was meant for us
|
|
41
|
+
- `Destination` — nothing checks it was meant for *this* gateway host
|
|
42
|
+
- `InResponseTo` — nothing binds the response to a request we sent
|
|
43
|
+
- `NotBefore` / `NotOnOrAfter` — the assertion's own validity window is ignored
|
|
44
|
+
|
|
45
|
+
Git history: the block arrived in commit `4bcee38` "Final changes for Compass SAML" and ends with
|
|
46
|
+
`echo` + `exit;` — it is debug code that was never switched on.
|
|
47
|
+
|
|
48
|
+
**Why this compounds:** `_Model_True_ClientAuthentication::getAuthenticatedSsoUser()` matches on
|
|
49
|
+
**email only**, with no issuer check. So any IdP the gateway already trusts (Compass, Prudential,
|
|
50
|
+
Office Depot) can assert a `@togatech.com` email and be handed the internal user.
|
|
51
|
+
|
|
52
|
+
The per-client IdP verification certs already exist in source as base64 blobs (Compass, Office
|
|
53
|
+
Depot, Prudential, RUMCSI, Agilant) — they are what a re-enabled verifier would load. Verification
|
|
54
|
+
is per-client; each IdP's signing cert must match its own client.
|
|
55
|
+
|
|
56
|
+
### Gap 2 — RelayState is encrypted but not authenticated
|
|
57
|
+
|
|
58
|
+
`_String::encryptWithKey()` (`_underscore/String.php` ~line 277) is **AES-256-CBC with no MAC and
|
|
59
|
+
an attacker-supplied IV**. A security review demonstrated that flipping IV bits rewrites the
|
|
60
|
+
plaintext `time` field **without the key**. So the `abs()` TTL fix (see
|
|
61
|
+
[gateway environments & redirect safety](./gateway-environments-and-redirect-safety.md)) does not
|
|
62
|
+
stop replay on its own. There is also **no nonce and no single-use marking** — a RelayState can be
|
|
63
|
+
presented any number of times.
|
|
64
|
+
|
|
65
|
+
**Fix direction:** AES-256-GCM, or an HMAC verified with `hash_equals()` **before** decrypting,
|
|
66
|
+
plus a server-side single-use nonce. Changing `encryptWithKey()` affects every caller — it is also
|
|
67
|
+
the `?saml=` handoff and the api2 encrypted-user-uuid auth, so this is not a local change.
|
|
68
|
+
|
|
69
|
+
## Gotchas
|
|
70
|
+
|
|
71
|
+
- **Do not describe the gateway as "authenticating" anyone to a client or an auditor.** Until Gap 1
|
|
72
|
+
is closed it authenticates the *originating TOGa request* (via RelayState), not the *asserting
|
|
73
|
+
IdP*.
|
|
74
|
+
- **Fixing the RelayState TTL is not replay protection.** Gap 2 makes the timestamp attacker-
|
|
75
|
+
controlled. Treat the TTL fix as hygiene, not as a control.
|
|
76
|
+
- **These are not "known and accepted" risks** — they are undone work. Do not close a ticket
|
|
77
|
+
citing this doc as sign-off.
|
|
78
|
+
|
|
79
|
+
## Change history
|
|
80
|
+
- 2026-09-25 — Created; recorded that `/acs` verifies no IdP signature and no Audience/Destination/InResponseTo/NotBefore/NotOnOrAfter, and that RelayState uses unauthenticated AES-256-CBC with an attacker-supplied IV (rgirish)
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: SAML gateway per-environment hosts & redirect safety (samlHostUrl / safeRedirectUrl)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: saml
|
|
5
|
+
project: SAML SSO Gateway
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- _underscore/Model/Core/ClientAuthentication.php
|
|
13
|
+
- saml/Controller/Index.php
|
|
14
|
+
- saml/.ebextensions/git.json
|
|
15
|
+
related:
|
|
16
|
+
- ../architecture.md
|
|
17
|
+
- ./downstream-integration-contract.md
|
|
18
|
+
- ./acs-assertion-trust-gaps.md
|
|
19
|
+
- ../workflows/rotating-the-saml-sp-certificate.md
|
|
20
|
+
- ../../_underscore/features/core-domains-app-host-registry.md
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
Which SAML gateway host serves which environment, and how `/acs` decides where it is allowed to send the login handoff. Open when SSO works in production but not in beta/sandbox, or when touching any redirect in the gateway.
|
|
24
|
+
|
|
25
|
+
## Summary
|
|
26
|
+
|
|
27
|
+
Two things used to be wrong here and both are fixed (2026-09-25, branch `TRUE-81993`):
|
|
28
|
+
|
|
29
|
+
1. **One gateway per environment, not one gateway.** `saml.beta.togahub.com` was publishing
|
|
30
|
+
**production's** entityID and ACS URL, because the beta entry in the host map pointed at
|
|
31
|
+
`https://saml.togahub.com`. An IdP configured from beta's `/meta` would post to production.
|
|
32
|
+
2. **`/acs` redirected to whatever string `Core.Domains.domain` held**, unvalidated. That is an
|
|
33
|
+
open redirect on a **credential delivery target** — the `?saml=` payload exchanges for real
|
|
34
|
+
tokens at `POST /auth/encrypted-user-uuid`.
|
|
35
|
+
|
|
36
|
+
Both now go through two helpers on `_Model_Core_ClientAuthentication`:
|
|
37
|
+
`samlHostUrl(string $environmentSlug): string` and `safeRedirectUrl(string $url): ?string`.
|
|
38
|
+
|
|
39
|
+
## How it works
|
|
40
|
+
|
|
41
|
+
### The gateway host map
|
|
42
|
+
|
|
43
|
+
`_underscore/Model/Core/ClientAuthentication.php` (~line 5):
|
|
44
|
+
|
|
45
|
+
```php
|
|
46
|
+
const SAML_ASSERTION_CONSUMER_SERVICE_HOST_URL_BY_ENVIRONMENT = [
|
|
47
|
+
'dev' => 'http://saml',
|
|
48
|
+
'beta' => 'https://saml.beta.togahub.com',
|
|
49
|
+
'sandbox-client' => 'https://saml.client.sandbox.togahub.com',
|
|
50
|
+
'production' => 'https://saml.togahub.com'
|
|
51
|
+
];
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
This single map drives **both** the outbound AuthnRequest ACS URL **and** what `/meta` publishes
|
|
55
|
+
as entityID + ACS — so a wrong entry silently cross-wires two environments.
|
|
56
|
+
|
|
57
|
+
Read it via `samlHostUrl($environmentSlug)`, never by array index. It throws when the environment
|
|
58
|
+
has no gateway. Before this, the bare index meant **23 of the 27 `Core.Environments` rows**
|
|
59
|
+
produced a PHP warning plus a malformed URL instead of a clean error. The `saml` call site wraps
|
|
60
|
+
it in try/catch because `saml()` calls `set_exception_handler(null)` — an uncaught throw renders a
|
|
61
|
+
raw PHP exception page to the browser.
|
|
62
|
+
|
|
63
|
+
### Deploy map — only THREE gateway hosts exist
|
|
64
|
+
|
|
65
|
+
27 environments in `Core.Environments`, 5 config files in `saml/Config/`, **3 live hosts**, each in
|
|
66
|
+
a **different AWS account**:
|
|
67
|
+
|
|
68
|
+
| Host | EB environment | AWS account | Region | Branch (CodePipeline) |
|
|
69
|
+
|---|---|---|---|---|
|
|
70
|
+
| `saml.togahub.com` | `saml-production` | 654654170868 | us-east-1 | `_production` |
|
|
71
|
+
| `saml.beta.togahub.com` | `true-saml-beta` | 502614707982 | us-west-2 | `_beta` |
|
|
72
|
+
| `saml.client.sandbox.togahub.com` | `saml-sandbox-client` | 975050298201 | us-east-1 | `_sandbox-client` |
|
|
73
|
+
|
|
74
|
+
- A 4th pipeline, `SAML-Production` in account 502614707982, deploys to `true-saml-beta`'s sibling
|
|
75
|
+
`true-saml-production` — **that EB environment does not exist**. The pipeline is dead. It is also
|
|
76
|
+
the only pipeline that pulls `_underscore` through CodePipeline; the three live hosts get
|
|
77
|
+
`_underscore` at deploy time via `.ebextensions/git.json`.
|
|
78
|
+
- `true-saml-beta` was **9 commits behind** as of 2026-09-25 (last deploy 2026-09-04). Beta is not
|
|
79
|
+
kept current automatically — check before you debug a beta-only SSO problem.
|
|
80
|
+
- Beta account 502614707982 has **no `toga/ssl/*` secrets and no cross-account grant**, so beta
|
|
81
|
+
`/acs` will fail `Code = 5` the moment it is used. See the cert-rotation workflow's
|
|
82
|
+
cross-account section.
|
|
83
|
+
|
|
84
|
+
### `safeRedirectUrl()` — structure validation, not an allowlist
|
|
85
|
+
|
|
86
|
+
```php
|
|
87
|
+
public static function safeRedirectUrl(string $url): ?string
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Returns a **rebuilt** `scheme://host[:port][/path]`, or `null`. It rejects userinfo, query,
|
|
91
|
+
fragment, non-`http(s)` schemes, malformed hosts, and path traversal. It **preserves a plain path**
|
|
92
|
+
because `https://tools.togatech.com/sso` is the only `Core.Domains` row with a path — and it is an
|
|
93
|
+
SSO domain.
|
|
94
|
+
|
|
95
|
+
**Redirect to the return value, never to the argument.** Checking the host and then appending
|
|
96
|
+
`'?saml='` to the *stored* string lets a stored path/query steal the handoff. Verified: a stored
|
|
97
|
+
value of `https://app.togahub.com/redirect?next=https://evil.net&x=` passes a host check and
|
|
98
|
+
forwards the payload.
|
|
99
|
+
|
|
100
|
+
### Domain ownership is the authorization
|
|
101
|
+
|
|
102
|
+
`singleSignOnServiceUrl()` takes `$domainUuid` straight from the query string and makes it the
|
|
103
|
+
RelayState redirect target. Nothing verified it belonged to the same client/environment. Three
|
|
104
|
+
guards now throw `_Exception_Validation` (HTTP 400 — **not** base `Exception`, which is a 500 plus
|
|
105
|
+
a developer alert):
|
|
106
|
+
|
|
107
|
+
1. unknown domain,
|
|
108
|
+
2. domain not owned by this `ClientAuthentication`'s client + environment,
|
|
109
|
+
3. domain fails `safeRedirectUrl()`.
|
|
110
|
+
|
|
111
|
+
**There is deliberately no host allowlist — do not add one.** Two attempts were built and rejected:
|
|
112
|
+
|
|
113
|
+
- A hardcoded suffix allowlist would have **blocked 24 of the 48 live SSO domains**, including
|
|
114
|
+
Rate's production SSO (`hometechsupport.rate.com`) and every `agilantsolutions.com` internal tool.
|
|
115
|
+
- Moving the list to a `Core.Parameters` row (`SAML_REDIRECT_ALLOWED_HOSTS`) was rejected too — it
|
|
116
|
+
is the same list, one indirection away, and it duplicates authorization that `Core.Domains`
|
|
117
|
+
already holds.
|
|
118
|
+
|
|
119
|
+
**Final design:** `safeRedirectUrl()` validates URL *structure* only. The existence of a row in
|
|
120
|
+
`Core.Domains`, owned by the right client and environment, **is** the authorization. Net diff fell
|
|
121
|
+
165 → 98 → 63 lines.
|
|
122
|
+
|
|
123
|
+
**Tradeoff accepted, and it is real:** write access to `Core.Domains` equals control of where a
|
|
124
|
+
login payload lands. `Core.Domains.domain` is a plain `FIELD_CHAR` with no validation and no
|
|
125
|
+
`isActive` column, and it already holds `http://localhost`, two ngrok URLs, and a malformed
|
|
126
|
+
`'https://\040store.gamma.togacommerce.com'`. See the Client_True "Base" role gotcha below.
|
|
127
|
+
|
|
128
|
+
### RelayState TTL
|
|
129
|
+
|
|
130
|
+
The replay check was written backwards:
|
|
131
|
+
|
|
132
|
+
```php
|
|
133
|
+
if ($relayState->time <= (time() + self::MAX_TIME_TO_LIVE)) // true for EVERY past timestamp
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
A captured RelayState never expired. Now:
|
|
137
|
+
|
|
138
|
+
```php
|
|
139
|
+
if (abs(time() - $relayState->time) <= self::MAX_TIME_TO_LIVE)
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
**This alone does not stop replay** — RelayState is encrypted but not authenticated, so the `time`
|
|
143
|
+
field can be rewritten without the key. See [ACS assertion trust gaps](./acs-assertion-trust-gaps.md).
|
|
144
|
+
|
|
145
|
+
## Gotchas
|
|
146
|
+
|
|
147
|
+
- **⚠ `Core.Domains` write access = redirect control, and in `Client_True` almost everyone has it.**
|
|
148
|
+
`Core.Records` id 1 exposes `Domains` as REST route `domains`, and `AclRecordPermissions` grants
|
|
149
|
+
Super User (`coreRoleId` 3) full CRUD. In `Client_True` the role **named** "Base" maps to
|
|
150
|
+
`coreRoleId` 3, so roughly **2,826 active users** can `POST`/`PUT` `/domains`. `Client_Compass`
|
|
151
|
+
maps its "Base" to `coreRoleId` 1 (Public) and keeps a separate SuperUser role with 115 people —
|
|
152
|
+
so this looks like a `Client_True` misconfiguration, not the intended model. Needs its own ticket.
|
|
153
|
+
- **A config file in `saml/Config/` does not mean a gateway exists.** 5 configs, 3 hosts.
|
|
154
|
+
- **Credentials are committed in this repo.** `saml/Config/production.ini` (DB password, AWS access
|
|
155
|
+
key, NetSuite creds) and `.ebextensions/git.json` (a GitHub PAT). Documented in the architecture
|
|
156
|
+
doc's gotchas; treat all of them as compromised until rotated.
|
|
157
|
+
|
|
158
|
+
## Change history
|
|
159
|
+
- 2026-09-25 — Added `samlHostUrl()` (throws instead of a bare index that broke 23 of 27 environments), fixed beta's ACS host pointing at production, added `safeRedirectUrl()` closing the `/acs` open redirect, fixed the backwards RelayState TTL check, added `$domainUuid` ownership guards; recorded the deploy map and the decision to use `Core.Domains` ownership instead of an allowlist (rgirish)
|
|
@@ -6,8 +6,8 @@ project: TOGa Commerce
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: [tcox, apeterson]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: [tcox, apeterson, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- toga2-commerce/src/lib/session.ts
|
|
13
13
|
- toga2-commerce/src/hooks/useAuthenticationFlow.ts
|
|
@@ -24,6 +24,8 @@ related:
|
|
|
24
24
|
- ../architecture.md
|
|
25
25
|
- ../../toga25-supply/features/sso-redirect-and-session-gating.md
|
|
26
26
|
- ../../../standards/frontend.md
|
|
27
|
+
- ../../../clients/compass-usa/features/saml-sso.md
|
|
28
|
+
- ../../saml/features/gateway-environments-and-redirect-safety.md
|
|
27
29
|
---
|
|
28
30
|
|
|
29
31
|
How commerce decides whether a visitor is signed in, shows the SSO panel or the credential form, and keeps a token refresh bound to one identity. Open for "blank page at /", "SSO never fires", intermittent forced logout, or cross-user residue.
|
|
@@ -73,6 +75,7 @@ api2 **rotates** the refresh token. Concurrent 401s each spent a token the other
|
|
|
73
75
|
- **⚠ Clearing a persisted key without resetting the store in memory does nothing** — the next `set()` writes the old user's state straight back.
|
|
74
76
|
- **⚠ `zu-cart-store` is not a real key.** The cart persists under **`zu-cart`** (`src/stores/useCartStoreZu.ts`). The wrong name in the clear list left the previous user's cart alive through a 401 logout.
|
|
75
77
|
- **⚠ Open — Compass Canada SSO is broken in production.** `https://compasscanada.togacommerce.com/home` shows the credential form though the client is SSO; the lookup returns no match. Candidates in order: no `Domains` row matching the exact `https://compasscanada.togacommerce.com` string; no `ClientAuthentications` row for that clientId+environmentId; `type` not the literal `SSO`; or the PUBLIC role lacking `AclFieldPermissions` to read `ClientAuthentications.type` (the call runs pre-login on the public token, so a null field fails silently). Diagnostic SQL handed over, **not yet run**. The production build is plain `vite build`, so `MODE` is `production` and the query asks for `Environments.slug` `PRODUCTION`.
|
|
78
|
+
- **⚠ `/domains` is cached — `Core.Records` id 1 carries `ttlCache = 15`.** After inserting or changing `Domains` / `ClientAuthentications` rows, the API can keep serving the old "no SSO" answer, so an immediate retest looks like the rows did not take. Suspected cause of Compass Canada SSO still showing `/login` on sandbox-client on 2026-09-25 after the rows were confirmed inserted.
|
|
76
79
|
- **⚠ Open — toga-blox has the same public-token bug.** `toga-blox-npm/src/api/auth.ts` `fetchPublicToken` writes the anonymous token to `accessToken`; toga25-supply works around it in three places instead of fixing it. Needs its own ticket.
|
|
77
80
|
- **⚠ Open — the SSO URL's host is not constrained** to the tenant's configured IdP domain (scheme only). No allowlist source exists today.
|
|
78
81
|
- **⚠ Open — `AuthContext` still monkey-patches `window.history` `pushState`/`replaceState`.** Sentry patches history too, so the cleanup can clobber it; supply's poll-on-interval + focus is the replacement. Also `login()` still swallows errors after tokens are written.
|
|
@@ -83,5 +86,6 @@ api2 **rotates** the refresh token. Concurrent 401s each spent a token the other
|
|
|
83
86
|
- **No per-session key prefix, and tokens stay in `localStorage`.** Renaming every zustand `persist` name silently orphans existing users' state on deploy and touches every call site; supply's cheaper answer is plain `localStorage.clear()` on logout since every key commerce writes is session state. httpOnly cookies would need backend work plus CSRF protection that does not exist — api2 is Bearer-only. Matches `2.0/standards/frontend.md` §12.
|
|
84
87
|
|
|
85
88
|
## Change history
|
|
89
|
+
- 2026-09-25 — Recorded that `GET /domains` is cached (`Core.Records` id 1, `ttlCache = 15`), the likely reason freshly inserted SSO rows still fall through to `/login` (rgirish)
|
|
86
90
|
- 2026-09-22 — Public token moved out of `localStorage`; `hasSession()` + identity epoch + `AUTH_STORAGE_KEYS` added; SSO sign-in screen replaces the auto IdP jump; open-redirect and SSO-URL-scheme holes closed; single-flight refresh. TRUE-81991 (`e546db3d`). (apeterson)
|
|
87
91
|
- 2026-09-21 — Blank page at `/` fixed: gate on stored `user.uuid`, `AuthRedirect` renders the spinner instead of `null`. Branch `TRUE-82172` (`1eebdd96`, `3dc53d9a`), **not merged or deployed**. (tcox)
|
|
@@ -30,6 +30,7 @@
|
|
|
30
30
|
| [Error-Issue Auto-Resolution & Reopen (frequency-decay lifecycle)](features/error-issue-auto-resolution.md) | Issue-level OPEN→RESOLVED lifecycle on the escalation cron: auto-resolves an Issue after events decay to zero for a baseline-proportional quiet duration, auto-r |
|
|
31
31
|
| [Etilize Catalog Item Import & Refresh](features/etilize-catalog-item-import.md) | Client-generic catalog onboarding: ingest an item catalog from an S3 CSV and enrich each item from the Etilize datasheet, writing directly to `DB_CLIENT`; open |
|
|
32
32
|
| [Etilize Item Translation Import](features/etilize-item-translation-import.md) | Imports non-English item + feature text from Etilize into the client's `ItemTranslations` (and feature sidecar tables), writing directly to `DB_CLIENT`; open wh |
|
|
33
|
+
| [Logs retention cleanup worker (Infrastructure/Logs/Cleanup — Logs_<Tenant>.Api pruning)](features/logs-retention-cleanup.md) | Daily batched pruning of `Logs_<Tenant>.Api` (GET 30d, other methods 90d); open when changing log retention, adding a table to prune, or when an old Api log row |
|
|
33
34
|
| [Monitoring Framework (Orchestrator + Child Monitors)](features/monitoring-framework.md) | DB-driven monitoring framework: one orchestrator applies anti-flap + notification rules to any registered "is X healthy?" child check; open when adding a monito |
|
|
34
35
|
| [Bidirectional NetSuite ↔ Forecast ↔ ClickUp opportunity stage sync — BUILT (stage map, echo guard, claim split)](features/netsuite-clickup-opportunity-bidirectional-design.md) | Two-way NetSuite↔ClickUp opportunity STAGE sync (shared `_Component_Opportunity_StageMap`, actor echo guard, claim split, stale-claim takeover); open before tou |
|
|
35
36
|
| [NetSuite Integrations Monitor (Monitor/Operations/NetsuiteIntegrations)](features/netsuite-integrations-monitor.md) | Cross-client health check that flags stuck NetSuite↔2.0 integrations; open when a NetSuite integration hangs or when building an all-client OneUptime sweep. |
|
|
@@ -6,8 +6,8 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: [jcardinal, ajean]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: [jcardinal, ajean, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Controller/Index.php
|
|
13
13
|
- worker2/Worker/
|
|
@@ -102,7 +102,9 @@ Split the action on `/`: last segment = method; everything before = class segmen
|
|
|
102
102
|
|
|
103
103
|
## Cleanup cron
|
|
104
104
|
|
|
105
|
-
`_Worker_Infrastructure_Worker_Cleanup`: `WorkerJobs()` deletes `isSuccess=1` rows >90 days (≤1000/run); `WebhookLogs()` deletes `Logs.Webhook` >90 days. Two daily CronJobs at 2 AM.
|
|
105
|
+
- `_Worker_Infrastructure_Worker_Cleanup`: `WorkerJobs()` deletes `isSuccess=1` rows >90 days (≤1000/run); `WebhookLogs()` deletes `Logs.Webhook` >90 days. Two daily CronJobs at 2 AM.
|
|
106
|
+
- `Infrastructure/Email/Cleanup`: `Email` / `EmailAttachment` >90 days, 03:00 daily (prod CronJobs id 36), one unbatched multi-table DELETE.
|
|
107
|
+
- `Infrastructure/Logs/Cleanup`: batched pruning of every `Logs_<Tenant>.Api` (GET 30d, writes 90d), 04:00 CT. Audit-trail tables are not pruned. **Not deployed yet (TRUE-81902).** See [Logs retention cleanup](features/logs-retention-cleanup.md).
|
|
106
108
|
|
|
107
109
|
## Cron actions must return a result on BOTH paths
|
|
108
110
|
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Logs retention cleanup worker (Infrastructure/Logs/Cleanup — Logs_<Tenant>.Api pruning)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: draft
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Infrastructure/Logs/Cleanup.php
|
|
13
|
+
- dbchanges2/Core/2026-09-11a - CronJobs_LogsCleanup_insert.sql
|
|
14
|
+
related:
|
|
15
|
+
- ../architecture.md
|
|
16
|
+
- ./platform-cache-cleanup.md
|
|
17
|
+
- ../../api2/features/request-logging.md
|
|
18
|
+
- ../../_underscore/features/record-change-audit-log.md
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
Daily batched pruning of `Logs_<Tenant>.Api` (GET 30d, other methods 90d); open when changing log retention, adding a table to prune, or when an old Api log row is missing.
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
`_Worker_Infrastructure_Logs_Cleanup::Run(int $maxAgeDays = 90, int $getMaxAgeDays = 30)`, action
|
|
26
|
+
`Infrastructure/Logs/Cleanup`. Deletes old rows from **every** `Logs_<Tenant>.Api`:
|
|
27
|
+
|
|
28
|
+
| Rows | Kept for |
|
|
29
|
+
|---|---|
|
|
30
|
+
| `method = GET` | 30 days (`getMaxAgeDays`) |
|
|
31
|
+
| POST / PUT / PATCH / DELETE | 90 days (`maxAgeDays`) |
|
|
32
|
+
|
|
33
|
+
**Status (2026-09-25): branch `TRUE-81902`, not merged or deployed.** The CronJobs row is not in prod
|
|
34
|
+
or dev-sandbox `Core.CronJobs` yet. Why: `Logs.Api` is ~96% of platform storage (1,644 GB of the
|
|
35
|
+
1,705 GB logs cluster). GETs are read noise; writes are kept longer for troubleshooting.
|
|
36
|
+
|
|
37
|
+
**Not pruned (decided):**
|
|
38
|
+
- `Record` / `RecordField` / `CustomRecordField` (audit trail) — kept until an archive policy exists.
|
|
39
|
+
- `Email` / `EmailAttachment` — stay with the existing `Infrastructure/Email/Cleanup` job (prod
|
|
40
|
+
CronJobs id 36, 03:00 daily, 90 days, one unbatched multi-table DELETE). Not folded into this job
|
|
41
|
+
on purpose.
|
|
42
|
+
|
|
43
|
+
This replaces `LOG-RETENTION-90-DAY-PLAN.md`'s "90 days for every table, audit trail included".
|
|
44
|
+
|
|
45
|
+
## How it works
|
|
46
|
+
|
|
47
|
+
1. **Cron** (`dbchanges2/Core/2026-09-11a - CronJobs_LogsCleanup_insert.sql`): daily 04:00 CT,
|
|
48
|
+
`maxExecutionTime` 18000, parameters `{"maxAgeDays": 90, "getMaxAgeDays": 30}`, `isActive=1`,
|
|
49
|
+
guarded by `NOT EXISTS` on `action` so a re-run inserts nothing.
|
|
50
|
+
2. **Cutoffs are computed once per run**, then applied to each `Logs_<Tenant>` DB. The database name
|
|
51
|
+
is checked with a regex before it goes into SQL.
|
|
52
|
+
3. **Table-driven.** The `TABLES` const holds rows of
|
|
53
|
+
`[table, parentTable, parentKey, dateColumn, method, age-param-name]`. To prune another table,
|
|
54
|
+
add a row — no new code.
|
|
55
|
+
- **Children before parents** (FKs have no cascade). If a child hits the batch cap, its parent is
|
|
56
|
+
**deferred** to the next run.
|
|
57
|
+
- The **all-methods Api row runs before the GET row**, so the GET pass does not rescan rows the
|
|
58
|
+
90-day pass would delete anyway.
|
|
59
|
+
4. **Batched delete:** `DELETE ... LIMIT 5000`, **no `ORDER BY`**, up to `MAX_BATCHES_PER_TABLE`
|
|
60
|
+
(2000) batches per table, 100 ms sleep between batches.
|
|
61
|
+
5. **Returns a JSON summary** to `WorkerJobs.output`: cutoffs, deleted rows per db/label,
|
|
62
|
+
`truncated`, `deferred` — per the
|
|
63
|
+
[return-on-both-paths rule](../architecture.md#cron-actions-must-return-a-result-on-both-paths).
|
|
64
|
+
|
|
65
|
+
## Gotchas
|
|
66
|
+
|
|
67
|
+
- **No `ORDER BY id` on the batched DELETE** — it forces a filesort on a huge table.
|
|
68
|
+
- **Watch item:** the GET pass re-scans 30–90-day non-GET rows on every batch via `Api_dtStamp_IDX`.
|
|
69
|
+
If GET-pass time grows or `truncated` shows up in the output, add a composite `(method, dtStamp)`
|
|
70
|
+
index on `Api` via `dbchanges2/Logs_Client/`. Not needed now.
|
|
71
|
+
- **A missing Api log row older than 30 days (GET) / 90 days (write) is not proof the request never
|
|
72
|
+
happened** — once deployed, it was pruned. See [request logging](../../api2/features/request-logging.md).
|
|
73
|
+
- **Scheduling is a dbchanges2 change** — the job does not run in an environment until the CronJobs
|
|
74
|
+
SQL is applied there.
|
|
75
|
+
|
|
76
|
+
## Change history
|
|
77
|
+
- 2026-09-25 — Created: Api-only pruning (GET 30d / others 90d), audit trail and Email left out; on branch TRUE-81902, not deployed (rgirish)
|
|
78
|
+
|
|
79
|
+
## Related
|
|
80
|
+
|
|
81
|
+
- [Worker (worker2) Architecture](../architecture.md)
|
|
82
|
+
- [Platform Cache Cleanup](./platform-cache-cleanup.md)
|
|
83
|
+
- [V2 request logging](../../api2/features/request-logging.md)
|
|
84
|
+
- [Record change audit log](../../_underscore/features/record-change-audit-log.md)
|
|
@@ -6,8 +6,8 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["jcardinal", "mhammontree", "bala"]
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["jcardinal", "mhammontree", "bala", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Monitor/Compass.php
|
|
13
13
|
- worker2/Worker/Client/Compass.php
|
|
@@ -31,6 +31,7 @@ related:
|
|
|
31
31
|
- ../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md
|
|
32
32
|
- ../../../clients/staples/features/oneuptime-sftp-folder-monitors.md
|
|
33
33
|
- ../../../clients/compass-usa/features/oneuptime-ma-refresh-order-monitor.md
|
|
34
|
+
- ../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
|
|
34
35
|
---
|
|
35
36
|
|
|
36
37
|
OneUptime push-metric monitor pattern for worker2 (2.0): a self-contained cron action decides pass/fail and POSTs a string token to a OneUptime Incoming Request monitor; open when building/tuning a worker2 monitor, its thresholds, or the OneUptime import JSON.
|
|
@@ -41,8 +42,8 @@ A second, **OneUptime-reporting** monitoring pattern for the 2.0 worker2 tier, p
|
|
|
41
42
|
`App_SystemMonitor_Compass` monitors. A worker2 cron action runs a self-contained check, decides
|
|
42
43
|
pass/fail itself, and POSTs a JSON metric body to a OneUptime "Incoming Request" monitor — replacing
|
|
43
44
|
the 1.0 tier's email alerts. The class is `abstract class _Worker_Monitor_Compass` and now carries a
|
|
44
|
-
**suite of
|
|
45
|
-
checks). Designed to be **client-agnostic** — a reusable template for future monitors and clients.
|
|
45
|
+
**suite of 12 monitors** covering the Compass USA / Office Depot integration (S3, DB, mailbox, and
|
|
46
|
+
NetSuite SuiteQL checks). Designed to be **client-agnostic** — a reusable template for future monitors and clients.
|
|
46
47
|
|
|
47
48
|
Distinct from the DB-driven email-orchestrator [Monitoring Framework](./monitoring-framework.md)
|
|
48
49
|
(`_Worker_Monitor` orchestrator + `_Worker_Monitors_<Name>` children + `Core.Monitors` table, sends
|
|
@@ -69,7 +70,7 @@ disallows `const` at function scope), plus a `$push` closure using `_ApiRequest`
|
|
|
69
70
|
method, so the class scales cleanly as monitors accumulate. The metric POST runs with **logging
|
|
70
71
|
disabled** and **`throwExceptionsOnFailure` disabled**, so a failed ping never fails the worker job.
|
|
71
72
|
|
|
72
|
-
### The
|
|
73
|
+
### The 12 monitors
|
|
73
74
|
|
|
74
75
|
Each measures one Compass/Office Depot integration metric and pushes a decided alarm token.
|
|
75
76
|
|
|
@@ -126,6 +127,11 @@ Each measures one Compass/Office Depot integration metric and pushes a decided a
|
|
|
126
127
|
'MA%'`), so this monitor exists to close that carve-out. Status-page name *"MA Employee Refresh Orders
|
|
127
128
|
(Office Depot)"*. DEGRADED >10 waiting / >32h oldest, HIGH >20 / >72h. Full derivation in
|
|
128
129
|
[Compass: MA refresh-order monitor](../../../clients/compass-usa/features/oneuptime-ma-refresh-order-monitor.md).
|
|
130
|
+
- **NetsuiteDuplicateSalesOrders** (NetSuite SuiteQL, added 2026-09-25) — Office Depot POs with more than
|
|
131
|
+
one **open** NetSuite SO in the last 7 days. `Core.CronJobs` id 80, `*/15`. **First monitor that queries
|
|
132
|
+
NetSuite instead of a Toga DB** (via `_Component_Api_Netsuite::send()`, `QUERY_SUITEQL`), because the
|
|
133
|
+
duplicates exist only in NetSuite. Details:
|
|
134
|
+
[Compass: NetSuite duplicate SO monitor](../../../clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md).
|
|
129
135
|
|
|
130
136
|
### Payload contract & OneUptime criteria (critical — OneUptime cannot compare numbers)
|
|
131
137
|
|
|
@@ -551,3 +557,6 @@ production deploy.**
|
|
|
551
557
|
- [Monitoring Framework](./monitoring-framework.md) — the parallel DB-driven, email-alert monitoring pattern
|
|
552
558
|
- [Cloud S3 helpers](../../_underscore/features/cloud-s3-helpers.md) — `_Cloud::getS3Objects()` used to list the EDI bucket
|
|
553
559
|
- [OneUptime 1.0 worker uptime monitoring](../../../1.0/apps/worker/features/oneuptime-worker-uptime-monitoring.md) — the 1.0 push-heartbeat predecessor
|
|
560
|
+
|
|
561
|
+
## Change history
|
|
562
|
+
- 2026-09-25 — Added `NetsuiteDuplicateSalesOrders`, the first monitor that queries NetSuite (SuiteQL) instead of a Toga DB. (rgirish)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -20,11 +20,11 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
20
20
|
## 2.0 framework
|
|
21
21
|
|
|
22
22
|
- **_underscore** (_Underscore) _(framework core)_ — 90 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
23
|
-
- **worker2** (Worker) —
|
|
23
|
+
- **worker2** (Worker) — 80 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
24
24
|
- **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
25
25
|
- **dbchanges2** (Database Changes) _(framework core)_ — 19 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
26
26
|
- **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
27
|
-
- **saml** (SAML SSO Gateway) —
|
|
27
|
+
- **saml** (SAML SSO Gateway) — 9 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
|
|
28
28
|
- **apiproxy** (API Proxy) — 1 doc(s) → [2.0/apps/apiproxy/INDEX.md](2.0/apps/apiproxy/INDEX.md)
|
|
29
29
|
- **toga2-view** (TOGa View Frontend) — 12 doc(s) → [2.0/apps/toga2-view/INDEX.md](2.0/apps/toga2-view/INDEX.md)
|
|
30
30
|
- **toga2-hub** (TOGa Hub) — 2 doc(s) → [2.0/apps/toga2-hub/INDEX.md](2.0/apps/toga2-hub/INDEX.md)
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
title: Compass Canada
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
apps:
|
|
5
|
+
- saml
|
|
5
6
|
- _underscore
|
|
6
7
|
- api2
|
|
7
8
|
- toga2-supply
|
|
@@ -17,8 +18,8 @@ project: _Underscore
|
|
|
17
18
|
client: compass-canada
|
|
18
19
|
type: profile
|
|
19
20
|
status: active
|
|
20
|
-
updated: 2026-09-
|
|
21
|
-
owners: [jcardinal, bala, tcox, apeterson, ajean, mhammontree]
|
|
21
|
+
updated: 2026-09-25
|
|
22
|
+
owners: [jcardinal, bala, tcox, apeterson, ajean, mhammontree, rgirish]
|
|
22
23
|
files: []
|
|
23
24
|
related:
|
|
24
25
|
- ../compass-usa/profile.md
|
|
@@ -37,6 +38,7 @@ related:
|
|
|
37
38
|
- features/item-fulfillment-tableviews.md
|
|
38
39
|
- ../../2.0/apps/toga2-commerce/features/expedited-shipping-gating.md
|
|
39
40
|
- ../../2.0/apps/toga2-commerce/features/sso-redirect-and-session-gating.md
|
|
41
|
+
- ../compass-usa/features/saml-sso.md
|
|
40
42
|
- ../../2.0/apps/_underscore/features/item-fulfillment-stage-lifecycle-and-order-status.md
|
|
41
43
|
- ../../2.0/apps/_underscore/features/surface-resolver.md
|
|
42
44
|
---
|
|
@@ -20,11 +20,13 @@
|
|
|
20
20
|
| [Compass ODP EDI File Retention & Import Retry (cron 3a)](features/odp-edi-file-retention-and-retry.md) | 1.0 | When cron 3a keeps vs deletes an ODP EDI file in S3, and why an api2 outage used to silently lose orders; open for lost ODP imports, retry behaviour, or the sti |
|
|
21
21
|
| [Compass: ODP / Strategic Systems ASN CSV email import (and the scientific-notation tracking-number loss)](features/odp-strategic-systems-asn-csv-import.md) | 1.0 | The 1.0 worker cron that reads Office Depot / Strategic Systems ASN CSVs from email, why it routes by column count, and why a tracking number can arrive as `8.7 |
|
|
22
22
|
| [Compass: MA refresh-order monitor (Monitor/Compass/MaRefreshOrderExceptions)](features/oneuptime-ma-refresh-order-monitor.md) | 2.0 | worker2 OneUptime push monitor counting Compass MA employee-refresh orders whose ODP SO has no NetSuite order yet; open when working on Compass MA monitoring or |
|
|
23
|
+
| [Compass: NetSuite duplicate sales-order monitor (Monitor/Compass/NetsuiteDuplicateSalesOrders)](features/oneuptime-netsuite-duplicate-sales-orders-monitor.md) | 2.0 | worker2 OneUptime push monitor that queries NetSuite (SuiteQL) for Office Depot POs with more than one open NetSuite sales order; open when working on Compass d |
|
|
23
24
|
| [Compass USA — Fulfilled vs Partially Fulfilled (order total kept; per-line rule built then rejected)](features/order-fulfillment-status-per-line.md) | 2.0 | Why Compass USA shows Fulfilled on partly-shipped orders and why the per-line fix was built, measured, then reverted; open before touching Compass USA `_status` |
|
|
24
25
|
| [People File Import admin page (tools /compass/people-file-import → worker2)](features/people-file-import-admin-page.md) | 1.0 | Browser page that lists the Compass PEOPLE files on SFTP and runs the importer on demand; open before changing the page or the importer's listing/guard code. |
|
|
25
26
|
| [Compass PEOPLE-File User Lifecycle (duplicate accounts, reactivation grace window, raw-SQL deactivation)](features/people-file-user-lifecycle.md) | 2.0 | The nightly Compass PEOPLE-file cron's Users-row lifecycle (insert/reactivate/deactivate), why it creates duplicate accounts, and its raw-SQL/index traps; open |
|
|
26
27
|
| [Persona Model & Levy-Sector Gating (worker2 PEOPLE cron)](features/persona-model-and-levy-gating.md) | 2.0 | Compass USA catalogue visibility is driven by **personas** in `Client_Compass`. |
|
|
27
28
|
| [Compass Sales-Order Line-Number Renumbering (and why it hangs off the SO hooks)](features/sales-order-line-renumbering.md) | 2.0 | How Compass keeps SO lines numbered 1..N with no gaps, why the renumber rides the parent SO hooks, and the two-phase SQL that avoids a duplicate-key fatal; open |
|
|
29
|
+
| [Compass SAML SSO (Entra) — identity mapping, the @togatech.com bypass, and per-environment enablement](features/saml-sso.md) | 2.0 | How Compass users are matched from an Entra SAML assertion, the hardcoded `@togatech.com` test bypass that is live in production, and what it takes to turn SSO |
|
|
28
30
|
| [Stranded Approval Reassignment (repointing approvals off dead duplicate Compass Users rows)](features/stranded-approval-reassignment.md) | 2.0 | The engine that repoints approvals stranded on dead duplicate Compass Users rows onto the live account; open when approvals are stuck on a re-hired employee's o |
|
|
29
31
|
| [Compass: where a `TrackingNumbers` row comes from (invoice sync, NOT the ASN) and how it reaches a customer](features/tracking-number-origin-and-asn-link.md) | 1.0 | Why a Compass TrackingNumbers row is born from the invoice sync (not the ASN) and why `dtCreated` is the wrong clock for any shipping notification/SLA; open bef |
|
|
30
32
|
| [Compass USA](profile.md) | 2.0 | Compass USA client profile: platforms, vendors/integrations, and the key features/traps across its 1.0+2.0 supply-chain commerce stack; open first when starting |
|
package/knowledge/clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Compass: NetSuite duplicate sales-order monitor (Monitor/Compass/NetsuiteDuplicateSalesOrders)"
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: compass-usa
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/Worker/Monitor/Compass.php
|
|
13
|
+
- worker/crons/toga2/compass/workflow/5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php
|
|
14
|
+
- dbchanges2/Core/2026-09-25b - Compass NetSuite Duplicate Sales Orders Monitor CronJob.sql
|
|
15
|
+
related:
|
|
16
|
+
- ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
|
|
17
|
+
- ../workflows/odp-order-pipeline-to-netsuite.md
|
|
18
|
+
- ../../../1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md
|
|
19
|
+
- ./oneuptime-ma-refresh-order-monitor.md
|
|
20
|
+
- ../profile.md
|
|
21
|
+
---
|
|
22
|
+
worker2 OneUptime push monitor that queries NetSuite (SuiteQL) for Office Depot POs with more than one open NetSuite sales order; open when working on Compass duplicate NetSuite SOs, cron 5 stamp failures, or any worker2 monitor that must query NetSuite.
|
|
23
|
+
|
|
24
|
+
## Summary
|
|
25
|
+
`Monitor/Compass/NetsuiteDuplicateSalesOrders` (`_Worker_Monitor_Compass::NetsuiteDuplicateSalesOrders()`) alarms when any Office Depot PO has more than one **open** NetSuite SO. It follows the shared [OneUptime push-metric pattern](../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md) — read that for payload contract and provisioning.
|
|
26
|
+
|
|
27
|
+
- **First worker2 OneUptime monitor that queries NetSuite, not a Toga DB.** Cron 5 duplicates exist ONLY in NetSuite (`Client_Compass` has one SO row), so a DB monitor cannot see them.
|
|
28
|
+
- It is the backstop for cron 5's unconfirmed forked stamp (see [pipeline gotcha](../workflows/odp-order-pipeline-to-netsuite.md#gotchas)).
|
|
29
|
+
- `Core.CronJobs` **id 80**, `*/15 * * * *`, `maxExecutionTime` 120, `parameters` NULL. Seed: `dbchanges2/Core/2026-09-25b - Compass NetSuite Duplicate Sales Orders Monitor CronJob.sql`. Run time ~6 s.
|
|
30
|
+
- Status page: Compass group, suggested name **"Office Depot Duplicate Orders in NetSuite"**.
|
|
31
|
+
- Push URL = in-method `$ONEUPTIME_URL` local (credential; never record the value).
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
### Query (SuiteQL via `_Component_Api_Netsuite::send()`)
|
|
36
|
+
POST `QUERY_SUITEQL` with header `Prefer: transient`:
|
|
37
|
+
- `transaction.type = 'SalesOrd'`, `entity = 35581` ("ODP Veyer (B2B)" — cron 5 raises every SO on it).
|
|
38
|
+
- `otherrefnum LIKE '________-____%'` (ODP PO shape).
|
|
39
|
+
- `createddate >= SYSDATE - 7`.
|
|
40
|
+
- `GROUP BY otherrefnum HAVING COUNT(*) > 1`; per group count **open** = status not `'H'` (Closed) and not voided.
|
|
41
|
+
|
|
42
|
+
Coverage: cron 5 is the only automated NetSuite SO creator for Compass. worker2 `Netsuite/SalesOrder/CreateNetSuite` is not used for Compass; `worker/crons/toga2/compass/edi/2_create_so_netsuite.php` is unscheduled dead code. MA orders are hand-keyed on the same entity.
|
|
43
|
+
|
|
44
|
+
### Payload + alarm
|
|
45
|
+
- `alarm:"HIGH"` when any PO has `openSalesOrderCount > 1` (even one).
|
|
46
|
+
- Groups whose extras are all Closed count as `cleanedDuplicateCount` (info only), so the alarm clears once ops closes the extra NetSuite SOs.
|
|
47
|
+
- Pushes `offendingPurchaseOrders` (comma list) + `duplicates[]` (max 20).
|
|
48
|
+
- SuiteQL failure → `status:"error"`, `error:"netsuite_query_failed"`; message capped at 500 chars because `send()` embeds the raw NetSuite body in its exception.
|
|
49
|
+
|
|
50
|
+
### OneUptime criteria
|
|
51
|
+
| Criterion | Result |
|
|
52
|
+
|---|---|
|
|
53
|
+
| Body contains `"alarm":"HIGH"` | Same status as sibling Compass Sales Order Transmissions HIGH (orange/Degraded; status id copied from that monitor's export) + incident titled `{{requestBody.offendingPurchaseOrders}}` |
|
|
54
|
+
| Body contains `"status":"error"` | Offline + incident |
|
|
55
|
+
| Received in 16 min + `"alarm":"OK"` | Online |
|
|
56
|
+
| Not received in 35 min | Offline (imported `isEnabled:false`; enable after first push) |
|
|
57
|
+
|
|
58
|
+
No 20-min Degraded criterion — the source export had no Degraded status id. Import JSON was generated from a real export of the sibling monitor; not committed (import artifact).
|
|
59
|
+
|
|
60
|
+
## Gotchas
|
|
61
|
+
- **⚠ The ODP-PO-shape filter is required.** Hand-keyed MA orders on entity 35581 carry junk `otherrefnum` values that repeat on purpose (`TBD` x7, a person's name x5 in Aug 2026). Without the filter they false-alarm.
|
|
62
|
+
- **Cron 5 joins several ODP POs into one `otherrefnum` with `" + "`** (e.g. `42019207-5125 + 42019108-5125`). Such a row still matches the shape filter and groups on the full joined string.
|
|
63
|
+
- **HIGH path not yet exercised end to end** (as of 2026-09-25). Only the OK path is proven in prod.
|
|
64
|
+
|
|
65
|
+
## Change history
|
|
66
|
+
- 2026-09-25 — Created and deployed to prod; verified cronJobId 80 runs 11:45–12:45 all `isSuccess=1` ("0 open duplicate PO(s), 11 already cleaned in the last 7 days, alarm=OK; OneUptime push ok"). (rgirish)
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Compass SAML SSO (Entra) — identity mapping, the @togatech.com bypass, and per-environment enablement
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: saml
|
|
5
|
+
project: SAML SSO Gateway
|
|
6
|
+
client: compass-usa
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-25
|
|
10
|
+
owners: ["rgirish"]
|
|
11
|
+
files:
|
|
12
|
+
- _underscore/Model/Compass/ClientAuthentication.php
|
|
13
|
+
- saml/Controller/Index.php
|
|
14
|
+
related:
|
|
15
|
+
- ../../../2.0/apps/saml/features/downstream-integration-contract.md
|
|
16
|
+
- ../../../2.0/apps/saml/features/gateway-environments-and-redirect-safety.md
|
|
17
|
+
- ../../../2.0/apps/saml/features/acs-assertion-trust-gaps.md
|
|
18
|
+
- ../../compass-canada/profile.md
|
|
19
|
+
- ../../../2.0/apps/toga2-commerce/features/sso-redirect-and-session-gating.md
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
How Compass users are matched from an Entra SAML assertion, the hardcoded `@togatech.com` test bypass that is live in production, and what it takes to turn SSO on for a Compass tenant in a non-production environment.
|
|
23
|
+
|
|
24
|
+
## Summary
|
|
25
|
+
|
|
26
|
+
Compass overrides the base SSO mapper in `_Model_Compass_ClientAuthentication::getAuthenticatedSsoUser()`.
|
|
27
|
+
It matches users on **`c_hrEmpUsername`** from the SAML NameID — not on email, which is what the
|
|
28
|
+
base `_Model_True_ClientAuthentication` does.
|
|
29
|
+
|
|
30
|
+
Compass_Usa and Compass_Canada share the **same gateway and the same Entra app** per environment,
|
|
31
|
+
so enabling a second Compass tenant on an environment is a data change, not a code change.
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
### Identity mapping
|
|
36
|
+
|
|
37
|
+
`_underscore/Model/Compass/ClientAuthentication.php` — the NameID is looked up against
|
|
38
|
+
`Users.c_hrEmpUsername`.
|
|
39
|
+
|
|
40
|
+
### ⚠ The `@togatech.com` bypass (live in production)
|
|
41
|
+
|
|
42
|
+
```php
|
|
43
|
+
if (stripos($nameId, '@togatech.com') !== false) {
|
|
44
|
+
$user = new _Model_Compass_User(4273);
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Any NameID containing `@togatech.com` returns hardcoded user id 4273**, regardless of who logged
|
|
48
|
+
in. The comment in the code says "return gina's user if testing on devteam" — but there is no
|
|
49
|
+
environment guard, so it runs in **production** too.
|
|
50
|
+
|
|
51
|
+
Worse, the id is not stable across tenants, because 4273 is a per-client-database id:
|
|
52
|
+
|
|
53
|
+
| Tenant | User id 4273 is |
|
|
54
|
+
|---|---|
|
|
55
|
+
| `Client_Compass` (sandbox-client) | `GINA.PRINCE@COMPASS-USA.COM` |
|
|
56
|
+
| `Client_CompassCanada` | `mikhail.prime@icedistricthospitality.com` |
|
|
57
|
+
|
|
58
|
+
This is why a TOGa tester logging into Compass appears as someone else. It is also a production
|
|
59
|
+
backdoor: combined with the missing IdP signature check (see
|
|
60
|
+
[ACS assertion trust gaps](../../../2.0/apps/saml/features/acs-assertion-trust-gaps.md)), any
|
|
61
|
+
trusted IdP asserting a `@togatech.com` NameID lands on a real Compass user. Needs its own ticket;
|
|
62
|
+
**do not remove it casually** — confirm nobody's test flow depends on it first.
|
|
63
|
+
|
|
64
|
+
### Enabling SSO for a Compass tenant on a non-production environment
|
|
65
|
+
|
|
66
|
+
The two rows that matter are `Core.Domains` and `Core.ClientAuthentications`. For Compass_Canada on
|
|
67
|
+
`sandbox-client` (2026-09-25), the `Domains` rows for
|
|
68
|
+
`compasscanada.client.sandbox.togacommerce.com` and `.togasupply.com` **already existed**; only the
|
|
69
|
+
two `ClientAuthentications` rows were missing.
|
|
70
|
+
|
|
71
|
+
- SQL: `dbchanges2/Core/2026-09-25a - CompassCanada SandboxClient SSO.sql`
|
|
72
|
+
- Resulting row: `ClientAuthentications` id 11, `clientId` 43, `environmentId` 24, `type` `SSO`
|
|
73
|
+
|
|
74
|
+
**Status: not yet working.** The rows are confirmed inserted in the client-sandbox `Core`, but the
|
|
75
|
+
frontend still redirects to `/login`. Leading suspect: `Core.Records` id 1 (`/domains`) has
|
|
76
|
+
`ttlCache = 15`, so the API may be serving a cached "no SSO" response. Unresolved — do not treat
|
|
77
|
+
Canada sandbox-client SSO as done.
|
|
78
|
+
|
|
79
|
+
### How the frontend finds the SSO config
|
|
80
|
+
|
|
81
|
+
`toga2-commerce/src/hooks/useAuthenticationFlow.ts` (~lines 88-134) calls `GET /domains` filtered on
|
|
82
|
+
**both** its own origin **and** `Environments.slug` = its own Vite build `MODE`, joining
|
|
83
|
+
`ClientAuthentications` on `clientId` + `environmentId`. If `type == 'SSO'` it calls the scripted
|
|
84
|
+
API `/client-authentications/singleSignOnServiceUrl` and redirects.
|
|
85
|
+
|
|
86
|
+
**A build whose `MODE` does not match the environment slug falls through to `/login` with no
|
|
87
|
+
error.** That is the first thing to check on "SSO just shows the password form".
|
|
88
|
+
|
|
89
|
+
## Gotchas
|
|
90
|
+
|
|
91
|
+
- **⚠ User id 4273 is not the same person in every Compass tenant.** Never copy a Compass user id
|
|
92
|
+
between `Client_Compass` and `Client_CompassCanada`.
|
|
93
|
+
- **The `@togatech.com` bypass has no environment guard** — it is live in production.
|
|
94
|
+
- **`/domains` is cached (`ttlCache = 15` on `Core.Records` id 1).** After inserting SSO rows, an
|
|
95
|
+
immediate frontend test can still see the old "no SSO" answer.
|
|
96
|
+
- **Both Compass tenants share one Entra app per environment** — so a gateway/metadata mistake on
|
|
97
|
+
one tenant breaks both.
|
|
98
|
+
|
|
99
|
+
## Change history
|
|
100
|
+
- 2026-09-25 — Created; documented the `c_hrEmpUsername` mapper, the hardcoded `@togatech.com` → user 4273 bypass live in production, the Compass_Canada sandbox-client SSO enablement SQL (rows inserted, frontend still on `/login`, `ttlCache` suspected), and how the commerce frontend resolves an SSO config (rgirish)
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
title: Compass USA
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
apps:
|
|
5
|
+
- saml
|
|
5
6
|
- _underscore
|
|
6
7
|
- api2
|
|
7
8
|
- toga2-supply
|
|
@@ -20,10 +21,11 @@ project: _Underscore
|
|
|
20
21
|
client: compass-usa
|
|
21
22
|
type: profile
|
|
22
23
|
status: active
|
|
23
|
-
updated: 2026-09-
|
|
24
|
+
updated: 2026-09-25
|
|
24
25
|
owners: [jcardinal, bala, tcox, apeterson, dfranks, akhokhani, ajean, rgirish]
|
|
25
26
|
files: []
|
|
26
27
|
related:
|
|
28
|
+
- features/saml-sso.md
|
|
27
29
|
- ../../2.0/apps/_underscore/features/sales-order-po-number-sourcing.md
|
|
28
30
|
- features/persona-model-and-levy-gating.md
|
|
29
31
|
- features/people-file-user-lifecycle.md
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: compass-usa
|
|
7
7
|
type: workflow
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: ["rgirish", "bala", "dfranks", "jcardinal", "mhammontree"]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/toga2/compass/workflow/1_transmit_compass_sales_orders_to_mits.php
|
|
@@ -30,6 +30,7 @@ related:
|
|
|
30
30
|
- clients/compass-usa/features/mits-po-transmission-to-vendors.md
|
|
31
31
|
- clients/compass-usa/features/mits-po-to-so-item-linking.md
|
|
32
32
|
- clients/compass-usa/profile.md
|
|
33
|
+
- clients/compass-usa/features/oneuptime-netsuite-duplicate-sales-orders-monitor.md
|
|
33
34
|
---
|
|
34
35
|
The end-to-end Compass ODP order → NetSuite pipeline through the numbered 1.0 worker crons, plus the "this order never reached NetSuite" diagnostic method; open for any ODP-to-NetSuite tracing or cron-mechanics question.
|
|
35
36
|
|
|
@@ -139,13 +140,17 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
|
|
|
139
140
|
- **⚠ Do NOT expect a 30-minute wait — branch (a) is normal.** If you assume half an hour is normal, a genuinely stalled transmission looks fine for 30 minutes. An order that has all its 850s back and is still untransmitted **two `*/10` runs later** is already abnormal — go look at the bridge rows (`PurchaseOrders_SalesOrders` / `SalesOrders_PurchaseOrders`) rather than waiting. The 30-minute timeout is the exception, reserved for orders where something never comes back.
|
|
140
141
|
- **⚠ The 30-minute clock runs from the Office Depot PO's `dtCreated`** — the moment we imported the 850, not when the Compass order was raised. So re-importing an old EDI file **restarts the timer**, and a recovered order waits another 30 minutes.
|
|
141
142
|
- **⚠ A late 850 produces a SECOND NetSuite sales order.** If a PO's 850 arrives *after* the Compass SO has already transmitted, the new ODP SO is picked up on its own and creates a **separate** NetSuite SO rather than joining the first.
|
|
142
|
-
- **⚠ Duplicate NetSuite SOs with NO duplicate in Toga = the dead-connection stamp race (root-caused 2026-09-22, FIXED in `worker` `afce32ca`).** Symptom: Compass PO `42031181-1127` produced **three** NetSuite SOs (290875/290876/290877), each exactly **10 minutes** apart
|
|
143
|
+
- **⚠ Duplicate NetSuite SOs with NO duplicate in Toga = the dead-connection stamp race (root-caused 2026-09-22, FIXED in `worker` `afce32ca`).** Symptom: Compass PO `42031181-1127` produced **three** NetSuite SOs (290875/290876/290877), each exactly **10 minutes** apart. **`Client_Compass.PurchaseOrders` and `SalesOrders` each had exactly ONE row.** The duplication is only in NetSuite, so counting Toga rows will tell you nothing is wrong — watch it with the [NetSuite duplicate SO monitor](../features/oneuptime-netsuite-duplicate-sales-orders-monitor.md).
|
|
144
|
+
- **True scope (SuiteQL on prod NetSuite, 2026-09-25):** **22** Office Depot POs had 2–5 NetSuite SOs each, **2026-09-15 → 2026-09-22** (not 15 dups / 11 POs / Sep 21-22 as first recorded). Last duplicate 2026-09-22 14:53 EST; fix `afce32ca` landed 16:05 EST; zero since. Baseline 2026-08-01 → 09-14: zero ODP-PO-shaped repeats. Sep 18–22 groups are cleaned (1 open copy, rest Closed).
|
|
145
|
+
- **⚠ OPEN — not investigated:** 9 POs from Sep 15–17 have 2–3 copies that are **all status "Billed"** — possible double invoicing to Office Depot, not confirmed: `42000479-5910` (3), `41974840-1214` (3), `41997932-1127`, `41979931-5910`, `41977620-6869`, `41974464-1214`, `41972814-1080`, `41995223-1135`, `41994912-5910`.
|
|
143
146
|
- **Mechanism.** Cron 5 holds one MySQL connection across the slow NetSuite SOAP `add()`. `wait_timeout` on the prod client cluster is **180 seconds**. A kit order's `add()` runs past that, so the connection is dead when the stamp `UPDATE SalesOrders SET c_dtTransmittedToNetsuite ...` runs (`crons/.../5_create_netsuite_sales_orders_from_office_depot_purchase_orders.php:433`). The order exists in NetSuite but is **unstamped**, so the next `*/10` run re-sends it.
|
|
144
147
|
- **Proof, not inference.** `Logs.Issue` **769** (reference `W0`), *"mysqli_select_db(): MySQL server has gone away"*, first seen 2026-09-15, 33 occurrences; trace `query.php:168 → database.php:431 → cron 5 :433`. **All 15** duplicate NetSuite SOs are followed by a gone-away event 8-10 s later, zero misses; the 10 SOs that stuck have no matching error. **Logs are CDT, NetSuite is EST — apply the 1-hour offset before comparing timestamps.**
|
|
145
148
|
- **Why kits.** `PC-KIT-OPT2` (10 expanded lines, all $12.58) was 10 of the 11 duplicated POs. The part number does not cause this — kits are the slow path, so they are the orders whose connection times out.
|
|
146
|
-
- **Fix (deployed in cron 5).** (a) `App_Database::
|
|
149
|
+
- **Fix (deployed in cron 5, `afce32ca`).** (a) The stamp runs via `App_Database::queryForked($sql, $clientToga2DatabaseLink)` in a separate process — **there is NO `keepAlive()` call** (corrected 2026-09-25; the first write-up said there was). (b) A `register_shutdown_function()` that emails devteam if the process dies with an order created in NetSuite but unstamped. Full mechanics: [1.0 MySQL Connection Lifecycle](../../../1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md).
|
|
150
|
+
- **⚠ Remaining gap: the forked stamp is NOT confirmed** (the code comment says so), and the shutdown alarm is disarmed right after the fork. A failed forked write leaves the order in NetSuite, unstamped, with no alert → possible duplicate on the next run. The [NetSuite duplicate SO monitor](../features/oneuptime-netsuite-duplicate-sales-orders-monitor.md) is the backstop.
|
|
151
|
+
- **Decision 2026-09-25 — monitor only, no more prevention in cron 5.** Untaken option: set a NetSuite `externalId` on the SO so NetSuite rejects a second create. Declined because the script already has fixes for this.
|
|
147
152
|
- **⚠ The try/catch cron 5 added on Sep 21 to prevent exactly this is DEAD CODE.** Commit `fe4d110d` wrapped the stamp in a `try`/`catch` specifically to stop duplicates. It cannot fire: 1.0 turns the "gone away" **warning** into a fatal `exit` inside the error handler, so it never unwinds to a `catch`. Evidence: 33 `Logs.Issue` 769 occurrences, **zero** matching rows in `Logs_Compass.Email`. Only the shutdown handler runs after that `exit`.
|
|
148
|
-
- **⚠ The Sep 21 push is what made a latent bug visible.** `fe4d110d` flipped this cron from `"active": 0` to `"active": 1` in `schedules/cron.worker.sync.json`, and `85eb8784` turned it into a 450-second bounded loop (`$startTime = time() + 450`) running every 10 minutes. The stamp race had existed since 2026-09-15
|
|
153
|
+
- **⚠ The Sep 21 push is what made a latent bug visible.** `fe4d110d` flipped this cron from `"active": 0` to `"active": 1` in `schedules/cron.worker.sync.json`, and `85eb8784` turned it into a 450-second bounded loop (`$startTime = time() + 450`) running every 10 minutes. The stamp race had existed since 2026-09-15 (9 POs duplicated Sep 15–17 even while the schedule was off); two days of scheduled running then produced most of the rest (22 POs total, see above). **Before re-enabling any long-dormant cron that writes to an external system, check its stamp path.**
|
|
149
154
|
- **⚠ The ODP PO number lands in `otherRefNum`, which NetSuite global search does NOT index.** Cron 5 sets `$salesOrder->otherRefNum = substr($rowSalesOrders['PurchaseOrderNumbers'], 0, 45)` (`:194`). Typing the Office Depot PO number into NetSuite's **global search box returns nothing** — the field is populated, it is simply not searchable. That silence reads exactly like "the order never reached NetSuite," and it is not. **Open the sales order by internal id instead:** take `c_netsuiteInternalSalesOrderId` from the **Office Depot** `SalesOrders` row (`customerId = 1`). Verified 2026-08-31 on ODP POs `41849836-1079` / `41849745-1127` / `41849655-5125` → SO internal ids `7437998` / `7437898` / `7438098`.
|
|
150
155
|
- **⚠ Do not infer the schedule from transmission times.** Some transmissions cluster at **:00 and :30**, which reads like a 30-minute cron — it is not. The schedule is **`*/10`** (re-verified in `worker/schedules/cron.worker.sync.json` 2026-08-28); that clustering is the timeout branch of the release gate firing on the orders that can only release that way. Orders released by branch (a) do **not** cluster — they land on whichever `*/10` run follows their last 850 (e.g. `13:20:46`).
|
|
151
156
|
- **⚠ Office Depot only sends an 850 back for what they buy FROM us (confirmed with ODP 2026-08-28).** A single Compass sales order can carry **two different kinds** of Compass PO, and **both** have `vendorId` = Office Depot:
|
|
@@ -158,9 +163,11 @@ The NetSuite writeback lands on the **downstream ODP SalesOrder (customerId=1)**
|
|
|
158
163
|
- **Editing cron recipient lists — a grep hit is not a live hit.** Email recipients are hardcoded in the cron scripts, and **several dormant copies of the same script exist** under `crons/toga2/compass/edi/` and `crons/toga2/compass/DOA_edi_odp_flow/`. Before changing a recipient list, cross-check the file against `worker/schedules/*.json` for a referencing entry whose `active` is not `0`. Only the scheduled copies matter.
|
|
159
164
|
|
|
160
165
|
## Change history
|
|
161
|
-
- 2026-09-
|
|
166
|
+
- 2026-09-25 — Corrected the fix description (forked stamp via `queryForked`, no `keepAlive`; stamp unconfirmed); widened duplicate scope to 22 POs Sep 15–22; flagged 9 all-Billed POs as open; recorded monitor-only decision and the `NetsuiteDuplicateSalesOrders` backstop. (rgirish)
|
|
167
|
+
- 2026-09-22 — Root-caused the duplicate NetSuite SOs (15 across 11 POs, Sep 21-22) as the dead-MySQL-connection stamp race, not a Toga duplicate; fixed cron 5 with a forked stamp + a `register_shutdown_function` alert (`worker` `afce32ca`). Recorded that the Sep 21 try/catch is dead code and that `fe4d110d` re-enabling the cron is what surfaced the latent bug. (rgirish)
|
|
162
168
|
|
|
163
169
|
## Related
|
|
170
|
+
- [Compass: NetSuite duplicate SO monitor](../features/oneuptime-netsuite-duplicate-sales-orders-monitor.md)
|
|
164
171
|
- [1.0 MySQL Connection Lifecycle — keepAlive / reconnect](../../../1.0/apps/library/features/db-connection-lifecycle-and-reconnect.md)
|
|
165
172
|
- [Compass Order Lifecycle & Data-Integrity Invariants](order-lifecycle-and-data-integrity.md)
|
|
166
173
|
- [Compass MITS PO Transmission to Vendors](../features/mits-po-transmission-to-vendors.md)
|
|
@@ -4,6 +4,6 @@
|
|
|
4
4
|
|-----|-----------|---------|
|
|
5
5
|
| [NYCDOE 2.0 ASN SFTP Ingestion + Parsing (Sftp/Import → Sftp/Parse → Case/ServiceRequest)](features/asn-sftp-ingestion-2-0.md) | 2.0 | 2.0 rewrite of DOE's vendor-SFTP ASN intake (TRUE-80519 Phases 1–2): per-manufacturer fetch to `Files`, header-validated parse into one Case per customer PO + o |
|
|
6
6
|
| [NYCDOE EDI 850 Receive & 997 Acknowledgement](features/edi-850-receive-and-997-ack.md) | 1.0 | DOE's EDI 850-in → NetSuite SO + inline 997-out ack; open for the 997 build, its "always accept" stub, or a DOE ack complaint. |
|
|
7
|
-
| [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send,
|
|
7
|
+
| [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, the held→complete close clo |
|
|
8
8
|
| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | Full DOE ServiceNow/ASN integration map — ticket mirror, ASN → NetSuite SO/TOGa Desk pipeline, outbound status/POD, EDI; open for any DOE cron/pipeline work or |
|
|
9
9
|
| [New York City Department of Education](profile.md) | 1.0 | NYC DOE client profile (1.0 worker tier live; 2.0 ASN rewrite in progress), ServiceNow + EDI; open for DOE's platform scope, endpoints, API clients, and where i |
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-25
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/send_ticket_updates.php
|
|
@@ -23,7 +23,7 @@ related:
|
|
|
23
23
|
- ../../../1.0/apps/worker/architecture.md
|
|
24
24
|
---
|
|
25
25
|
|
|
26
|
-
Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, and the outbound cron's selection/throttle model; open before changing DOE hold logic, the selection query, or "which SNOW field is authoritative."
|
|
26
|
+
Bidirectional DOE ticket hold-status sync (ServiceNow ⇄ TOGaDesk), the four defects that let holds silently revert or never re-send, the held→complete close clobber (TRUE-82332), and the outbound cron's selection/throttle model; open before changing DOE hold/close logic, the selection query, or "which SNOW field is authoritative."
|
|
27
27
|
|
|
28
28
|
## Summary
|
|
29
29
|
|
|
@@ -73,6 +73,17 @@ When a DOE repair order is on hold, `send_ticket_updates.php` pushes **the full
|
|
|
73
73
|
- The "Assigned → In Progress" auto-start block is still guarded so it never pushes `state:2` over a held order (TRUE-79922). Named constants: `SN_STATE_IN_PROGRESS` / `SN_STATE_ON_HOLD` / `SN_HOLD_REASON_DEFAULT` in `send_ticket_updates.php`.
|
|
74
74
|
- The existing **empty-body patch-response check** (~L493) still **throws**, so any remaining rejection surfaces loudly. Keep it.
|
|
75
75
|
|
|
76
|
+
### Close vs. hold-release — held→complete never resolved (TRUE-82332)
|
|
77
|
+
**Bug (silent, 2026-09-18 → fix):** for a COMPLETE order the close block sets `state` 6 (8 for `STATUS_COMPLETE_EXCEPTION`) + `close_code` + `close_notes`. The later hold-release block (`!$orderIsOnHold && $snState == 'On Hold'` → `state` 2) **overwrote** it with 2 (In Progress). The PATCH succeeded, `dtSynced` advanced to the newest note, `close_code` now matched ServiceNow (so the `close_code != remote` check never rebuilt the close), and the order left the queue (COMPLETE is eligible only while notes > `dtSynced`). No error, no alert.
|
|
78
|
+
- **Why it surfaced 2026-09-18:** before TRUE-82129 hold pushes did not reach ServiceNow, so almost nothing was On Hold remotely at completion. After it, every held→complete order hit this.
|
|
79
|
+
- **Evidence:** 45 incidents 2026-09-18..09-25, each one close PATCH with `"state":2`, none with 6/8; non-held closes in the same window sent 6. Example INC2131474: GET `Logs.API` id 173203251 "On Hold" → PATCH id 173203252 (2026-09-22 15:08:32) `close_code` "Repaired" + `state` 2 → next GET "In Progress", no more traffic.
|
|
80
|
+
- **Fix (worker `a04c5452a`, branch TRUE-82332, not deployed as of 2026-09-25):** (a) the release block sets `state` 2 only when `!isset($payload['state'])`; (b) the close block also fires when the ServiceNow state is not in `$serviceNowClosedStatesArray`, even if `close_code` matches — so a re-queued stuck ticket gets its close re-sent.
|
|
81
|
+
- **Rule: a later block in the payload builder must not overwrite a `state` an earlier block already set.** Close outranks release.
|
|
82
|
+
- **No PATCH storm:** the earlier "complete + SN Resolved/Closed/Canceled" check retires the ticket before the close block; `dtSynced` moves only after a successful PATCH.
|
|
83
|
+
- **OPEN:** not confirmed ServiceNow accepts On Hold → Resolved (6) in one step. If rejected: empty-body 200 → `$failedPatches`, `dtSynced` not moved, retried every 6-min run, alert email max 1/hr, until someone steps in.
|
|
84
|
+
|
|
85
|
+
**Recovering tickets already stuck (one-off, run AFTER the worker deploy):** they left the queue, so re-queue them — set `Common.NYCDOETickets.dtSynced` = newest `repair_order_notes.notesdate` − 1 second, pinned in the `WHERE` on `id` + `ticketNumber` + current `dtSynced`. Only the newest note becomes "unsent"; verify it is already in ServiceNow (`referenceId` set / not internal) so no duplicate `work_notes`. **Run it before the deploy and it re-sends `state` 2 and drops them again.** Tickets since closed by hand in ServiceNow are harmless: the "both closed" check retires them with one GET, no PATCH. One-off nudge — not a `dbchanges` file.
|
|
86
|
+
|
|
76
87
|
> **DECISION (TRUE-81044) — do NOT map TogaDesk hold reasons onto ServiceNow hold-reason codes.** `hold_reason` stays pinned to `10` ("Other") and the human-readable reason is carried entirely in `u_other_reason_on_hold` / `work_notes`, sourced from TogaDesk. Rationale: ServiceNow's own states are not reliable, so mapping our six `HOLD_*` constants onto their code set buys nothing and adds a second source of truth. **Do not re-propose a reason-code mapping.**
|
|
77
88
|
|
|
78
89
|
### Outbound selection, watermark & throttle (TRUE-82027 current shape)
|
|
@@ -122,7 +133,7 @@ All three `error_log()` calls were removed (the per-ticket technician sys_id mes
|
|
|
122
133
|
- **The invalid-assignment-group email had never sent, ever** — the block built the email and never called `$email->send()` (every sibling DOE cron does, e.g. `2_send_serials_to_netsuite.php:346`). Added, behind the same throttle on its own key `system|ticket_sync_invalid_group_alert`.
|
|
123
134
|
|
|
124
135
|
### Verifying a TRUE-82027-class change (no sandbox — verified in production)
|
|
125
|
-
There is **no sandbox for the SNOW integration**; verification happens in prod. Deploy order is **dbchanges → library → worker**, and dbchanges `_production` does **not** auto-apply — the migration lands on the next worker EB deploy. After deploy, check legacy `Logs.API`
|
|
136
|
+
There is **no sandbox for the SNOW integration**; verification happens in prod. Deploy order is **dbchanges → library → worker**, and dbchanges `_production` does **not** auto-apply — the migration lands on the next worker EB deploy. After deploy, check legacy `Logs.API` (query shapes in *Reference facts* — `LIKE` scans time out) for:
|
|
126
137
|
1. A held order PATCHing `state:3` with **no new note**.
|
|
127
138
|
2. INC2087375's `u_eta` moving from 2026-09-07 to 2026-09-17.
|
|
128
139
|
3. `SELECT COUNT(*) FROM Common.NYCDOETickets WHERE dtLastChecked IS NOT NULL` climbing.
|
|
@@ -160,7 +171,8 @@ Making DOE holds persist took a change in **three repos**, with a **deploy-order
|
|
|
160
171
|
The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets`) is **intentionally kept** as a failover for SNOW API unreliability — not removed by this work.
|
|
161
172
|
|
|
162
173
|
### Reference facts (DOE sync debugging)
|
|
163
|
-
- **SNOW request/response logs:** logged to the legacy `Logs` schema, **`API` table** (`App_Model_Logs_API`, `db_logs`, `_databaseNameOverride 'Logs'`). `requestPayload` / `responsePayload` are queryable CHAR columns — the primary debugging avenue.
|
|
174
|
+
- **SNOW request/response logs:** logged to the legacy `Logs` schema, **`API` table** (`App_Model_Logs_API`, `db_logs`, `_databaseNameOverride 'Logs'`). `requestPayload` / `responsePayload` are queryable CHAR columns — the primary debugging avenue. Time column is **`dtStamp`** (not `dtCreated`). The custom DOE endpoint base is **`/api/nycd3/`** on prod host **`supportadmin.schools.nyc`** (not `nycdoeprod.service-now.com`).
|
|
175
|
+
- **Query shape that works (TRUE-82332):** `endpoint LIKE '%nycd3%...'` and broad `requestPayload LIKE` scans time out (>300s) on the MCP. Use exact matches: PATCH `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/data/tasks/INCxxxx'`; GET `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/v2/data/tasks?domain=diit%2Calm&filter=sys_class_name%3Dincident%5Enumber%3DINCxxxx'`. For sweeps: `FORCE INDEX (dtStamp)`, windows ≤2 days, plus `sourceJob` / `method` filters. The state:3↔state:2 oscillation in TRUE-79922 was proven from this table (INC2074899, INC2185183 showed `state:3` then `state:2` pushed seconds apart in one run).
|
|
164
176
|
- **`NYCDOETickets` (`App_Model_Common_NYCDOETicket`, `db_common` / legacy `Common`)** stores the inbound SNOW record in `rawData`, a **`FIELDTYPE_BLOBSTORAGE`** field (S3 bucket `asifiles`, key `prod/Common/NYCDOETickets/rawData/{id}`) — **NOT a queryable DB column.**
|
|
165
177
|
- **⚠ `repair_orders.dtUpdated` is skewed — never build a selection query on it.** It is written exactly **240 minutes behind** the clock used by `repair_order_notes.notesdate` and `NYCDOETickets.dtSynced`. **Do NOT compensate with `+240 MINUTE`** — 240 is almost certainly UTC-vs-Eastern and becomes 300 at the DST change (2026-11-01), so a hardcoded offset breaks twice a year. TRUE-82027 sidestepped it by removing `dtUpdated` from the query. Mechanism + the second (720-minute) skew: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
|
|
166
178
|
- **⚠ `Logs.API` history is capped at 30 days from 2026-09-17** (Jeff, sprint review) and existing logs are being purged. This investigation depended on history far older than that (9,914 PATCHes back to 2026-08-20; a hold last pushed 22 days earlier; the 2,392-call storm). **Slow-burn bugs like this can no longer be traced backwards — they must be caught by alerting.** NYCDOE stays 1.0 so its logs stay in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0. Agreed: hold the purge until TRUE-82027 is verified in production.
|
|
@@ -181,14 +193,16 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
|
|
|
181
193
|
- **Diagnosing which writer dropped a hold:** `repair_order_history` (`repairid, userid, dtStamp, note`) records *user-driven* status changes. A hold reverting with **no** history row between the hold and the revert = a raw-`UPDATE` writer (`updateStatus`/`qqStatus`), **not** the worker inbound cron (`process_tickets.php` contains no reference to `ORDER_ASSIGNED_AWAITING_SCHEDULING`). Note that **since TRUE-80060** a *status-changing* `class.repair.php::addNotes` now **calls** `updateStatus()` (so it too can move the cache via that raw `UPDATE`); a **pure comment** still writes nothing to `repair_orders.status`. This history-gap technique is how TRUE-80060 was pinned to `qqStatus()` rather than the SNOW round-trip.
|
|
182
194
|
- **⚠ OUTAGE (TRUE-82129, 2026-09-18) — `send_ticket_updates.php` died on its FIRST ticket of EVERY run, for a whole day.** `App_Database::fetchOne()` (like `fetchRow()` / `numRows()`) takes its result **by reference** (`&$res`, `library/app/database.php` L336/L352/L362). TRUE-82027 (commit `b63fb0136`) passed `App_Database::query(...)` into it **inline**, which raises the notice *"Only variables should be passed by reference"* — and 1.0's error handler **terminates the run on a notice**. Live from the 2026-09-18 10:18 deploy until fixed. **Fix:** assign the `query()` result to a variable first. Symptom set worth recognizing: zero ServiceNow calls after the deploy, `Common.CronJobExecutions.dtCheckOut` NULL on every run, and exactly **one** `NYCDOETickets.dtLastChecked` stamp per 6-minute run. General 1.0 trap — see the 1.0 back-end standard.
|
|
183
195
|
- **⚠ "The outbound cron re-asserts On Hold every run" (business rule 2) was only true in theory between 2026-09-18 10:18 and the TRUE-82129 fix** — the cron was dying on its first ticket. Check the cron actually completes before trusting that guarantee.
|
|
184
|
-
- **OPEN QUESTION — `dtSynced` is not a reliable outbound-only signal.** The 12 reported incidents (INC2298009, INC2298110, INC2297667, INC2299269-77) had `Common.NYCDOETickets.dtSynced` advance from ~2026-09-17/18 05:00–09:46 to ~08:57–09:00 on 2026-09-18 **while the outbound cron was not running**. Suspected `process_tickets.php` (inbound also writes `dtSynced`) but **not confirmed**. Confirm before reading `dtSynced` as proof of an outbound push.
|
|
185
|
-
- **⚠ SECURITY — hardcoded live ServiceNow credentials in `library/app/api/nycdoev2.php` / `nycdoe.php`.** Found 2026-09-18, not fixed; detail in [ServiceNow / ASN Integration](servicenow-integration.md).
|
|
196
|
+
- **OPEN QUESTION — `dtSynced` is not a reliable outbound-only signal.** The 12 reported incidents (INC2298009, INC2298110, INC2297667, INC2299269-77) had `Common.NYCDOETickets.dtSynced` advance from ~2026-09-17/18 05:00–09:46 to ~08:57–09:00 on 2026-09-18 **while the outbound cron was not running**. Suspected `process_tickets.php` (inbound also writes `dtSynced`) but **not confirmed**. More evidence (2026-09-25): some rows have `dtSynced` LATER than their newest note (INC2244995: `dtSynced` 2026-09-25 06:20 vs newest note 09-21 16:04), which the outbound note watermark alone cannot produce. Confirm before reading `dtSynced` as proof of an outbound push.
|
|
197
|
+
- **⚠ SECURITY — hardcoded live ServiceNow credentials in `library/app/api/nycdoev2.php` / `nycdoe.php`.** Found 2026-09-18, not fixed; a test-repo diagnostic script also embeds the same credential; detail in [ServiceNow / ASN Integration](servicenow-integration.md).
|
|
198
|
+
- **KNOWN REMAINING GAP (parked, TRUE-82332) — `u_technician` re-pushed with no change.** The technician email comparison appears never to match, so passes with nothing new still PATCH `u_technician` alone (INC2131474: two such PATCHes at 16:06:30 / 16:06:31). Wasted calls; not fixed.
|
|
186
199
|
- **KNOWN REMAINING GAP (needs a follow-up ticket):** both `qqStatus()` hold gates live inside the `type IN (ONSITE_REPAIR, ONSITE_SERVICE)` branch. The `DEPOT_REPAIR` branch (`repairorder.php` from ~L799) has **no hold gate at all**, so a `DEPOT_REPAIR` order set to any hold status is still silently recomputed out of hold by `updateStatus()` — same class of bug, untouched by TRUE-80060.
|
|
187
200
|
- **KNOWN REMAINING GAP (needs a follow-up ticket) — inbound note reconciliation is paged too small.** `process_tickets.php`'s inbound comment reconciliation fetches only `page_size=50, page_number=1` of the SNOW journal, then **deletes any local note not in that set** (the remainder-delete ~L1326–1330). An order with **>50** comments/work_notes can have local notes silently deleted. It is also inconsistent with `addNotes`, which fetches `page_size=100`.
|
|
188
201
|
- **KNOWN REMAINING GAP — `buildDoubleFieldArray` collapses unsynced notes.** In `process_tickets`, `buildDoubleFieldArray` keys on `IFNULL(referenceId,0)`, so **all** NULL-`referenceId` notes collapse to key `'0'`, making the remainder-delete handle multiple unsynced notes inconsistently.
|
|
189
202
|
- `qqStatus()` collapses all recognized hold sub-statuses to base `HOLD` in `repair_orders.status`; the specific hold **sub-reason persists only in `repair_order_notes.newstatus`**. Any status badge / report / filter reading `repair_orders.status` directly cannot distinguish the six hold variants after a recompute.
|
|
190
203
|
|
|
191
204
|
## Change history
|
|
205
|
+
- 2026-09-25 — TRUE-82332: hold-release no longer overwrites the close `state`; close re-sent when SN not closed; stuck-ticket re-queue recipe; `Logs.API` query shapes. (mhammontree)
|
|
192
206
|
- 2026-09-18 — TRUE-82129: fixed the by-reference `fetchOne()` notice that killed every run; batch loop + run caps replaced LIMIT 1 + per-ticket sleep; alert emails de-PII'd and escaped; unmapped-group email now grouped per run. (mhammontree)
|
|
193
207
|
- 2026-09-17 — TRUE-82027: outbound push runs once per order per pass; `dtLastChecked` split from `dtSynced`; failure alerting instead of a throw. (mhammontree)
|
|
194
208
|
- 2026-08-20 — TRUE-81044: `state` + `hold_reason` + `u_eta` + `u_other_reason_on_hold` pushed together; stopped writing `u_status_task`. (mhammontree)
|
|
@@ -157,7 +157,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
|
|
|
157
157
|
|
|
158
158
|
## Gotchas
|
|
159
159
|
|
|
160
|
-
- **⚠ SECURITY (cso review 2026-09-18 — NOT fixed, needs its own ticket): live production ServiceNow credentials are hardcoded in committed PHP.** `library/app/api/nycdoev2.php` and `library/app/api/nycdoe.php` hold the ServiceNow **`clientSecret` and `refreshToken` as literals** (stage values too). Treat as compromised: move them to config/env and **rotate with DOE**. Separately, `Vision_Log.ErrorLog` rows dump `$_SERVER` including live AWS access key id / secret in plain text — its own ticket, same rotation treatment. **No credential values are recorded here or anywhere in the KB.**
|
|
160
|
+
- **⚠ SECURITY (cso review 2026-09-18 — NOT fixed, needs its own ticket): live production ServiceNow credentials are hardcoded in committed PHP.** `library/app/api/nycdoev2.php` and `library/app/api/nycdoe.php` hold the ServiceNow **`clientSecret` and `refreshToken` as literals** (stage values too). A test-repo diagnostic script (committed 2026-06-18) embeds the same prod credential — one rotation covers both. Treat as compromised: move them to config/env and **rotate with DOE**. Separately, `Vision_Log.ErrorLog` rows dump `$_SERVER` including live AWS access key id / secret in plain text — its own ticket, same rotation treatment. **No credential values are recorded here or anywhere in the KB.**
|
|
161
161
|
|
|
162
162
|
- **⚠ OPEN BUG / NAMED PATTERN — "Lenovo Off-Layout ASN File" (silent column shift + unit collapse).** Investigated 2026-08-11 (read-only; **no code fix exists**). Lenovo sends NYCDOE ASNs in **two different column layouts**, and `library/app/asnprocessor/lenovo.php` picks its column mapping by **column COUNT with no header-name validation** (`:110`, `if ($colCount >= 30`), so the off-layout file is parsed with the wrong mapping and silently mis-read.
|
|
163
163
|
- **The two layouts.** The normal daily feed `DOE-Lenovo-ASN-MM-DD-YYYY.CSV` (~2 MB) has **30** columns and matches the code's mapping. A second file, `DOE-Open-Lenovo-MM-DD-YYYY.csv` (**lowercase** `.csv`, ~45 KB), has **31** columns in a **different order**: `Contact Email` moves from column 29 to column **6**, and a trailing `Full Shipment` column is appended. Everything from `Street Address` onward is therefore read **one column too early**. The "extra comma" repair heuristic at `:119-127` does not catch this (unchanged since March 2026 — commits `3442ddcf`, `6956dfbf`, `864cf073`, so not a regression).
|
|
@@ -240,6 +240,7 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
|
|
|
240
240
|
|
|
241
241
|
- **The queue `dedupeKey` format is load-bearing.** A 2026 incident (WR260236464 duplicate repair order) was caused by dedupeKey format drift (4-part keys with serial vs 3-part without) letting old rows re-import; Stage 2 then created a sibling item row (frozen original not reusable), and Stage 5's per-item-row serial scoping duplicated the unit → duplicate RO. Any change to the key format needs a backfill or transitional matcher. Forensic fingerprint: a Unit with `trackingNumberId IS NULL` was added by Stage 5 from NetSuite, not by the importer.
|
|
242
242
|
- **Two dedupe strategies coexist:** SFTP path = INSERT+catch (race-safe); API path = check-then-insert (unique index backstop).
|
|
243
|
+
- **Held→complete incidents never resolving in ServiceNow** = the TRUE-82332 close clobber; mechanism, fix and re-queue recipe in [Hold-Status Sync](hold-status-sync.md).
|
|
243
244
|
- **Timezone traps:** worker runs `America/Chicago`; `repair_orders.dtEta` is stored **Eastern**; ServiceNow `u_eta` is sent as **UTC** `YYYY-MM-DD HH:MM:SS` (converted via `App_Date::convertTimezone()`, only sent when first 19 chars differ from remote).
|
|
244
245
|
- **EDI PO104 leading-zero suppression** (go-live 2026-06-07): DOE's IBM-based translator sends `.94` instead of `0.94` — exposure is in `App_Edi::translatePurchaseOrder850()`, not the cron.
|
|
245
246
|
- **Installation schedule reports skip weekends but not holidays** — empty output near a holiday is expected, not a bug. Empty also just means no ROs with `dtEta` on the exact target date.
|
|
@@ -261,4 +262,5 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
|
|
|
261
262
|
- Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
|
|
262
263
|
|
|
263
264
|
## Change history
|
|
265
|
+
- 2026-09-25 — TRUE-82332: cross-link to the close-clobber fix; test-repo copy of the ServiceNow credential noted (mhammontree)
|
|
264
266
|
- 2026-09-25 — Corrected "no 2.0 tenant" (TRUE-80519 2.0 ASN rewrite now exists); DOE-Open-Lenovo marked retired (mhammontree)
|
|
@@ -37,9 +37,8 @@ NYC DOE — TOGA client whose **live** integration runs in the **1.0 worker tier
|
|
|
37
37
|
## Endpoints
|
|
38
38
|
| System | Endpoint | Notes |
|
|
39
39
|
|---|---|---|
|
|
40
|
-
| ServiceNow prod | `https://
|
|
41
|
-
| ServiceNow
|
|
42
|
-
| ServiceNow disposals | `https://supportadmin.schools.nyc` (stage `nycdoedev3`) | disposal service requests |
|
|
40
|
+
| ServiceNow prod (live) | `https://supportadmin.schools.nyc` (stage `nycdoedev3.service-now.com`) | `App_Api_NYCDOEV2` — all live INC/RITM GET/PATCH under `/api/nycd3/`, incl. disposals; V1 `App_Api_NYCDOE` uses the same prod host (stage `supportadminstg.schools.nyc`) |
|
|
41
|
+
| ServiceNow legacy | `https://nycdoeprod.service-now.com` (stage `nycdoedev2`) | only the dormant `download_tickets*.php` — not live |
|
|
43
42
|
| DOE EDI SFTP | `transfer.schools.nyc` (SSH key `doedimedu.pem`) | EDI 850 in, invoices out |
|
|
44
43
|
| Vendor SFTP feeds | Apple / Lexmark / Lenovo (Lenovo: `ftp.asisystem.com`) | legacy ASN file ingestion |
|
|
45
44
|
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
title: "TOGA Technology"
|
|
3
3
|
framework: "2.0"
|
|
4
4
|
apps:
|
|
5
|
+
- saml
|
|
5
6
|
- _underscore
|
|
6
7
|
- tools
|
|
7
8
|
- worker2
|
|
@@ -14,11 +15,13 @@ project: _Underscore
|
|
|
14
15
|
client: true
|
|
15
16
|
type: profile
|
|
16
17
|
status: active
|
|
17
|
-
updated: 2026-09-
|
|
18
|
-
owners: [jcardinal, kyalamarthi, tcox]
|
|
18
|
+
updated: 2026-09-25
|
|
19
|
+
owners: [jcardinal, kyalamarthi, tcox, rgirish]
|
|
19
20
|
files: []
|
|
20
21
|
related:
|
|
21
22
|
- clients/true/features/users-personas-data-model.md
|
|
23
|
+
- ../../2.0/apps/saml/features/gateway-environments-and-redirect-safety.md
|
|
24
|
+
- ../../2.0/apps/_underscore/features/core-domains-app-host-registry.md
|
|
22
25
|
---
|
|
23
26
|
|
|
24
27
|
"True" — TOGa's internal/staff tenant (`Client_True`, identifier `True`, default email SSO mapper); open for its internal apps (Tools, sprint dashboard, togatech site) and its Talos-pricing internal-tenant role.
|
|
@@ -37,6 +40,26 @@ The public **TOGA Technology website** (repo `togatech`, standalone React SPA, a
|
|
|
37
40
|
|
|
38
41
|
The **Tools** app (1.0; repo `tools`) authenticates True users via **SSO** through `saml.togahub.com` and reads the `Client_True` DB **read-only** to gate its internal tooling by staff persona (see `1.0/apps/tools/`). Its gateway return domain is registered in `Core.Domains` (`uuid 2927bc15-e347-4358-a430-fb28f9446d27`, `clientId 1`, `appId 30`, env 1, `https://tools.togatech.com/sso`).
|
|
39
42
|
|
|
43
|
+
## ⚠ "Base" maps to Super User in Client_True
|
|
44
|
+
|
|
45
|
+
In `Client_True` the role **named `Base`** maps to **`coreRoleId` 3 = Super User**. Every other
|
|
46
|
+
tenant we checked does not do this: `Client_Compass` maps its `Base` to `coreRoleId` 1 (Public) and
|
|
47
|
+
keeps a separate SuperUser role with 115 people. This looks like a misconfiguration, not the
|
|
48
|
+
intended model.
|
|
49
|
+
|
|
50
|
+
The concrete consequence found 2026-09-25: `Core.Records` id 1 exposes `Domains` as the REST route
|
|
51
|
+
`domains`, and `AclRecordPermissions` grants Super User full create/read/update/delete. So roughly
|
|
52
|
+
**2,826 active `Client_True` users can `POST`/`PUT` `/domains`.**
|
|
53
|
+
|
|
54
|
+
That matters more than it used to, because `Core.Domains` ownership is now the authorization for
|
|
55
|
+
where the SAML gateway delivers a login handoff — see
|
|
56
|
+
[SAML gateway per-environment hosts & redirect safety](../../2.0/apps/saml/features/gateway-environments-and-redirect-safety.md).
|
|
57
|
+
Write access to `Core.Domains` is control over where a credential payload lands. Needs its own
|
|
58
|
+
ticket.
|
|
59
|
+
|
|
60
|
+
**Before reading any `Client_True` ACL grant, check what `coreRoleId` the role name actually maps
|
|
61
|
+
to — do not trust the name.**
|
|
62
|
+
|
|
40
63
|
## Talos Pricing — the internal tenant
|
|
41
64
|
|
|
42
65
|
True is an **internal tenant** in the Talos Pricing platform (`TalosClients.isInternal = 1`, formal name **TOGA Technology**), not a paying client. It holds nearly all Talos usage and, via `defaultTenant`, all untagged AWS Bedrock spend, so it **must** exist as a contract row or there is no platform unit price at all. It is **costed and feeds the unit price** but is **excluded from fee recommendations** — modelling it as a zero-revenue contract made margin compute as 0%, parking it permanently in "Needs attention" and dragging blended portfolio margin negative. A contract row was created and signed 2026-08-03. Measured July 2026: $549.18 tagged of a $1,013.76 total. See [talos-pricing-automation](../../2.0/apps/worker2/features/talos-pricing-automation.md).
|
package/package.json
CHANGED