toga-ai 1.0.910 → 1.0.912
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/INDEX.md +1 -0
- package/knowledge/1.0/apps/library/features/api-request-logging-and-retention.md +89 -0
- package/knowledge/1.0/apps/library/features/netsuite-suiteql-api-reference.md +5 -3
- package/knowledge/2.0/apps/dbchanges2/features/rerunnable-additive-inserts.md +11 -1
- package/knowledge/2.0/standards/secrets-and-build-credentials.md +73 -0
- package/knowledge/INDEX.md +2 -2
- package/knowledge/clients/adyen/INDEX.md +1 -1
- package/knowledge/clients/adyen/profile.md +36 -2
- package/knowledge/clients/nycdoe/features/hold-status-sync.md +71 -2
- package/knowledge/clients/nycdoe/features/servicenow-integration.md +4 -3
- package/knowledge/clients/quad/INDEX.md +1 -0
- package/knowledge/clients/quad/profile.md +5 -1
- package/knowledge/clients/quad/workflows/granting-all-personas-and-global-admin.md +86 -0
- package/knowledge/clients/quad/workflows/granting-internal-admin-navigation.md +5 -0
- package/package.json +1 -1
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
|-----|---------|
|
|
5
5
|
| [Library (1.0 Framework) Architecture](architecture.md) | The 1.0 `App_` framework core: bootstrap/autoloader, the `app/` (`App_`) + `browser/` (`Browser_`) class trees, and vendored packages; open for how any 1.0 app |
|
|
6
6
|
| [Address Validation Gateway (App_Api_OfficeDepot::validateAddress + USPS fallback)](features/address-validation-gateway.md) | The shared 1.0 address-validation gateway (`App_Api_OfficeDepot::validateAddress` → USPS fallback); open when standardizing/verifying a typed address before sav |
|
|
7
|
+
| [API Request Logging & Retention (App_ApiTransaction → Logs.API / Logs_<Tenant>)](features/api-request-logging-and-retention.md) | Where every outbound API call is logged (1.0 `Logs.API` vs the 2.0 `Logs_<Tenant>` cluster), how the routing is chosen, and the 30-day retention trim; open befo |
|
|
7
8
|
| [Where a new App_ class goes — the app/ folder IS a behavioral contract](features/app-class-placement-base-contracts.md) | Where a new `App_` class goes — each `app/` folder's base class carries real inherited behavior; open before creating any class under `app/model|client|api`. |
|
|
8
9
|
| [App_Sso — Reusable 1.0 SSO Initiation (SP-initiated SAML via saml.togahub.com)](features/app-sso-initiation.md) | Reusable 1.0 SP-initiated SAML SSO via `saml.togahub.com` (`App_Sso::initiate`); the canonical doc to follow for any "add SSO to `<1.0 app>`" request. |
|
|
9
10
|
| [Cron Execution Monitoring (App_Framework check-in/out → CronJobExecutions)](features/cron-execution-monitoring.md) | 1.0 cron check-in/out lifecycle via `App_Framework::cronInitialization()`/`cronFinished()` → `Common.CronJobExecutions`; open to monitor a 1.0 cron or read its |
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "API Request Logging & Retention (App_ApiTransaction → Logs.API / Logs_<Tenant>)"
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: library
|
|
5
|
+
project: Library
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-29
|
|
10
|
+
owners: ["mhammontree"]
|
|
11
|
+
files:
|
|
12
|
+
- library/app/apitransaction.php
|
|
13
|
+
- library/app/model/logs/api.php
|
|
14
|
+
- library/app/model/toga2/logs/api.php
|
|
15
|
+
- worker/crons/maintenance/maintenance_daily.php
|
|
16
|
+
related:
|
|
17
|
+
- error-capture-1-0.md
|
|
18
|
+
- ../architecture.md
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
Where every outbound API call is logged (1.0 `Logs.API` vs the 2.0 `Logs_<Tenant>` cluster), how the routing is chosen, and the 30-day retention trim; open before changing log routing, retention, or debugging past-window history.
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
`App_ApiTransaction` writes one row per outbound API call. It has **two** storage paths — the legacy 1.0 `Logs`.`API` table and the 2.0 per-tenant `Logs_<Tenant>` cluster — and picks between them per run. Retention is enforced only on the 1.0 side, by a daily worker cron, at **30 days**.
|
|
26
|
+
|
|
27
|
+
## How it works
|
|
28
|
+
|
|
29
|
+
### Which log DB a call lands in
|
|
30
|
+
`library/app/apitransaction.php:377`:
|
|
31
|
+
```php
|
|
32
|
+
if (App_Registry::varIsSet('toga2LogsDatabase')) { /* App_Model_Toga2_Logs_Api */ }
|
|
33
|
+
else { /* App_Model_Logs_API → legacy Logs.API */ }
|
|
34
|
+
```
|
|
35
|
+
- 2.0 routing is **opt-in per cron**. Only 10 crons set it, all Prudential, each hardcoding
|
|
36
|
+
`App_Registry::set('toga2LogsDatabase', 'Logs_Prudential')`.
|
|
37
|
+
- Every other `crons/toga2/*` cron falls through to the 1.0 table — **this is why 2.0 client traffic
|
|
38
|
+
lands in the 1.0 `Logs.API`**.
|
|
39
|
+
|
|
40
|
+
### Retention (1.0 only)
|
|
41
|
+
`worker/crons/maintenance/maintenance_daily.php` trims the log tables nightly.
|
|
42
|
+
- `$daysToKeep_APILog` = **30** (was 90; cut 2026-09-28, TRUE-82049, commit `89ef44f35`).
|
|
43
|
+
- The `DELETE FROM Logs.API` is **chunked**: 25,000 rows per statement in a do/while loop with a
|
|
44
|
+
1s sleep, still bounded overall by `$maxLimitForDeletes`. Every *other* table this cron trims is
|
|
45
|
+
a single statement — `Logs.API` is the exception because it is ~12x the next biggest table
|
|
46
|
+
(`Vision_Log.Log`, 45.85 GB), and a single 5,000,000-row DELETE holds locks for minutes and lags
|
|
47
|
+
read replicas.
|
|
48
|
+
- **The one-time backlog purge (~20M rows) is separate and not done** as of 2026-09-29.
|
|
49
|
+
|
|
50
|
+
### Storage facts (measured 2026-09-28)
|
|
51
|
+
- 1.0 `Logs`.`API`: **565.27 GB / 34,464,186 rows**, growing **~6.3 GB/day**. `dtStamp` is the
|
|
52
|
+
indexed date column; `sourceJob` names the writing cron.
|
|
53
|
+
- Top writers by bytes (2-day sample): `sync_togasupply_compass_usa.php` 2.53 GB (128 KB/row),
|
|
54
|
+
`periodic_forecast_discrepancy_fix.php` 1.63 GB, `sync_togasupply_hh.php` 0.84 GB,
|
|
55
|
+
`sync_togasupply_elite.php` 0.53 GB, `compass/workflow/5_create_netsuite_sales_order.php`
|
|
56
|
+
1,581 KB/row, `infrastructure/system_monitors.php` 3,040 KB/row.
|
|
57
|
+
- 2.0 logs cluster (`writer.logs.database.togahub.com`): **~1.67 TB** across 36 `Logs_<Tenant>`
|
|
58
|
+
DBs, oldest rows **2025-08-24** (~13 months — *not* the 90 days believed), and **no scheduled
|
|
59
|
+
DB events trimming it**. Largest: `Logs_Wje` 372.80 GB, `Logs_Elite` 228.99 GB,
|
|
60
|
+
`Logs_Compass` 185.88 GB, `Logs_Nychh` 129.71 GB, `Logs_Prudential` 125.32 GB.
|
|
61
|
+
`Logs_Northwell` is 43.59 GB from only 194,643 rows = 224 KB/row.
|
|
62
|
+
|
|
63
|
+
### Decision
|
|
64
|
+
**`Logs.API` is forensics-only; 30 days is enough** (Jeff, 2026-09-17). Debugging that needs
|
|
65
|
+
older history is no longer possible — slow-burn bugs must be caught by **alerting**, not
|
|
66
|
+
reconstructed after the fact.
|
|
67
|
+
|
|
68
|
+
## Gotchas
|
|
69
|
+
|
|
70
|
+
- **DELETE does not return space to disk on InnoDB.** After the backlog purge the file stays
|
|
71
|
+
~565 GB with `DATA_FREE` growing. Reclaiming disk needs `OPTIMIZE TABLE` (~565 GB spare
|
|
72
|
+
required) or a copy-and-swap — a separate decision, not part of the retention change.
|
|
73
|
+
- **Do NOT move 2.0 client logging to the 2.0 cluster without solving retention there first.**
|
|
74
|
+
It would shift ~7 GB/day onto a cluster that is already bigger and has **no working retention**.
|
|
75
|
+
Deliberately descoped from TRUE-82049. Nobody has found the 2.0 archiving script — it is not in
|
|
76
|
+
`worker2`, `api2`, or `_underscore`.
|
|
77
|
+
- **`App_Model_Toga2_Logs_Api` has NO `save()` override.** The 1.0 `App_Model_Logs_API::save()`
|
|
78
|
+
deliberately wraps `parent::save()` in try/catch because a long-running cron can outlive its
|
|
79
|
+
`db_logs` connection (MySQL "server has gone away"), which once aborted whole imports mid-run.
|
|
80
|
+
Routing traffic to the 2.0 model without adding that guard reintroduces that failure.
|
|
81
|
+
- **The 2.0 model has a `source` column the 2.0 branch never sets** — moving a cron to 2.0 logging
|
|
82
|
+
loses the writing-cron name (`sourceJob`'s equivalent).
|
|
83
|
+
- **Cap payload size when adding new logging.** Team precedent: worker2's
|
|
84
|
+
`MAX_LOG_PAYLOAD_LENGTH = 8000` with a `...[truncated]` marker
|
|
85
|
+
(`Worker/Netsuite/Opportunity.php:148`, `Worker/Clickup/Opportunity.php:45`). Rows of
|
|
86
|
+
1,500–3,000 KB are what built the 565 GB table.
|
|
87
|
+
|
|
88
|
+
## Change history
|
|
89
|
+
- 2026-09-29 — Created: `Logs.API` retention cut 90→30 days and chunked DELETE (TRUE-82049); dual 1.0/2.0 routing and 2.0-cluster storage facts recorded. (mhammontree)
|
|
@@ -6,8 +6,8 @@ project: Library
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
10
|
-
owners: ["dfranks", "jcardinal", "kyalamarthi", "ajean"]
|
|
9
|
+
updated: 2026-09-28
|
|
10
|
+
owners: ["dfranks", "jcardinal", "kyalamarthi", "ajean", "rgirish"]
|
|
11
11
|
files:
|
|
12
12
|
- library/app/api/netsuite/rest.php
|
|
13
13
|
- library/ssl/netsuite_ec_key.pem
|
|
@@ -110,7 +110,9 @@ Status codes (`transaction.status` letter):
|
|
|
110
110
|
|
|
111
111
|
Header columns present (✓ confirmed in SuiteQL): `id`, `tranid`, `trandate`, `lastmodifieddate`, `type`, `status`, `entity`, `employee`, `leadsource`, `terms`, `memo`, `otherrefnum`, `foreigntotal`/`total` (both return the same value), `foreignamountunpaid` (AR balance — invoices only; NULL for SO/credit memo/cash sale), `duedate`, `shipmethod`, `opportunity`, `shippingaddress` (numeric id → join `transactionShippingAddress`), and custom fields (`CUSTBODY_END_CUSTOMER`, `CUSTBODY_STOCKING_ORDER`, …).
|
|
112
112
|
|
|
113
|
-
Header fields NOT in SuiteQL (✗ — require a per-id REST GET): `shippingCost`, `taxTotal`, `subTotal`, `createdFrom` (also derivable from `previousTransactionLineLink`, but non-deterministically — see shim doc),
|
|
113
|
+
Header fields NOT in SuiteQL (✗ — require a per-id REST GET): `shippingCost`, `taxTotal`, `subTotal`, `createdFrom` (also derivable from `previousTransactionLineLink`, but non-deterministically — see shim doc), and the **structured** `shippingAddress` object (SuiteQL gives only the id).
|
|
114
|
+
|
|
115
|
+
Tracking numbers in SuiteQL (verified 2026-09-28): `transaction.trackingnumberlist` (on the IF) / `linkedtrackingnumberlist` (on the SO) return **`trackingnumber` record ids**, not the numbers — resolve with `SELECT id, trackingnumber FROM trackingnumber WHERE id IN (...)`. `trackingnumbermap` returned nothing. The REST per-id GET below also has them.
|
|
114
116
|
|
|
115
117
|
### `transactionline` table
|
|
116
118
|
|
|
@@ -6,7 +6,7 @@ project: Database Changes
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-28
|
|
10
10
|
owners: [bala, kyalamarthi, apeterson, ajean, jcardinal, rgirish]
|
|
11
11
|
files:
|
|
12
12
|
- dbchanges2/Client_Compass/2026-08-26a - CompassCreativeStudioPersona.sql
|
|
@@ -182,6 +182,8 @@ the item-fulfillment bridges, which sit side by side and do **not** agree:
|
|
|
182
182
|
| `ItemFulfillmentItems_TrackingNumbers` | yes — `(itemFulfillmentItemId, trackingNumberId)` |
|
|
183
183
|
| `ItemFulfillmentItemUnits_TrackingNumbers` | yes — `(itemFulfillmentItemUnitId, trackingNumberId)` |
|
|
184
184
|
| `ItemFulfillmentItemUnits` | **no** — and it already holds **254** duplicate `(itemFulfillmentItemId, unitId)` rows as a result |
|
|
185
|
+
| `Users_Personas` | yes — `(userId, personaId)` (`userId_personaId_UNIQUE`, `Client/2026-09-22-BLANK_CLIENT_DATABASE.sql:11410`) |
|
|
186
|
+
| `Users_Roles` | yes — `(userId, roleId)` |
|
|
185
187
|
|
|
186
188
|
The last row is the whole point: the table with no unique key had already accumulated duplicates
|
|
187
189
|
that nobody noticed, because nothing ever errored.
|
|
@@ -198,6 +200,13 @@ WHERE
|
|
|
198
200
|
ur.id IS NULL;
|
|
199
201
|
```
|
|
200
202
|
|
|
203
|
+
**Not every table has a `dtCreated` to set.** The "always write `dtCreated = NOW()`" rule below
|
|
204
|
+
applies only where the column exists. The user bridges do **not** have one —
|
|
205
|
+
`_Model_Client_Users_Persona` (`_underscore/Model/Client/Users/Persona.php`) declares exactly
|
|
206
|
+
`(id, uuid, userId, personaId)`, and `Users_Roles` is the same shape — so a hand-written INSERT names
|
|
207
|
+
only the uuid and the two ids. Read the model (or the blank-client schema) before adding a timestamp
|
|
208
|
+
column that does not exist.
|
|
209
|
+
|
|
201
210
|
**A guarded insert that inserts zero rows is a success, not a wasted file** — see
|
|
202
211
|
[Client-DB grant migrations are re-runnable NO-OPs](../../_underscore/features/acl-permission-chain.md).
|
|
203
212
|
|
|
@@ -505,6 +514,7 @@ an INSERT that names only `uuid`/`tableViewId`/`recordFieldId`/`isVisible`/`inde
|
|
|
505
514
|
[FIELD_SQL calculated fields](../../_underscore/features/calculated-sql-fields.md).
|
|
506
515
|
|
|
507
516
|
## Change history
|
|
517
|
+
- 2026-09-28 — `Users_Personas` / `Users_Roles` added to the composite-unique table (both unique on the id pair, so a guarded re-run is a clean no-op), plus the counterpoint to the dtCreated rule: these user bridges have no timestamp column at all. (apeterson)
|
|
508
518
|
- 2026-09-23 — Added the NULL-unsafe flag-guard trap (`col = 0` matches no NULL row; use `COALESCE(col, 0) = 0`), found when a bundles TableViewFields migration updated 0 rows; flagged the 2026-08-26 `EnableTableViewFieldFilters` files as carrying the same hole. (apeterson)
|
|
509
519
|
- 2026-09-22 — Added multi-tier chain backfill rules from the Compass tracking/serial backfills: re-run the same file until the verify count is 0 (each pass moves data up one tier; both files needed 3 passes); `GROUP BY` the target key pair instead of `SELECT DISTINCT` when many downstream rows roll up to one upstream row (bundles tripped a bridge UNIQUE 6-to-1); measure the safe/ambiguous/no-match split before writing bulk DML and skip the ambiguous rows (4,900 broken links = 1,982 safe + 0 ambiguous + 2,918 no-match). Sharpened the unique-key rule with measured composite keys across the four item-fulfillment tables — `ItemFulfillmentItemUnits` has none and already carries 254 duplicates. (rgirish)
|
|
510
520
|
- 2026-09-21 — Added three lessons from the platform-wide `Users.contactId` NOT NULL backfill: a guarded
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Where Secrets Live (AWS Secrets Manager, Build Credentials)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
project: _Underscore
|
|
5
|
+
client: shared
|
|
6
|
+
type: standard
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-09-28
|
|
9
|
+
owners: [jcardinal]
|
|
10
|
+
related:
|
|
11
|
+
- frontend-deploy.md
|
|
12
|
+
- ../apps/_underscore/features/cloud-secrets-manager.md
|
|
13
|
+
- ../apps/api2/architecture.md
|
|
14
|
+
- ../apps/toga25-blox/architecture.md
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
Every secret a machine (build, CI, app) reads lives in one place, AWS Secrets Manager in the production account; open before adding, reading, or rotating any credential.
|
|
18
|
+
|
|
19
|
+
## Rule
|
|
20
|
+
|
|
21
|
+
- One place: AWS Secrets Manager, **production account `654654170868`, `us-east-1`**. No secret values in config files, `.env`, source, CI YAML, or committed anywhere.
|
|
22
|
+
- Secrets are **JSON**. Value key is `token` (or named keys when a secret holds several related values).
|
|
23
|
+
- Never write a secret's actual value into a knowledge doc, commit, log, URL, or chat — location and name only.
|
|
24
|
+
|
|
25
|
+
## Naming
|
|
26
|
+
|
|
27
|
+
- Folder-style prefix by **who reads it**: `build/` = anything a build/CI job may read (Amplify, GitHub Actions).
|
|
28
|
+
- Other existing prefixes: `toga/ssl/*` (SAML SP certs), `talos/<env>/*`, `sftp/<client>/*`.
|
|
29
|
+
- Current `build/` secrets:
|
|
30
|
+
- `build/github-read` — fine-grained GitHub token on the shared deployment user `agilantgit`, resource owner org `agilantsolutions`, Contents read-only on `toga25-blox` / `toga25-desk` / `toga25-iq` / `toga2-template`, 1-year expiry — renew before it expires.
|
|
31
|
+
- `build/fontawesome` — FontAwesome Pro package token.
|
|
32
|
+
|
|
33
|
+
## Env var name for local dev / CI
|
|
34
|
+
|
|
35
|
+
Upper-case the secret name, `/` → `_`, append `_<KEY>` upper-cased.
|
|
36
|
+
|
|
37
|
+
- `build/fontawesome` + key `token` → `BUILD_FONTAWESOME_TOKEN`. Read by the app's `.npmrc`. Each developer sets it once: `setx BUILD_FONTAWESOME_TOKEN "<token>"`, then open a new terminal.
|
|
38
|
+
- `build/github-read` → `BUILD_GITHUB_READ_TOKEN`.
|
|
39
|
+
|
|
40
|
+
## Cross-account access (prod-stored, non-prod reader)
|
|
41
|
+
|
|
42
|
+
Secret is encrypted with KMS key `alias/toga-secrets` (key policy lets non-prod account `975050298201` decrypt only via Secrets Manager) plus a resource policy on the secret naming the reader roles. Readers must use the **full ARN**: `arn:aws:secretsmanager:us-east-1:654654170868:secret:<name>`. Same pattern as `toga/ssl/*` (read by non-prod `aws-elasticbeanstalk-ec2-role`).
|
|
43
|
+
|
|
44
|
+
### Build readers today (non-prod `975050298201`)
|
|
45
|
+
|
|
46
|
+
- `amplify-platform-build` — Amplify service role; inline policy `platform-build-secrets-read` grants `secretsmanager:GetSecretValue` on prod `secret:build/*` + `kms:Decrypt` on `toga-secrets` via Secrets Manager. Attached to Amplify app `toga25-desk` (`d311sdoq12ddne`).
|
|
47
|
+
- `github-actions-toga25-blox` — GitHub OIDC provider (`token.actions.githubusercontent.com`), scoped to repo `agilantsolutions/toga25-blox` only, same policy shape. Used by `.github/workflows/apps-build.yml`.
|
|
48
|
+
- **Open item:** production Amplify builds need their own `amplify-platform-build` role in the **prod** account (same-account read — no cross-account sharing needed there).
|
|
49
|
+
|
|
50
|
+
## Adding a secret / a reader
|
|
51
|
+
|
|
52
|
+
- **New build secret:** create under `build/`, encrypt with `alias/toga-secrets`, add the same resource policy (the two non-prod roles above). No IAM change needed — both roles already allow `build/*`.
|
|
53
|
+
- **New non-prod reader role:** add its ARN to each shared secret's resource policy.
|
|
54
|
+
|
|
55
|
+
## Rotation
|
|
56
|
+
|
|
57
|
+
- Rotate at the source (GitHub, FontAwesome, etc.), update the value in Secrets Manager. Builds pick it up next run; developers update their local env var.
|
|
58
|
+
- **Open item:** old FontAwesome + npmjs tokens are committed in `toga25-desk` / `toga25-iq` / `toga2-template` git history — rotate once every app reads `build/fontawesome`. Legacy apps `toga25-supply` / `toga2-commerce` still use the current (unrotated) token, so rotating now would break them first.
|
|
59
|
+
- Any credential that has ever been committed must be treated as compromised and rotated — deleting the line does not remove it from history. See [worker2 committed secrets debt register](../apps/worker2/features/committed-secrets-and-config-debt.md) and [api2 architecture Security note](../apps/api2/architecture.md#security-note-committed-secrets) for the existing backlog this standard is meant to stop growing.
|
|
60
|
+
|
|
61
|
+
## Never
|
|
62
|
+
|
|
63
|
+
- Commit a token.
|
|
64
|
+
- Put a token in a URL, log, or chat.
|
|
65
|
+
- Give a build role access to all secrets — a build runs hundreds of npm packages; scope to `build/*` or narrower.
|
|
66
|
+
|
|
67
|
+
## Related
|
|
68
|
+
|
|
69
|
+
- [2.0 Frontend Deployment (Vite + AWS Amplify)](frontend-deploy.md)
|
|
70
|
+
- [_Cloud Secrets Manager helper](../apps/_underscore/features/cloud-secrets-manager.md) — the app-side (PHP) reader; this standard covers the AWS-side layout and build/CI readers
|
|
71
|
+
- [API (api2) Architecture — Security note](../apps/api2/architecture.md#security-note-committed-secrets)
|
|
72
|
+
- [toga25-blox Architecture](../apps/toga25-blox/architecture.md)
|
|
73
|
+
</content>
|
package/knowledge/INDEX.md
CHANGED
|
@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
4
4
|
|
|
5
5
|
## 1.0 framework
|
|
6
6
|
|
|
7
|
-
- **library** (Library) _(framework core)_ —
|
|
7
|
+
- **library** (Library) _(framework core)_ — 29 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
|
|
8
8
|
- **worker** (Worker) — 40 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
|
|
9
9
|
- **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
|
|
10
10
|
- **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
|
|
@@ -22,7 +22,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
22
22
|
- **_underscore** (_Underscore) _(framework core)_ — 90 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
23
23
|
- **worker2** (Worker) — 80 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
24
24
|
- **api2** (API) — 26 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
25
|
-
- **dbchanges2** (Database Changes) _(framework core)_ —
|
|
25
|
+
- **dbchanges2** (Database Changes) _(framework core)_ — 22 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
26
26
|
- **toga2-supply** (TOGa Supply) — 9 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
27
27
|
- **saml** (SAML SSO Gateway) — 9 doc(s) → [2.0/apps/saml/INDEX.md](2.0/apps/saml/INDEX.md)
|
|
28
28
|
- **apiproxy** (API Proxy) — 1 doc(s) → [2.0/apps/apiproxy/INDEX.md](2.0/apps/apiproxy/INDEX.md)
|
|
@@ -2,4 +2,4 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Framework | Summary |
|
|
4
4
|
|-----|-----------|---------|
|
|
5
|
-
| [Adyen](profile.md) | 2.0 | Adyen live TOGa Supply client (`Client_Adyen`) on the 1.0 NetSuite→TOGa Supply sync, status draft; open for the
|
|
5
|
+
| [Adyen](profile.md) | 2.0 | Adyen live TOGa Supply client (`Client_Adyen`) on the 1.0 NetSuite→TOGa Supply sync, status draft; open for the prod tracking-API test orders + SR↔NetSuite matc |
|
|
@@ -11,21 +11,27 @@ project: Worker
|
|
|
11
11
|
client: adyen
|
|
12
12
|
type: profile
|
|
13
13
|
status: draft
|
|
14
|
-
updated: 2026-09-
|
|
14
|
+
updated: 2026-09-28
|
|
15
15
|
owners: ["rgirish"]
|
|
16
16
|
files:
|
|
17
17
|
- worker/crons/toga2/netsuite/sync_togasupply_adyen.php
|
|
18
18
|
- library/app/api/toga2.php
|
|
19
19
|
- _underscore/Model/Client/Item.php
|
|
20
|
+
- _underscore/Trait/Netsuite/SalesOrder.php
|
|
21
|
+
- _underscore/Trait/Netsuite/ItemFulfillment.php
|
|
22
|
+
- dbchanges2/Client_Adyen/2026-09-22a - AdyenApiTestSeed.sql
|
|
23
|
+
- dbchanges2/Client_Adyen/2026-09-28a - LinkTestOrdersToNetsuite.sql
|
|
24
|
+
- dbchanges2/Client_Adyen/2026-09-28b - RemoveUnmatchedTestOrders.sql
|
|
20
25
|
related:
|
|
21
26
|
- ../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md
|
|
27
|
+
- ../../1.0/apps/library/features/netsuite-suiteql-api-reference.md
|
|
22
28
|
- ../growrk/features/transfer-order-flow.md
|
|
23
29
|
- ../../2.0/apps/_underscore/features/acl-permission-chain.md
|
|
24
30
|
- ../../2.0/apps/dbchanges2/workflows/client-schema-drift-audit.md
|
|
25
31
|
- ../../2.0/apps/dbchanges2/workflows/nonprod-metadata-drift-repair.md
|
|
26
32
|
---
|
|
27
33
|
|
|
28
|
-
Adyen live TOGa Supply client (`Client_Adyen`) on the 1.0 NetSuite→TOGa Supply sync, status draft; open for the `isFulfillable`
|
|
34
|
+
Adyen live TOGa Supply client (`Client_Adyen`) on the 1.0 NetSuite→TOGa Supply sync, status draft; open for the prod tracking-API test orders + SR↔NetSuite matching, the `isFulfillable` column gap, the beta=client-sandbox env trap, and the ACL findings.
|
|
29
35
|
|
|
30
36
|
## Summary
|
|
31
37
|
|
|
@@ -36,6 +42,7 @@ Adyen live TOGa Supply client (`Client_Adyen`) on the 1.0 NetSuite→TOGa Supply
|
|
|
36
42
|
|
|
37
43
|
- **NetSuite → TOGa Supply sync.** Per-client thin wrapper over the shared engine — full mechanics on [per-client sync](../../1.0/apps/worker/features/netsuite-togasupply-per-client-sync.md).
|
|
38
44
|
- **Transfer orders ENABLED** (`IS_ENABLED_INTEGRATION_TRANSFER_ORDERS = true`). Adyen's NetSuite transfer orders arrive **as NetSuite sales orders**, classified by `TRANSFER_ORDER_DETECTION_MODE__ZERO_DOLLAR_HOLD` — `holdInvoice === true` **AND** `total === 0`, same rule as GroWrk (see [GroWrk transfer order flow](../growrk/features/transfer-order-flow.md)). Mode is now declared explicitly in Adyen's launcher; `App_Api_Toga2::isTransferOrder()` throws if it is missing, so **`library` and `worker` must deploy together** for Adyen.
|
|
45
|
+
- **IDs:** `Core.Clients` id **48**; NetSuite customer **31054** ("4950 Adyen").
|
|
39
46
|
- **`Client_Adyen`** is one of the client DBs with surface-layer / ACL drift noted on the [ACL permission chain](../../2.0/apps/_underscore/features/acl-permission-chain.md) and [surface-layer schema](../../2.0/apps/dbchanges2/features/surface-layer-schema.md) docs.
|
|
40
47
|
|
|
41
48
|
## Gotchas
|
|
@@ -48,6 +55,32 @@ Adyen live TOGa Supply client (`Client_Adyen`) on the 1.0 NetSuite→TOGa Supply
|
|
|
48
55
|
- **⚠ Adyen's working beta calls hit `client-sandbox`, NOT `dev-sandbox`.** Calls go to `api.beta.togahub.com` with a JWT carrying `"env":"beta"`, which maps to the MCP environment **`client-sandbox`**. `Core.Domains` in client-sandbox shows `api.beta.togahub.com` with `env = 'beta'` and **`clientId = NULL`** — a shared API domain, so the tenant comes from the **JWT's client uuid**, not the hostname.
|
|
49
56
|
- Comparing **dev-sandbox against prod** compares two environments the developer is not using — it produced **three consecutive wrong answers** in the 2026-09-08 session. Decode the token's `env` + `id.client.uuid` before any environment diff — recipe in [non-prod metadata drift, step 1](../../2.0/apps/dbchanges2/workflows/nonprod-metadata-drift-repair.md#steps).
|
|
50
57
|
|
|
58
|
+
### Prod tracking-API test orders — linked to real NetSuite (2026-09-28)
|
|
59
|
+
|
|
60
|
+
Adyen tests the tracking API on **prod**. `2026-09-22a - AdyenApiTestSeed.sql` seeded dummy SalesOrders / ItemFulfillments / TrackingNumbers (fake `1ZADYEN…` numbers). Before the NetSuite sync runs they had to become real, or the sync would create **duplicate SOs**.
|
|
61
|
+
|
|
62
|
+
- **`2026-09-28a - LinkTestOrdersToNetsuite.sql` (RAN on prod):** sets `SalesOrders.c_netsuiteInternalSalesOrderId` + `number` = NS `tranid` + `dateOrder`; `ItemFulfillments.c_netsuiteInternalItemFulfillmentId` + `number` + `dateItemFulfillment`; swaps fake tracking for real FedEx numbers, status `PENDING`. Verified: 22 SOs linked, 21 IFs + 42 tracking numbers real.
|
|
63
|
+
- **`2026-09-28b - RemoveUnmatchedTestOrders.sql` (RAN on prod):** deletes the dummy SO/PO/IF chain for SOs with no NS match. ServiceRequests kept.
|
|
64
|
+
- **Why this prevents duplicates:** the sync matches existing rows by `c_netsuiteInternalSalesOrderId` / `c_netsuiteInternalItemFulfillmentId` (`_underscore/Trait/Netsuite/SalesOrder.php`, `ItemFulfillment.php`). It will replace the dummy `SalesOrderItems` (itemId 6).
|
|
65
|
+
- **⚠ Leftovers:** dummy POs `PO-SR%` and vendor `V-TEST-001` still exist for the linked SOs.
|
|
66
|
+
|
|
67
|
+
**⚠ The Adyen NetSuite sync has never run.** `Parameters` cursors stuck at `2026-08-20 00:00:00|0`, the `Customers` row has no `c_netsuiteInternalCustomerId`, no `Locations`. Set these up before expecting sync data.
|
|
68
|
+
|
|
69
|
+
**How to match an Adyen SR to its NetSuite SO — there is NO shared key.**
|
|
70
|
+
|
|
71
|
+
- Adyen's reference number is `ServiceRequests.c_externalReferenceId` (e.g. 48753831); `SalesOrders.serviceRequestId` → `ServiceRequests.id`. NetSuite does not store it.
|
|
72
|
+
- Match SR contact name + ship-to address against NS SO `otherrefnum` (holds the employee name) / shipping attention + addr1.
|
|
73
|
+
- NS Adyen reclaim SOs: memo "Employee Leaving" / "Contractor Leaving", item `SVC-FS-OFFBOARD-ADN`, one IF each with **2 FedEx packages**. First number (87x/875/876) = outbound, 792x = return label — **this outbound/return split is an assumption, not confirmed.**
|
|
74
|
+
- NS tracking-number lookup in SuiteQL: see [SuiteQL reference → `transaction` table](../../1.0/apps/library/features/netsuite-suiteql-api-reference.md).
|
|
75
|
+
|
|
76
|
+
**Known data gaps (2026-09-28):**
|
|
77
|
+
|
|
78
|
+
1. SR100024 duplicates SR100021 (same Adyen ref 48748808) — a lookup by that ref returns **2 SRs**.
|
|
79
|
+
2. SR100026 → NS SO 290879, not fulfilled yet.
|
|
80
|
+
3. SR100027–30 have no NS SO.
|
|
81
|
+
4. NS SO 288261 has no SR in Toga.
|
|
82
|
+
5. SR100000/01/02/25 are Adyen test SRs with no NS order.
|
|
83
|
+
|
|
51
84
|
### Open defect — `Client_Adyen.Items` missing `isFulfillable` (prod AND beta)
|
|
52
85
|
|
|
53
86
|
- Root cause of the 2026-09-08 "works in sandbox, not in production" report; **NOT an ACL problem**. Adyen is the **only** tenant missing the column.
|
|
@@ -91,3 +124,4 @@ The `AclRecordPermissions` prod-vs-dev-sandbox route-name diff from that session
|
|
|
91
124
|
- Formal/legal client name and account owner — recorded here simply as **Adyen**.
|
|
92
125
|
- Which repos beyond the `apps:` list are genuinely in Adyen's scope.
|
|
93
126
|
- Whether Adyen has client-specific models/features that deserve their own docs.
|
|
127
|
+
- Confirm with Adyen/ops: which FedEx package is outbound vs return; whether SR100024 should be closed; who sets up the NetSuite sync (customer id, Locations, cursors).
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-29
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/send_ticket_updates.php
|
|
@@ -16,6 +16,7 @@ files:
|
|
|
16
16
|
- library/app/model/common/nycdoeticket.php
|
|
17
17
|
- dbchanges/Common/MH/2026-09-16.sql
|
|
18
18
|
- library/app/api/nycdoev2.php
|
|
19
|
+
- worker/crons/sync/nycdoe/import_inc.php
|
|
19
20
|
- togadesk/desk/includes/classes/class.repair.php
|
|
20
21
|
related:
|
|
21
22
|
- servicenow-integration.md
|
|
@@ -101,6 +102,60 @@ Current selection query:
|
|
|
101
102
|
- The standalone assignment-group block was **deleted** — the main payload already diffs `assignment_group`, so it was a duplicate GET, a second code path, and the source of the lockout stamp.
|
|
102
103
|
- Debug pin already in the file: the commented `#AND NYCDOETickets.ticketNumber = '...'` line scopes a pass to one ticket.
|
|
103
104
|
|
|
105
|
+
### Assignment-group scoping — what counts as OUR ticket (rule, Mark 2026-09-28)
|
|
106
|
+
**We only care about an INC or RITM while it is assigned to one of OUR ServiceNow assignment
|
|
107
|
+
groups. If it is not, drop it and never re-check it. NEVER add another vendor's group to
|
|
108
|
+
`Common.NYCDOETicketsDepartments`** to "make it sync" — a group that is not ours means DOE
|
|
109
|
+
reassigned the incident away from ASI.
|
|
110
|
+
|
|
111
|
+
**⚠ `Common.NYCDOETickets.isValidDepartmentName` means the OPPOSITE of its name.** `1` =
|
|
112
|
+
**EXCLUDED from processing**; `0`/`NULL` = processed. Every write of `1` is on a skip path
|
|
113
|
+
(invalid site code, unmapped assignment group, or SD QA — deliberately skipped at
|
|
114
|
+
`process_tickets.php:392`), and all four DOE selections pick up only `= 0 OR IS NULL`. The
|
|
115
|
+
companion column `departmentName` carries the human reason (`INVALID_SITE_CODE`,
|
|
116
|
+
`Invalid Address Resolution`, or the actual ServiceNow group name). Reading the column name
|
|
117
|
+
literally gives you the exact opposite of the truth. Writers: `process_tickets.php` (348, 392,
|
|
118
|
+
1520), `import_asn.php:104`, `import_inc.php:85`, `send_ticket_updates.php:235`,
|
|
119
|
+
`send_request_item_updates.php:56`. A pure rename across 13 sites would be safe, but must not be
|
|
120
|
+
bundled with a behavior change.
|
|
121
|
+
|
|
122
|
+
**The drop is self-healing — which is why no new column is needed.** `import_inc.php:82-86` loads
|
|
123
|
+
the existing row by `referenceSysId` and unconditionally resets
|
|
124
|
+
`dtProcessed = null; isValidDepartmentName = null; departmentName = null;` on every re-import
|
|
125
|
+
(same in `import_asn.php:148`). While a ticket sits in a non-ASI group it never matches the
|
|
126
|
+
importer filter, so it stays dropped at zero cost; if DOE reassigns it back to ASI its
|
|
127
|
+
`sys_updated_on` moves, it re-imports, and the flag clears automatically. A purpose-built new
|
|
128
|
+
column would NOT be reset by either import path and would make the drop irreversible — reusing
|
|
129
|
+
`isValidDepartmentName` is the correct choice, not a compromise.
|
|
130
|
+
|
|
131
|
+
**⚠ OPEN DEFECT (backfilled by hand, CODE FIX NOT DONE — needs its own ticket).**
|
|
132
|
+
`send_ticket_updates.php` (~L375) treats an unmapped group as "someone forgot a mapping" and
|
|
133
|
+
emails devteam@togatech.com hourly saying *"Add the group to that table to let them sync."* That
|
|
134
|
+
advice is wrong. Measured cost: 15 tickets across 9 other-vendor groups (NOC Field, ATLAS Tier III
|
|
135
|
+
Support, SD Nagarro Queens, OASIS L3 Support, NGSIS/GAM Support, Central User Support, Field Ops
|
|
136
|
+
Queens, Field Ops Bronx, IET Support) re-GET every 6 minutes = **~3,600 wasted ServiceNow calls/day
|
|
137
|
+
and ~236 emails since 2026-09-18**. Those 15 INCs were set by hand in production to
|
|
138
|
+
`isValidDepartmentName = 1` with the real group in `departmentName`, and confirmed excluded.
|
|
139
|
+
**Agreed approach for the fix ticket (CTO-reviewed) — an ordered predicate:**
|
|
140
|
+
1. Group **is** in `NYCDOETicketsDepartments` → process normally (this is what keeps SD QA working).
|
|
141
|
+
2. Group is not in the table but **starts with `ASI`** → keep the throttled email; this is the only
|
|
142
|
+
case the current advice is right for.
|
|
143
|
+
3. Neither → **drop silently.**
|
|
144
|
+
Do **not** close the repair orders — write one `repair_order_notes` line and send a one-time digest
|
|
145
|
+
to DOE ops. `send_request_item_updates.php` needs the same treatment for RITMs.
|
|
146
|
+
|
|
147
|
+
**⚠ Three competing definitions of "our DOE assignment groups" — they disagree.**
|
|
148
|
+
- **(a) `Common.NYCDOETicketsDepartments`** — 14 rows: ASI Bronx, ASI Brooklyn, ASI Manhattan,
|
|
149
|
+
ASI Queens, ASI Staten Island, ASI General, ASI Calls, ASI Reconnect, ASI Research, ASI DOA,
|
|
150
|
+
ASI TM, SD QA (`isImportEnabled=0`), ASI Disposal, ASI PCS Delivery. **This one is correct.**
|
|
151
|
+
- **(b) `download_tickets_incidents.php:80`** — a hardcoded 12-group ServiceNow query list that
|
|
152
|
+
includes "ASI Enhanced", which exists nowhere else. The file is referenced by **no**
|
|
153
|
+
`schedules/*.json` → **dead code; delete it in its own commit.**
|
|
154
|
+
- **(c) `App_Api_NYCDOEV2::listIncItems()` (`library/app/api/nycdoev2.php:912`)** — the **live**
|
|
155
|
+
importer filter, a prefix match `assignment_groupSTARTSWITHASI^stateIN1,2,3,10`. This is what
|
|
156
|
+
`import_inc.php` actually uses.
|
|
157
|
+
**Risk:** a future group named "ASI Something" imports under (c) then is dropped under (a).
|
|
158
|
+
|
|
104
159
|
### Batch loop, run caps & per-ticket isolation (TRUE-82129 current shape)
|
|
105
160
|
Before this the cron ran the selection query **once per ticket under `LIMIT 1`** with a `sleep(1)` after every ticket — one ticket per 6-minute run, ~10 days for a full pass over the queue.
|
|
106
161
|
- **One `LIMIT 250` (`BATCH_SIZE`) read into a PHP array, consumed with `array_shift`, refilled only when empty.** The query is `type=ALL`, ~35,021 rows, `Using temporary; Using filesort` (it drives off `managed_service_orders`, so `Common.NYCDOETickets.idx_dtLastChecked` is unusable at this join order). A filesort already sorts every surviving row before returning the top N, **so a bigger LIMIT costs nothing extra**.
|
|
@@ -138,6 +193,19 @@ There is **no sandbox for the SNOW integration**; verification happens in prod.
|
|
|
138
193
|
2. INC2087375's `u_eta` moving from 2026-09-07 to 2026-09-17.
|
|
139
194
|
3. `SELECT COUNT(*) FROM Common.NYCDOETickets WHERE dtLastChecked IS NOT NULL` climbing.
|
|
140
195
|
|
|
196
|
+
**Answering "did the alert fire, and did the mail actually send?" — two tables, not an inbox.**
|
|
197
|
+
An inbox check alone is misleading; the DB is decisive.
|
|
198
|
+
- **Did the alert fire?** marker rows in `Core.AdvanceShippingNoticeQueue`, `dedupeKey`
|
|
199
|
+
`system|ticket_sync_patch_failure_alert` and `system|ticket_sync_invalid_group_alert`.
|
|
200
|
+
- **Did the mail send?** `Logs.Emails` — columns `dtSent`, `isSuccess`, `error`, `subject`, `to`,
|
|
201
|
+
`sourceJob`. ⚠ the table is **`Emails`**, not `Email`, and the timestamp is **`dtSent`**, not
|
|
202
|
+
`dtStamp`.
|
|
203
|
+
|
|
204
|
+
**TRUE-82027 verified healthy in production 2026-09-28:** `dtLastChecked` + `idx_dtLastChecked`
|
|
205
|
+
present; ~2,784 active tickets cycling in under an hour at `RECHECK_INTERVAL_MINUTES=6`; PATCH
|
|
206
|
+
volume ~44,000/day at 443,706 x HTTP 200 against only 6 non-200 over 10 days (3x 502, 1x 429);
|
|
207
|
+
reproduction case INC2087375 back in rotation and syncing.
|
|
208
|
+
|
|
141
209
|
### RITM cron — deliberately unchanged by TRUE-81044
|
|
142
210
|
`send_request_item_updates.php` already pushes native `state:"8"` (On Hold) and **never wrote** `u_status_task`, so TRUE-81044's ask was already satisfied there. It carries a defensive `HOLD_*` guard on the scheduled-update path so it can't push `state:"-14"` while held; its reverse-hold logic still **reads** `u_status_task == 'On Hold'`.
|
|
143
211
|
> **OPEN QUESTION:** ServiceNow's communication was scoped to **Incident** state. We did **not** extend `hold_reason` / `u_eta` / `u_other_reason_on_hold` to the RITM payload without confirmation. Whether the RITM On Hold transition carries the same mandatory-set requirement is unconfirmed — ask before changing the RITM payload.
|
|
@@ -175,7 +243,7 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
|
|
|
175
243
|
- **Query shape that works (TRUE-82332):** `endpoint LIKE '%nycd3%...'` and broad `requestPayload LIKE` scans time out (>300s) on the MCP. Use exact matches: PATCH `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/data/tasks/INCxxxx'`; GET `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/v2/data/tasks?domain=diit%2Calm&filter=sys_class_name%3Dincident%5Enumber%3DINCxxxx'`. For sweeps: `FORCE INDEX (dtStamp)`, windows ≤2 days, plus `sourceJob` / `method` filters. The state:3↔state:2 oscillation in TRUE-79922 was proven from this table (INC2074899, INC2185183 showed `state:3` then `state:2` pushed seconds apart in one run).
|
|
176
244
|
- **`NYCDOETickets` (`App_Model_Common_NYCDOETicket`, `db_common` / legacy `Common`)** stores the inbound SNOW record in `rawData`, a **`FIELDTYPE_BLOBSTORAGE`** field (S3 bucket `asifiles`, key `prod/Common/NYCDOETickets/rawData/{id}`) — **NOT a queryable DB column.**
|
|
177
245
|
- **⚠ `repair_orders.dtUpdated` is skewed — never build a selection query on it.** It is written exactly **240 minutes behind** the clock used by `repair_order_notes.notesdate` and `NYCDOETickets.dtSynced`. **Do NOT compensate with `+240 MINUTE`** — 240 is almost certainly UTC-vs-Eastern and becomes 300 at the DST change (2026-11-01), so a hardcoded offset breaks twice a year. TRUE-82027 sidestepped it by removing `dtUpdated` from the query. Mechanism + the second (720-minute) skew: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
|
|
178
|
-
- **⚠ `Logs.API` history is capped at 30 days
|
|
246
|
+
- **⚠ `Logs.API` history is capped at 30 days — SHIPPED 2026-09-28 (TRUE-82049).** This investigation depended on history far older than that (9,914 PATCHes back to 2026-08-20; a hold last pushed 22 days earlier; the 2,392-call storm). **Slow-burn bugs like this can no longer be traced backwards — they must be caught by alerting.** NYCDOE stays 1.0 so its logs stay in legacy `Logs.API`. Mechanics, storage facts and the not-yet-run backlog purge: [API Request Logging & Retention](../../../1.0/apps/library/features/api-request-logging-and-retention.md).
|
|
179
247
|
- **Worker prod code cannot run on a Windows dev box.** The DB/S3 data layer fetches EC2 instance-metadata (IMDSv2) credentials and fails off-EC2. Investigate with the toga DB MCP and the `Logs.API` table instead of running the crons locally.
|
|
180
248
|
|
|
181
249
|
## Gotchas
|
|
@@ -202,6 +270,7 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
|
|
|
202
270
|
- `qqStatus()` collapses all recognized hold sub-statuses to base `HOLD` in `repair_orders.status`; the specific hold **sub-reason persists only in `repair_order_notes.newstatus`**. Any status badge / report / filter reading `repair_orders.status` directly cannot distinguish the six hold variants after a recompute.
|
|
203
271
|
|
|
204
272
|
## Change history
|
|
273
|
+
- 2026-09-29 — Assignment-group scoping rule + inverted `isValidDepartmentName` + the three competing group definitions; TRUE-82027 verified in prod (marker rows + `Logs.Emails` method). (mhammontree)
|
|
205
274
|
- 2026-09-25 — TRUE-82332: hold-release no longer overwrites the close `state`; close re-sent when SN not closed; stuck-ticket re-queue recipe; `Logs.API` query shapes. (mhammontree)
|
|
206
275
|
- 2026-09-18 — TRUE-82129: fixed the by-reference `fetchOne()` notice that killed every run; batch loop + run caps replaced LIMIT 1 + per-ticket sleep; alert emails de-PII'd and escaped; unmapped-group email now grouped per run. (mhammontree)
|
|
207
276
|
- 2026-09-17 — TRUE-82027: outbound push runs once per order per pass; `dtLastChecked` split from `dtSynced`; failure alerting instead of a throw. (mhammontree)
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-29
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/import_asn.php
|
|
@@ -62,7 +62,7 @@ The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents +
|
|
|
62
62
|
| Table | Purpose |
|
|
63
63
|
|---|---|
|
|
64
64
|
| `NYCDOETickets` | Local ticket mirror (keyed `referenceSysId`; `type` = INCIDENT / RITM) |
|
|
65
|
-
| `NYCDOETicketsDepartments` | Allowlist of
|
|
65
|
+
| `NYCDOETicketsDepartments` | Allowlist of OUR assignment groups — others are dropped and never re-checked. ⚠ Two other places define this list differently, and `NYCDOETickets.isValidDepartmentName` is **inverted** (`1` = excluded): see [Hold-Status Sync](hold-status-sync.md) → *Assignment-group scoping* |
|
|
66
66
|
| `NYCDOELocations` | Site-id → address mirror (backs ASN ship-to resolution) |
|
|
67
67
|
| `AdvanceShippingNoticeQueue` | Stage-1 landing zone; **UNIQUE index `idx_dedupeKey`** |
|
|
68
68
|
| `AdvanceShippingNotices` / `…Items` / `…Units` | ASN header (vendor+PO) → per-part items (`qtyOrder`) → individual units |
|
|
@@ -130,7 +130,7 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘ [UNIQUE ded
|
|
|
130
130
|
|
|
131
131
|
### Debugging & DB topology (DOE 1.0 sync)
|
|
132
132
|
The 1.0 worker cannot run on a dev/Windows box; use the toga DB MCP + `Logs.API` instead of running prod code locally.
|
|
133
|
-
- **⚠ `Logs.API` retention is 30 days
|
|
133
|
+
- **⚠ `Logs.API` retention is 30 days — SHIPPED 2026-09-28 (TRUE-82049)**; mechanics, storage facts and the not-yet-run backlog purge in [API Request Logging & Retention](../../../1.0/apps/library/features/api-request-logging-and-retention.md) — so it is no longer a *historical* debugging avenue, only a recent-window one. Bugs that burn slowly (a hold not re-pushed for 22 days; 9,914 PATCHes traced back to 2026-08-20) can no longer be reconstructed after the fact and must be caught by **alerting** instead. NYCDOE stays 1.0, so its logs remain in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0.
|
|
134
134
|
- **`repair_orders.dtUpdated` and `.lastupdateddate` are both time-skewed in `TOGaDeskSupport`** — never select or diff on them, and never compensate with a fixed offset. Detail: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
|
|
135
135
|
- **`dbchanges` does not describe the real 1.0 schema** — some live columns were created outside the repo. Verify indexes/columns against the database, not the repo: [Authoring & Shipping a 1.0 dbchanges SQL File](../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
|
|
136
136
|
- **`db_core` alias → legacy (V1) environment, schema `Core`** — holds the FLAT tables `AdvanceShippingNotices` / `AdvanceShippingNoticeItems` / `AdvanceShippingNoticeUnits` (columns incl. `togadeskRepairOrderId`, `serialNumber`, `netSuiteInternalSalesOrderId`, `netSuiteInternalPurchaseOrderId`, `customerPurchaseOrder`). The 1.0 cron reads **legacy/Core**.
|
|
@@ -262,5 +262,6 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
|
|
|
262
262
|
- Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
|
|
263
263
|
|
|
264
264
|
## Change history
|
|
265
|
+
- 2026-09-29 — Assignment-group scoping corrected (inverted `isValidDepartmentName`, three competing group lists) with detail in hold-status-sync; `Logs.API` 30-day retention shipped. (mhammontree)
|
|
265
266
|
- 2026-09-25 — TRUE-82332: cross-link to the close-clobber fix; test-repo copy of the ServiceNow credential noted (mhammontree)
|
|
266
267
|
- 2026-09-25 — Corrected "no 2.0 tenant" (TRUE-80519 2.0 ASN rewrite now exists); DOE-Open-Lenovo marked retired (mhammontree)
|
|
@@ -7,4 +7,5 @@
|
|
|
7
7
|
| [Quad: PO + ASN email importer (import_po_and_asn.php)](features/po-asn-email-import.md) | 1.0 | Quad's inbox-driven PO/ASN CSV importer and its 2026-08 failure classes; open when a Quad PO/ASN email jams the mailbox, fails EV-12, or you touch `import_po_an |
|
|
8
8
|
| [Quad Graphics](profile.md) | 2.0 | Quad Graphics is a TOGA 2.0 client on the `_underscore` platform, prod schema `Client_Quad`. |
|
|
9
9
|
| [Quad Graphics — reversing the 30-item catalog fold (storefront-assortment recovery)](workflows/catalog-fold-reversal.md) | 2.0 | How the 2026-08-31 catalog-blind item merge damaged Quad's storefront and how the reversal was built; open when assessing/repairing catalog-fold damage on a two |
|
|
10
|
+
| [Giving a Quad user every persona plus the `Global Admin` role](workflows/granting-all-personas-and-global-admin.md) | 2.0 | The two-table grant for "give internal user X everything on Quad" — every `Personas` row + `Global Admin` — and why it does **not** produce a combined catalogue |
|
|
10
11
|
| [Giving Quad's internal TOGA staff extra toga25-supply nav (the `Agilant - Administrators` role pattern)](workflows/granting-internal-admin-navigation.md) | 2.0 | How to surface extra 2.5 Supply pages to internal TOGA staff on a tenant without showing them to the client's own users — Quad's first internal-admin role; open |
|
|
@@ -23,6 +23,7 @@ related:
|
|
|
23
23
|
- ../../2.0/apps/toga2-commerce/features/storefront-persona-gating-and-home-counts.md
|
|
24
24
|
- ./workflows/catalog-fold-reversal.md
|
|
25
25
|
- ./workflows/granting-internal-admin-navigation.md
|
|
26
|
+
- ./workflows/granting-all-personas-and-global-admin.md
|
|
26
27
|
- ./features/po-asn-email-import.md
|
|
27
28
|
- ./features/enter-po-details-vendor-selection-ev12.md
|
|
28
29
|
- ../../2.0/apps/_underscore/features/sales-order-po-number-sourcing.md
|
|
@@ -63,7 +64,10 @@ Client-specific DB change-sets live in `dbchanges2/Client_Quad/`.
|
|
|
63
64
|
- **Quad is the ONLY storefront tenant with the persona switcher**
|
|
64
65
|
(`determineShouldShowPersonaSwitcher` returns true for QUAD alone). With it on, `getLoginSettings`
|
|
65
66
|
sets `_personaIds` to a **single** persona (the saved default, else the first) — so a Quad user
|
|
66
|
-
never sees a combined view of their personas.
|
|
67
|
+
never sees a combined view of their personas. Granting a user **all** personas therefore only makes
|
|
68
|
+
them **selectable one at a time** in the switcher; it does not widen what they see at once
|
|
69
|
+
([all personas + Global Admin](./workflows/granting-all-personas-and-global-admin.md)).
|
|
70
|
+
`Client_Quad` also has **no persona 24 at all** (not
|
|
67
71
|
on a user, not on a location, the persona does not exist), so the Compass "persona 24" storefront
|
|
68
72
|
hack never touched Quad. See
|
|
69
73
|
[Storefront persona gating](../../2.0/apps/toga2-commerce/features/storefront-persona-gating-and-home-counts.md).
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Giving a Quad user every persona plus the `Global Admin` role
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: dbchanges2
|
|
5
|
+
project: Database Changes
|
|
6
|
+
client: quad
|
|
7
|
+
type: workflow
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-28
|
|
10
|
+
owners: ["apeterson"]
|
|
11
|
+
files:
|
|
12
|
+
- dbchanges2/Client_Quad/2026-09-28b - GrantAllPersonasToNSchenkel.sql
|
|
13
|
+
related:
|
|
14
|
+
- ../profile.md
|
|
15
|
+
- ./granting-internal-admin-navigation.md
|
|
16
|
+
- ../../compass-usa/workflows/granting-persona-bundle-access.md
|
|
17
|
+
- ../../../2.0/apps/_underscore/features/effective-persona-resolution.md
|
|
18
|
+
- ../../../2.0/apps/toga2-commerce/features/storefront-persona-gating-and-home-counts.md
|
|
19
|
+
- ../../../2.0/apps/dbchanges2/features/rerunnable-additive-inserts.md
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
The two-table grant for "give internal user X everything on Quad" — every `Personas` row + `Global Admin` — and why it does **not** produce a combined catalogue view on this tenant.
|
|
23
|
+
|
|
24
|
+
## Summary
|
|
25
|
+
|
|
26
|
+
A recurring request shape: *give internal user X everything on tenant Y*. On Quad it is **two
|
|
27
|
+
guarded, re-runnable INSERTs**, both entirely inside `Client_Quad` (no `Core` join — separate prod
|
|
28
|
+
clusters):
|
|
29
|
+
|
|
30
|
+
1. **`Users_Personas`** — one row per `Client_Quad.Personas` row the user does not already hold
|
|
31
|
+
(`Users` resolved by `email`, `CROSS JOIN Personas`), uuid from the house inline `RANDOM_BYTES()`
|
|
32
|
+
v4 expression.
|
|
33
|
+
2. **`Users_Roles`** — the tenant's `Global Admin` role.
|
|
34
|
+
|
|
35
|
+
Both are additive: the user keeps every persona and role they already hold (roles resolve as a
|
|
36
|
+
union — [ACL permission chain](../../../2.0/apps/_underscore/features/acl-permission-chain.md)).
|
|
37
|
+
Nothing is removed, so this is not a reversible-by-re-run change; removing access is a separate,
|
|
38
|
+
destructive job.
|
|
39
|
+
|
|
40
|
+
## Steps
|
|
41
|
+
|
|
42
|
+
1. **Guard `Users_Personas` on `(userId, personaId)`** — that *is* the real unique key
|
|
43
|
+
(`userId_personaId_UNIQUE`, verified against
|
|
44
|
+
`dbchanges2/Client/2026-09-22-BLANK_CLIENT_DATABASE.sql:11410`), so a guarded re-run is a true
|
|
45
|
+
no-op rather than a duplicate-key abort.
|
|
46
|
+
2. **Resolve the role BY NAME, not by id** — `INNER JOIN Roles r ON r.name = 'Global Admin'`. On
|
|
47
|
+
Quad that is id **7** today, but ids drift per environment; the name keeps one file correct
|
|
48
|
+
everywhere.
|
|
49
|
+
3. **Name only `uuid`, `userId`, `personaId` / `roleId`.** `Users_Personas` and `Users_Roles` carry
|
|
50
|
+
**no `dtCreated`** — see the gotcha below.
|
|
51
|
+
4. **End with two verify SELECTs** — the user's persona list ordered by
|
|
52
|
+
`CAST(number AS UNSIGNED)` (`Personas.number` is a string column) and their role list.
|
|
53
|
+
5. **Hand the file to the developer to run.** Verify in the app by logging out and back in.
|
|
54
|
+
|
|
55
|
+
### The 2026-09-28 Quad migration
|
|
56
|
+
`dbchanges2/Client_Quad/2026-09-28b - GrantAllPersonasToNSchenkel.sql` — branch `TRUE-82247`,
|
|
57
|
+
**not committed and not yet run in any environment.** Grants `nschenkel@togatech.com` every
|
|
58
|
+
`Client_Quad` persona plus `Global Admin`.
|
|
59
|
+
|
|
60
|
+
Context: NSchenkel is one of the three `@togatech.com` Quad users deliberately left **off** the new
|
|
61
|
+
`Agilant - Administrators` role created the same day
|
|
62
|
+
([internal-admin navigation](./granting-internal-admin-navigation.md) — that role went to
|
|
63
|
+
`devteam@togatech.com` only). This grant gives a different, wider reach: the tenant's own
|
|
64
|
+
`Global Admin`, not the nav-only internal role.
|
|
65
|
+
|
|
66
|
+
## Gotchas
|
|
67
|
+
|
|
68
|
+
- **⚠ On Quad, "all personas" does NOT mean "sees everything at once".** Quad is the **only**
|
|
69
|
+
toga2-commerce tenant with the **persona switcher**
|
|
70
|
+
(`determineShouldShowPersonaSwitcher` returns true for `QUAD` alone), and with it on
|
|
71
|
+
`getLoginSettings` sets `_personaIds` to a **single** persona (the saved default, else the first).
|
|
72
|
+
So this grant makes every persona **selectable, one at a time** — it does not build the combined
|
|
73
|
+
catalogue view the same grant would give on Compass / Compass Canada. Expect the reporter to say
|
|
74
|
+
"the storefront did not change". See
|
|
75
|
+
[Storefront persona gating](../../../2.0/apps/toga2-commerce/features/storefront-persona-gating-and-home-counts.md).
|
|
76
|
+
- **This touches only source 1 of the three effective-persona sources.** `Locations_Personas` on the
|
|
77
|
+
user's own location and on the 5-hop sector are untouched — so a "who has persona N" check still
|
|
78
|
+
has to resolve all three. See
|
|
79
|
+
[Effective persona resolution](../../../2.0/apps/_underscore/features/effective-persona-resolution.md).
|
|
80
|
+
- **`Global Admin` is not an internal-only role on Quad.** Role 7 is held by 7 users, five of them
|
|
81
|
+
`@quad.com` — fine for granting *to* one internal user, never a place to hang an internal-only
|
|
82
|
+
override.
|
|
83
|
+
|
|
84
|
+
## Change history
|
|
85
|
+
- 2026-09-28 — Created; documents the `Users_Personas` + `Users_Roles` all-personas grant written for
|
|
86
|
+
NSchenkel (not yet run) and the Quad persona-switcher caveat (apeterson)
|
|
@@ -16,6 +16,7 @@ related:
|
|
|
16
16
|
- ../../../2.0/apps/toga25-supply/features/side-navigation-and-default-route.md
|
|
17
17
|
- ../../../2.0/apps/_underscore/features/acl-permission-chain.md
|
|
18
18
|
- ../../../2.0/apps/dbchanges2/features/rerunnable-additive-inserts.md
|
|
19
|
+
- ./granting-all-personas-and-global-admin.md
|
|
19
20
|
---
|
|
20
21
|
|
|
21
22
|
How to surface extra 2.5 Supply pages to internal TOGA staff on a tenant without showing them to the client's own users — Quad's first internal-admin role; open for "internal user can't see page X" on any tenant.
|
|
@@ -45,6 +46,8 @@ staff. That is a tenant-wide behavior change dressed up as a per-user fix.
|
|
|
45
46
|
3. **Create a dedicated `Agilant - Administrators` role** in the client DB if the tenant has none,
|
|
46
47
|
and assign only the internal users the developer names. Quad got **`devteam@togatech.com` only**;
|
|
47
48
|
the other three `@togatech.com` Quad users (JWise, NSchenkel, rgreen) were deliberately left off.
|
|
49
|
+
NSchenkel was instead granted the tenant's own `Global Admin` role (a different, wider reach) —
|
|
50
|
+
[all personas + Global Admin](./granting-all-personas-and-global-admin.md).
|
|
48
51
|
4. **Add role-scoped `SurfaceOverrides` rows** (`surfaceId` 39, the element ids, `IS_VISIBLE` = `'1'`,
|
|
49
52
|
`personaId`/`languageId` NULL) for that new role. Nothing else — no `AclRecordPermissions`, no
|
|
50
53
|
`AclFieldPermissions`, so the migration cannot widen data access.
|
|
@@ -77,5 +80,7 @@ Quad's starting state for contrast: **one** nav override total (element 109 `sal
|
|
|
77
80
|
migration there.
|
|
78
81
|
|
|
79
82
|
## Change history
|
|
83
|
+
- 2026-09-28 — NSchenkel (left off this role) granted `Global Admin` + all personas instead, in a
|
|
84
|
+
separate file; linked. (apeterson)
|
|
80
85
|
- 2026-09-28 — Created; Quad's first `Agilant - Administrators` role + Items/Vendor Items nav
|
|
81
86
|
overrides written (not yet run), scoped away from `Global Admin` role 7 (apeterson)
|
package/package.json
CHANGED