toga-ai 1.0.911 → 1.0.912
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/library/INDEX.md +1 -0
- package/knowledge/1.0/apps/library/features/api-request-logging-and-retention.md +89 -0
- package/knowledge/INDEX.md +1 -1
- package/knowledge/clients/nycdoe/features/hold-status-sync.md +71 -2
- package/knowledge/clients/nycdoe/features/servicenow-integration.md +4 -3
- package/package.json +1 -1
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
|-----|---------|
|
|
5
5
|
| [Library (1.0 Framework) Architecture](architecture.md) | The 1.0 `App_` framework core: bootstrap/autoloader, the `app/` (`App_`) + `browser/` (`Browser_`) class trees, and vendored packages; open for how any 1.0 app |
|
|
6
6
|
| [Address Validation Gateway (App_Api_OfficeDepot::validateAddress + USPS fallback)](features/address-validation-gateway.md) | The shared 1.0 address-validation gateway (`App_Api_OfficeDepot::validateAddress` → USPS fallback); open when standardizing/verifying a typed address before sav |
|
|
7
|
+
| [API Request Logging & Retention (App_ApiTransaction → Logs.API / Logs_<Tenant>)](features/api-request-logging-and-retention.md) | Where every outbound API call is logged (1.0 `Logs.API` vs the 2.0 `Logs_<Tenant>` cluster), how the routing is chosen, and the 30-day retention trim; open befo |
|
|
7
8
|
| [Where a new App_ class goes — the app/ folder IS a behavioral contract](features/app-class-placement-base-contracts.md) | Where a new `App_` class goes — each `app/` folder's base class carries real inherited behavior; open before creating any class under `app/model|client|api`. |
|
|
8
9
|
| [App_Sso — Reusable 1.0 SSO Initiation (SP-initiated SAML via saml.togahub.com)](features/app-sso-initiation.md) | Reusable 1.0 SP-initiated SAML SSO via `saml.togahub.com` (`App_Sso::initiate`); the canonical doc to follow for any "add SSO to `<1.0 app>`" request. |
|
|
9
10
|
| [Cron Execution Monitoring (App_Framework check-in/out → CronJobExecutions)](features/cron-execution-monitoring.md) | 1.0 cron check-in/out lifecycle via `App_Framework::cronInitialization()`/`cronFinished()` → `Common.CronJobExecutions`; open to monitor a 1.0 cron or read its |
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "API Request Logging & Retention (App_ApiTransaction → Logs.API / Logs_<Tenant>)"
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: library
|
|
5
|
+
project: Library
|
|
6
|
+
client: shared
|
|
7
|
+
type: feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-29
|
|
10
|
+
owners: ["mhammontree"]
|
|
11
|
+
files:
|
|
12
|
+
- library/app/apitransaction.php
|
|
13
|
+
- library/app/model/logs/api.php
|
|
14
|
+
- library/app/model/toga2/logs/api.php
|
|
15
|
+
- worker/crons/maintenance/maintenance_daily.php
|
|
16
|
+
related:
|
|
17
|
+
- error-capture-1-0.md
|
|
18
|
+
- ../architecture.md
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
Where every outbound API call is logged (1.0 `Logs.API` vs the 2.0 `Logs_<Tenant>` cluster), how the routing is chosen, and the 30-day retention trim; open before changing log routing, retention, or debugging past-window history.
|
|
22
|
+
|
|
23
|
+
## Summary
|
|
24
|
+
|
|
25
|
+
`App_ApiTransaction` writes one row per outbound API call. It has **two** storage paths — the legacy 1.0 `Logs`.`API` table and the 2.0 per-tenant `Logs_<Tenant>` cluster — and picks between them per run. Retention is enforced only on the 1.0 side, by a daily worker cron, at **30 days**.
|
|
26
|
+
|
|
27
|
+
## How it works
|
|
28
|
+
|
|
29
|
+
### Which log DB a call lands in
|
|
30
|
+
`library/app/apitransaction.php:377`:
|
|
31
|
+
```php
|
|
32
|
+
if (App_Registry::varIsSet('toga2LogsDatabase')) { /* App_Model_Toga2_Logs_Api */ }
|
|
33
|
+
else { /* App_Model_Logs_API → legacy Logs.API */ }
|
|
34
|
+
```
|
|
35
|
+
- 2.0 routing is **opt-in per cron**. Only 10 crons set it, all Prudential, each hardcoding
|
|
36
|
+
`App_Registry::set('toga2LogsDatabase', 'Logs_Prudential')`.
|
|
37
|
+
- Every other `crons/toga2/*` cron falls through to the 1.0 table — **this is why 2.0 client traffic
|
|
38
|
+
lands in the 1.0 `Logs.API`**.
|
|
39
|
+
|
|
40
|
+
### Retention (1.0 only)
|
|
41
|
+
`worker/crons/maintenance/maintenance_daily.php` trims the log tables nightly.
|
|
42
|
+
- `$daysToKeep_APILog` = **30** (was 90; cut 2026-09-28, TRUE-82049, commit `89ef44f35`).
|
|
43
|
+
- The `DELETE FROM Logs.API` is **chunked**: 25,000 rows per statement in a do/while loop with a
|
|
44
|
+
1s sleep, still bounded overall by `$maxLimitForDeletes`. Every *other* table this cron trims is
|
|
45
|
+
a single statement — `Logs.API` is the exception because it is ~12x the next biggest table
|
|
46
|
+
(`Vision_Log.Log`, 45.85 GB), and a single 5,000,000-row DELETE holds locks for minutes and lags
|
|
47
|
+
read replicas.
|
|
48
|
+
- **The one-time backlog purge (~20M rows) is separate and not done** as of 2026-09-29.
|
|
49
|
+
|
|
50
|
+
### Storage facts (measured 2026-09-28)
|
|
51
|
+
- 1.0 `Logs`.`API`: **565.27 GB / 34,464,186 rows**, growing **~6.3 GB/day**. `dtStamp` is the
|
|
52
|
+
indexed date column; `sourceJob` names the writing cron.
|
|
53
|
+
- Top writers by bytes (2-day sample): `sync_togasupply_compass_usa.php` 2.53 GB (128 KB/row),
|
|
54
|
+
`periodic_forecast_discrepancy_fix.php` 1.63 GB, `sync_togasupply_hh.php` 0.84 GB,
|
|
55
|
+
`sync_togasupply_elite.php` 0.53 GB, `compass/workflow/5_create_netsuite_sales_order.php`
|
|
56
|
+
1,581 KB/row, `infrastructure/system_monitors.php` 3,040 KB/row.
|
|
57
|
+
- 2.0 logs cluster (`writer.logs.database.togahub.com`): **~1.67 TB** across 36 `Logs_<Tenant>`
|
|
58
|
+
DBs, oldest rows **2025-08-24** (~13 months — *not* the 90 days believed), and **no scheduled
|
|
59
|
+
DB events trimming it**. Largest: `Logs_Wje` 372.80 GB, `Logs_Elite` 228.99 GB,
|
|
60
|
+
`Logs_Compass` 185.88 GB, `Logs_Nychh` 129.71 GB, `Logs_Prudential` 125.32 GB.
|
|
61
|
+
`Logs_Northwell` is 43.59 GB from only 194,643 rows = 224 KB/row.
|
|
62
|
+
|
|
63
|
+
### Decision
|
|
64
|
+
**`Logs.API` is forensics-only; 30 days is enough** (Jeff, 2026-09-17). Debugging that needs
|
|
65
|
+
older history is no longer possible — slow-burn bugs must be caught by **alerting**, not
|
|
66
|
+
reconstructed after the fact.
|
|
67
|
+
|
|
68
|
+
## Gotchas
|
|
69
|
+
|
|
70
|
+
- **DELETE does not return space to disk on InnoDB.** After the backlog purge the file stays
|
|
71
|
+
~565 GB with `DATA_FREE` growing. Reclaiming disk needs `OPTIMIZE TABLE` (~565 GB spare
|
|
72
|
+
required) or a copy-and-swap — a separate decision, not part of the retention change.
|
|
73
|
+
- **Do NOT move 2.0 client logging to the 2.0 cluster without solving retention there first.**
|
|
74
|
+
It would shift ~7 GB/day onto a cluster that is already bigger and has **no working retention**.
|
|
75
|
+
Deliberately descoped from TRUE-82049. Nobody has found the 2.0 archiving script — it is not in
|
|
76
|
+
`worker2`, `api2`, or `_underscore`.
|
|
77
|
+
- **`App_Model_Toga2_Logs_Api` has NO `save()` override.** The 1.0 `App_Model_Logs_API::save()`
|
|
78
|
+
deliberately wraps `parent::save()` in try/catch because a long-running cron can outlive its
|
|
79
|
+
`db_logs` connection (MySQL "server has gone away"), which once aborted whole imports mid-run.
|
|
80
|
+
Routing traffic to the 2.0 model without adding that guard reintroduces that failure.
|
|
81
|
+
- **The 2.0 model has a `source` column the 2.0 branch never sets** — moving a cron to 2.0 logging
|
|
82
|
+
loses the writing-cron name (`sourceJob`'s equivalent).
|
|
83
|
+
- **Cap payload size when adding new logging.** Team precedent: worker2's
|
|
84
|
+
`MAX_LOG_PAYLOAD_LENGTH = 8000` with a `...[truncated]` marker
|
|
85
|
+
(`Worker/Netsuite/Opportunity.php:148`, `Worker/Clickup/Opportunity.php:45`). Rows of
|
|
86
|
+
1,500–3,000 KB are what built the 565 GB table.
|
|
87
|
+
|
|
88
|
+
## Change history
|
|
89
|
+
- 2026-09-29 — Created: `Logs.API` retention cut 90→30 days and chunked DELETE (TRUE-82049); dual 1.0/2.0 routing and 2.0-cluster storage facts recorded. (mhammontree)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -4,7 +4,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
4
4
|
|
|
5
5
|
## 1.0 framework
|
|
6
6
|
|
|
7
|
-
- **library** (Library) _(framework core)_ —
|
|
7
|
+
- **library** (Library) _(framework core)_ — 29 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
|
|
8
8
|
- **worker** (Worker) — 40 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
|
|
9
9
|
- **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
|
|
10
10
|
- **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-29
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/send_ticket_updates.php
|
|
@@ -16,6 +16,7 @@ files:
|
|
|
16
16
|
- library/app/model/common/nycdoeticket.php
|
|
17
17
|
- dbchanges/Common/MH/2026-09-16.sql
|
|
18
18
|
- library/app/api/nycdoev2.php
|
|
19
|
+
- worker/crons/sync/nycdoe/import_inc.php
|
|
19
20
|
- togadesk/desk/includes/classes/class.repair.php
|
|
20
21
|
related:
|
|
21
22
|
- servicenow-integration.md
|
|
@@ -101,6 +102,60 @@ Current selection query:
|
|
|
101
102
|
- The standalone assignment-group block was **deleted** — the main payload already diffs `assignment_group`, so it was a duplicate GET, a second code path, and the source of the lockout stamp.
|
|
102
103
|
- Debug pin already in the file: the commented `#AND NYCDOETickets.ticketNumber = '...'` line scopes a pass to one ticket.
|
|
103
104
|
|
|
105
|
+
### Assignment-group scoping — what counts as OUR ticket (rule, Mark 2026-09-28)
|
|
106
|
+
**We only care about an INC or RITM while it is assigned to one of OUR ServiceNow assignment
|
|
107
|
+
groups. If it is not, drop it and never re-check it. NEVER add another vendor's group to
|
|
108
|
+
`Common.NYCDOETicketsDepartments`** to "make it sync" — a group that is not ours means DOE
|
|
109
|
+
reassigned the incident away from ASI.
|
|
110
|
+
|
|
111
|
+
**⚠ `Common.NYCDOETickets.isValidDepartmentName` means the OPPOSITE of its name.** `1` =
|
|
112
|
+
**EXCLUDED from processing**; `0`/`NULL` = processed. Every write of `1` is on a skip path
|
|
113
|
+
(invalid site code, unmapped assignment group, or SD QA — deliberately skipped at
|
|
114
|
+
`process_tickets.php:392`), and all four DOE selections pick up only `= 0 OR IS NULL`. The
|
|
115
|
+
companion column `departmentName` carries the human reason (`INVALID_SITE_CODE`,
|
|
116
|
+
`Invalid Address Resolution`, or the actual ServiceNow group name). Reading the column name
|
|
117
|
+
literally gives you the exact opposite of the truth. Writers: `process_tickets.php` (348, 392,
|
|
118
|
+
1520), `import_asn.php:104`, `import_inc.php:85`, `send_ticket_updates.php:235`,
|
|
119
|
+
`send_request_item_updates.php:56`. A pure rename across 13 sites would be safe, but must not be
|
|
120
|
+
bundled with a behavior change.
|
|
121
|
+
|
|
122
|
+
**The drop is self-healing — which is why no new column is needed.** `import_inc.php:82-86` loads
|
|
123
|
+
the existing row by `referenceSysId` and unconditionally resets
|
|
124
|
+
`dtProcessed = null; isValidDepartmentName = null; departmentName = null;` on every re-import
|
|
125
|
+
(same in `import_asn.php:148`). While a ticket sits in a non-ASI group it never matches the
|
|
126
|
+
importer filter, so it stays dropped at zero cost; if DOE reassigns it back to ASI its
|
|
127
|
+
`sys_updated_on` moves, it re-imports, and the flag clears automatically. A purpose-built new
|
|
128
|
+
column would NOT be reset by either import path and would make the drop irreversible — reusing
|
|
129
|
+
`isValidDepartmentName` is the correct choice, not a compromise.
|
|
130
|
+
|
|
131
|
+
**⚠ OPEN DEFECT (backfilled by hand, CODE FIX NOT DONE — needs its own ticket).**
|
|
132
|
+
`send_ticket_updates.php` (~L375) treats an unmapped group as "someone forgot a mapping" and
|
|
133
|
+
emails devteam@togatech.com hourly saying *"Add the group to that table to let them sync."* That
|
|
134
|
+
advice is wrong. Measured cost: 15 tickets across 9 other-vendor groups (NOC Field, ATLAS Tier III
|
|
135
|
+
Support, SD Nagarro Queens, OASIS L3 Support, NGSIS/GAM Support, Central User Support, Field Ops
|
|
136
|
+
Queens, Field Ops Bronx, IET Support) re-GET every 6 minutes = **~3,600 wasted ServiceNow calls/day
|
|
137
|
+
and ~236 emails since 2026-09-18**. Those 15 INCs were set by hand in production to
|
|
138
|
+
`isValidDepartmentName = 1` with the real group in `departmentName`, and confirmed excluded.
|
|
139
|
+
**Agreed approach for the fix ticket (CTO-reviewed) — an ordered predicate:**
|
|
140
|
+
1. Group **is** in `NYCDOETicketsDepartments` → process normally (this is what keeps SD QA working).
|
|
141
|
+
2. Group is not in the table but **starts with `ASI`** → keep the throttled email; this is the only
|
|
142
|
+
case the current advice is right for.
|
|
143
|
+
3. Neither → **drop silently.**
|
|
144
|
+
Do **not** close the repair orders — write one `repair_order_notes` line and send a one-time digest
|
|
145
|
+
to DOE ops. `send_request_item_updates.php` needs the same treatment for RITMs.
|
|
146
|
+
|
|
147
|
+
**⚠ Three competing definitions of "our DOE assignment groups" — they disagree.**
|
|
148
|
+
- **(a) `Common.NYCDOETicketsDepartments`** — 14 rows: ASI Bronx, ASI Brooklyn, ASI Manhattan,
|
|
149
|
+
ASI Queens, ASI Staten Island, ASI General, ASI Calls, ASI Reconnect, ASI Research, ASI DOA,
|
|
150
|
+
ASI TM, SD QA (`isImportEnabled=0`), ASI Disposal, ASI PCS Delivery. **This one is correct.**
|
|
151
|
+
- **(b) `download_tickets_incidents.php:80`** — a hardcoded 12-group ServiceNow query list that
|
|
152
|
+
includes "ASI Enhanced", which exists nowhere else. The file is referenced by **no**
|
|
153
|
+
`schedules/*.json` → **dead code; delete it in its own commit.**
|
|
154
|
+
- **(c) `App_Api_NYCDOEV2::listIncItems()` (`library/app/api/nycdoev2.php:912`)** — the **live**
|
|
155
|
+
importer filter, a prefix match `assignment_groupSTARTSWITHASI^stateIN1,2,3,10`. This is what
|
|
156
|
+
`import_inc.php` actually uses.
|
|
157
|
+
**Risk:** a future group named "ASI Something" imports under (c) then is dropped under (a).
|
|
158
|
+
|
|
104
159
|
### Batch loop, run caps & per-ticket isolation (TRUE-82129 current shape)
|
|
105
160
|
Before this the cron ran the selection query **once per ticket under `LIMIT 1`** with a `sleep(1)` after every ticket — one ticket per 6-minute run, ~10 days for a full pass over the queue.
|
|
106
161
|
- **One `LIMIT 250` (`BATCH_SIZE`) read into a PHP array, consumed with `array_shift`, refilled only when empty.** The query is `type=ALL`, ~35,021 rows, `Using temporary; Using filesort` (it drives off `managed_service_orders`, so `Common.NYCDOETickets.idx_dtLastChecked` is unusable at this join order). A filesort already sorts every surviving row before returning the top N, **so a bigger LIMIT costs nothing extra**.
|
|
@@ -138,6 +193,19 @@ There is **no sandbox for the SNOW integration**; verification happens in prod.
|
|
|
138
193
|
2. INC2087375's `u_eta` moving from 2026-09-07 to 2026-09-17.
|
|
139
194
|
3. `SELECT COUNT(*) FROM Common.NYCDOETickets WHERE dtLastChecked IS NOT NULL` climbing.
|
|
140
195
|
|
|
196
|
+
**Answering "did the alert fire, and did the mail actually send?" — two tables, not an inbox.**
|
|
197
|
+
An inbox check alone is misleading; the DB is decisive.
|
|
198
|
+
- **Did the alert fire?** marker rows in `Core.AdvanceShippingNoticeQueue`, `dedupeKey`
|
|
199
|
+
`system|ticket_sync_patch_failure_alert` and `system|ticket_sync_invalid_group_alert`.
|
|
200
|
+
- **Did the mail send?** `Logs.Emails` — columns `dtSent`, `isSuccess`, `error`, `subject`, `to`,
|
|
201
|
+
`sourceJob`. ⚠ the table is **`Emails`**, not `Email`, and the timestamp is **`dtSent`**, not
|
|
202
|
+
`dtStamp`.
|
|
203
|
+
|
|
204
|
+
**TRUE-82027 verified healthy in production 2026-09-28:** `dtLastChecked` + `idx_dtLastChecked`
|
|
205
|
+
present; ~2,784 active tickets cycling in under an hour at `RECHECK_INTERVAL_MINUTES=6`; PATCH
|
|
206
|
+
volume ~44,000/day at 443,706 x HTTP 200 against only 6 non-200 over 10 days (3x 502, 1x 429);
|
|
207
|
+
reproduction case INC2087375 back in rotation and syncing.
|
|
208
|
+
|
|
141
209
|
### RITM cron — deliberately unchanged by TRUE-81044
|
|
142
210
|
`send_request_item_updates.php` already pushes native `state:"8"` (On Hold) and **never wrote** `u_status_task`, so TRUE-81044's ask was already satisfied there. It carries a defensive `HOLD_*` guard on the scheduled-update path so it can't push `state:"-14"` while held; its reverse-hold logic still **reads** `u_status_task == 'On Hold'`.
|
|
143
211
|
> **OPEN QUESTION:** ServiceNow's communication was scoped to **Incident** state. We did **not** extend `hold_reason` / `u_eta` / `u_other_reason_on_hold` to the RITM payload without confirmation. Whether the RITM On Hold transition carries the same mandatory-set requirement is unconfirmed — ask before changing the RITM payload.
|
|
@@ -175,7 +243,7 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
|
|
|
175
243
|
- **Query shape that works (TRUE-82332):** `endpoint LIKE '%nycd3%...'` and broad `requestPayload LIKE` scans time out (>300s) on the MCP. Use exact matches: PATCH `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/data/tasks/INCxxxx'`; GET `endpoint = 'https://supportadmin.schools.nyc/api/nycd3/v2/data/tasks?domain=diit%2Calm&filter=sys_class_name%3Dincident%5Enumber%3DINCxxxx'`. For sweeps: `FORCE INDEX (dtStamp)`, windows ≤2 days, plus `sourceJob` / `method` filters. The state:3↔state:2 oscillation in TRUE-79922 was proven from this table (INC2074899, INC2185183 showed `state:3` then `state:2` pushed seconds apart in one run).
|
|
176
244
|
- **`NYCDOETickets` (`App_Model_Common_NYCDOETicket`, `db_common` / legacy `Common`)** stores the inbound SNOW record in `rawData`, a **`FIELDTYPE_BLOBSTORAGE`** field (S3 bucket `asifiles`, key `prod/Common/NYCDOETickets/rawData/{id}`) — **NOT a queryable DB column.**
|
|
177
245
|
- **⚠ `repair_orders.dtUpdated` is skewed — never build a selection query on it.** It is written exactly **240 minutes behind** the clock used by `repair_order_notes.notesdate` and `NYCDOETickets.dtSynced`. **Do NOT compensate with `+240 MINUTE`** — 240 is almost certainly UTC-vs-Eastern and becomes 300 at the DST change (2026-11-01), so a hardcoded offset breaks twice a year. TRUE-82027 sidestepped it by removing `dtUpdated` from the query. Mechanism + the second (720-minute) skew: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
|
|
178
|
-
- **⚠ `Logs.API` history is capped at 30 days
|
|
246
|
+
- **⚠ `Logs.API` history is capped at 30 days — SHIPPED 2026-09-28 (TRUE-82049).** This investigation depended on history far older than that (9,914 PATCHes back to 2026-08-20; a hold last pushed 22 days earlier; the 2,392-call storm). **Slow-burn bugs like this can no longer be traced backwards — they must be caught by alerting.** NYCDOE stays 1.0 so its logs stay in legacy `Logs.API`. Mechanics, storage facts and the not-yet-run backlog purge: [API Request Logging & Retention](../../../1.0/apps/library/features/api-request-logging-and-retention.md).
|
|
179
247
|
- **Worker prod code cannot run on a Windows dev box.** The DB/S3 data layer fetches EC2 instance-metadata (IMDSv2) credentials and fails off-EC2. Investigate with the toga DB MCP and the `Logs.API` table instead of running the crons locally.
|
|
180
248
|
|
|
181
249
|
## Gotchas
|
|
@@ -202,6 +270,7 @@ The SNOW round-trip note reconciliation (delete-and-reinsert in `process_tickets
|
|
|
202
270
|
- `qqStatus()` collapses all recognized hold sub-statuses to base `HOLD` in `repair_orders.status`; the specific hold **sub-reason persists only in `repair_order_notes.newstatus`**. Any status badge / report / filter reading `repair_orders.status` directly cannot distinguish the six hold variants after a recompute.
|
|
203
271
|
|
|
204
272
|
## Change history
|
|
273
|
+
- 2026-09-29 — Assignment-group scoping rule + inverted `isValidDepartmentName` + the three competing group definitions; TRUE-82027 verified in prod (marker rows + `Logs.Emails` method). (mhammontree)
|
|
205
274
|
- 2026-09-25 — TRUE-82332: hold-release no longer overwrites the close `state`; close re-sent when SN not closed; stuck-ticket re-queue recipe; `Logs.API` query shapes. (mhammontree)
|
|
206
275
|
- 2026-09-18 — TRUE-82129: fixed the by-reference `fetchOne()` notice that killed every run; batch loop + run caps replaced LIMIT 1 + per-ticket sleep; alert emails de-PII'd and escaped; unmapped-group email now grouped per run. (mhammontree)
|
|
207
276
|
- 2026-09-17 — TRUE-82027: outbound push runs once per order per pass; `dtLastChecked` split from `dtSynced`; failure alerting instead of a throw. (mhammontree)
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-09-
|
|
9
|
+
updated: 2026-09-29
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/import_asn.php
|
|
@@ -62,7 +62,7 @@ The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents +
|
|
|
62
62
|
| Table | Purpose |
|
|
63
63
|
|---|---|
|
|
64
64
|
| `NYCDOETickets` | Local ticket mirror (keyed `referenceSysId`; `type` = INCIDENT / RITM) |
|
|
65
|
-
| `NYCDOETicketsDepartments` | Allowlist of
|
|
65
|
+
| `NYCDOETicketsDepartments` | Allowlist of OUR assignment groups — others are dropped and never re-checked. ⚠ Two other places define this list differently, and `NYCDOETickets.isValidDepartmentName` is **inverted** (`1` = excluded): see [Hold-Status Sync](hold-status-sync.md) → *Assignment-group scoping* |
|
|
66
66
|
| `NYCDOELocations` | Site-id → address mirror (backs ASN ship-to resolution) |
|
|
67
67
|
| `AdvanceShippingNoticeQueue` | Stage-1 landing zone; **UNIQUE index `idx_dedupeKey`** |
|
|
68
68
|
| `AdvanceShippingNotices` / `…Items` / `…Units` | ASN header (vendor+PO) → per-part items (`qtyOrder`) → individual units |
|
|
@@ -130,7 +130,7 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘ [UNIQUE ded
|
|
|
130
130
|
|
|
131
131
|
### Debugging & DB topology (DOE 1.0 sync)
|
|
132
132
|
The 1.0 worker cannot run on a dev/Windows box; use the toga DB MCP + `Logs.API` instead of running prod code locally.
|
|
133
|
-
- **⚠ `Logs.API` retention is 30 days
|
|
133
|
+
- **⚠ `Logs.API` retention is 30 days — SHIPPED 2026-09-28 (TRUE-82049)**; mechanics, storage facts and the not-yet-run backlog purge in [API Request Logging & Retention](../../../1.0/apps/library/features/api-request-logging-and-retention.md) — so it is no longer a *historical* debugging avenue, only a recent-window one. Bugs that burn slowly (a hold not re-pushed for 22 days; 9,914 PATCHes traced back to 2026-08-20) can no longer be reconstructed after the fact and must be caught by **alerting** instead. NYCDOE stays 1.0, so its logs remain in legacy `Logs.API`; Compass/Elite/WJE move their API logging to 2.0.
|
|
134
134
|
- **`repair_orders.dtUpdated` and `.lastupdateddate` are both time-skewed in `TOGaDeskSupport`** — never select or diff on them, and never compensate with a fixed offset. Detail: [Field-Service Dispatch](../../../1.0/apps/togadesk/features/field-service-dispatch.md).
|
|
135
135
|
- **`dbchanges` does not describe the real 1.0 schema** — some live columns were created outside the repo. Verify indexes/columns against the database, not the repo: [Authoring & Shipping a 1.0 dbchanges SQL File](../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
|
|
136
136
|
- **`db_core` alias → legacy (V1) environment, schema `Core`** — holds the FLAT tables `AdvanceShippingNotices` / `AdvanceShippingNoticeItems` / `AdvanceShippingNoticeUnits` (columns incl. `togadeskRepairOrderId`, `serialNumber`, `netSuiteInternalSalesOrderId`, `netSuiteInternalPurchaseOrderId`, `customerPurchaseOrder`). The 1.0 cron reads **legacy/Core**.
|
|
@@ -262,5 +262,6 @@ All **live** DOE business logic is 1.0. The only other 2.0 code is the in-progre
|
|
|
262
262
|
- Local deep-dive (full WR260236464 case study with record IDs): `doe_service_now_integration.md` in Mark's WWW workspace root; also the `service-now-integration` skill (`asn-import-flow.md`)
|
|
263
263
|
|
|
264
264
|
## Change history
|
|
265
|
+
- 2026-09-29 — Assignment-group scoping corrected (inverted `isValidDepartmentName`, three competing group lists) with detail in hold-status-sync; `Logs.API` 30-day retention shipped. (mhammontree)
|
|
265
266
|
- 2026-09-25 — TRUE-82332: cross-link to the close-clobber fix; test-repo copy of the ServiceNow credential noted (mhammontree)
|
|
266
267
|
- 2026-09-25 — Corrected "no 2.0 tenant" (TRUE-80519 2.0 ASN rewrite now exists); DOE-Open-Lenovo marked retired (mhammontree)
|
package/package.json
CHANGED