toga-ai 1.0.37 → 1.0.38
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/knowledge/INDEX.md
CHANGED
|
@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
5
5
|
## 1.0 framework
|
|
6
6
|
|
|
7
7
|
- **library** (Library) _(framework core)_ — 2 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
|
|
8
|
-
- **worker** (Worker) —
|
|
8
|
+
- **worker** (Worker) — 2 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
|
|
9
9
|
|
|
10
10
|
## 2.0 framework
|
|
11
11
|
|
|
@@ -20,4 +20,5 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
20
20
|
- **compass-canada** → [clients/compass-canada/INDEX.md](clients/compass-canada/INDEX.md)
|
|
21
21
|
- **compass-usa** → [clients/compass-usa/INDEX.md](clients/compass-usa/INDEX.md)
|
|
22
22
|
- **elite** → [clients/elite/INDEX.md](clients/elite/INDEX.md)
|
|
23
|
+
- **nycdoe** → [clients/nycdoe/INDEX.md](clients/nycdoe/INDEX.md)
|
|
23
24
|
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Client: nycdoe
|
|
2
|
+
|
|
3
|
+
| Doc | Framework | Summary | Files |
|
|
4
|
+
|-----|-----------|---------|-------|
|
|
5
|
+
| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents + RITMs) into local tables, turns vendor shipment notices into NetSuite Sales Orde | worker/crons/sync/nycdoe/import_asn.php, worker/crons/sync/nycdoe/import_inc.php, worker/crons/sync/nycdoe/legacy_import_asn.php, worker/crons/sync/nycdoe/legacy_process_asn_queue.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/1_send_asn_to_netsuite.php, worker/crons/sync/nycdoe/2_send_serials_to_netsuite.php, worker/crons/sync/nycdoe/3_create_installation_ticket.php, worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/send_request_item_updates.php, worker/crons/sync/nycdoe/send_nycdoe_proof_of_delivery.php, worker/crons/sync/nycdoe/sync_nycdoe_locations.php, worker/crons/sync/nycdoe/receive_edi_purchase_orders.php, worker/crons/sync/nycdoe/send_edi_open_invoices.php, worker/crons/notifications/nycdoe/, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.notification.json, library/app/api/nycdoe.php, library/app/api/nycdoev2.php, library/app/asnprocessor/manufacturer.php, library/app/edi.php |
|
|
6
|
+
| [NYC DOE (New York City Department of Education)](profile.md) | 1.0 | NYC DOE (New York City Department of Education) is a TOGA client whose entire integration runs in the **1.0 worker tier** (~30 cron scripts under `worker/crons/ | |
|
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: NYCDOE ServiceNow / ASN Integration
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: worker
|
|
5
|
+
project: Worker
|
|
6
|
+
client: nycdoe
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-06-10
|
|
10
|
+
owners: [mhammontree]
|
|
11
|
+
files:
|
|
12
|
+
- worker/crons/sync/nycdoe/import_asn.php
|
|
13
|
+
- worker/crons/sync/nycdoe/import_inc.php
|
|
14
|
+
- worker/crons/sync/nycdoe/legacy_import_asn.php
|
|
15
|
+
- worker/crons/sync/nycdoe/legacy_process_asn_queue.php
|
|
16
|
+
- worker/crons/sync/nycdoe/process_tickets.php
|
|
17
|
+
- worker/crons/sync/nycdoe/1_send_asn_to_netsuite.php
|
|
18
|
+
- worker/crons/sync/nycdoe/2_send_serials_to_netsuite.php
|
|
19
|
+
- worker/crons/sync/nycdoe/3_create_installation_ticket.php
|
|
20
|
+
- worker/crons/sync/nycdoe/send_ticket_updates.php
|
|
21
|
+
- worker/crons/sync/nycdoe/send_request_item_updates.php
|
|
22
|
+
- worker/crons/sync/nycdoe/send_nycdoe_proof_of_delivery.php
|
|
23
|
+
- worker/crons/sync/nycdoe/sync_nycdoe_locations.php
|
|
24
|
+
- worker/crons/sync/nycdoe/receive_edi_purchase_orders.php
|
|
25
|
+
- worker/crons/sync/nycdoe/send_edi_open_invoices.php
|
|
26
|
+
- worker/crons/notifications/nycdoe/
|
|
27
|
+
- worker/schedules/cron.worker.sync.json
|
|
28
|
+
- worker/schedules/cron.worker.notification.json
|
|
29
|
+
- library/app/api/nycdoe.php
|
|
30
|
+
- library/app/api/nycdoev2.php
|
|
31
|
+
- library/app/asnprocessor/manufacturer.php
|
|
32
|
+
- library/app/edi.php
|
|
33
|
+
related:
|
|
34
|
+
- ../profile.md
|
|
35
|
+
- ../../../1.0/apps/worker/architecture.md
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## Summary
|
|
39
|
+
|
|
40
|
+
The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents + RITMs) into
|
|
41
|
+
local tables, turns vendor shipment notices into NetSuite Sales Orders and TogaDesk
|
|
42
|
+
installation repair orders (the **ASN pipeline** — the heart of the integration), and pushes
|
|
43
|
+
ticket state / ETA / proof-of-delivery back to ServiceNow. A parallel EDI surface exchanges
|
|
44
|
+
850 POs and invoices over DOE's SFTP.
|
|
45
|
+
|
|
46
|
+
> **Schedule files are the source of truth for what runs** (`worker/schedules/
|
|
47
|
+
> cron.worker.sync.json`, `cron.worker.notification.json`). Header comments inside the
|
|
48
|
+
> scripts are frequently stale about cadence. A script on disk but absent from the schedule
|
|
49
|
+
> never runs.
|
|
50
|
+
|
|
51
|
+
## Core local tables (`db_common` unless noted)
|
|
52
|
+
|
|
53
|
+
| Table | Purpose |
|
|
54
|
+
|---|---|
|
|
55
|
+
| `NYCDOETickets` | Local ticket mirror (keyed `referenceSysId`; `type` = INCIDENT / RITM) |
|
|
56
|
+
| `NYCDOETicketsDepartments` | Allowlist of valid assignment groups — others are not processed |
|
|
57
|
+
| `NYCDOELocations` | Site-id → address mirror (backs ASN ship-to resolution) |
|
|
58
|
+
| `AdvanceShippingNoticeQueue` | Stage-1 landing zone; **UNIQUE index `idx_dedupeKey`** |
|
|
59
|
+
| `AdvanceShippingNotices` / `…Items` / `…Units` | ASN header (vendor+PO) → per-part items (`qtyOrder`) → individual units |
|
|
60
|
+
| `managed_service_orders`, `repair_orders` (`db_togadesk`) | TogaDesk-side fulfillment (one MSO per ASN; `nycDoeTicketId` links back to the RITM) |
|
|
61
|
+
|
|
62
|
+
## Integration surfaces (live schedule)
|
|
63
|
+
|
|
64
|
+
| Surface | Script | Schedule |
|
|
65
|
+
|---|---|---|
|
|
66
|
+
| Incidents in | `import_inc.php` (500/batch into `NYCDOETickets`) | every 15 min |
|
|
67
|
+
| RITMs in (ASN Stage 1a) | `import_asn.php` via `App_Api_NYCDOEV2::listRequestItems(500)` | every 15 min |
|
|
68
|
+
| Vendor SFTP in (ASN Stage 1b) | `legacy_import_asn.php` — "Will NOT be turned off" | every 5 min |
|
|
69
|
+
| Incident → TogaDesk ticket | `process_tickets.php` (client 16; throws if >200 unprocessed — flood valve) | every 15 min |
|
|
70
|
+
| Queue → ASN records (Stage 2) | `legacy_process_asn_queue.php` | every 15 min |
|
|
71
|
+
| NetSuite SO create (Stage 3, **freeze**) | `1_send_asn_to_netsuite.php` | every 2 h at :07 |
|
|
72
|
+
| NetSuite PO/serials (Stage 4) | `2_send_serials_to_netsuite.php` | every 2 h at :27 |
|
|
73
|
+
| TogaDesk repair orders (Stage 5) | `3_create_installation_ticket.php` | every 5 min |
|
|
74
|
+
| NetSuite Item Fulfillment (Stage 6) | `4_create_ns_item_fulfillment.php` (**paused**, `active: 0`); `5_DOA_…` unscheduled | not running |
|
|
75
|
+
| ETA → ServiceNow | `send_ticket_updates.php` | every 6 min |
|
|
76
|
+
| RITM state → ServiceNow | `send_request_item_updates.php` | every 5 min |
|
|
77
|
+
| Proof of delivery → ServiceNow | `send_nycdoe_proof_of_delivery.php` | every 5 min |
|
|
78
|
+
| Locations sync | `sync_nycdoe_locations.php` | daily 3:00 AM |
|
|
79
|
+
| Site-id monitor | `email_notification_siteid.php` | daily 7:01 AM |
|
|
80
|
+
| EDI 850 POs in → NetSuite SO + 997 ack | `receive_edi_purchase_orders.php` (`senderId == 'NYCDOE'` only) | 9:30 AM & 3:30 PM |
|
|
81
|
+
| EDI invoices out (NetSuite saved search 2108) | `send_edi_open_invoices.php` | every 2 h at :13 |
|
|
82
|
+
| Installation schedule reports | `notifications/nycdoe/send_installation_schedules_doe.php` (3 business days ahead, weekends skipped, **holidays NOT skipped**), daily + current-week variants, weekly part-orders | 5–8 AM daily/Mondays |
|
|
83
|
+
|
|
84
|
+
**Unscheduled / dormant:** `download_tickets*.php` (superseded by `import_inc.php` /
|
|
85
|
+
`import_asn.php`), `DOA_*` variants, `doe_install_fix.php`,
|
|
86
|
+
`send_installation_schedules.php` (base variant). Note `sync/syncro/download_tickets.php`
|
|
87
|
+
in the schedule is the **Syncro** integration, not NYCDOE. A second
|
|
88
|
+
`crons/infrastructure/import_asn.php` also exists — confirm which variant before editing.
|
|
89
|
+
|
|
90
|
+
## The ASN pipeline
|
|
91
|
+
|
|
92
|
+
```
|
|
93
|
+
ServiceNow API ──(import_asn.php, SERIALIZED ONLY)──┐
|
|
94
|
+
├─► AdvanceShippingNoticeQueue ──(legacy_process_asn_queue.php)──►
|
|
95
|
+
Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘ [UNIQUE dedupeKey]
|
|
96
|
+
|
|
97
|
+
AdvanceShippingNotices ─► Items (qtyOrder) ─► Units (1/serial; 1 per non-serial item row)
|
|
98
|
+
│ │ │
|
|
99
|
+
│ 1_send_asn_to_netsuite.php 3_create_installation_ticket.php
|
|
100
|
+
│ ▼ ▼
|
|
101
|
+
│ NetSuite Sales Order TogaDesk repair orders (1 per Unit)
|
|
102
|
+
│ ★ FREEZE qtyOrder ★
|
|
103
|
+
│ │
|
|
104
|
+
▼ 2_send_serials_to_netsuite.php
|
|
105
|
+
TogaDesk "Receiving Summary" → "#received / #ordered" (denominator = SUM(qtyOrder))
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
- **Stage 1a (API):** vendor detected by string match on `u_asset_mfg` (apple=1, acer=2,
|
|
109
|
+
lenovo=3, lexmark=4; unknown → Lexmark). Serial 'S'-prefix stripped for Lexmark/Lenovo.
|
|
110
|
+
Ship-to resolved via `App_Api_NYCDOEV2::resolveShippingAddress()`; on failure a tracking
|
|
111
|
+
`NYCDOETicket` is recorded but **ASN creation is skipped**.
|
|
112
|
+
`dedupeKey = vendorId|PO|DOE-part|serialNumber`. Dedup is SELECT-before-INSERT with
|
|
113
|
+
per-row try/catch (unique index is the race backstop).
|
|
114
|
+
- **Stage 1b (SFTP):** the only source of **non-serialized** items (in practice the Lexmark
|
|
115
|
+
feed). Serial-less lines of the same PO+part are summed per file into one queue row with
|
|
116
|
+
`dedupeKey = vendorId|PO|DOE-part|<file token>`. Dedup is INSERT + catch "Duplicate entry"
|
|
117
|
+
(race-safe).
|
|
118
|
+
- **Stage 2:** queue → ASN/Items/Units/TrackingNumbers in a transaction. Serialized: one
|
|
119
|
+
Unit per serial. Non-serialized: the file's quantity is a **cumulative** total —
|
|
120
|
+
`unsent item qtyOrder = cumulative − SUM(committed qtyOrder)`; post-freeze growth becomes
|
|
121
|
+
a **fresh item row**, a decrease fires a discrepancy email and changes nothing. Exactly
|
|
122
|
+
one Unit per serial-less item row regardless of quantity. Failed rows retry 5× then alert
|
|
123
|
+
`devteam@goagilant.com`.
|
|
124
|
+
- **Stage 3 (THE FREEZE):** creates the NetSuite SO and stamps
|
|
125
|
+
`AdvanceShippingNoticeItems.netSuiteInternalSalesOrderId`. **Once stamped, that item's
|
|
126
|
+
`qtyOrder` is immutable** — there is no update path to a sent SO line; growth must be a
|
|
127
|
+
new item row → new SO line.
|
|
128
|
+
- **Stage 4:** reconciles the NetSuite **PO** (`netSuiteInternalPurchaseOrderId` — not the
|
|
129
|
+
SO id) and sends serial inventory assignments.
|
|
130
|
+
- **Stage 5:** for units `WHERE togadeskRepairOrderId IS NULL`, reads serials from NetSuite
|
|
131
|
+
item receipts (so TogaDesk RO serials are **driven by NetSuite**, not the ASN unit),
|
|
132
|
+
creates/reuses TogaDesk manufacturers/models/assets/serial-numbers/MSO and creates **a
|
|
133
|
+
new repair_orders row unconditionally per loop iteration** — one RO per Unit (a 50-cable
|
|
134
|
+
line = 1 unit = 1 RO). Existing-serial lookup is scoped **per item row** (`$asnItemId`).
|
|
135
|
+
- **"#/N received" UI** (`togadesk/desk/template/pages/managedserviceorders/view.php`):
|
|
136
|
+
denominator = `SUM(qtyOrder)` per part (committed + unsent); numerator = Units with
|
|
137
|
+
`togadeskRepairOrderId IS NOT NULL`.
|
|
138
|
+
|
|
139
|
+
## Critical business rules (SME-confirmed — re-confirm, don't infer)
|
|
140
|
+
|
|
141
|
+
1. **An ASN file is a CUMULATIVE restatement** of a PO+part total, not a delta (50 grows to
|
|
142
|
+
75; the file says "75").
|
|
143
|
+
2. **Match non-serialized items on customer PO + part number** — tracking/waybill numbers
|
|
144
|
+
are NOT reliable.
|
|
145
|
+
3. **ServiceNow only delivers serialized items.** Non-serialized comes only via vendor SFTP
|
|
146
|
+
— this is why both ingestion paths exist.
|
|
147
|
+
4. **The NetSuite freeze:** `netSuiteInternalSalesOrderId IS NOT NULL` ⇒ `qtyOrder` locked.
|
|
148
|
+
Growth = new item row; decrease = discrepancy alert, never a reduction.
|
|
149
|
+
5. **One unit per non-serialized item row** — a 50-cable line is 1 unit and 1 repair order.
|
|
150
|
+
6. **The `-RPL` PO suffix** forces a brand-new ASN/MSO/RO chain for replacements.
|
|
151
|
+
|
|
152
|
+
## Gotchas / known issues
|
|
153
|
+
|
|
154
|
+
- **The queue `dedupeKey` format is load-bearing.** A 2026 incident (WR260236464 duplicate
|
|
155
|
+
repair order) was caused by dedupeKey format drift (4-part keys with serial vs 3-part
|
|
156
|
+
without) letting old rows re-import; Stage 2 then created a sibling item row (frozen
|
|
157
|
+
original not reusable), and Stage 5's per-item-row serial scoping duplicated the unit →
|
|
158
|
+
duplicate RO. Any change to the key format needs a backfill or transitional matcher.
|
|
159
|
+
Forensic fingerprint: a Unit with `trackingNumberId IS NULL` was added by Stage 5 from
|
|
160
|
+
NetSuite, not by the importer.
|
|
161
|
+
- **Two dedupe strategies coexist:** SFTP path = INSERT+catch (race-safe); API path =
|
|
162
|
+
check-then-insert (unique index backstop).
|
|
163
|
+
- **Timezone traps:** worker runs `America/Chicago`; `repair_orders.dtEta` is stored
|
|
164
|
+
**Eastern**; ServiceNow `u_eta` is sent as **UTC** `YYYY-MM-DD HH:MM:SS` (converted via
|
|
165
|
+
`App_Date::convertTimezone()`, only sent when first 19 chars differ from remote).
|
|
166
|
+
- **EDI PO104 leading-zero suppression** (go-live 2026-06-07): DOE's IBM-based translator
|
|
167
|
+
sends `.94` instead of `0.94` — exposure is in `App_Edi::translatePurchaseOrder850()`,
|
|
168
|
+
not the cron.
|
|
169
|
+
- **Installation schedule reports skip weekends but not holidays** — empty output near a
|
|
170
|
+
holiday is expected, not a bug. Empty also just means no ROs with `dtEta` on the exact
|
|
171
|
+
target date.
|
|
172
|
+
- **Stage cadences differ** (5 min / 15 min / 6 min / 2 h) — "missing" data downstream is
|
|
173
|
+
often just a stage that hasn't ticked yet.
|
|
174
|
+
- **Stage 6 is not currently running** (`4_` paused with `active: 0`, `5_` unscheduled).
|
|
175
|
+
- TRUE-77354: Stage 5 lookups changed `LIKE` → `=` to stop `_`/`%` wildcard matches in
|
|
176
|
+
serials.
|
|
177
|
+
|
|
178
|
+
## Operating rules when changing this integration
|
|
179
|
+
|
|
180
|
+
1. **Trace the full pipeline downstream before fixing** — the constraint that makes an edit
|
|
181
|
+
unsafe usually lives downstream (the freeze point and the consumer/display query).
|
|
182
|
+
2. **Ask the SME for business-data semantics — never guess** (cumulative vs delta, identity
|
|
183
|
+
keys, what counts as one unit, whether totals can decrease).
|
|
184
|
+
3. All ServiceNow calls go through `App_Api_NYCDOEV2` public static methods.
|
|
185
|
+
4. Never mutate a frozen item's quantity; carry deltas forward as new rows.
|
|
186
|
+
5. With no staging environment, verify with a standalone arithmetic simulation plus a read
|
|
187
|
+
of the consumer query; `php -l` every touched file.
|
|
188
|
+
|
|
189
|
+
## Related docs
|
|
190
|
+
|
|
191
|
+
- [NYC DOE client profile](../profile.md)
|
|
192
|
+
- [Worker (1.0) architecture](../../../1.0/apps/worker/architecture.md) — cron dispatch,
|
|
193
|
+
role election, schedule files
|
|
194
|
+
- Local deep-dive (full WR260236464 case study with record IDs):
|
|
195
|
+
`doe_service_now_integration.md` in Mark's WWW workspace root; also the
|
|
196
|
+
`service-now-integration` skill (`asn-import-flow.md`)
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: NYC DOE (New York City Department of Education)
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
project: Worker
|
|
5
|
+
client: nycdoe
|
|
6
|
+
type: profile
|
|
7
|
+
status: active
|
|
8
|
+
updated: 2026-06-10
|
|
9
|
+
owners: [mhammontree]
|
|
10
|
+
files: []
|
|
11
|
+
related:
|
|
12
|
+
- features/servicenow-integration.md
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Summary
|
|
16
|
+
NYC DOE (New York City Department of Education) is a TOGA client whose entire integration
|
|
17
|
+
runs in the **1.0 worker tier** (~30 cron scripts under `worker/crons/sync/nycdoe/` and
|
|
18
|
+
`worker/crons/notifications/nycdoe/`, on the `sync` and `notification` worker roles).
|
|
19
|
+
ServiceNow is DOE's system of record for tickets (Incidents and RITMs); TOGA mirrors those
|
|
20
|
+
locally, drives fulfillment through TogaDesk and NetSuite, and pushes status / ETA /
|
|
21
|
+
proof-of-delivery back to ServiceNow. A parallel EDI relationship (850 POs in, invoices out)
|
|
22
|
+
runs over DOE's SFTP.
|
|
23
|
+
|
|
24
|
+
## Platforms & data
|
|
25
|
+
- **1.0 worker tier only** — no 2.0 footprint. Local mirror tables live in `db_common`
|
|
26
|
+
(`NYCDOETickets`, `NYCDOELocations`, `AdvanceShippingNotice*`).
|
|
27
|
+
- **TogaDesk:** DOE is client id **16** (`db_togadesk`) — repair orders, managed service
|
|
28
|
+
orders, and the "#received / #ordered" Receiving Summary UI.
|
|
29
|
+
- **NetSuite:** Sales Orders, PO reconciliation, item receipts, Item Fulfillments, invoices
|
|
30
|
+
(SuiteTalk toolkit in `library`).
|
|
31
|
+
|
|
32
|
+
## Endpoints
|
|
33
|
+
| System | Endpoint | Notes |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| ServiceNow prod | `https://nycdoeprod.service-now.com` | Incidents + RITMs |
|
|
36
|
+
| ServiceNow stage | `https://nycdoedev2.service-now.com` | used when `App_Registry::inTestMode()` |
|
|
37
|
+
| ServiceNow disposals | `https://supportadmin.schools.nyc` (stage `nycdoedev3`) | disposal service requests |
|
|
38
|
+
| DOE EDI SFTP | `transfer.schools.nyc` (SSH key `doedimedu.pem`) | EDI 850 in, invoices out |
|
|
39
|
+
| Vendor SFTP feeds | Apple / Lexmark / Lenovo (Lenovo: `ftp.asisystem.com`) | legacy ASN file ingestion |
|
|
40
|
+
|
|
41
|
+
## API clients
|
|
42
|
+
- `App_Api_NYCDOE` (`library/app/api/nycdoe.php`) — V1 client (legacy ticket downloaders).
|
|
43
|
+
- `App_Api_NYCDOEV2` (`library/app/api/nycdoev2.php`) — centralized V2 client. **All
|
|
44
|
+
ServiceNow calls must go through its public static methods** — never call endpoints
|
|
45
|
+
directly from a cron.
|
|
46
|
+
|
|
47
|
+
## Key features (this client)
|
|
48
|
+
- [ServiceNow / ASN Integration](features/servicenow-integration.md) — the full integration
|
|
49
|
+
map: ticket sync, the ASN pipeline (the core flow), outbound status sync, EDI, and the
|
|
50
|
+
SME-confirmed business rules.
|
package/package.json
CHANGED