toga-ai 1.0.775 → 1.0.777
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/2.0/apps/saml/architecture.md +8 -2
- package/knowledge/2.0/apps/saml/workflows/rotating-the-saml-sp-certificate.md +67 -10
- package/knowledge/INDEX.md +1 -1
- package/knowledge/clients/nycdoe/INDEX.md +1 -0
- package/knowledge/clients/nycdoe/features/edi-850-receive-and-997-ack.md +108 -0
- package/knowledge/clients/nycdoe/features/servicenow-integration.md +5 -2
- package/knowledge/clients/nycdoe/profile.md +2 -1
- package/package.json +1 -1
|
@@ -6,7 +6,7 @@ project: SAML SSO Gateway
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-04
|
|
10
10
|
owners: ["rgirish", "jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- saml/index.php
|
|
@@ -49,7 +49,7 @@ Legacy `/?meta`, `/?acs`, `/?sls` query-style paths are rewritten automatically.
|
|
|
49
49
|
2. **Version + time check** — RelayState JSON: `v` (currently `1`), `time`, `domain` UUID, optional `urlParameters`. Validated against `MAX_TIME_TO_LIVE` (300s).
|
|
50
50
|
3. **Parse SAMLResponse** — base64-decoded, deserialized via LightSaml.
|
|
51
51
|
4. **Status check** — proceeds only on `STATUS_SUCCESS`.
|
|
52
|
-
5. **Assertion decryption
|
|
52
|
+
5. **Assertion decryption (dual-credential during cert overlap).** `/acs` builds two `X509Credential`s from the SP encryption keys in `_underscore/Assets/ssl/` (new + old) and calls `$reader->decryptMultiAssertion([$credentialNew, $credentialOld], $ctx)`. LightSaml's `decryptMulti()` tries each credential and only re-throws once all fail (fails closed), so an assertion encrypted to either the new or old SP key decrypts during a rotation overlap. See the SP cert-rotation workflow.
|
|
53
53
|
6. **Resolve client + environment** — `_Model_Core_Domain` by RelayState `domain` UUID → `_Model_Core_Client` + `_Model_Core_Environment`. Dev on real AWS instance downgrades to beta.
|
|
54
54
|
7. **Dynamic DB registration** — SQL joins `Clients`, `Databases`, `DatabaseHosts`, `Environments` for the environment slug. Calls `_Database::register()` + `connect()` for `DB_CLIENT`, `DB_CLIENT_LOGS`, `DB_CLIENT_ARCHIVE`.
|
|
55
55
|
8. **Identity mapping** — `_Model_<ClientIdentifier>_ClientAuthentication::getAuthenticatedSsoUser($assertion)`. Wrapped in try/catch; exceptions sent to Sentry.
|
|
@@ -59,6 +59,10 @@ Legacy `/?meta`, `/?acs`, `/?sls` query-style paths are rewritten automatically.
|
|
|
59
59
|
### Failure responses
|
|
60
60
|
`Invalid SAMLResponse (Code = 1–4)`, `SAML Authentication Failed`, `User Authentication Failed`.
|
|
61
61
|
|
|
62
|
+
## /meta SP credential (signing vs encryption)
|
|
63
|
+
|
|
64
|
+
`/meta` publishes TWO independent `KeyDescriptor`s: `use="signing"` and `use="encryption"`. They can carry **different** certs, which lets the SP encryption cert rotate without touching the signing cert. The private key that signs our **outbound** AuthnRequests lives in `_underscore` (`_Model_Core_ClientAuthentication::PATH_TO_PRIVATE_KEY`) and is **global — one key for all client IdPs**, so the signing cert cannot flip until every IdP has loaded the new metadata. Cert paths are set via class constants in `saml/Controller/Index.php` (`PATH_TO_SIGNING_CERTIFICATE`, `PATH_TO_ENCRYPTION_CERTIFICATE_NEW/OLD` + matching private-key constants — paths only, no key material). Full procedure: SP cert-rotation workflow.
|
|
65
|
+
|
|
62
66
|
## Per-client SSO extension pattern
|
|
63
67
|
|
|
64
68
|
Identity mapping lives in `_underscore/Model/<ClientIdentifier>/ClientAuthentication.php`.
|
|
@@ -90,6 +94,7 @@ To onboard a new SSO client, add the class in `_underscore` — not in this repo
|
|
|
90
94
|
- **`_Database::register()` auto-starts a lazy transaction (since Apr 2 2026).** Any code that registers DBs then writes must call `_Database::transactionCommit()` before exiting — otherwise MySQL silently rolls back all writes on connection close. The `/acs` handler calls `transactionCommit()` after `getAuthenticatedSsoUser()` succeeds and before the redirect. See `_underscore/Database.php:48`.
|
|
91
95
|
- **SAML signature verification is commented out — this is the repo's top security priority.** The IdP X509 cert verification block in the `/acs` flow is disabled, so the gateway accepts any well-formed, status-success `SAMLResponse` without proving it was signed by the expected IdP. The only remaining gate is the encrypted RelayState, which authenticates the *originating TOGa request* — not the *asserting IdP*. Threat model: an attacker who can craft or replay a `SAMLResponse` (or a malicious/compromised IdP) can forge an identity assertion for any user, since nothing binds the assertion to a trusted signing key. Re-enabling per-IdP X509 signature verification is the single highest-impact security fix for this repo.
|
|
92
96
|
- **Plaintext secrets in source — move to AWS SSM Parameter Store.** `saml/Config/production.ini` stores credentials in plaintext, and a Sentry DSN is hardcoded in `saml/Controller/Index.php`. Both are checked into the repo. Migrate these to AWS SSM Parameter Store (SecureString) and load at runtime; the source tree should reference parameter names only, never literal credential values.
|
|
97
|
+
- **`/meta` cert lines must be `trim()`-ed (CRLF).** The `.crt` files in `_underscore/Assets/ssl` are CRLF-encoded; the cert-format code does `explode("\n", ...)` and indents each line, so without a per-line `trim()` a stray `\r` ends up on every base64 line inside `<ds:X509Certificate>`. Trim each line in both the signing and encryption blocks.
|
|
93
98
|
- **SLS not implemented.** `/sls` falls through to the default banner.
|
|
94
99
|
- **`set_exception_handler(null)` at top of `saml()`.** Unhandled exceptions output raw PHP errors. All exceptions from `getAuthenticatedSsoUser()` are caught and sent to Sentry.
|
|
95
100
|
|
|
@@ -97,3 +102,4 @@ To onboard a new SSO client, add the class in `_underscore` — not in this repo
|
|
|
97
102
|
- 2026-06-11 — Added `transactionCommit()` before redirect; wrapped `getAuthenticatedSsoUser()` in try/catch with Sentry; echo+exit on auth failure (rgirish)
|
|
98
103
|
- 2026-06-25 — Sharpened signature-verification gotcha with explicit threat model and flagged it as the repo's top security priority; documented plaintext secrets in `production.ini` + hardcoded Sentry DSN in `Controller/Index.php` with SSM Parameter Store recommendation (jcardinal)
|
|
99
104
|
- 2026-08-27 — Documented non-prod shared-ALB self-registration postdeploy hook (`060_...sh` + `ebs/register_instance_to_shared_application_load_balancer.php`); skips `production`, requires `aws/aws-sdk-php` + two IAM actions; cross-linked worker2 ALB auto-registration feature (jcardinal)
|
|
105
|
+
- 2026-09-04 — `/acs` now dual-credential decrypt (`decryptMultiAssertion([new, old])`) for SP encryption-cert overlap; documented `/meta`'s two KeyDescriptors (signing vs encryption) and the global outbound-signing key; added the CRLF `/meta` cert-line `trim()` gotcha. (jcardinal)
|
|
@@ -6,7 +6,7 @@ project: SAML SSO Gateway
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: workflow
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-04
|
|
10
10
|
owners: ["jcardinal"]
|
|
11
11
|
files:
|
|
12
12
|
- _underscore/Assets/ssl/togahub_com.crt
|
|
@@ -32,16 +32,58 @@ cert), see the SSL certificate trust-model strategy standard.
|
|
|
32
32
|
|
|
33
33
|
## How the SP credential is used
|
|
34
34
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
35
|
+
The SP cert/key is used in **four** places — three in `saml`, one in `_underscore`:
|
|
36
|
+
|
|
37
|
+
- **(a) `/meta` signing `KeyDescriptor`** — `saml/Controller/Index.php` publishes the cert IdPs
|
|
38
|
+
use to verify our outbound AuthnRequest signature. Read fresh via `file_get_contents` on each
|
|
39
|
+
request (no server-side cache).
|
|
40
|
+
- **(b) `/meta` encryption `KeyDescriptor`** — the cert IdPs use to encrypt assertions to us.
|
|
41
|
+
- **(c) `/acs` assertion decryption** — `saml/Controller/Index.php` decrypts the inbound
|
|
42
|
+
encrypted assertion with the matching private key.
|
|
43
|
+
- **(d) Outbound AuthnRequest signing (GLOBAL)** — `_Model_Core_ClientAuthentication`
|
|
44
|
+
(`_underscore/Model/Core/ClientAuthentication.php`, `PATH_TO_PRIVATE_KEY`, around lines
|
|
45
|
+
11 / 95 / 107) signs the redirect-binding AuthnRequest query string with XMLSecLibs
|
|
46
|
+
`RSA_SHA256`. **One key signs the login requests we send to ALL client IdPs — you cannot sign
|
|
47
|
+
per-client.** This is why signing and encryption rotate on different schedules (see
|
|
48
|
+
*Encryption-only rotation*).
|
|
49
|
+
|
|
42
50
|
- **IdPs cache metadata up to 7 days** (`cacheDuration PT604800S`), so a cert change is not
|
|
43
51
|
guaranteed to be picked up by every IdP immediately after deploy.
|
|
44
52
|
|
|
53
|
+
## Encryption-only rotation with an overlap window (2026-09-04)
|
|
54
|
+
|
|
55
|
+
Because a cert change is not picked up by every IdP at once, the safe rotation splits the two
|
|
56
|
+
uses and **rotates encryption first, signing later**:
|
|
57
|
+
|
|
58
|
+
- **Signing stays on the OLD cert.** If `/meta` advertised the new signing cert (or (d) signed
|
|
59
|
+
with the new key) before every IdP loaded the new metadata, any IdP that verifies our
|
|
60
|
+
AuthnRequest signature could **reject our login requests**. Signing flips only after all IdPs
|
|
61
|
+
are on the new metadata — a later, coordinated step.
|
|
62
|
+
- **Encryption rotates now, with both keys accepted.** `/meta` presents TWO independent
|
|
63
|
+
`KeyDescriptor`s: `use="signing"` = current/old cert, `use="encryption"` = new cert. Two
|
|
64
|
+
independent KeyDescriptors is valid SAML metadata, so encryption can rotate without touching
|
|
65
|
+
signing.
|
|
66
|
+
- **`/acs` accepts EITHER encryption key during overlap.** It builds two `X509Credential`s
|
|
67
|
+
(new + old) and calls `$reader->decryptMultiAssertion([$credentialNew, $credentialOld], $ctx)`.
|
|
68
|
+
In vendored LightSaml, `decryptMulti()` tries each credential and only re-throws once all fail
|
|
69
|
+
— so it fails closed. This lets an IdP that has re-encrypted to the new cert and one still on
|
|
70
|
+
the old cert both succeed.
|
|
71
|
+
- **Class constants (paths only, no key material)** in `saml/Controller/Index.php` replaced the
|
|
72
|
+
old single pair: `PATH_TO_SIGNING_CERTIFICATE`; `PATH_TO_ENCRYPTION_CERTIFICATE_NEW` +
|
|
73
|
+
`PATH_TO_ENCRYPTION_PRIVATE_KEY_NEW`; `PATH_TO_ENCRYPTION_CERTIFICATE_OLD` +
|
|
74
|
+
`PATH_TO_ENCRYPTION_PRIVATE_KEY_OLD`.
|
|
75
|
+
- **What forced it:** old cert `togahub_com.crt` (O=Agilant Solutions) expires **2026-09-16**;
|
|
76
|
+
new cert `togahub_com_2036-08-27.crt` (O=TOGA Technology, Inc.) is valid to 2036.
|
|
77
|
+
|
|
78
|
+
### Remaining steps / follow-ups (not yet done)
|
|
79
|
+
1. **Flip signing to the new cert** once every IdP has loaded the new metadata — advertise the
|
|
80
|
+
new signing cert in `/meta` and switch (d) to the new key. Coordinate the date.
|
|
81
|
+
2. **Log which SP key decrypted** (new vs old, identifier only — never assertion data) so you can
|
|
82
|
+
see which IdPs have cut over and when it is safe to drop the old key.
|
|
83
|
+
3. **Wrap the `/acs` decrypt in try/catch** and echo `Invalid SAMLResponse (Code = 5)` to match
|
|
84
|
+
the existing coded failure responses.
|
|
85
|
+
4. **Dedupe the two `/meta` cert-format blocks** into a shared helper (non-blocking).
|
|
86
|
+
|
|
45
87
|
## Current credential (as of 2026-08-27, staged — cutover pending)
|
|
46
88
|
|
|
47
89
|
A dedicated **self-signed** replacement was generated to decouple the SP credential from the
|
|
@@ -88,6 +130,14 @@ short-lived DigiCert/Thawte TLS wildcard that had been reused as the SP cert (se
|
|
|
88
130
|
6. **Deploy the cert+key to the other bilateral services** that use the same SP credential —
|
|
89
131
|
AS2 (AWS Transfer Family) and any other pinned endpoint — importing both cert and key.
|
|
90
132
|
|
|
133
|
+
### Gotcha — trim each cert line (CRLF)
|
|
134
|
+
|
|
135
|
+
The `.crt` files in `_underscore/Assets/ssl` are **CRLF-encoded**. The `/meta` cert-format code
|
|
136
|
+
does `explode("\n", ...)` then indents each line. Without a per-line `trim()`, a stray carriage
|
|
137
|
+
return (`\r`) is embedded on every base64 line inside `<ds:X509Certificate>` in the metadata XML.
|
|
138
|
+
Fix: `trim()` each line in the cert-format code — applied to both the signing and encryption
|
|
139
|
+
blocks. (Latent bug that pre-dated this change in the single original block.)
|
|
140
|
+
|
|
91
141
|
## Background — why the change
|
|
92
142
|
|
|
93
143
|
The old `_underscore/Assets/ssl/togahub_com.crt` was a **reused DigiCert/Thawte multi-domain
|
|
@@ -105,5 +155,12 @@ every pinning partner to reinstall on each renewal. The dedicated self-signed 10
|
|
|
105
155
|
`_underscore`-then-`saml` cutover ordering. Discovered the prior cert was a reused short-lived
|
|
106
156
|
DigiCert wildcard TLS cert, which is the reason for the switch. Cutover itself still pending.
|
|
107
157
|
(jcardinal)
|
|
108
|
-
|
|
109
|
-
|
|
158
|
+
- 2026-09-04 — Documented the **encryption-only rotation** actually built in
|
|
159
|
+
`saml/Controller/Index.php`: `/meta` now presents two `KeyDescriptor`s (signing=old,
|
|
160
|
+
encryption=new) and `/acs` uses `decryptMultiAssertion([new, old])` so both encryption keys
|
|
161
|
+
work during overlap; new path-only class constants replaced the single cert/key pair.
|
|
162
|
+
Recorded that outbound AuthnRequest signing uses one GLOBAL key for all IdPs, which is why
|
|
163
|
+
signing stays on the old cert until every IdP loads the new metadata. Added the CRLF
|
|
164
|
+
per-line-`trim()` gotcha for `/meta` cert formatting, and the open follow-ups. Change is
|
|
165
|
+
working-tree only (deploys via `saml` on `_production`); Rate is testing against
|
|
166
|
+
`saml-production`. (jcardinal)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -5,7 +5,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
5
5
|
## 1.0 framework
|
|
6
6
|
|
|
7
7
|
- **library** (Library) _(framework core)_ — 24 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
|
|
8
|
-
- **worker** (Worker) —
|
|
8
|
+
- **worker** (Worker) — 34 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
|
|
9
9
|
- **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
|
|
10
10
|
- **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
|
|
11
11
|
- **togadesk** (TOGa Desk) — 13 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Framework | Summary | Files |
|
|
4
4
|
|-----|-----------|---------|-------|
|
|
5
|
+
| [NYCDOE EDI 850 Receive & 997 Acknowledgement](features/edi-850-receive-and-997-ack.md) | 1.0 | DOE sends purchase orders as **EDI 850** files over their SFTP; TOGA turns each one into a NetSuite Sales Order and sends back an **EDI 997** functional acknowl | worker/crons/sync/nycdoe/receive_edi_purchase_orders.php, worker/schedules/cron.worker.sync.json, library/app/edi.php |
|
|
5
6
|
| [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | DOE ticket **hold** status must round-trip between ServiceNow (SNOW) and TOGaDesk and **stay held** — holds are SLA-bearing in both systems. | worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/send_request_item_updates.php, library/app/model/togadesk/repairorder.php, library/app/api/nycdoev2.php, togadesk/desk/includes/classes/class.repair.php |
|
|
6
7
|
| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents + RITMs) into local tables, turns vendor shipment notices into NetSuite Sales Orde | worker/crons/sync/nycdoe/import_asn.php, worker/crons/sync/nycdoe/import_inc.php, worker/crons/sync/nycdoe/legacy_import_asn.php, worker/crons/sync/nycdoe/legacy_process_asn_queue.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/1_send_asn_to_netsuite.php, worker/crons/sync/nycdoe/2_send_serials_to_netsuite.php, worker/crons/sync/nycdoe/3_create_installation_ticket.php, worker/crons/sync/nycdoe/test_multi_po_receipt_resolution.php, worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/send_request_item_updates.php, worker/crons/sync/nycdoe/send_nycdoe_proof_of_delivery.php, worker/crons/sync/nycdoe/sync_nycdoe_locations.php, worker/crons/sync/nycdoe/receive_edi_purchase_orders.php, worker/crons/sync/nycdoe/send_edi_open_invoices.php, worker/crons/notifications/nycdoe/, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.notification.json, library/app/api/nycdoe.php, library/app/api/nycdoev2.php, library/app/nycdoe.php, library/app/asnprocessor/manufacturer.php, library/app/asnprocessor/apple.php, library/app/asnprocessor/lenovo.php, library/app/asnprocessor/lexmark.php, library/app/asnprocessor/acer.php, library/app/edi.php, library/app/netsuite.php, dbchanges/Core/SK/, dbchanges/TOGaDeskSupport/SK/, worker2/Worker/Monitor/Nycdoe.php |
|
|
7
8
|
| [New York City Department of Education](profile.md) | 1.0 | NYC DOE (New York City Department of Education) is a TOGA client whose entire integration runs in the **1.0 worker tier** (~30 cron scripts under `worker/crons/ | |
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: NYCDOE EDI 850 Receive & 997 Acknowledgement
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: worker
|
|
5
|
+
project: Worker
|
|
6
|
+
client: nycdoe
|
|
7
|
+
type: client-feature
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-09-04
|
|
10
|
+
owners: [mhammontree]
|
|
11
|
+
files:
|
|
12
|
+
- worker/crons/sync/nycdoe/receive_edi_purchase_orders.php
|
|
13
|
+
- worker/schedules/cron.worker.sync.json
|
|
14
|
+
- library/app/edi.php
|
|
15
|
+
related:
|
|
16
|
+
- servicenow-integration.md
|
|
17
|
+
- ../profile.md
|
|
18
|
+
- ../../../1.0/apps/worker/architecture.md
|
|
19
|
+
- ../../../1.0/apps/library/architecture.md
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## Summary
|
|
23
|
+
|
|
24
|
+
DOE sends purchase orders as **EDI 850** files over their SFTP; TOGA turns each one into a
|
|
25
|
+
NetSuite Sales Order and sends back an **EDI 997** functional acknowledgement. There is
|
|
26
|
+
**no separate 997 cron** — the 997 is built and uploaded inline at the end of the 850
|
|
27
|
+
receiver, so anything that aborts the receiver also means DOE never gets an ack.
|
|
28
|
+
|
|
29
|
+
This doc owns the 850-in / 997-out surface. The broader DOE ticket + ASN pipeline lives in
|
|
30
|
+
[servicenow-integration.md](servicenow-integration.md).
|
|
31
|
+
|
|
32
|
+
## How it works
|
|
33
|
+
|
|
34
|
+
1. **Cron:** `worker/crons/sync/nycdoe/receive_edi_purchase_orders.php`, schedule
|
|
35
|
+
`30 9,15 * * *` (9:30 AM and 3:30 PM) in `worker/schedules/cron.worker.sync.json`.
|
|
36
|
+
2. Pulls every file from `/outbound` on DOE's SFTP `transfer.schools.nyc` (SSH key file
|
|
37
|
+
`library/ssl/doe/doedimedu.pem`; username and key live in the
|
|
38
|
+
[DOE profile](../profile.md) / on disk — never in this doc).
|
|
39
|
+
3. `App_Edi::translatePurchaseOrder850()` parses each file. Only files whose
|
|
40
|
+
`senderId == 'NYCDOE'` are processed.
|
|
41
|
+
4. Creates the NetSuite Sales Order (DOE customer internal id **21797**) and attaches the
|
|
42
|
+
raw 850 file.
|
|
43
|
+
5. **Then** builds the 997 via
|
|
44
|
+
`App_Edi::generateAcknowledgements997FromPurchaseOrders850()` (`library/app/edi.php`,
|
|
45
|
+
~line 348) and `put`s it to `/inbound` as `997_<original 850 filename>`
|
|
46
|
+
(cron ~lines 338-350).
|
|
47
|
+
6. Logs **both** the 850 and the 997 to `App_Model_FileLog` (`db_logs`, table
|
|
48
|
+
`Logs.FileLog`, `legacy` environment) under job `EDI_NYCDOE_PO`, then deletes the 850
|
|
49
|
+
from DOE's SFTP.
|
|
50
|
+
|
|
51
|
+
### 997 segment structure
|
|
52
|
+
|
|
53
|
+
`ISA` / `GS(FA)`, then per PO: `ST(997)`, `AK1(PO, groupStartIdentifier)`,
|
|
54
|
+
`AK2(850, transactionSetIdentifier)`, `AK5(A)`, `AK9(A,1,1,1)`, `SE(6)`; then `GE` / `IEA`.
|
|
55
|
+
Transaction set control number starts at **1001** and increments per PO. Default
|
|
56
|
+
separators: `~` element, `\n` segment.
|
|
57
|
+
|
|
58
|
+
Field mapping (verified against 850 FileLog id 880810):
|
|
59
|
+
|
|
60
|
+
| 997 segment | Source in the 850 |
|
|
61
|
+
|---|---|
|
|
62
|
+
| `AK1~PO~2` | the 850's `GS~PO~NYCDOE~2127360111~…~2~X~004010` |
|
|
63
|
+
| `AK2~850~004204317` | the 850's `ST~850~004204317` |
|
|
64
|
+
| (PO number, elsewhere) | `BEG~00~KN~WR270037765` |
|
|
65
|
+
|
|
66
|
+
## Gotchas
|
|
67
|
+
|
|
68
|
+
- **The cron header comment is STALE.** It says "INTENDED TO BE RUN EVERY HOUR". It runs
|
|
69
|
+
twice a day. `worker/schedules/cron.worker.sync.json` is the source of truth.
|
|
70
|
+
- **The 997 generator is an "always accept" stub — still OPEN.**
|
|
71
|
+
`library/app/edi.php` ~lines 393-412 hardcode `AK5 = 'A'` and `AK9 = 'A','1','1','1'`, so
|
|
72
|
+
every 997 reports "accepted, 1 transaction set" no matter what the 850 contained. It can
|
|
73
|
+
never report a reject or a real transaction count. Out of scope for TRUE-81757; needs its
|
|
74
|
+
own ticket if DOE ever audits the acks beyond `AK2`.
|
|
75
|
+
- **An exception in the NetSuite loop skips the 997 entirely** and leaves the 850 on SFTP,
|
|
76
|
+
so the file re-processes next run and DOE gets no ack at all. That fail-loud abort is
|
|
77
|
+
correct for 1.0 sync/cron code, but it is a distinct cause from a *malformed* 997 —
|
|
78
|
+
check which one you have before touching the generator.
|
|
79
|
+
- **`generateAcknowledgements997FromPurchaseOrders850()` has exactly one production
|
|
80
|
+
caller** (this DOE cron). The other two callers are dev scratch scripts under
|
|
81
|
+
`test/@jeff/edi/` and `test/@pinal/netSuites/`, so a change to it is DOE-only in effect.
|
|
82
|
+
- **`sync/prudential/receive_edi_purchase_orders.php` does NOT use this 997 function.** Do
|
|
83
|
+
not assume the two EDI receivers share the ack path.
|
|
84
|
+
- **EDI PO104 leading-zero suppression** — DOE's IBM translator sends `.94` not `0.94`;
|
|
85
|
+
exposure is in `App_Edi::translatePurchaseOrder850()`, not the cron. See
|
|
86
|
+
[servicenow-integration.md](servicenow-integration.md).
|
|
87
|
+
|
|
88
|
+
## Debugging a DOE 997 complaint
|
|
89
|
+
|
|
90
|
+
1. Query `Logs.FileLog` on the `legacy` environment for job `EDI_NYCDOE_PO`.
|
|
91
|
+
2. Match the pair: the 997 row's `fileName` is `997_` + the 850's `fileName` (e.g. id
|
|
92
|
+
`880810` = the 850, `880811` = its 997).
|
|
93
|
+
3. `fileData` is a BLOB — read it with `CONVERT(fileData USING latin1)` to see the EDI text.
|
|
94
|
+
4. The 850 is deleted from DOE's SFTP after a successful run, but the **raw file data is
|
|
95
|
+
retained in `Logs.FileLog`**, so a 997 can be regenerated from the stored 850 without
|
|
96
|
+
asking DOE to resend.
|
|
97
|
+
|
|
98
|
+
## Change history
|
|
99
|
+
- 2026-09-04 — **Fixed (TRUE-81757): `AK2` sent the wrong transaction set id.**
|
|
100
|
+
`App_Edi::generateAcknowledgements997FromPurchaseOrders850()` (`library/app/edi.php`
|
|
101
|
+
line 400) hardcoded `'855'`; changed to `'850'` — the 997 acknowledges an inbound **850**
|
|
102
|
+
Purchase Order, so `AK201` must be `850`. Not an isolated event: the line has read `'855'`
|
|
103
|
+
since the function was first written (commit `3136e813`), and every 997 in `Logs.FileLog`
|
|
104
|
+
back to at least 2026-03-20 shows `AK2~855` (ids 880811, 868639, 814727, 810405) — DOE's
|
|
105
|
+
translator silently accepted it for months and complained only once. **Business decision
|
|
106
|
+
(Eric): fix going forward only** — no back-fix and no resend of historical 997s.
|
|
107
|
+
Also documented the full 850-in/997-out flow, the always-accept `AK5`/`AK9` stub (still
|
|
108
|
+
open), and the `Logs.FileLog` debugging path. (mhammontree)
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
9
|
+
updated: 2026-09-04
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/import_asn.php
|
|
@@ -42,6 +42,7 @@ files:
|
|
|
42
42
|
- worker2/Worker/Monitor/Nycdoe.php
|
|
43
43
|
related:
|
|
44
44
|
- ../profile.md
|
|
45
|
+
- edi-850-receive-and-997-ack.md
|
|
45
46
|
- ../../../2.0/apps/worker2/features/oneuptime-worker2-monitoring.md
|
|
46
47
|
- hold-status-sync.md
|
|
47
48
|
- ../../../1.0/apps/worker/architecture.md
|
|
@@ -90,7 +91,7 @@ ticket state / ETA / proof-of-delivery back to ServiceNow. A parallel EDI surfac
|
|
|
90
91
|
| Proof of delivery → ServiceNow | `send_nycdoe_proof_of_delivery.php` | every 5 min |
|
|
91
92
|
| Locations sync | `sync_nycdoe_locations.php` | daily 3:00 AM |
|
|
92
93
|
| Site-id monitor | `email_notification_siteid.php` | daily 7:01 AM |
|
|
93
|
-
| EDI 850 POs in → NetSuite SO + 997 ack | `receive_edi_purchase_orders.php` (`senderId == 'NYCDOE'` only) | 9:30 AM & 3:30 PM |
|
|
94
|
+
| EDI 850 POs in → NetSuite SO + 997 ack | `receive_edi_purchase_orders.php` (`senderId == 'NYCDOE'` only) — see [edi-850-receive-and-997-ack.md](edi-850-receive-and-997-ack.md) | 9:30 AM & 3:30 PM |
|
|
94
95
|
| EDI invoices out (NetSuite saved search 2108) | `send_edi_open_invoices.php` | every 2 h at :13 |
|
|
95
96
|
| Installation schedule reports | `notifications/nycdoe/send_installation_schedules_doe.php` (3 business days ahead, weekends skipped, **holidays NOT skipped**), daily + current-week variants, weekly part-orders | 5–8 AM daily/Mondays |
|
|
96
97
|
|
|
@@ -647,6 +648,7 @@ the client's `apps:` scope solely so this monitor loads, and its presence is **n
|
|
|
647
648
|
of the consumer query; `php -l` every touched file.
|
|
648
649
|
|
|
649
650
|
## Change history
|
|
651
|
+
- 2026-09-04 — Split the EDI 850-in / 997-ack surface into its own doc, [edi-850-receive-and-997-ack.md](edi-850-receive-and-997-ack.md) (TRUE-81757 `AK2~855`→`850` fix, always-accept `AK5`/`AK9` stub, `Logs.FileLog` debugging). This doc keeps the ticket/ASN pipeline. (mhammontree)
|
|
650
652
|
- 2026-08-18 — Repaired stranded ASN `26644` / customer PO `WR260175404-RPL` (**data-only, no code
|
|
651
653
|
in `worker` or `dbchanges`**): item `54552` (`DOE-62C5GAR1US`, unit `604254`, serial `VKVY4802`)
|
|
652
654
|
had been `pending` with a NULL `togadeskRepairOrderId` since 2026-07-02. Repointed
|
|
@@ -757,6 +759,7 @@ the client's `apps:` scope solely so this monitor loads, and its presence is **n
|
|
|
757
759
|
## Related docs
|
|
758
760
|
|
|
759
761
|
- [NYC DOE client profile](../profile.md)
|
|
762
|
+
- [NYCDOE EDI 850 Receive & 997 Acknowledgement](edi-850-receive-and-997-ack.md) — 850-in/997-out
|
|
760
763
|
- [Worker (1.0) architecture](../../../1.0/apps/worker/architecture.md) — cron dispatch,
|
|
761
764
|
role election, schedule files
|
|
762
765
|
- Local deep-dive (full WR260236464 case study with record IDs):
|
|
@@ -11,11 +11,12 @@ project: Worker
|
|
|
11
11
|
client: nycdoe
|
|
12
12
|
type: profile
|
|
13
13
|
status: active
|
|
14
|
-
updated: 2026-
|
|
14
|
+
updated: 2026-09-04
|
|
15
15
|
owners: [mhammontree, sking]
|
|
16
16
|
files: []
|
|
17
17
|
related:
|
|
18
18
|
- features/servicenow-integration.md
|
|
19
|
+
- features/edi-850-receive-and-997-ack.md
|
|
19
20
|
---
|
|
20
21
|
|
|
21
22
|
## Summary
|
package/package.json
CHANGED