toga-ai 1.0.473 → 1.0.475
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/knowledge/1.0/apps/dbchanges/INDEX.md +5 -0
- package/knowledge/1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md +149 -0
- package/knowledge/1.0/apps/worker/INDEX.md +1 -1
- package/knowledge/1.0/apps/worker/architecture.md +18 -3
- package/knowledge/2.0/apps/worker2/INDEX.md +2 -1
- package/knowledge/2.0/apps/worker2/architecture.md +26 -1
- package/knowledge/2.0/apps/worker2/features/platform-cache-cleanup.md +5 -1
- package/knowledge/2.0/apps/worker2/features/team-sprint-management.md +15 -1
- package/knowledge/2.0/apps/worker2/workflows/php-runtime-upgrade-dependency-audit.md +196 -0
- package/knowledge/INDEX.md +2 -1
- package/knowledge/clients/nycdoe/INDEX.md +1 -1
- package/knowledge/clients/nycdoe/features/servicenow-integration.md +97 -6
- package/knowledge/clients/nycdoe/profile.md +6 -2
- package/knowledge/clients/tow-foundation/features/receipt-processing.md +14 -2
- package/knowledge/registry.json +7 -0
- package/package.json +1 -1
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
# dbchanges (Database Changes) — 1.0 knowledge
|
|
2
|
+
|
|
3
|
+
| Doc | Summary | Files |
|
|
4
|
+
|-----|---------|-------|
|
|
5
|
+
| [Authoring & Shipping a 1.0 dbchanges SQL File](workflows/authoring-and-shipping-sql-files.md) | `dbchanges` is the **1.0** (legacy/V1) schema-and-data change repository — the 1.0 sibling of 2.0's `dbchanges2`. | dbchanges/index.php, dbchanges/Core/, worker/crons/infrastructure/execute_dbchanges.php, worker/.ebextensions/030_dbchanges.config |
|
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Authoring & Shipping a 1.0 dbchanges SQL File
|
|
3
|
+
framework: "1.0"
|
|
4
|
+
repo: dbchanges
|
|
5
|
+
project: Database Changes
|
|
6
|
+
client: shared
|
|
7
|
+
type: workflow
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-29
|
|
10
|
+
owners: [sking]
|
|
11
|
+
files:
|
|
12
|
+
- dbchanges/index.php
|
|
13
|
+
- dbchanges/Core/
|
|
14
|
+
- worker/crons/infrastructure/execute_dbchanges.php
|
|
15
|
+
- worker/.ebextensions/030_dbchanges.config
|
|
16
|
+
related:
|
|
17
|
+
- ../../worker/architecture.md
|
|
18
|
+
- ../../../../clients/nycdoe/features/servicenow-integration.md
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Summary
|
|
22
|
+
|
|
23
|
+
`dbchanges` is the **1.0** (legacy/V1) schema-and-data change repository — the 1.0 sibling of
|
|
24
|
+
2.0's `dbchanges2`. Unlike `dbchanges2` it ships **its own runner**, `dbchanges/index.php`, which
|
|
25
|
+
is invoked by the worker tier. The runner's parsing model is crude and its error path is
|
|
26
|
+
**fail-stop for the whole queue**, so *how* you write the `.sql` file decides whether your change
|
|
27
|
+
(and everyone else's queued behind it) applies at all.
|
|
28
|
+
|
|
29
|
+
Two rules dominate everything below:
|
|
30
|
+
|
|
31
|
+
1. **A chunk that contains only comments aborts the entire queue.** Put all commentary *before*
|
|
32
|
+
the final statement; never end a commented line with a semicolon.
|
|
33
|
+
2. **`_production` does NOT auto-apply.** Merged production data changes land on the next worker
|
|
34
|
+
**deploy**, not within minutes.
|
|
35
|
+
|
|
36
|
+
> `dbchanges/Core/*` targets the **legacy (V1) `Core` schema** — not the 2.0 prod `Core`. Confirm
|
|
37
|
+
> the target environment before writing anything.
|
|
38
|
+
|
|
39
|
+
## How the runner parses a file (`dbchanges/index.php`)
|
|
40
|
+
|
|
41
|
+
- Each `.sql` file is read whole and split on the **literal `";\n"`** (semicolon + newline)
|
|
42
|
+
— `index.php` ~L140. There is no SQL lexer: no awareness of strings, comments, or delimiters.
|
|
43
|
+
- Every non-empty chunk is sent to MySQL through a single **`mysqli_query()`** (~L145). Multi-
|
|
44
|
+
statement execution is not used.
|
|
45
|
+
- On error (~L146–170) the runner writes to `Logs.Errors` and, on the prod path, **`exit()`s** —
|
|
46
|
+
which **blocks every dbchanges file queued behind it** until the offending file is fixed.
|
|
47
|
+
- Each file runs **inside a transaction**: `mysqli_autocommit($link, false)` is set before the
|
|
48
|
+
`_dbchanges` bookkeeping INSERT, and autocommit is only re-enabled (implicit COMMIT) after the
|
|
49
|
+
whole file succeeds. So on error the statement **and** its `_dbchanges` row roll back together,
|
|
50
|
+
and the file is retried on the next run.
|
|
51
|
+
|
|
52
|
+
### Consequence: the "empty query" trap
|
|
53
|
+
|
|
54
|
+
A chunk consisting solely of `--` comment lines comes back as MySQL error **1065 "Query was
|
|
55
|
+
empty"** — a hard failure that stops the queue. This is easy to trigger accidentally by pasting
|
|
56
|
+
verification queries or explanatory notes *after* the last statement.
|
|
57
|
+
|
|
58
|
+
```sql
|
|
59
|
+
-- CORRECT: all commentary lives above the statement, one trailing statement.
|
|
60
|
+
-- Repoint ASN items 53919/53933 to the received SO/PO chain.
|
|
61
|
+
UPDATE ...;
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
```sql
|
|
65
|
+
-- WRONG: the trailing comment block becomes its own chunk → error 1065 → queue stops.
|
|
66
|
+
UPDATE ...;
|
|
67
|
+
-- Verify with:
|
|
68
|
+
-- SELECT * FROM AdvanceShippingNoticeItems WHERE id IN (53919, 53933);
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
All 13 pre-existing files under `dbchanges/Core/` comply with this. Follow them.
|
|
72
|
+
|
|
73
|
+
## Expressing an all-or-nothing guard (the only pattern that works)
|
|
74
|
+
|
|
75
|
+
Procedural guards are **not expressible** in a dbchanges file:
|
|
76
|
+
|
|
77
|
+
- `IF` / `SIGNAL` / conditional `COMMIT`-`ROLLBACK` on `ROW_COUNT()` are legal only inside a
|
|
78
|
+
stored routine.
|
|
79
|
+
- A routine body cannot be shipped anyway: its internal semicolons are shredded by the `";\n"`
|
|
80
|
+
split, and `DELIMITER` is a **client** directive that `mysqli` does not understand.
|
|
81
|
+
- An explicit `COMMIT` is **actively harmful** — it lands the data change independently of the
|
|
82
|
+
`_dbchanges` bookkeeping row, breaking retry-on-failure.
|
|
83
|
+
|
|
84
|
+
The working pattern is a **single-statement inline preflight guard**: a derived-table subquery
|
|
85
|
+
that counts the rows in the expected *before* state and requires an exact count, so the statement
|
|
86
|
+
is a no-op unless reality matches the assumption. The **derived-table wrapper is load-bearing** —
|
|
87
|
+
it is what avoids MySQL error **1093** ("can't specify target table for update in FROM clause").
|
|
88
|
+
Validated on **MySQL 8.0.42**.
|
|
89
|
+
|
|
90
|
+
```sql
|
|
91
|
+
UPDATE SomeTable t
|
|
92
|
+
SET t.col = <new>
|
|
93
|
+
WHERE t.id IN (<ids>)
|
|
94
|
+
AND (SELECT cnt FROM (
|
|
95
|
+
SELECT COUNT(*) AS cnt FROM SomeTable
|
|
96
|
+
WHERE id IN (<ids>) AND col = <expected-old>
|
|
97
|
+
) AS preflight) = <expected-row-count>;
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
## Which branches actually auto-apply
|
|
101
|
+
|
|
102
|
+
| Path | Trigger | Timing |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| `execute_dbchanges` cron (`worker/crons/infrastructure/execute_dbchanges.php`) | branch matches `_%` **and** `config.<env>.ini` exists in the repo (env = branch minus the leading `_`) | every **2 minutes** |
|
|
105
|
+
| Elastic Beanstalk container command (`worker/.ebextensions/030_dbchanges.config`) | worker **deploy** | on next deploy |
|
|
106
|
+
|
|
107
|
+
The repo ships `config.alpha/beta/demo/hotfix/prod/stage/test/worker.ini`, therefore:
|
|
108
|
+
|
|
109
|
+
- **Auto-apply within ~2 min of a push:** `_alpha`, `_beta`, `_demo`, `_hotfix`, `_stage`.
|
|
110
|
+
- **`_production` does NOT auto-apply** — it would need `config.production.ini`, which does not
|
|
111
|
+
exist (the shipped file is `config.prod.ini`). Production applies via the EB deploy container
|
|
112
|
+
command instead, i.e. **on the next worker deploy**. Plan comms and expectations accordingly.
|
|
113
|
+
|
|
114
|
+
## Steps
|
|
115
|
+
|
|
116
|
+
1. Confirm the target schema/environment (`Core/` = legacy V1 `Core`).
|
|
117
|
+
2. Branch per the git-workflow rules (`fix/...`, `feature/...`) — never commit to `_production`.
|
|
118
|
+
3. Write **one file, commentary first, statements last**, no trailing comment chunk, no explicit
|
|
119
|
+
`COMMIT`, and no `DELIMITER`/stored-routine constructs.
|
|
120
|
+
4. Wrap any risky data change in the derived-table preflight guard above so a drifted before-state
|
|
121
|
+
is a no-op rather than a wrong write.
|
|
122
|
+
5. Open a PR into `_production` (or the target `_<env>` branch).
|
|
123
|
+
6. After merge: `_<env>` branches apply within ~2 min; **`_production` waits for the next worker
|
|
124
|
+
deploy** — verify with a read-only query afterwards rather than assuming.
|
|
125
|
+
|
|
126
|
+
## Gotchas
|
|
127
|
+
|
|
128
|
+
- **A comment-only chunk stops the whole queue** (error 1065 + `exit()` on the prod path). Your
|
|
129
|
+
malformed file blocks unrelated teammates' migrations, so this is a team-wide failure, not a
|
|
130
|
+
local one.
|
|
131
|
+
- **No `DELIMITER`, no stored routines, no explicit `COMMIT`.** See above.
|
|
132
|
+
- **Error 1093** is why the preflight subquery must be wrapped in a derived table.
|
|
133
|
+
- **Retry semantics are a feature**: because the data change and the `_dbchanges` row share a
|
|
134
|
+
transaction, a failing file re-runs next cycle. Do not "help" it with a manual COMMIT.
|
|
135
|
+
- **⚠ SECURITY — hardcoded GitHub token.** `worker/crons/infrastructure/execute_dbchanges.php`
|
|
136
|
+
contains a **plaintext GitHub personal access token** in the small config block near the top of
|
|
137
|
+
the file (~L14–18), and interpolates it into a `shell_exec()` git-clone command line — which
|
|
138
|
+
also exposes it in the worker host's process listing. Treat the token as **compromised**:
|
|
139
|
+
rotate it, move it to config/env (`config.<env>.ini`), and pass it without embedding it in an
|
|
140
|
+
argv string. No credential value is recorded here by design.
|
|
141
|
+
|
|
142
|
+
## Change history
|
|
143
|
+
- 2026-07-29 — Repo onboarded to the KB (1.0 `dbchanges`, framework core sibling of `dbchanges2`).
|
|
144
|
+
Documented the `index.php` `";\n"` split + `mysqli_query()` model and the comment-only-chunk /
|
|
145
|
+
error-1065 queue abort, the per-file transaction & retry semantics, the derived-table preflight
|
|
146
|
+
guard as the only workable all-or-nothing pattern (verified MySQL 8.0.42), and which `_%`
|
|
147
|
+
branches actually auto-apply (`_production` does **not** — it lands on the next worker deploy).
|
|
148
|
+
Flagged the plaintext GitHub token in `worker/crons/infrastructure/execute_dbchanges.php` for
|
|
149
|
+
rotation. (sking)
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Summary | Files |
|
|
4
4
|
|-----|---------|-------|
|
|
5
|
-
| [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |
|
|
5
|
+
| [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config, worker/crons/infrastructure/execute_dbchanges.php, worker/.ebextensions/030_dbchanges.config |
|
|
6
6
|
| [Compass MA Sales Order Exception Report](features/compass-ma-sales-order-exception-report.md) | A worker cron that emails operations the "Compass Refresh Exception Report" — Compass `MA%` sales orders whose corresponding Office Depot (ODP) sales order has | worker/crons/toga2/compass/workflow/7_generate_ma_sales_order_exception_report.php |
|
|
7
7
|
| [Compass Partial In-Transit & Delivered Emails (per package)](features/compass-partial-in-transit-delivered-emails.md) | Compass USA and Compass Canada send a **per-package** in-transit email (and a matching delivered email) instead of one email listing the whole order. | worker/crons/toga2/compass/update_salesorder_status_from_odp.php, worker/crons/toga2/compasscanada/workflow/3_update_salesorder_status_from_grand_and_toy.php |
|
|
8
8
|
| [Elite TOGA 2.0 → TOGaDeskSupport Standalone Attachment Sync](features/elite-togadesk-attachment-sync.md) | `sync_togadesk_elite_attachments.php` is a standalone cron (every 5 minutes) that syncs file attachments from TOGA 2.0 into TOGaDeskSupport for Elite. | worker/crons/toga2/elite/sync_togadesk_elite_attachments.php, worker/crons/toga2/elite/test_sync_togadesk_elite_attachments.php |
|
|
@@ -6,8 +6,8 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-
|
|
10
|
-
owners: [jcardinal]
|
|
9
|
+
updated: 2026-07-29
|
|
10
|
+
owners: [jcardinal, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/index.php
|
|
13
13
|
- worker/_/app/framework.php
|
|
@@ -15,8 +15,11 @@ files:
|
|
|
15
15
|
- worker/schedules/
|
|
16
16
|
- worker/ebs/cron.worker.php
|
|
17
17
|
- worker/.ebextensions/035_cron.worker.config
|
|
18
|
+
- worker/crons/infrastructure/execute_dbchanges.php
|
|
19
|
+
- worker/.ebextensions/030_dbchanges.config
|
|
18
20
|
related:
|
|
19
21
|
- ../library/architecture.md
|
|
22
|
+
- ../dbchanges/workflows/authoring-and-shipping-sql-files.md
|
|
20
23
|
---
|
|
21
24
|
|
|
22
25
|
## Summary
|
|
@@ -190,7 +193,7 @@ The worker box is assembled from several repos, not just this one. `ebs/git.json
|
|
|
190
193
|
|---|---|---|
|
|
191
194
|
| `library` | `/var/www/library` | 1.0 framework core (`_.php`, all `App_*`). |
|
|
192
195
|
| `resources` | `…/resources` | Front-end assets (not used by crons). |
|
|
193
|
-
| `dbchanges` | `/var/www/dbchanges` | DB migration scripts. |
|
|
196
|
+
| `dbchanges` | `/var/www/dbchanges` | DB migration scripts **+ their runner (`index.php`)**. Applied by the 2-min `crons/infrastructure/execute_dbchanges.php` cron on `_%` branches with a matching `config.<env>.ini`, and by `.ebextensions/030_dbchanges.config` on deploy. Authoring rules: [Authoring & Shipping a 1.0 dbchanges SQL File](../dbchanges/workflows/authoring-and-shipping-sql-files.md). |
|
|
194
197
|
| `togadesk` | `…/ontrack` | TOGaDesk app — note `cron.worker.togadesk.json` runs `../ontrack/crons/tickets_prod.php` and `monitoring_prod.php` **out of the sibling repo**. |
|
|
195
198
|
| `_underscore` | `/var/www/_underscore` | **2.0 framework core** — present because the `toga2/` client integrations reach into 2.0 (TOGa-2 / TOGaSupply). This is a genuine 1.0↔2.0 bridge living inside a 1.0 app. |
|
|
196
199
|
|
|
@@ -205,6 +208,18 @@ web face (`mvc/` GET routes for login/logout/404); the tier's real work is the c
|
|
|
205
208
|
|
|
206
209
|
## Conventions & gotchas
|
|
207
210
|
|
|
211
|
+
- **`dbchanges` auto-apply is branch-gated, and `_production` is NOT auto-applied.**
|
|
212
|
+
`crons/infrastructure/execute_dbchanges.php` runs every 2 minutes but only for branches matching
|
|
213
|
+
`_%`, deriving the env by stripping the leading `_` and requiring `config.<env>.ini` in the
|
|
214
|
+
dbchanges repo. Present configs mean `_alpha`/`_beta`/`_demo`/`_hotfix`/`_stage` apply within
|
|
215
|
+
~2 min of a push, while `_production` does not (it would need `config.production.ini`; the
|
|
216
|
+
shipped file is `config.prod.ini`) — production migrations land via
|
|
217
|
+
`.ebextensions/030_dbchanges.config` on the **next worker deploy**.
|
|
218
|
+
- **⚠ SECURITY — plaintext GitHub token in `crons/infrastructure/execute_dbchanges.php`.** A
|
|
219
|
+
hardcoded GitHub personal access token sits in the config block near the top of the file
|
|
220
|
+
(~L14–18) and is interpolated into a `shell_exec()` git-clone command line, exposing it in the
|
|
221
|
+
worker host's process listing. Treat as **compromised**: rotate, move to `config.<env>.ini`/env,
|
|
222
|
+
and stop passing it in argv. (No credential value is recorded in the KB.)
|
|
208
223
|
- **Schedules are the source of truth for what runs.** A `.php` under `crons/` that no schedule
|
|
209
224
|
references is dormant (and may be `DOA/`). Set `active: 0` to disable without deleting.
|
|
210
225
|
- **All cron paths are relative to `crons/`** except the `../ontrack/...` jobs that run from the
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
| Doc | Summary | Files |
|
|
4
4
|
|-----|---------|-------|
|
|
5
|
-
| [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php |
|
|
5
|
+
| [Worker (worker2) Architecture](architecture.md) | Worker (repo `worker2`) is an AWS Elastic Beanstalk **Worker Tier** application that processes background jobs. | worker2/Controller/Index.php, worker2/Worker/, worker2/LambdaFunctions/, _underscore/Worker.php, worker2/composer.json |
|
|
6
6
|
| [Deploy-Time Auto-Registration to the Shared ALB Target Group (non-production)](features/alb-target-group-auto-registration.md) | TOGA does **not** pay for EB-managed load-balancer registration, so an EB instance is normally **not** added to its environment's ALB target group — a fresh or | worker2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, worker2/ebs/register_instance_to_shared_application_load_balancer.php, worker2/.platform/hooks/prebuild/_shared/040-write-instance-id.sh, worker2/.platform/hooks/prebuild/_shared/041-write-region.sh, worker2/.platform/hooks/postdeploy/015_install_composer.sh, api2/.platform/hooks/postdeploy/060_register_instance_to_shared_application_load_balancer.sh, api2/ebs/register_instance_to_shared_application_load_balancer.php |
|
|
7
7
|
| [Automated PR Merger — Concurrent Force-Push Clobber Race](features/automated-pr-merger-force-push-race.md) | The automated PR merger `_Worker_Team_GitHub::Merge` (`worker2` `Worker/Team/Github.php`) merges approved PRs to `_production` by **force-pushing from a clone t | Worker/Team/Github.php |
|
|
8
8
|
| [ClickUp Connectivity Watchdog](features/clickup-connectivity-watchdog.md) | A cron watchdog that emails when the ClickUp integration looks disconnected during business hours. | worker2/Worker/Clickup/Health.php, worker2/Database/ClickupHealthWatchdog.sql |
|
|
@@ -34,4 +34,5 @@
|
|
|
34
34
|
| [Teams Meeting Transcript Export](features/teams-transcript-export.md) | > **SUPERSEDED (2026-07-09) — the S3-staging model below is history.** `Export` is now a thin > **GRAPH-DIRECT** cron poller: it no longer archives raw VTT to ` | worker2/Worker/Team/Transcripts.php, worker2/Config/production.ini, worker2/Database/TeamsTranscriptExports.sql, dbchanges2/Core/2026-06-18a - Teams Transcript Export schedule.sql |
|
|
35
35
|
| [VAPI Webhook Handler (worker2 — AI-BDR end-of-call processing)](features/vapi-webhook-handler.md) | `_Worker_Vapi` ([worker2/Worker/Vapi.php](worker2/Worker/Vapi.php)) is the **PHP side of the AI-BDR call loop** — the webhook that receives VAPI's end-of-call r | worker2/Worker/Vapi.php, worker2/Worker/Ai/Bdr/Vapi.php |
|
|
36
36
|
| [WJE Freshservice Sync (worker2)](features/wje-freshservice-sync.md) | WJE ("WJE IT", helpdesk `wje.freshservice.com`) is a **Freshservice**-based help-desk client whose tickets, contacts, assets, groups, categories, and canned res | worker2/Worker/Wje.php, _underscore/Component/Api/Wje/Wje.php, _underscore/Model/Wje/Ticket.php, _underscore/Model/Wje/TicketNote.php, _underscore/Model/Wje/Contact.php, _underscore/Model/Wje/Unit.php, _underscore/Model/Wje/TicketTeam.php, _underscore/Model/Wje/TicketCategory.php, _underscore/Model/Wje/AssetType.php, _underscore/Model/Wje/PredefinedReply.php, library/app/api/wje.php, worker/crons/toga2/wje/import_supporting_records.php, worker/crons/toga2/wje/sync_togasupply_wje.php, worker/crons/notifications/reports/wje/wje_common.php, library/app/systemmonitor/wje.php, dbchanges2/Client_Wje/2024-10-04 - WjeOnboarding.sql |
|
|
37
|
+
| [PHP Runtime Upgrade on Elastic Beanstalk (worker2 8.3 → 8.5 + PhpSpreadsheet 1.x → 3.x)](workflows/php-runtime-upgrade-dependency-audit.md) | The procedure used to move worker2 from **PHP 8.3 to PHP 8.5** on Elastic Beanstalk, and the dependency work that had to land first. | worker2/composer.json, worker2/composer.lock, worker2/Worker/Team/Sprint.php, worker2/Worker/Client/TowFoundation/ProcessReceipts.php, worker2/Worker/Forecast/Import.php |
|
|
37
38
|
| [Ticket → ClickUp Pseudocode Planning (Talos-grounded)](workflows/ticket-to-pseudocode-planning.md) | A repeatable procedure for turning a ClickUp ticket into a reviewed, formatted implementation plan posted back to the ticket's `📝 Pseudocode` custom field. | test/@dave/clickup_md2delta.js |
|
|
@@ -6,17 +6,19 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: architecture
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-29
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Controller/Index.php
|
|
13
13
|
- worker2/Worker/
|
|
14
14
|
- worker2/LambdaFunctions/
|
|
15
15
|
- _underscore/Worker.php
|
|
16
|
+
- worker2/composer.json
|
|
16
17
|
related:
|
|
17
18
|
- ./features/creating-worker-actions.md
|
|
18
19
|
- ./features/alb-target-group-auto-registration.md
|
|
19
20
|
- ../_underscore/features/async-query-execution.md
|
|
21
|
+
- ./workflows/php-runtime-upgrade-dependency-audit.md
|
|
20
22
|
---
|
|
21
23
|
|
|
22
24
|
## Summary
|
|
@@ -32,6 +34,18 @@ processes background jobs. It's a `_underscore` 2.0 app (`index.php` is just
|
|
|
32
34
|
**MySQL is the source of truth; SQS is delivery only.** All job state lives in
|
|
33
35
|
`Core.WorkerJobs`. Every worker invocation reads from that table and writes its result back.
|
|
34
36
|
|
|
37
|
+
**PHP runtime baseline: 8.5 on both environments.** `_production` and `_sandbox-dev` both run
|
|
38
|
+
*PHP 8.5 on 64bit Amazon Linux 2023 / platform 4.13.4* (production was moved off 8.3 on
|
|
39
|
+
2026-07-29). `vendor/` is gitignored, so a dependency that is incompatible with the environment's
|
|
40
|
+
PHP **only ever fails at EB deploy time**, never locally — and if two environments run different
|
|
41
|
+
PHP, identical code deploys to one and fails on the other. worker2's root `require` therefore
|
|
42
|
+
carries an explicit upper bound, `"php": ">=8.2 <8.6"`, so the next platform jump fails pointing
|
|
43
|
+
at *our* requirement instead of at a confusing transitive library error; `composer update` is run
|
|
44
|
+
on the **lowest** supported runtime so one lock is valid everywhere. Never silence such a failure
|
|
45
|
+
with a `config.platform.php` pin or a downgrade to a version that merely *permits* the new PHP —
|
|
46
|
+
that trades a loud deploy failure for silent runtime breakage. Full procedure:
|
|
47
|
+
[PHP Runtime Upgrade on Elastic Beanstalk](./workflows/php-runtime-upgrade-dependency-audit.md).
|
|
48
|
+
|
|
35
49
|
**Production vs. non-production inbound differ.** The SQS/Lambda job pipeline above describes
|
|
36
50
|
**production**. **Non-production worker2 environments do not incorporate SQS at all** — they exist
|
|
37
51
|
for **manual invocation over HTTP** (hence `index.php` dispatch and `.platform/httpd/conf.d/`, and
|
|
@@ -44,6 +58,17 @@ passes through SQS** — a queue-based reproduction of a prod issue will not wor
|
|
|
44
58
|
listener rules and security groups must restrict non-prod to internal/VPN sources, and the
|
|
45
59
|
HTTP-triggered job endpoints must enforce auth.
|
|
46
60
|
|
|
61
|
+
**Critical rules:** MySQL is the source of truth and SQS is delivery only — all job state lives in
|
|
62
|
+
`Core.WorkerJobs`, so never treat a queue message as the record of a job. Every worker/webhook
|
|
63
|
+
endpoint must return **HTTP 200 on every path, including failure** (failures are recorded in
|
|
64
|
+
`WorkerJobs`): there is no DLQ and the visibility timeout is 3600s, so a single 500 becomes a
|
|
65
|
+
poison-message storm. Workers are `abstract class _Worker_<Name>` with `public static` entry
|
|
66
|
+
methods and are invoked **only** through `_Worker::runTask(action, parameters)` — never call a
|
|
67
|
+
worker method directly. This repo is PHP and Python only: never add a `.sql` file here, as all
|
|
68
|
+
schema changes and cron seeds belong to `dbchanges2`. Both environments run **PHP 8.5**, and the
|
|
69
|
+
repo's own root `require` (`">=8.2 <8.6"`) is the deliberate binding upper bound — never pin
|
|
70
|
+
`config.platform.php` or downgrade a package merely to make a deploy install on a newer runtime.
|
|
71
|
+
|
|
47
72
|
## AWS infrastructure
|
|
48
73
|
|
|
49
74
|
| Component | Notes |
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-29
|
|
10
10
|
owners: [jcardinal]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Platform/Cache.php
|
|
@@ -18,6 +18,7 @@ related:
|
|
|
18
18
|
- ./creating-worker-actions.md
|
|
19
19
|
- ../../_underscore/features/per-client-database-connections.md
|
|
20
20
|
- ../architecture.md
|
|
21
|
+
- ../workflows/php-runtime-upgrade-dependency-audit.md
|
|
21
22
|
---
|
|
22
23
|
|
|
23
24
|
## Summary
|
|
@@ -123,6 +124,9 @@ environment.
|
|
|
123
124
|
re-reads its own just-emptied tables from the query cache and mis-decides what is left.
|
|
124
125
|
|
|
125
126
|
## Change history
|
|
127
|
+
- 2026-07-29 — Verified `Platform/Cache/Truncate` working on **PHP 8.5** after worker2's EB
|
|
128
|
+
runtime upgrade from 8.3 (both environments now 8.5, AL2023 platform 4.13.4). No code change.
|
|
129
|
+
See [PHP Runtime Upgrade on Elastic Beanstalk](../workflows/php-runtime-upgrade-dependency-audit.md). (jcardinal)
|
|
126
130
|
- 2026-07-28 — Added second action `_Worker_Platform_Cache::Truncate()` on the same class: manual /
|
|
127
131
|
administrative, deliberately **not** scheduled. Removes the entire cross-client cache
|
|
128
132
|
(`DELETE FROM Tables` + cascade, defensive `DELETE FROM Tables_Clients` for orphaned cursors, then
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: shared
|
|
7
7
|
type: feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-29
|
|
10
10
|
owners: ["jcardinal", "kyalamarthi"]
|
|
11
11
|
files:
|
|
12
12
|
- worker2/Worker/Team/Sprint.php
|
|
@@ -17,6 +17,7 @@ related:
|
|
|
17
17
|
- ./creating-worker-actions.md
|
|
18
18
|
- ./clickup-work-type-automation.md
|
|
19
19
|
- ./clickup-project-routing.md
|
|
20
|
+
- ../workflows/php-runtime-upgrade-dependency-audit.md
|
|
20
21
|
---
|
|
21
22
|
|
|
22
23
|
## Summary
|
|
@@ -222,6 +223,13 @@ backfills `workTypeAtLock` the first time a task is seen).
|
|
|
222
223
|
|
|
223
224
|
## Gotchas
|
|
224
225
|
|
|
226
|
+
- **PhpSpreadsheet coordinates are ARRAYS now (3.x).** All `*ByColumnAndRow` methods were
|
|
227
|
+
removed in PhpSpreadsheet 2.0; `Sprint.php` had **828** of them (the bulk of the repo's 856)
|
|
228
|
+
and they are all now the array form — `->setCellValue([$col, $row], $v)`,
|
|
229
|
+
`->getStyle([$c1, $r1, $c2, $r2])`, `->mergeCells([$c1, $r1, $c2, $r2])`. Do not reintroduce
|
|
230
|
+
the old signatures when copying older report code into this file; the full mapping and the
|
|
231
|
+
safe transform method are in
|
|
232
|
+
[PHP Runtime Upgrade on Elastic Beanstalk](../workflows/php-runtime-upgrade-dependency-audit.md#step-5--migrate-the-call-sites-bycolumnandrow-removed-in-20).
|
|
225
233
|
- **Single source file, huge methods.** `SprintEnd` alone is ~2,700 lines; `SprintLock` and
|
|
226
234
|
`SprintDaily` are each ~1,000. Almost all report layout/formatting is inline. Treat the
|
|
227
235
|
method boundaries (and the constants block at the top) as the map.
|
|
@@ -363,6 +371,12 @@ middleware defaults every tile/chart endpoint to it when no `?sprint=` is suppli
|
|
|
363
371
|
|
|
364
372
|
## Change history
|
|
365
373
|
|
|
374
|
+
- 2026-07-29 — Migrated all **828** `*ByColumnAndRow` call sites in `Sprint.php` to
|
|
375
|
+
PhpSpreadsheet's array-coordinate API as part of the phpspreadsheet 1.30.2 → 3.10.7 upgrade
|
|
376
|
+
required by the worker2 PHP 8.5 runtime move. No report behavior change intended; the
|
|
377
|
+
array-range forms of `getStyle`/`mergeCells` were verified empirically against 3.10.7
|
|
378
|
+
(including an Xlsx save/load round-trip). See
|
|
379
|
+
[PHP Runtime Upgrade on Elastic Beanstalk](../workflows/php-runtime-upgrade-dependency-audit.md). (jcardinal)
|
|
366
380
|
- 2026-07-24 — Productionized the dashboard: the metric definitions above are now served by six
|
|
367
381
|
api2 Record Scripts on `_Model_Team_Sprint` (see
|
|
368
382
|
[Sprint Dashboard API](../../api2/features/sprint-dashboard-api.md)), retiring the Express
|
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: PHP Runtime Upgrade on Elastic Beanstalk (worker2 8.3 → 8.5 + PhpSpreadsheet 1.x → 3.x)
|
|
3
|
+
framework: "2.0"
|
|
4
|
+
repo: worker2
|
|
5
|
+
project: Worker
|
|
6
|
+
client: shared
|
|
7
|
+
type: workflow
|
|
8
|
+
status: active
|
|
9
|
+
updated: 2026-07-29
|
|
10
|
+
owners: [jcardinal]
|
|
11
|
+
files:
|
|
12
|
+
- worker2/composer.json
|
|
13
|
+
- worker2/composer.lock
|
|
14
|
+
- worker2/Worker/Team/Sprint.php
|
|
15
|
+
- worker2/Worker/Client/TowFoundation/ProcessReceipts.php
|
|
16
|
+
- worker2/Worker/Forecast/Import.php
|
|
17
|
+
related:
|
|
18
|
+
- ../architecture.md
|
|
19
|
+
- ../features/team-sprint-management.md
|
|
20
|
+
- ../features/platform-cache-cleanup.md
|
|
21
|
+
- ../../../1.0/apps/tools/workflows/deploy-to-elastic-beanstalk-al2023.md
|
|
22
|
+
- ../../../clients/tow-foundation/features/receipt-processing.md
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Summary
|
|
26
|
+
|
|
27
|
+
The procedure used to move worker2 from **PHP 8.3 to PHP 8.5** on Elastic Beanstalk, and the
|
|
28
|
+
dependency work that had to land first. It is written as a reusable checklist because every
|
|
29
|
+
TOGA repo will face the same sequence as EB platform branches advance.
|
|
30
|
+
|
|
31
|
+
Trigger: an EB `app-deploy` to `_sandbox-dev` failed at **"Install composer dependencies"** with
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
Your lock file does not contain a compatible set of packages...
|
|
35
|
+
phpoffice/phpspreadsheet is locked to version 1.30.2 ... requires php >=7.4.0 <8.5.0
|
|
36
|
+
-> your php version (8.5.8) does not satisfy that requirement.
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Outcome: worker2 now runs **phpspreadsheet 3.10.7** on a single lock valid on both 8.3 and 8.5,
|
|
40
|
+
and **both** EB environments (`_production` and `_sandbox-dev`) run PHP 8.5 on Amazon Linux 2023
|
|
41
|
+
platform 4.13.4.
|
|
42
|
+
|
|
43
|
+
## Step 1 — Diagnose the environment, not the branch
|
|
44
|
+
|
|
45
|
+
`_production` and `_sandbox-dev` were **byte-identical** in `composer.json`, `composer.lock`
|
|
46
|
+
(both phpspreadsheet 1.30.2), and all 21 `.platform/` + `.ebextensions/` files. The only
|
|
47
|
+
difference was the EB environment's **PHP runtime**:
|
|
48
|
+
|
|
49
|
+
| Environment | Platform |
|
|
50
|
+
|---|---|
|
|
51
|
+
| `_production` | PHP **8.3** running on 64bit Amazon Linux 2023/4.13.4 |
|
|
52
|
+
| `_sandbox-dev` | PHP **8.5** running on 64bit Amazon Linux 2023/4.13.4 |
|
|
53
|
+
|
|
54
|
+
Same platform branch, different PHP. The developer deleted and re-branched `_sandbox-dev` from
|
|
55
|
+
`_production` and the failure **reproduced identically** — re-branching cannot fix an
|
|
56
|
+
environment/runtime mismatch. When a deploy fails on one environment and not another with
|
|
57
|
+
identical code, diff the **environment**.
|
|
58
|
+
|
|
59
|
+
`vendor/` is gitignored in worker2, so this class of failure only ever surfaces at EB deploy
|
|
60
|
+
time, never locally.
|
|
61
|
+
|
|
62
|
+
## Step 2 — Audit the WHOLE lock for upper PHP bounds
|
|
63
|
+
|
|
64
|
+
Composer reports only **"Problem 1"**. Before sizing a runtime upgrade, walk every locked
|
|
65
|
+
package's `require.php` for an upper bound. Of worker2's **36** locked packages, exactly **one**
|
|
66
|
+
carried a bound below 8.5 — `phpoffice/phpspreadsheet`. Everything else
|
|
67
|
+
(`aws/aws-sdk-php`, `sentry/sentry`, guzzle, `phpseclib`, `phpmailer`, `symfony/*`,
|
|
68
|
+
`markbaker/*`) was open-ended or 8.5-inclusive. Do not fix the first reported package and
|
|
69
|
+
assume you are done.
|
|
70
|
+
|
|
71
|
+
## Step 3 — Read the library's version-line policy, not just its constraint
|
|
72
|
+
|
|
73
|
+
PhpSpreadsheet lines, all released 2026-07-12 (verified on Packagist + GitHub Releases):
|
|
74
|
+
|
|
75
|
+
| Line | `require.php` | PHP 8.5? |
|
|
76
|
+
|---|---|---|
|
|
77
|
+
| 1.30.6 | `>=7.4.0 <8.5.0` | **deliberately excluded** (1.30.1+ tightened from 1.30.0) |
|
|
78
|
+
| 2.4.7 | `>=8.1.0 <8.6.0` | yes, but capped at 8.6 |
|
|
79
|
+
| 3.10.7 | `^8.1` | yes, no upper cap |
|
|
80
|
+
| 5.9.0 | `^8.2` | current line |
|
|
81
|
+
|
|
82
|
+
**Trap:** 1.30.0's `^7.4 || ^8.0` technically *permits* 8.5, and maintainers narrowed 1.30.1+
|
|
83
|
+
to `<8.5.0` **on purpose**. So downgrading to 1.30.0 — or pinning `config.platform.php` — makes
|
|
84
|
+
`composer install` succeed while the code runs on an explicitly unsupported runtime. That
|
|
85
|
+
converts a loud deploy failure into silent runtime breakage. There is a composer-only way to
|
|
86
|
+
make 1.x *install* on both 8.3 and 8.5; there is **no** composer-only way to make it *safe*.
|
|
87
|
+
|
|
88
|
+
## Step 4 — Pick the target: `^3.10`, not 2.4 and not 5.x
|
|
89
|
+
|
|
90
|
+
- **`^3.10`** (resolved 3.10.7) — supports 8.3 and 8.5 with **no 8.6 cap**, and its breaking
|
|
91
|
+
surface is limited to the 2.0 removals (Step 5).
|
|
92
|
+
- **2.4.7** rejected: its `<8.6.0` cap would force a repeat migration at PHP 8.6.
|
|
93
|
+
- **5.x** rejected: adds 4.0 behavioral changes (DataValidation stored per-worksheet not
|
|
94
|
+
per-cell; CSV reader stops auto-detecting Mac line endings; HTML writer emits `TRUE`/`FALSE`
|
|
95
|
+
instead of `1`/empty string; Xlsx writer `forceFullCalc` default flips) and 5.0 changes
|
|
96
|
+
(external images need `setAllowExternalImages(true)`; `DefaultValueBinder` binds integers
|
|
97
|
+
>15 digits as strings) — real risk to report output for **zero gain** on the 8.5 question.
|
|
98
|
+
|
|
99
|
+
Also add an explicit **upper** bound to the repo's own root require:
|
|
100
|
+
|
|
101
|
+
```json
|
|
102
|
+
"php": ">=8.2 <8.6"
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
3.10.7 itself allows up to `<9.0`, so the repo bound is the deliberate binding constraint —
|
|
106
|
+
when a future EB platform lands PHP 8.6, composer fails pointing at **worker2's own**
|
|
107
|
+
requirement rather than at a confusing transitive library error.
|
|
108
|
+
|
|
109
|
+
**Resolve the lock on the lowest runtime you must support.** The lock was produced locally on
|
|
110
|
+
PHP 8.2.12 (below both 8.3 and 8.5), which yields the most **conservative** package set — valid
|
|
111
|
+
on 8.3 and 8.5 alike, so no `config.platform.php` pin was needed.
|
|
112
|
+
|
|
113
|
+
`ezyang/htmlpurifier` is **dropped** as a transitive dependency going 1.x → 3.x. Safe here
|
|
114
|
+
(zero direct usage in worker2) — check it in any other repo doing this upgrade.
|
|
115
|
+
|
|
116
|
+
## Step 5 — Migrate the call sites (`*ByColumnAndRow` removed in 2.0)
|
|
117
|
+
|
|
118
|
+
PhpSpreadsheet 2.0 removed **all** deprecated `*ByColumnAndRow` methods ("All deprecated things
|
|
119
|
+
have been removed"). **856** call sites migrated across 3 files: `Worker/Team/Sprint.php` (828),
|
|
120
|
+
`Worker/Client/TowFoundation/ProcessReceipts.php` (20), `Worker/Forecast/Import.php` (8).
|
|
121
|
+
|
|
122
|
+
The mapping — coordinates become an **array**:
|
|
123
|
+
|
|
124
|
+
```php
|
|
125
|
+
->setCellValueByColumnAndRow($c, $r, $v) => ->setCellValue([$c, $r], $v)
|
|
126
|
+
->setCellValueExplicitByColumnAndRow($c, $r, $v, $t) => ->setCellValueExplicit([$c, $r], $v, $t)
|
|
127
|
+
->getCellByColumnAndRow($c, $r) => ->getCell([$c, $r])
|
|
128
|
+
->getStyleByColumnAndRow($c1, $r1, $c2, $r2) => ->getStyle([$c1, $r1, $c2, $r2])
|
|
129
|
+
->mergeCellsByColumnAndRow($c1, $r1, $c2, $r2) => ->mergeCells([$c1, $r1, $c2, $r2])
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
Per-method counts: `setCellValue` 447, `getStyle` 341, `mergeCells` 53, `getCell` 14,
|
|
133
|
+
`setCellValueExplicit` 1.
|
|
134
|
+
|
|
135
|
+
**Method gotcha:** a naive regex breaks on nested calls and on commas inside string literals.
|
|
136
|
+
The migration used a paren/quote/comment-aware transform that splits only **top-level** commas
|
|
137
|
+
and **validates each call's arity** against an expected set, skipping and reporting anomalies
|
|
138
|
+
rather than silently rewriting. Result: 856/856 converted, 0 skipped, 0 anomalies, `php -l`
|
|
139
|
+
clean on all three files, zero residual `ByColumnAndRow` in the repo.
|
|
140
|
+
|
|
141
|
+
**Verified empirically against installed 3.10.7** (not assumed from docs): 4-element
|
|
142
|
+
`getStyle([1,1,3,1])` really does style the full range (A1 and C1 both bold), the 2-element
|
|
143
|
+
`getStyle([2,2])` targets a single cell, `mergeCells([1,5,3,5])` yields `A5:C5`, and both styles
|
|
144
|
+
and merges survive an Xlsx save/load round-trip.
|
|
145
|
+
|
|
146
|
+
## Step 6 — Bump the runtime as its own isolated change
|
|
147
|
+
|
|
148
|
+
The safe sequence, and the reason it is safe:
|
|
149
|
+
|
|
150
|
+
1. Make the code run on **both** runtimes first — one lock valid on 8.3 and 8.5. `_production`
|
|
151
|
+
therefore stayed deployable to the old 8.3 environment throughout the transition.
|
|
152
|
+
2. Prove it on `_sandbox-dev` (already 8.5).
|
|
153
|
+
3. **Then** bump `_production`'s EB platform 8.3 → 8.5 as an isolated, independently
|
|
154
|
+
revertible change.
|
|
155
|
+
|
|
156
|
+
Never a combined code+runtime jump — if it breaks you cannot tell which half did it. The
|
|
157
|
+
alternative of downgrading `_sandbox-dev` to 8.3 was explicitly declined: it hides the problem
|
|
158
|
+
that production will hit next.
|
|
159
|
+
|
|
160
|
+
Post-upgrade verification: `Platform/Cache/Truncate` confirmed working on PHP 8.5 (see
|
|
161
|
+
[Platform Cache Cleanup](../features/platform-cache-cleanup.md)).
|
|
162
|
+
|
|
163
|
+
## Open risk — pre-existing composer advisories (own ticket)
|
|
164
|
+
|
|
165
|
+
**Unrelated to this work and not introduced by it.** `composer audit` reports **16
|
|
166
|
+
medium-severity** advisories across 4 packages, principally:
|
|
167
|
+
|
|
168
|
+
| Package | Locked | Fixed in | Advisories |
|
|
169
|
+
|---|---|---|---|
|
|
170
|
+
| `guzzlehttp/guzzle` | 7.10.0 | >= 7.15.1 | CVE-2026-59883, CVE-2026-55767, CVE-2026-55568 |
|
|
171
|
+
| `guzzlehttp/psr7` | 2.9.0 | >= 2.12.3 | CVE-2026-59882, CVE-2026-55766, CVE-2026-49214 |
|
|
172
|
+
|
|
173
|
+
Plus cookie-scope / `Referer` / `Proxy-Authorization` leakage advisories. Deliberately left out
|
|
174
|
+
of the 8.5 branch to keep it a single-concern change — **needs its own ticket.**
|
|
175
|
+
|
|
176
|
+
## Key rules
|
|
177
|
+
|
|
178
|
+
- **Identical code failing on one environment only = diff the environment.** Re-branching never
|
|
179
|
+
fixes a runtime mismatch.
|
|
180
|
+
- **Audit every locked package's PHP bound**, not just the one composer names.
|
|
181
|
+
- **Never pin `config.platform.php` or downgrade to a version that merely *permits* the new
|
|
182
|
+
PHP** to silence a deploy error — that trades a loud failure for a silent one.
|
|
183
|
+
- **Run `composer update` on the lowest runtime you must support** so one lock covers all
|
|
184
|
+
environments.
|
|
185
|
+
- **Carry an explicit upper PHP bound in the repo's own root `require`** so the next runtime
|
|
186
|
+
jump fails pointing at us.
|
|
187
|
+
- **Code first (dual-runtime), runtime second (isolated).**
|
|
188
|
+
|
|
189
|
+
## Change history
|
|
190
|
+
- 2026-07-29 — Initial capture. worker2 upgraded phpspreadsheet 1.30.2 → 3.10.7 (`^3.10`) and
|
|
191
|
+
root require `>=8.2` → `>=8.2 <8.6`; 856 `*ByColumnAndRow` call sites migrated to the array
|
|
192
|
+
coordinate form across `Team/Sprint.php`, `Client/TowFoundation/ProcessReceipts.php`, and
|
|
193
|
+
`Forecast/Import.php`; `ezyang/htmlpurifier` dropped as a transitive dep. Production EB
|
|
194
|
+
environment then bumped PHP 8.3 → 8.5 (AL2023 platform 4.13.4) as an isolated change — both
|
|
195
|
+
environments now 8.5. 16 pre-existing medium composer advisories (guzzle 7.10.0, psr7 2.9.0)
|
|
196
|
+
flagged for a separate ticket. (jcardinal)
|
package/knowledge/INDEX.md
CHANGED
|
@@ -6,6 +6,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
6
6
|
|
|
7
7
|
- **library** (Library) _(framework core)_ — 14 doc(s) → [1.0/apps/library/INDEX.md](1.0/apps/library/INDEX.md)
|
|
8
8
|
- **worker** (Worker) — 17 doc(s) → [1.0/apps/worker/INDEX.md](1.0/apps/worker/INDEX.md)
|
|
9
|
+
- **dbchanges** (Database Changes) _(framework core)_ — 1 doc(s) → [1.0/apps/dbchanges/INDEX.md](1.0/apps/dbchanges/INDEX.md)
|
|
9
10
|
- **worker1.5** (Worker 1.5) — 0 doc(s) → [1.0/apps/worker1.5/INDEX.md](1.0/apps/worker1.5/INDEX.md)
|
|
10
11
|
- **togadesk** (TOGa Desk) — 10 doc(s) → [1.0/apps/togadesk/INDEX.md](1.0/apps/togadesk/INDEX.md)
|
|
11
12
|
- **togaview** (TOGa View) — 6 doc(s) → [1.0/apps/togaview/INDEX.md](1.0/apps/togaview/INDEX.md)
|
|
@@ -18,7 +19,7 @@ _Auto-generated by `knowledge.js index`. Do not hand-edit._
|
|
|
18
19
|
## 2.0 framework
|
|
19
20
|
|
|
20
21
|
- **_underscore** (_Underscore) _(framework core)_ — 40 doc(s) → [2.0/apps/_underscore/INDEX.md](2.0/apps/_underscore/INDEX.md)
|
|
21
|
-
- **worker2** (Worker) —
|
|
22
|
+
- **worker2** (Worker) — 35 doc(s) → [2.0/apps/worker2/INDEX.md](2.0/apps/worker2/INDEX.md)
|
|
22
23
|
- **api2** (API) — 19 doc(s) → [2.0/apps/api2/INDEX.md](2.0/apps/api2/INDEX.md)
|
|
23
24
|
- **dbchanges2** (Database Changes) _(framework core)_ — 3 doc(s) → [2.0/apps/dbchanges2/INDEX.md](2.0/apps/dbchanges2/INDEX.md)
|
|
24
25
|
- **toga2-supply** (TOGa Supply) — 3 doc(s) → [2.0/apps/toga2-supply/INDEX.md](2.0/apps/toga2-supply/INDEX.md)
|
|
@@ -3,5 +3,5 @@
|
|
|
3
3
|
| Doc | Framework | Summary | Files |
|
|
4
4
|
|-----|-----------|---------|-------|
|
|
5
5
|
| [NYCDOE Ticket Hold-Status Sync (ServiceNow ⇄ TOGaDesk)](features/hold-status-sync.md) | 1.0 | DOE ticket **hold** status must round-trip between ServiceNow (SNOW) and TOGaDesk and **stay held** — holds are SLA-bearing in both systems. | worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/send_request_item_updates.php, library/app/model/togadesk/repairorder.php, library/app/api/nycdoev2.php, togadesk/desk/includes/classes/class.repair.php |
|
|
6
|
-
| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents + RITMs) into local tables, turns vendor shipment notices into NetSuite Sales Orde | worker/crons/sync/nycdoe/import_asn.php, worker/crons/sync/nycdoe/import_inc.php, worker/crons/sync/nycdoe/legacy_import_asn.php, worker/crons/sync/nycdoe/legacy_process_asn_queue.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/1_send_asn_to_netsuite.php, worker/crons/sync/nycdoe/2_send_serials_to_netsuite.php, worker/crons/sync/nycdoe/3_create_installation_ticket.php, worker/crons/sync/nycdoe/test_multi_po_receipt_resolution.php, worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/send_request_item_updates.php, worker/crons/sync/nycdoe/send_nycdoe_proof_of_delivery.php, worker/crons/sync/nycdoe/sync_nycdoe_locations.php, worker/crons/sync/nycdoe/receive_edi_purchase_orders.php, worker/crons/sync/nycdoe/send_edi_open_invoices.php, worker/crons/notifications/nycdoe/, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.notification.json, library/app/api/nycdoe.php, library/app/api/nycdoev2.php, library/app/asnprocessor/manufacturer.php, library/app/asnprocessor/apple.php, library/app/asnprocessor/lenovo.php, library/app/asnprocessor/lexmark.php, library/app/asnprocessor/acer.php, library/app/edi.php, library/app/netsuite.php |
|
|
6
|
+
| [NYCDOE ServiceNow / ASN Integration](features/servicenow-integration.md) | 1.0 | The NYCDOE/ServiceNow integration mirrors DOE's ServiceNow tickets (Incidents + RITMs) into local tables, turns vendor shipment notices into NetSuite Sales Orde | worker/crons/sync/nycdoe/import_asn.php, worker/crons/sync/nycdoe/import_inc.php, worker/crons/sync/nycdoe/legacy_import_asn.php, worker/crons/sync/nycdoe/legacy_process_asn_queue.php, worker/crons/sync/nycdoe/process_tickets.php, worker/crons/sync/nycdoe/1_send_asn_to_netsuite.php, worker/crons/sync/nycdoe/2_send_serials_to_netsuite.php, worker/crons/sync/nycdoe/3_create_installation_ticket.php, worker/crons/sync/nycdoe/test_multi_po_receipt_resolution.php, worker/crons/sync/nycdoe/send_ticket_updates.php, worker/crons/sync/nycdoe/send_request_item_updates.php, worker/crons/sync/nycdoe/send_nycdoe_proof_of_delivery.php, worker/crons/sync/nycdoe/sync_nycdoe_locations.php, worker/crons/sync/nycdoe/receive_edi_purchase_orders.php, worker/crons/sync/nycdoe/send_edi_open_invoices.php, worker/crons/notifications/nycdoe/, worker/schedules/cron.worker.sync.json, worker/schedules/cron.worker.notification.json, library/app/api/nycdoe.php, library/app/api/nycdoev2.php, library/app/asnprocessor/manufacturer.php, library/app/asnprocessor/apple.php, library/app/asnprocessor/lenovo.php, library/app/asnprocessor/lexmark.php, library/app/asnprocessor/acer.php, library/app/edi.php, library/app/netsuite.php, dbchanges/Core/SK/ |
|
|
7
7
|
| [New York City Department of Education](profile.md) | 1.0 | NYC DOE (New York City Department of Education) is a TOGA client whose entire integration runs in the **1.0 worker tier** (~30 cron scripts under `worker/crons/ | |
|
|
@@ -6,7 +6,7 @@ project: Worker
|
|
|
6
6
|
client: nycdoe
|
|
7
7
|
type: client-feature
|
|
8
8
|
status: active
|
|
9
|
-
updated: 2026-07-
|
|
9
|
+
updated: 2026-07-29
|
|
10
10
|
owners: [mhammontree, sking]
|
|
11
11
|
files:
|
|
12
12
|
- worker/crons/sync/nycdoe/import_asn.php
|
|
@@ -36,10 +36,12 @@ files:
|
|
|
36
36
|
- library/app/asnprocessor/acer.php
|
|
37
37
|
- library/app/edi.php
|
|
38
38
|
- library/app/netsuite.php
|
|
39
|
+
- dbchanges/Core/SK/
|
|
39
40
|
related:
|
|
40
41
|
- ../profile.md
|
|
41
42
|
- hold-status-sync.md
|
|
42
43
|
- ../../../1.0/apps/worker/architecture.md
|
|
44
|
+
- ../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md
|
|
43
45
|
---
|
|
44
46
|
|
|
45
47
|
## Summary
|
|
@@ -149,11 +151,36 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘ [UNIQUE ded
|
|
|
149
151
|
line = 1 unit = 1 RO). Existing-serial lookup is scoped **per item row** (`$asnItemId`).
|
|
150
152
|
Item receipts are now resolved **Sales-Order-wide** via
|
|
151
153
|
`App_NetSuite::listItemReceiptsCreatedFromSalesOrder($soId)` (fans out across ALL POs from
|
|
152
|
-
`listPurchaseOrdersCreatedFromSalesOrder`, dedupes receipts by `internalId`), cached
|
|
153
|
-
per
|
|
154
|
-
|
|
155
|
-
receipt
|
|
156
|
-
|
|
154
|
+
`listPurchaseOrdersCreatedFromSalesOrder`, dedupes receipts by `internalId`), cached
|
|
155
|
+
**per sales-order internal id** (`$itemReceiptsBySo`, keyed by `netSuiteInternalSalesOrderId`)
|
|
156
|
+
— a shared SO is still fetched only once — with the `getItemReceipt` detail cache kept per
|
|
157
|
+
(globally-unique) receipt `internalId`. Both the inner item query **and** the outer
|
|
158
|
+
ASN-candidate query require `netSuiteInternalSalesOrderId <> 0` (a sentinel guard mirroring
|
|
159
|
+
the existing PO guard) so a null/zero-SO row cannot poison the cache or match the wrong SO's
|
|
160
|
+
receipts. Self-healing: units previously stuck under an unstamped PO clear on subsequent
|
|
161
|
+
5-min cron runs.
|
|
162
|
+
- **Receipt resolution is SO-ONLY — the stamped PO id never finds a receipt** (traced
|
|
163
|
+
2026-07-29, `3_create_installation_ticket.php`: candidate query ~L248-278, per-item ~L311-341,
|
|
164
|
+
receipt resolution ~L354-360, part-number match ~L418-422, per-unit gate ~L511-535). The chain
|
|
165
|
+
is strictly `AdvanceShippingNoticeItems.netSuiteInternalSalesOrderId` →
|
|
166
|
+
`listPurchaseOrdersCreatedFromSalesOrder` (all POs created from that SO) →
|
|
167
|
+
`listItemReceiptsCreateFromPurchaseOrder` for each → dedupe (this is what
|
|
168
|
+
`listItemReceiptsCreatedFromSalesOrder` wraps, `library/app/netsuite.php` ~L739 / ~L762 /
|
|
169
|
+
~L797, details via `getItemReceipt` ~L1697). **`netSuiteInternalPurchaseOrderId` is NEVER used
|
|
170
|
+
to look up receipts** — it is only a non-null / non-zero **eligibility gate**. The schema is
|
|
171
|
+
misleading here: the PO column *looks* authoritative and is not.
|
|
172
|
+
- **Item receipts are never persisted.** They are fetched **live from NetSuite on every run**
|
|
173
|
+
(no table, no bridge row), and the receipt-line → ASN-line linkage is derived **at runtime by
|
|
174
|
+
part-number string match** after stripping the `DOE-` prefix and `_` characters. So NetSuite is
|
|
175
|
+
the only source of truth for what was received, and a part-number formatting drift breaks the
|
|
176
|
+
match silently.
|
|
177
|
+
- **MSO reuse is safe on re-run.** `doeCreateInstallationRepairOrder()` looks the managed service
|
|
178
|
+
order up by `advanceShippingNoticeId`, so re-running Stage 5 against an ASN that already has an
|
|
179
|
+
MSO **reuses** it rather than creating a duplicate.
|
|
180
|
+
- **The per-SO (not per-ASN) receipt cache is load-bearing.** An ASN can carry lines from
|
|
181
|
+
more than one NetSuite sales order; caching the receipt set once per ASN off the *first*
|
|
182
|
+
line's SO matches every other line against the wrong SO's receipts (FIXED 2026-07-22 — see
|
|
183
|
+
the multi-SO gotcha + change history). This is one level *above* the multi-PO-per-SO fix.
|
|
157
184
|
- **"#/N received" UI** (`togadesk/desk/template/pages/managedserviceorders/view.php`):
|
|
158
185
|
denominator = `SUM(qtyOrder)` per part (committed + unsent); numerator = Units with
|
|
159
186
|
`togadeskRepairOrderId IS NOT NULL`.
|
|
@@ -212,6 +239,37 @@ Vendor SFTP ───(legacy_import_asn.php, ser+non-ser)─┘ [UNIQUE ded
|
|
|
212
239
|
pairs to size live double-truck-roll exposure; then trace the numeric-PO↔WR cluster.
|
|
213
240
|
Investigation was **read-only** — no code or data changes made.
|
|
214
241
|
|
|
242
|
+
- **⚠ SYSTEMIC, UNALERTED — a manually edited NetSuite SO strands hardware forever.** If a
|
|
243
|
+
NetSuite sales order is manually edited **after** Stage 3 has already stamped the SO/PO ids,
|
|
244
|
+
NetSuite may spawn a **replacement SO + PO**. The ASN line then points at the replacement PO,
|
|
245
|
+
which is **unreceived** (the goods were received on the *original* SO/PO chain), so Stage 5
|
|
246
|
+
resolves zero receipts → matches no serials → stamps nothing → creates no installation ticket.
|
|
247
|
+
**Silently, forever**: no error, no retry escalation, no alert; the every-5-min cron just keeps
|
|
248
|
+
finding nothing while the hardware sits stranded. Because receipt resolution is SO-only (see
|
|
249
|
+
Stage 5), the stamped SO id is the single point of failure.
|
|
250
|
+
- **Same family as the cross-wired SO/PO stamp regression below** (diagnosed 2026-07-23, ASN
|
|
251
|
+
26215): both start with a **manual NetSuite edit** and both are invisible because receipt
|
|
252
|
+
resolution is SO-only. There, the stamped PO still held the real receipt (a union lookup
|
|
253
|
+
recovers it); here the whole stamped chain points at an unreceived replacement, so the fix is
|
|
254
|
+
to repoint the ASN lines at the received chain.
|
|
255
|
+
- **`email_notification_siteid.php` does NOT catch this** — its gate still assumes the older
|
|
256
|
+
pre-non-serialized shape and it lacks the `netSuiteInternalSalesOrderId <> 0` sentinel
|
|
257
|
+
exclusion.
|
|
258
|
+
- **Durable fix (not yet built):** detect an ASN item with a stamped SO whose POs yield **zero**
|
|
259
|
+
item receipts beyond a threshold age, and alert — that condition is exactly this failure class.
|
|
260
|
+
- **Recovery pattern (what we do today):** repoint the affected `AdvanceShippingNoticeItems` rows
|
|
261
|
+
back to the SO/PO chain the goods were actually **received** on, via a `dbchanges` data repair,
|
|
262
|
+
then let the existing `*/5` Stage 5 cron create the TOGa Desk tickets itself. **No code change
|
|
263
|
+
is needed** and MSO reuse (keyed on `advanceShippingNoticeId`) makes the re-run idempotent.
|
|
264
|
+
- **Worked incident — ASN `26280`, customer PO `S202620046` (repaired 2026-07-29).** SO `271613`
|
|
265
|
+
(internal `6920229`) was manually edited, spawning replacement SO `274520`
|
|
266
|
+
(internal `6997347`) + PO `165123` (internal `6997348`) for two monitor lines. The monitors
|
|
267
|
+
were physically received on the **original** chain — PO `164203` (internal `6924437`), item
|
|
268
|
+
receipt internal `6924439` — so PO `165123` stayed unreceived and Stage 5 found nothing. Repair
|
|
269
|
+
repointed `AdvanceShippingNoticeItems` rows `53919` and `53933` to SO `6920229` / PO `6924437`
|
|
270
|
+
(`dbchanges/Core/SK/2026-07-29-data-repair-asn26280.sql`, PR dbchanges#257, branch
|
|
271
|
+
`fix/asn26280-repoint-monitor-so` → `_production`). Note `_production` does **not** auto-apply —
|
|
272
|
+
see the [1.0 dbchanges authoring workflow](../../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
|
|
215
273
|
- **The queue `dedupeKey` format is load-bearing.** A 2026 incident (WR260236464 duplicate
|
|
216
274
|
repair order) was caused by dedupeKey format drift (4-part keys with serial vs 3-part
|
|
217
275
|
without) letting old rows re-import; Stage 2 then created a sibling item row (frozen
|
|
@@ -325,6 +383,16 @@ use the toga DB MCP + `Logs.API` instead of running prod code locally.
|
|
|
325
383
|
of the consumer query; `php -l` every touched file.
|
|
326
384
|
|
|
327
385
|
## Change history
|
|
386
|
+
- 2026-07-29 — Repaired stranded ASN `26280` / customer PO `S202620046` (data-only, no code): a
|
|
387
|
+
manually edited SO spawned a replacement SO/PO (`274520`/`165123`) that was never received, so
|
|
388
|
+
Stage 5 found zero receipts and never ticketed two monitor lines; repointed
|
|
389
|
+
`AdvanceShippingNoticeItems` `53919`/`53933` to the received chain SO `6920229` / PO `6924437`
|
|
390
|
+
and let the `*/5` cron ticket them (`dbchanges/Core/SK/2026-07-29-data-repair-asn26280.sql`,
|
|
391
|
+
PR dbchanges#257). Documented the underlying **systemic, unalerted stranding failure class**
|
|
392
|
+
(`email_notification_siteid.php` does not catch it) and the traced fact that Stage 5 resolves
|
|
393
|
+
item receipts **only** via `netSuiteInternalSalesOrderId` — `netSuiteInternalPurchaseOrderId` is
|
|
394
|
+
purely an eligibility gate — with receipts fetched live from NetSuite each run and linked by
|
|
395
|
+
part-number match after stripping `DOE-`/`_`. Read-only in `worker`/`library`. (sking)
|
|
328
396
|
- 2026-07-23 — Diagnosed a regression of the 2026-07-09 multi-PO fix: Stage 5's SO-wide receipt
|
|
329
397
|
resolution *replaced* the by-stamped-PO lookup, so a cross-wired stamp (an item's stamped PO
|
|
330
398
|
not a child of its stamped SO) silently drops install tickets with no error. Root cause of the
|
|
@@ -336,6 +404,29 @@ use the toga DB MCP + `Logs.API` instead of running prod code locally.
|
|
|
336
404
|
owed ROs; a permanent union fix in the cron was considered but deferred (warehouse process now
|
|
337
405
|
corrected by training). Added a regression gotcha + two debugging notes. No production code
|
|
338
406
|
changed. (mhammontree; SME sking)
|
|
407
|
+
- 2026-07-22 — Fixed multi-SO-per-ASN silently dropping install tickets: Stage 5 cached the
|
|
408
|
+
NetSuite item-receipt set once per ASN off the first line's SO, so lines belonging to any
|
|
409
|
+
other SO on the same ASN matched the wrong receipts and never got a RepairOrder/asset. Now
|
|
410
|
+
cached **per sales-order internal id** (`$itemReceiptsBySo`); added the
|
|
411
|
+
`netSuiteInternalSalesOrderId <> 0` sentinel guard to both the outer ASN-candidate and the
|
|
412
|
+
per-item queries. One level above the 2026-07-09 multi-PO-per-SO fix. Case: PO `S202668066`,
|
|
413
|
+
ASN `26641`, SO `7219713`+`7246833`. Verified by read-only regression probe
|
|
414
|
+
`test_multi_so_receipt_cache.php` (PASS). PR worker#1678 (`fix/nycdoe-multi-so-receipt-cache`).
|
|
415
|
+
(sking)
|
|
416
|
+
- 2026-07-10 — Stage 5 (`3_create_installation_ticket.php`) now creates install tickets for
|
|
417
|
+
**non-serialized** received lines, which were previously never ticketed (Stage 5 was entirely
|
|
418
|
+
serial-driven off NetSuite `inventoryAssignment`). Added line-level column
|
|
419
|
+
`AdvanceShippingNoticeItems.togadeskRepairOrderId` (db_core, migration `dbchanges/Core/SK/
|
|
420
|
+
2026-07-10.sql`) + field on `App_Model_Core_AdvanceShippingNoticeItem`; ONE RO per non-serial
|
|
421
|
+
line on first receipt with qty>0 (no partial-qty accumulation); null serial/tag (both nullable,
|
|
422
|
+
verified). Rewrote ASN + item selection queries JOIN→EXISTS/NOT EXISTS (mutually-exclusive
|
|
423
|
+
serialized vs non-serialized branches); serialization decided authoritatively via
|
|
424
|
+
`App_NetSuite::getItemIsSerializedByPartNumber()`; extracted shared
|
|
425
|
+
`doeCreateInstallationRepairOrder()`; hoisted the ASN-completion check to once-per-ASN counting
|
|
426
|
+
both unit and line non-tickets. Also fixed a pre-existing SQL-injection risk (NetSuite serials
|
|
427
|
+
interpolated into `IN (...)` unescaped → now `App_Database::sqlEscape` via `array_map`; added
|
|
428
|
+
`(int)` casts on `$asnId`/`$asnItemId`). Branch `feature/doe-stage3-nonserialized-install`;
|
|
429
|
+
3 PRs deploy order dbchanges → library → worker. (sking)
|
|
339
430
|
- 2026-07-09 — Fixed silent install-ticket drop when a NetSuite SO is fulfilled across
|
|
340
431
|
multiple POs: Stage 5 now resolves item receipts Sales-Order-wide via new helper
|
|
341
432
|
`App_NetSuite::listItemReceiptsCreatedFromSalesOrder` (fans across all POs from the SO,
|
|
@@ -5,12 +5,13 @@ apps:
|
|
|
5
5
|
- worker
|
|
6
6
|
- library
|
|
7
7
|
- togadesk
|
|
8
|
+
- dbchanges
|
|
8
9
|
project: Worker
|
|
9
10
|
client: nycdoe
|
|
10
11
|
type: profile
|
|
11
12
|
status: active
|
|
12
|
-
updated: 2026-07-
|
|
13
|
-
owners: [mhammontree]
|
|
13
|
+
updated: 2026-07-29
|
|
14
|
+
owners: [mhammontree, sking]
|
|
14
15
|
files: []
|
|
15
16
|
related:
|
|
16
17
|
- features/servicenow-integration.md
|
|
@@ -32,6 +33,9 @@ runs over DOE's SFTP.
|
|
|
32
33
|
orders, and the "#received / #ordered" Receiving Summary UI.
|
|
33
34
|
- **NetSuite:** Sales Orders, PO reconciliation, item receipts, Item Fulfillments, invoices
|
|
34
35
|
(SuiteTalk toolkit in `library`).
|
|
36
|
+
- **`dbchanges` (1.0):** DOE schema + **data-repair** changes ship here (`dbchanges/Core/` targets
|
|
37
|
+
the legacy/V1 `Core` schema). Stranded-ASN recoveries are data-only repairs — see the
|
|
38
|
+
[1.0 dbchanges authoring workflow](../../1.0/apps/dbchanges/workflows/authoring-and-shipping-sql-files.md).
|
|
35
39
|
|
|
36
40
|
## Endpoints
|
|
37
41
|
| System | Endpoint | Notes |
|
|
@@ -5,8 +5,8 @@ project: Worker
|
|
|
5
5
|
client: tow-foundation
|
|
6
6
|
type: client-feature
|
|
7
7
|
status: active
|
|
8
|
-
updated: 2026-07-
|
|
9
|
-
owners: ["rgirish"]
|
|
8
|
+
updated: 2026-07-29
|
|
9
|
+
owners: ["rgirish", "jcardinal"]
|
|
10
10
|
files:
|
|
11
11
|
- worker2/Worker/Client/TowFoundation.php
|
|
12
12
|
- worker2/Worker/Client/TowFoundation/ProcessReceipts.php
|
|
@@ -14,6 +14,7 @@ files:
|
|
|
14
14
|
- worker2/Worker/Client/TowFoundation/TowFoundationCategories.php
|
|
15
15
|
related:
|
|
16
16
|
- clients/tow-foundation/profile.md
|
|
17
|
+
- ../../../2.0/apps/worker2/workflows/php-runtime-upgrade-dependency-audit.md
|
|
17
18
|
---
|
|
18
19
|
|
|
19
20
|
## Summary
|
|
@@ -353,6 +354,13 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
|
|
|
353
354
|
|
|
354
355
|
## Gotchas / known issues
|
|
355
356
|
|
|
357
|
+
- **PhpSpreadsheet coordinates are ARRAYS now (3.x).** worker2 runs phpspreadsheet 3.10.7 on
|
|
358
|
+
PHP 8.5; every `*ByColumnAndRow` method was removed in PhpSpreadsheet 2.0. The 20 call sites
|
|
359
|
+
in `ProcessReceipts.php` use the array form (`->setCellValue([$col, $row], $v)`,
|
|
360
|
+
`->getStyle([$c1, $r1, $c2, $r2])`). Do not paste older QB-Excel layout code using the old
|
|
361
|
+
signatures. See
|
|
362
|
+
[PHP Runtime Upgrade on Elastic Beanstalk](../../../2.0/apps/worker2/workflows/php-runtime-upgrade-dependency-audit.md).
|
|
363
|
+
|
|
356
364
|
- **`$year` variable shadowing** — `Run()` uses `$year` as both a filter parameter and a
|
|
357
365
|
loop variable (line ~159: `$year = $receipt['year']`). After the loop `$year` holds the
|
|
358
366
|
last receipt's year, not the original filter value. Harmless now but fragile if the loop
|
|
@@ -465,6 +473,10 @@ Fatal errors send only to `NOTIFY_EMAIL_DEV` (no CC/BCC).
|
|
|
465
473
|
|
|
466
474
|
## Change history
|
|
467
475
|
|
|
476
|
+
- 2026-07-29 — Migrated the **20** `*ByColumnAndRow` call sites in `ProcessReceipts.php` to
|
|
477
|
+
PhpSpreadsheet's array-coordinate API (phpspreadsheet 1.30.2 → 3.10.7, required by worker2's
|
|
478
|
+
PHP 8.3 → 8.5 runtime move). No intended change to the QB Excel output. (jcardinal)
|
|
479
|
+
|
|
468
480
|
- 2026-07-17 — **Memo fidelity overhaul + in-sheet Review column + cycle-aware MoveBack + offline export.** All in `ProcessReceipts.php`. (1) FIXED: the AI was paraphrasing/inventing the payment memo — rewrote the `extractReceiptData()` prompt + `payment_memo` schema to transcribe a human-added note **verbatim** and return **`null`** (never infer) when none exists; removed the old "infer a memo" instruction (client-reported: `"Lucy Ball of Lone Pine Foundation"`→`"LB LPFoundation"`, and a fabricated Asana memo). (2) FIXED: statement `Notes` override silently failed when AI OCR'd the wrong **year** — `matchStatementCandidate()` now falls back to a month+day (year-ignoring) date match so the authoritative Notes memo still wins despite year drift. (3) BUILT: in-sheet `Review` column (`buildExcelRow`+`generateExcel`) flags `"VERIFY AMOUNT (no matching charge on statement)"` (e.g. Ololo Safari KES `$172,872` OCR'd vs USD `$1,340.09`) and `"ADD MEMO (no note found on receipt)"` — the client reviews the Excel, not the email, so unverified/blank rows must surface in the sheet. (4) BUILT: prompt now scans the **entire page** (margins/corners/header/beside address) for visually-distinct human notes (missed a highlighted Optimum "Telephone & Internet" box, a red Garelick "Board Meeting … Remaining Deposit", an orange Lucid "IT Software License"). (5) BUILT: strip a leading 1–2-caps-plus-colon initials tag (`"RF:"`) from the memo — enforced in the prompt **and** as a deterministic `stripInitialsPrefix()` backstop that leaves 3+-letter prefixes (`"Postage:"`) alone. (6) BUILT: cycle-aware `MoveBack($scopeToCycle=true)` + `cycleDateWindow()`/`archivedFileDate()` — restore now moves only files whose archived-filename date falls in the cycle window (Amex = day-after-prior-3rd → this-3rd), instead of dumping the whole Archive into one cycle. (7) BUILT: read-only `DownloadCycleFiles(person, billingCycle, destDir, year, match, includeArchive)` — downloads a person's cycle receipts + statement Excel(s) to a local dir (constrained inside `__DIR__`), walking Archive via `collectPersonArchiveFiles()`, falling back to all statements when none matches. (8) DISCOVERED: local `Run()` always ends in a non-fatal `Unknown database 'logs_towfoundation'` at `_Email->send()` (Excel is generated + uploaded before that step, so work succeeds); Nadia Alia's statements are raw Amex "Transaction Details" exports with no `Notes` column (blank `ADD MEMO` memos are correct for her; no statement covers `06-04..07-03`); Talos is currently pointed at beta (`api.beta.togaiq.com`). Code changes remain **uncommitted** on the `_production` working tree (`fix/towfoundation-memo-verbatim` branch proposed, not yet created). (rgirish)
|
|
469
481
|
- 2026-07-13 — **Read-only Preview action + Ligia map-key fix + July cycle findings.** (1) BUILT: `Preview(?year, ?person, ?billingCycle): string` — a true dry-run that reuses `walkReceiptsFolder()` and the same filters as `Run()` but only tallies (downloads/extracts/moves nothing, sends no email), returning pretty JSON with per-person receipt counts, cycles, fileTypes, up-to-3 sampleFiles, and hasStatement, plus a softer `personsWithNoReceipts` list. Read `persons[]` as the authoritative empty signal; `personsWithNoReceipts` is derived from statement-only folders and also surfaces stray top-level folders (e.g. "Processed") as cosmetic noise. (2) FIXED: `CLASS_MAP`/`PAYMENT_ACCOUNT_MAP` keys for Ligia Marroquin Soto were `"Ligia Marroquin"` (no "Soto") — never hit against the folder-derived `"Ligia Marroquin Soto"`, so her Class + Payment Account silently blanked (both maps fall back to `''` with no warning). Renamed keys to `"Ligia Marroquin Soto"` (Class ⇒ `Administration:Operations`, Payment Account ⇒ `AMEX Open Credit Card:Ligia Marroquin-Soto Amex CC`); broadened the person-folder-normalization gotcha — map keys MUST equal `folder − " CC receipts"`. (3) DISCOVERED: July Amex `"07-03-2026"` has 72 receipts across 9 people (Nadia Alia 43 …); Diane Sierpina, Brent Peterkin, Susan Ransden empty for July; Ryan Farrell + Magdalena Minta have July receipts but no statement → AI-inferred memos; **Michael Zuber Zander** has a receipts folder but is in neither map (and zero receipts) → left unmapped pending client-provided QB Class + Payment Account. Also recorded that SharePoint creds live in the `[sharepoint_towfoundation]` Config ini section, read via `_Config::sharepoint_towfoundation()`. (rgirish)
|
|
470
482
|
- 2026-07-13 — **Per-person QB Excel location fix + two production findings.** (1) FIXED: `uploadExcelToSharePoint()` now writes each person's generated QB Excel to that person's own `Credit Card Receipts/{Person} CC receipts/{Year}/3. QB Excel/` subfolder (matching the docblock and client expectation) instead of a single shared root `3. QB Excel/`; signature is now `(accessToken, driveId, personFolderName, year, excelName, tmpFile)`, fed by a `$personFolders` map populated first-write-wins in Pass 1 with an `error_log` fallback to `{personName}/{currentYear}`. (2) DISCOVERED (production-critical gotcha): the statement-Notes lookup is EXACT string equality between the statement filename (`cycleKey`) and the receipt's billing-cycle FOLDER name — no date/fuzzy fallback — so a card whose folder has no identically-named statement silently skips the authoritative Notes memo and falls back to AI inference with zero signal (confirmed for Ryan Farrell's Mastercard folder vs. an Amex-named statement); also noted `computeRefNo()` hard-codes cycle-end day `03`, giving Mastercard rows a `…03…` Ref No. (pre-existing, unchanged). (3) DECIDED: monthly production procedure is two explicit per-card `Run` invocations (Amex + Ryan Farrell's Mastercard), NOT a "MM-YYYY month sweep" — the sweep prototype was deliberately reverted because `PAYMENT_ACCOUNT_MAP` is keyed per-person not per-card and would misattribute the QB Payment Account for anyone holding two cards in a swept month; recorded production-safety facts (no dry-run — `limit` still archives + emails; file moves reversible via `MoveBack` but email is not recallable; recipients are compile-time `NOTIFY_*` constants). (rgirish)
|
package/knowledge/registry.json
CHANGED
package/package.json
CHANGED