toga-ai 1.0.0

package/knowledge/1.0/apps/worker/INDEX.md

@@ -0,0 +1,5 @@
+# worker (Worker) — 1.0 knowledge
+
+| Doc | Summary | Files |
+|-----|---------|-------|
+| [Worker (1.0 Framework) Architecture](architecture.md) | `worker` is the legacy (**1.0** `App_` framework) **background-job tier**. | worker/index.php, worker/_/app/framework.php, worker/crons/, worker/schedules/, worker/ebs/cron.worker.php, worker/.ebextensions/035_cron.worker.config |

package/knowledge/1.0/apps/worker/architecture.md

@@ -0,0 +1,223 @@
+---
+title: Worker (1.0 Framework) Architecture
+framework: "1.0"
+repo: worker
+project: Worker
+client: shared
+type: architecture
+status: active
+updated: 2026-06-08
+owners: [jcardinal]
+files:
+  - worker/index.php
+  - worker/_/app/framework.php
+  - worker/crons/
+  - worker/schedules/
+  - worker/ebs/cron.worker.php
+  - worker/.ebextensions/035_cron.worker.config
+related:
+  - ../library/architecture.md
+---
+
+## Summary
+
+`worker` is the legacy (**1.0** `App_` framework) **background-job tier**. It is a single
+codebase of ~550 standalone PHP CRON scripts (under `crons/`) that handle everything the web
+apps must not do inline: notifications/reports, database backups, third-party syncs (NetSuite,
+EDI partners, NYC DOE, ManageEngine, Syncro…), cloud-infrastructure maintenance, and the
+TOGa / TOGaDesk / TOGa-2 client integrations.
+
+In production the tier is **one Elastic Beanstalk environment running 7 instances**, each of
+which **self-elects a distinct role** (`notification`, `database`, `infrastructure`, `toga`,
+`togadesk`, `sync`, `catalog`) and runs only that role's slice of the schedule. Platform:
+**PHP 7.2 running on 64-bit Amazon Linux / 2.8.8**.
+
+> This repo is the "legacy worker" referred to in the 1.0 back-end standard's
+> *Asynchronous / Background Processing* note — long-running work is handed off here rather
+> than run inside a web request. It builds on the `library` core (see
+> [Library architecture](../library/architecture.md)).
+
+## How a job becomes a cron (the dispatch pipeline)
+
+There is **no in-process scheduler**. Jobs are plain PHP files invoked by the OS `crontab`,
+which is generated at deploy time:
+
+1. **`schedules/cron.<env>.json`** declares the jobs for an environment. Each entry is
+   `{ active, name, schedule (5-field cron expr), cron | command | url }`:
+   - `cron` → `php /var/www/html/crons/<path>` (the common case)
+   - `command` → run the raw shell command (e.g. `apachectl -k graceful`)
+   - `url` → `wget http://localhost<url>`
+   - `active: 0` disables the entry without deleting it.
+2. **`.ebextensions/035_cron.worker.config`** runs `ebs/cron.worker.php` as an EB
+   `container_command` on every deploy.
+3. **`ebs/cron.worker.php`** reads the `ENVIRONMENT` env var, loads `cron.<env>.json`, and for
+   the `worker` env *also* loads the **role-specific** `cron.worker.<role>.json` (see role
+   election below). It writes the assembled crontab into the EB appdeploy enact hook
+   `/opt/elasticbeanstalk/hooks/appdeploy/enact/01_cron.sh`, which installs it via
+   `crontab cronjobs`.
+4. The OS cron daemon then runs each `php /var/www/html/crons/<path>` on its schedule.
+
+So **to add/modify a scheduled job you edit a `schedules/*.json` file and redeploy** — the cron
+path is relative to `crons/`. Editing the `.php` alone does nothing until it is referenced by a
+schedule entry.
+
+### Schedule files
+
+| File | Purpose |
+|---|---|
+| `cron.worker.json` | **Base — runs on every production worker** regardless of role: `apachectl -k graceful` (closes DB connections) + `worker/worker_heartbeat.php` (every minute). |
+| `cron.worker.<role>.json` | Per-role job set. The 7 roles: `notification` (~88), `sync` (~96), `toga` (8), `togadesk` (14), `infrastructure` (12), `database` (3), `catalog` (2). |
+| `cron.{alpha,beta,demo,hotfix,sprint,stage,ui}.json` | Non-prod environments — each runs the *entire* job set on a single box (no role splitting). |
+
+## Production topology: 7 self-electing workers
+
+The 7 production instances share one image; **role assignment is dynamic, not baked in.** This
+is the most important and least obvious part of the architecture.
+
+- The roster lives in DB table **`Workers`** (`workerInstanceId`, `workerName`, `dtHeartbeat`)
+  in the **`Vision_Log`** database (config group `db_log`).
+- The canonical role list is **`App_Worker::$desiredWorkers`** (in `library`): `notification`,
+  `database`, `infrastructure`, `toga`, `togadesk`, `sync`, `catalog`. (`ebs/cron.worker.php`
+  carries its own near-duplicate copy of this list — keep the two in sync.)
+- **Role election (at deploy, in `ebs/cron.worker.php`):** the script connects to `Vision_Log`,
+  reads `Workers`, computes which roles are not yet claimed, and assigns this instance (keyed by
+  its EC2 instance-id) the **first unclaimed role** — `INSERT`/`UPDATE`-ing its `Workers` row.
+  It then appends that role's `cron.worker.<role>.json` to the crontab.
+- **Heartbeat / self-healing (`crons/worker/worker_heartbeat.php`, every minute on all 7):**
+  - Updates this instance's `dtHeartbeat = NOW()`.
+  - First time an instance is seen, tags the EC2 instance `Worker=<ROLE>-WORKER`, **associates
+    the role's dedicated Elastic IP** (`App_Worker::$associateWorkerWithElasticIpAllocationId`),
+    and emails ops that the role was instated.
+  - If any worker's heartbeat is stale (> **270 s** in the heartbeat; the deploy script uses a
+    separate **7-minute** cutoff), it **terminates that EC2 instance** via the AWS API and
+    deletes its `Workers` row. EB replaces the instance, which re-runs deploy, finds the now-open
+    role, and re-claims it → automatic self-recovery. Each instance staggers with a random
+    `sleep(rand(1,50))` to avoid races.
+- **`App_Worker::getWorkerName()`** returns the current instance's role; helpers
+  `isPrimaryWorker()` / `isPrimaryDbWorker()` (currently commented out) gated "primary-only"
+  jobs in an older model.
+
+> Net effect: roles are a **claim-the-first-free-slot pool**, and a dead worker's role is
+> reclaimed by its replacement within minutes. There is no static instance→role mapping to edit.
+
+## Anatomy of a cron script
+
+Every script is self-contained and follows this boilerplate:
+
+```php
+<?php
+require_once('_.php');                                   // library bootstrap (see below)
+App_Framework_Worker::initialize();                      // worker framework init
+App_Framework::cronInitialization($timeLimitSeconds);    // lock + logging + time limit
+
+// ... the job's work, written against App_Model / App_Database / App_Api / App_Email ...
+
+App_Framework::cronFinished();                           // log completion
+App_Database::closeConnections();
+```
+
+- **`App_Framework_Worker::initialize()`** (`_/app/framework.php`) extends the library
+  `App_Framework`, disables sessions, registers the vendored libraries this tier needs
+  (PHPExcel reader/writer, NetSuite toolkit, phpseclib, php-imap2), and initializes **Sentry**
+  (project `worker1`, environment = the EB env).
+- **`App_Framework::cronInitialization($timeLimitSeconds = 7200, $okayToPerformOnReadCluster =
+  false, $exitIfProcessAlreadyRunning = true)`** (in `library`):
+  - **Overlap guard** — `exitIfProcessRunning()` scans `ps -ef` for another PHP process running
+    the same script file and exits early if found (prevents a slow job from stacking on its next
+    tick). This is a process-table lock, not a file lock.
+  - `set_time_limit()`, `date_default_timezone_set('America/Chicago')`, optional read-cluster
+    routing.
+  - Logs a `CRON_START` row to the `Log` table (`db_log`) and a row to `CronJobExecutions`
+    (`db_common`) for monitoring.
+- **`cronFinished()`** logs completion; **`App_Database::closeConnections()`** must be called so
+  the connection is released before the next tick.
+- **Sentry cron monitors (newer jobs):** wrap the body with
+  `if (App_Framework::isProcessRunning()) exit;` then `App_Worker::checkIn('<monitor-slug>')`
+  … work … `App_Worker::checkOut()`. A missing `checkOut()` makes Sentry flag the run as missed.
+
+## Directory layout
+
+```
+worker/
+├── index.php              # thin web entry: require _.php; App_Framework_Worker::{initialize,renderIndex}
+├── _/app/                 # APPROOT marker + app-level App_ overrides
+│   ├── framework.php          # App_Framework_Worker (extends library App_Framework)
+│   └── frameworkindex.php     # App_FrameworkIndex_Worker (renders the minimal web index)
+├── crons/                 # ~553 standalone job scripts, grouped by domain (below)
+├── schedules/             # *.json crontab definitions (per env + per role)
+├── mvc/                   # minimal web routes (get / login / logout / 404) — worker has a thin web face
+├── common/                # header.php / footer.php for the web face
+├── ebs/                   # deploy-time provisioning scripts (run by .ebextensions)
+├── .ebextensions/         # EB platform config (apache, https, imap, s3fs, ldap, redis, composer, git, cron)
+├── config.<env>.ini       # per-environment DB connections + settings (one per dev/stage/prod box)
+├── composer.json/vendor/  # Composer deps (Sentry, AWS SDK, …) — note: 1.0 apps normally vendor via library
+└── assets/
+```
+
+`_.php` itself is **not committed** — it is the `library` framework bootstrap, made available on
+the box at deploy (library is git-cloned to `/var/www/library`; the `_/` folder is the
+`__APPROOT__` marker the autoloader walks up to find). App-level classes named `App_*` resolve to
+`_/app/...`, shadowing the library copies (standard 1.0 autoloader behavior).
+
+### `crons/` domains
+
+| Folder | ~Files | What it does |
+|---|---:|---|
+| `notifications/` | 115 | Emailed reports, reminders, campaigns, CSAT surveys, daily/weekly/monthly client reports (Office Depot, Walmart, NYC DOE, Canon, Newegg, InComm…). Also `infrastructure/send_emails.php` outbound mail queue + code-commit digests. |
+| `sync/` | 97 | Record syncing & reconciliation: NetSuite, EDI partners (Prudential, Canon, DOE), ManageEngine, Syncro, Active Directory, Staples, tracking-number reconciliation. |
+| `toga2/` | 162 | **TOGa-2 (2.0) client integrations** — per-client subfolders (`compass`, `compasscanada`, `prudential`, `aig`, `rate`, `wje`, `elite`, `startech`, `nychh`, `northwell`, `sap_ariba`, `bankofamerica`, `managelife`, `quad`…), the NetSuite `togasupply` syncs, and `forecast2/` sales/order/opportunity imports. |
+| `toga/` | 23 | Core TOGa (1.0) processors: order processor, Walmart e-comm/B2B, subscription billing, garbage collector, client sync. |
+| `infrastructure/` | 18 | DB cluster failover, AWS GuardDuty auto-terminate, domain documenter, `system_monitors.php`, sprint open/close automation, forecast2 imports. |
+| `vendorinvoiceretrieval/` | 18 | Automated vendor-invoice retrieval. |
+| `catalog/` | 17 | Catalog DB reconciliation (Office Depot inventory imports). |
+| `maintenance/` | 9 | Hourly/daily/weekly maintenance + TOGaDesk ticket archiving. |
+| `togadesk/` | 8 | TOGaDesk helpdesk: inactive-people cleanup, UMA customer/ticket sync, task reminders, repair-order status. |
+| `test_active_directory/` | 5 | AD integration test scripts. |
+| `database/` | 4 | Backups, restores, restore scheduling. |
+| `worker/` | 2 | `worker_heartbeat.php` (the self-healing heartbeat) + helper. |
+| `DOA/` | 75 | **Dead-On-Arrival** — deprecated jobs kept for reference, **not scheduled**. Per the 1.0 standard, do not build on `DOA_`-prefixed code. |
+
+## Cross-repo bridges (cloned at deploy via `ebs/git.json`)
+
+The worker box is assembled from several repos, not just this one. `ebs/git.json` clones:
+
+| Repo | Deployed to | Why |
+|---|---|---|
+| `library` | `/var/www/library` | 1.0 framework core (`_.php`, all `App_*`). |
+| `resources` | `…/resources` | Front-end assets (not used by crons). |
+| `dbchanges` | `/var/www/dbchanges` | DB migration scripts. |
+| `togadesk` | `…/ontrack` | TOGaDesk app — note `cron.worker.togadesk.json` runs `../ontrack/crons/tickets_prod.php` and `monitoring_prod.php` **out of the sibling repo**. |
+| `_underscore` | `/var/www/_underscore` | **2.0 framework core** — present because the `toga2/` client integrations reach into 2.0 (TOGa-2 / TOGaSupply). This is a genuine 1.0↔2.0 bridge living inside a 1.0 app. |
+
+## Deploy-time provisioning (`.ebextensions/` → `ebs/`)
+
+Container commands set up the box on each deploy: apache tuning, force HTTP→HTTPS, install IMAP,
+**mount S3 via s3fs**, export/cache folders, LDAP, php.ini, Redis, Composer, clone the git
+libraries above, permit library executables, **symbolic links** (`ebs/setup_symbolic_links.php`
+links paths from `links.<env>.json`, e.g. `ontrack/desk/uploads` → an S3-backed folder), run
+`dbchanges`, and finally generate the crontab. `index.php` exists only to give the box a trivial
+web face (`mvc/` GET routes for login/logout/404); the tier's real work is the crons.
+
+## Conventions & gotchas
+
+- **Schedules are the source of truth for what runs.** A `.php` under `crons/` that no schedule
+  references is dormant (and may be `DOA/`). Set `active: 0` to disable without deleting.
+- **All cron paths are relative to `crons/`** except the `../ontrack/...` jobs that run from the
+  sibling TOGaDesk repo.
+- **Always call `App_Database::closeConnections()`** at the end; rely on `cronInitialization`'s
+  overlap guard rather than rolling your own lock.
+- **Timezone is `America/Chicago`** everywhere; cron expressions are in that zone, but several
+  `togadesk/` UMA jobs are commented with their UTC equivalents — read the `name` carefully.
+- **Two stale-worker cutoffs exist** (270 s in the heartbeat, 7 min in the deploy script) and the
+  `$desiredWorkers` list is duplicated in `ebs/cron.worker.php` and `App_Worker` — keep them
+  aligned when changing roles.
+- **Per-role Elastic IPs** mean a role's outbound IP is stable across instance replacement —
+  relevant for partner allow-lists (EDI/SFTP endpoints).
+
+## ⚠ Security note — committed secrets
+
+Several files contain **live credentials checked into the repo**: AWS access keys
+(`ebs/cron.worker.php`, and in a comment in `crons/worker/worker_heartbeat.php`), GitHub PATs
+(`ebs/git.json`), and a Sentry DSN (`_/app/framework.php`). These should be rotated and moved to
+environment variables / SSM Parameter Store rather than living in source. (Values are
+deliberately not reproduced here.)